Integration with Azure DevOps Test Plans

From version 8.0.0 onwards, Katalon Studio can be natively integrated with Azure DevOps (ADO) - Test Plans. This integration will help you:

  1. Easily map Test Cases in ADO to automated Test Cases in Katalon Studio to know which Test Cases are automated.

  2. Automatically send Test Execution logs and reports from Katalon Studio to Test Run in ADO to get the test status and have sufficient materials for debugging.

Requirements

  • Katalon Studio version 8.0.
  • An active Katalon Studio Enterprise license.
  • Set up Azure DevOps.

Enable the Integration and Authenticate with Azure DevOps Organization

You need to enable ADO integration and authenticate your ADO to retrieve and map test artifacts between two systems and submit test results to ADO. Do as follows:

In Katalon Studio, go to Project > Settings > Integrations > Azure DevOps:

  1. Select Enable Intergration to enable Authentication section for editting. xs

  2. Enter the required credentials for Authentication. Your credentials are encrypted by default for security.

    • Server URL: https://dev.azure.com/{yourorganization}
    • Personal Access Token: your Personal Access Token. We recommend you to create a Personal Access Token with full-access scopes.
  3. Select Encrypt authentication data for security assurance.

  4. Click Connect to verify whether Azure DevOps is connected successfully.

Configure the Integration

After successfully authenticating with ADO, you can select an ADO project among those you have access to in the drop-down list of Project.

To configure the integration, do as follows:

  1. Select a Project for submitting the test run.

    • Select a fetched project in the Project drop-down list.

      Click Fetch Project to fetch the latest projects list.

    • The Test Artifacts Mapping and Submission Option fieldsets are expanded automatically. You can customize the settings in each section.

  2. Conduct Test Artifacts Mapping.

    • In the Execution Status Mapping, map Katalon Studio's status with Azure DevOps's status to match the test results in Katalon Studio with the test outcomes in ADO.

    • In the Test Configuration Mapping, map the Execution OS/Device and Execution Browser/App configured to run the test in Katalon Studio with the Test Configurations in Azure DevOps.

      Depending on what OS and platform using to execute the test, Katalon Studio will get the corresponding Azure Test Configuration and use it as a filter for its configured test points for the test run submission.

      You can also Add or Remove item(s) to customize the settings.

      What is Test Configuration?

      A Test Configuration is a combination of configuration variable values. Your configuration variables could be, for example, operating system, browser, CPU type, database. A configuration might be "Windows 8 + 32-bit CPU" or "Windows 10 + 64-bit CPU." Learn more

      What is Test Point?

      A test point is a unique combination of a test case, test suite, configuration, and tester. Test cases by themselves are not executable. When you add a test case to a test suite, test point(s) are generated. Learn more

  3. Configure Submission Options.

    • Select a fetched test plan in the drop-down list, the test run is submitted to ADO automatically.

      Click Fetch Test Plans to fetch the latest test plans list.

    • Enter the required Test Run Name.

      If you want to specify the Build ID for test run submitted from Katalon Studio to ADO, enter the Definition ID. During runtime, Katalon Studio uses this definition ID to get the latest build ID and pass it to the submitted test run on ADO.

    • Decide when and what to submit test results.

      If you want to submit test results for ADO test case ID when there are multiple test points returned, select Submit test results for multiple test points with the same test case ID.

  4. Click Apply and Close to save your settings.

Map test cases between Katalon Studio and Azure DevOps

In Katalon Studio:

  1. Double-click on a Test Case to open the test case view.

  2. Select Integrations tab > specify the Test Cases ID(s) of ADO (to map to more than one ID, separate them by a comma).

  3. Click Verify to check whether the test case id exists in ADO for mapping the test case(s) > Save.

Submit test run and test results after execution

Ensure that you have already taken the stated steps.

When the execution finishes, the test run is created, and test results are uploaded automatically to ADO in the format specified as below:

Dynamically changing test run’s information in CLI

You can change the test plan ID, test run name, and build number of a test run by using the following command-line.

Requirements

  • An active Katalon Runtime Engine license.
  • Katalon Runtime Engine v8.0.

Katalonc Command-line Option

Description

Mandatory?

-adoPlanId=<testplan id>

Id of the test plan used for submitting test run(s).

N

-adoTestRunName="text"

Create test run(s) on ADO with the specified name.

N

--info -adoDefinitionID=<DefinitionID>

Get the latest completed Build ID of the specified Definition ID and pass it to Test Run properties on ADO..

N

Troubleshoot common issues

Error Solution
Cannot create Test Results for Azure DevOps Test Case ID= due to multiple Test Points returned. Please check Test Points with Id = ; or allow sending Test Results anyway in Project Settings.

Feedback