Configure Azure DevOps Test Plans integration in Katalon Studio

Katalon Studio can natively integrate with the Azure Test Plans service of Azure DevOps (ADO). This integration helps you automatically submit test runs and results to ADO with release information (release stage and release ID), execution logs, reports, and images for analysis.

Before you proceed with the integration, below are key concepts you need to learn:

What is a test point? A test point is a unique combination of a test case, test suite, configuration, and tester. Test cases by themselves are not executable. A test point is generated when you add a test case to a test suite. To learn more about test points, see the Microsoft document: Execute tab.

What is a test configuration? A test configuration combines configuration variable values containing operating system information, browser, CPU type, database. For example, Windows 8 + 32-bit CPU or Windows 10 + 64-bit CPU. To learn more about the test configuration, see the Microsoft document: Test different configurations.

Requirements

Before you get started, ensure that you have the following requirements:

• An active Katalon Studio Enterprise license.
• Azure DevOps Server 2022 installed.
• A team collection set up.
• A Personal Access Token with full access permissions.

Enable Azure DevOps Integration in Katalon Studio

To retrieve your test artifacts and create new test results directly on Azure DevOps, you need to integrate and authenticate your project with Azure Server first.

Follow the steps to configure and authenticate your project with Azure Server in Katalon Studio:

1. In Katalon Studio, go to Project > Settings > Integrations > Azure DevOps.

2. In the dialog, check the Enable Integration box. The Authentication area can now be edited.

   1. Enter your credentials. Your credentials are encrypted by default.

   • Server URL:

     • For Azure DevOps Services (cloud): Use the format https://dev.azure.com/{'{'}yourorganization{'}'}.

     • For Azure DevOps Server 2022 (on-premises): Use the format http(s)://{'{'}instance{'}'}/{'{'}collection{'}'}.

       Examples:

       • http://10.3.6.153/DefaultCollection/
       • https://10.3.6.153/DefaultCollection/
       • https://ec2amaz-q46tu5e/DefaultCollection/

   • Personal Access Token: Your Personal Access Token. We recommend you create a Personal Access Token with full-access scopes. See Microsoft document: Use personal access tokens and Scopes.

   3. Click Connect. If the connection to the Azure server is successful, the Test Connection Succeeded message appears:

      

3. After successfully authenticating your project with the Azure Server, navigate to the Project and Test Plan sections and select from the dropdown an ADO project and one or more test plans that you have access to.

   

   To retrieve the latest projects list, click Fetch Project.

   

4. Navigate to the Test Artifacts Mapping and map test artifacts between Katalon Studio and ADO.

   1. In Execution Status Mapping, match test results in Katalon Studio with test outcomes in ADO.

   2. In Test Configuration Mapping, pair Execution OS/Device, Execution Browser/App, and Execution Profile in Katalon Studio with Test Configuration in Azure DevOps. This step is to map test cases with test points in ADO for result submission.

      

      You can click Add or Remove to add or remove one or more items in each line item at your convenience.

5. Navigate to the Submissions Options section and configure the following:

   1. Select a test plan for the test run to be submitted. Name the test run. By default, the Automatically submit test run option is checked.

      

   2. To add build and release information to test runs, enter the Build Definition ID or Release Definition ID (Release Definition ID was introduced in 8.1.0), then click Verify to confirm the value

      During runtime, Katalon Studio uses these pipeline definition IDs to get and pass the latest Build and Release to the corresponding properties of a test run.

      

   3. [Optional] Select Include attachments when submitting test run and decide what attachments to be sent together with a test run. You can select more than one:

      • Applicable to: Choose whether to include attachments For all test executions or Only for test executions with failed test case(s).
      • Attachment types: Check the boxes next to the attachment formats you want to receive, such as Log file(s), Screenshot(s), HTML Report, PDF Report, and CSV Report.
      

   4. [Optional] With the associated test case ID and test configurations, more than one test point might be returned. These test points share the same test case ID and test configurations, but can still differ depending on the test suite and Tester. In this case, to decide whether Katalon Studio submits test results regardless of the number of test points returned, select Submit test results for multiple test points with the same test case ID.

      

      If multiple test configurations are assigned, multiple test points will be created. Select Submit test results for multiple test points with the same test case ID to send test results to Azure DevOps Server 2022.

   5. [Optional] Select Override existing automation fields if you want to override or update Automated test name and Automated test storage fields regardless of existing values.

      

6. To save your settings, click Apply and Close.

Map test cases between Katalon Studio and Azure DevOps

To map test cases between Katalon Studio and Azure DevOps, do as follows:

1. In Azure Test Plans, open a test suite and note the test case IDs you want to map.

   

2. In Katalon Studio, open a test case and go to the Integration tab.

3. Enter one or more test case IDs of ADO. You can map a single test case ID in Katalon Studio to multiple ADO test cases by separating IDs with commas.

   Example: 12345, 67890

   note

   Test cases can be retrieved from test plans you configured in Project Settings in Katalon Studio.

4. To check whether the test case IDs are valid, click Verify.

   

5. Click Save to apply the mapping.

   note

   ADO test case IDs must belong to a test plan configured in Project Settings in Katalon Studio.

You can also dynamically assign test plan IDs, test run names, build IDs, and release definitions using command-line options when executing tests. See Azure DevOps integration arguments.

Once a mapped Katalon Studio test case is executed:

• A new test run is created in Azure DevOps.

• After test execution, Katalon Studio automatically updates Azure DevOps by creating a new test run and uploading results to the mapped test cases.

  

• The Automation status in the Steps section is updated to Automated.

  

• As of Katalon Studio 10.3.1, the Automated Test Name (the name of the executed Katalon Studio test case) and the Automated Test Storage (the ID of the test case) fields in the Associated Automation section depend on whether the Override existing automation fields is enabled or disabled.

  

  • If the Override existing automation fields is enabled, Katalon Studio fills in the fields when they are empty, and overwrites existing values with the latest executed test case details.
  • If the Override existing automation fields is disabled, Katalon Studio fills in the fields only when they are empty. Existing values are preserved and not overwritten.

Troubleshooting

Invalid or duplicate test case IDs are entered

When you enter a test case ID, you might encounter an invalid or duplicate test case ID error in the Event Log.

Azure DevOps enforces unique test case IDs across all test plans and suites, ensuring consistent identification and management of test cases throughout your projects. For more information, see the following ADO documentation.

To solve this issue, you can check the test case ID in question in the Event Log. To differentiate tests with similar objectives across different plans or releases, we recommend creating new test cases rather than reusing the same ID.

Cannot create test results for Azure DevOps Test Case ID due to multiple Test Points returned

When you cannot upload test results to Azure DevOps, you might encounter the following error in the Event Log: Cannot create test results for Azure DevOps Test Case ID=<test-case-ID> due to multiple Test Points returned.

To solve this issue, you can check the IDs of the returned test points in the Event Log to readjust the test configuration, or allow sending test results anyway in project settings.

Learn more with Katalon Academy

Go to the following course to learn how to integrate ADO with Katalon Studio: Integrating Katalon Studio with Azure DevOps.