Skip to main content

Configure Azure DevOps Test Plans integration in Katalon Studio

Katalon Studio can natively integrate with the Azure Test Plans service of Azure DevOps (ADO). This integration helps you automatically submit test runs and results to ADO with release information (release stage and release ID), execution logs, reports, and images for analysis.

Before you proceed with the integration, below are key concepts you need to learn:

What is a test point? A test point is a unique combination of a test case, test suite, configuration, and tester. Test cases by themselves are not executable. A test point is generated when you add a test case to a test suite. To learn more about test points, see the Microsoft document: Execute tab.

What is a test configuration? A test configuration combines configuration variable values containing operating system information, browser, CPU type, database. For example, Windows 8 + 32-bit CPU or Windows 10 + 64-bit CPU. To learn more about the test configuration, see the Microsoft document: Test different configurations.

Requirements​

Before you get started, ensure that you have the following requirements:

  • An active Katalon Studio Enterprise license.
  • Azure DevOps Server 2022 installed.
  • A team collection set up.
  • A Personal Access Token with full access permissions.

Enable Azure DevOps Integration in Katalon Studio​

To retrieve your test artifacts and create new test results directly on Azure DevOps, you need to integrate and authenticate your project with Azure Server first.

Follow the steps to configure and authenticate your project with Azure Server in Katalon Studio:

  1. In Katalon Studio, go to Project > Settings > Integrations > Azure DevOps.

    1. In the dialog, check the Enable Integration box. The Authentication area can now be edited.
    2. Enter your credentials. Your credentials are encrypted by default.

    • Server URL

      • For Azure DevOps Services (cloud): Use the format https://dev.azure.com/{'{'}yourorganization{'}'}.

      • For Azure DevOps Server 2022 (on-premises): Use the format http(s)://{'{'}instance{'}'}/{'{'}collection{'}'}.

        Examples:

        • http://10.3.6.153/DefaultCollection/
        • https://10.3.6.153/DefaultCollection/
        • https://ec2amaz-q46tu5e/DefaultCollection/

    • Personal Access Token: Your Personal Access Token. We recommend you create a Personal Access Token with full-access scopes. See Microsoft document: Use personal access tokens and Scopes.

    1. Click Connect. If the connection to the Azure server is successful, the Test Connection Succeeded message appears:

      Test Connection Successful

  2. After successfully authenticating your project with the Azure Server, navigate to the Project and Test Plan sections and select from the dropdown an ADO project and one or more test plans that you have access to.

    Select Project and Test Plan

    To retrieve the latest projects list, click Fetch Project.

    Fetch Projects

  3. Navigate to the Test Artifacts Mapping and Submission Option section and map test artifacts between Katalon Studio and ADO.

    1. In Execution Status Mapping, match test results in Katalon Studio with test outcomes in ADO.

    2. In Test Configuration Mapping, pair Execution OS/Device, Execution Browser/App, and Execution Profile in Katalon Studio with Test Configuration in Azure DevOps. This step is to map test cases with test points in ADO for result submission.

      Test Artifacts Mapping

      You can click Add or Remove to add or remove one or more items in each line item at your convenience.

  4. To submit test results to ADO, follow these steps:

    1. Select a test plan for the test run to be submitted. To retrieve the latest test plans list, click Fetch Test Plans. By default, the Automatically submit test run option is checked.

    2. Name the test run.

      Automatically submit test run and name test run

    3. [Optional] If you want to add build and release information to test runs, specify Build Definition ID and Release Definition ID respectively (Release Definition ID was introduced in 8.1.0).

      During runtime, Katalon Studio uses these pipeline definition IDs to get and pass the latest Build and Release to the corresponding properties of a test run.

    4. [Optional] Decide what attachments to be sent together with a test run.

    5. [Optional] With the associated test case ID and test configurations, more than one test point might be returned. These test points share the same test case ID and test configurations, but can still differ depending on the test suite and Tester. In this case, to decide whether Katalon Studio submits test results regardless of the number of test points returned, select Submit test results for multiple test points with the same test case ID.

      Select submission options

      If multiple test configurations are assigned, multiple test points will be created. Select Submit test results for multiple test points with the same test case ID to send test results to Azure DevOps Server 2022.

  5. To save your settings, click Apply and Close.

Map test cases between Katalon Studio and Azure DevOps​

To map test cases between Katalon Studio and Azure DevOps, do as follows:

  1. In Azure Test Plans, open a test suite and note the test case IDs you want to map.

    View test case IDs in Azure Test Plans

  2. In Katalon Studio, open a test case and go to the Integration tab.

  3. Enter one or more test case IDs of ADO. You can map a single test case ID in Katalon Studio to multiple ADO test cases by separating IDs with commas.

    Example: 12345, 67890

    note

    Test cases can be retrieved from test plans you configured in Project Settings in Katalon Studio.

  4. To check whether the test case IDs are valid, click Verify.

    Save your ADO test case

  5. Click Save to apply the mapping.

    note

    ADO test case IDs must belong to a test plan configured in Project Settings in Katalon Studio.

You can also dynamically assign test plan IDs, test run names, build IDs, and release definitions using command-line options when executing tests. See Azure DevOps integration arguments.

Once a mapped Katalon Studio test case is executed:

  • A new test run is created in Azure DevOps.

  • After test execution, Katalon Studio automatically updates Azure DevOps by creating a new test run and uploading results to the mapped test cases.

    ADO test results

  • The Automation status in the Steps section is updated to Automated.

    Automation status in ADO Steps section

  • The Automated Test Name (the name of the executed Katalon Studio test case) and** Automated Test Storage** (the ID of the test case) fields in the Associated Automation section are populated only if they are initially empty.

    Automation status in ADO Steps section

Expected Behavior for Associated Automation Fields

As of version 10.3.0, Katalon Studio now respects existing mappings in Azure DevOps by preserving the values in the Associated Automation fields of ADO test cases. Specifically:

  • When two ADO test cases are mapped to the same Katalon test case, both will initially reflect the same automation reference after execution.
  • If you later re-map one of those ADO test cases to a different Katalon test case and re-run the test, Katalon will not overwrite existing values in the Associated Automation fields unless they are empty.
  • This behavior ensures that previously linked test cases retain their automation references, avoiding unintended overwrites and allowing more controlled mapping changes.

Troubleshooting​

Invalid or duplicate test case IDs are entered​

When you enter a test case ID, you might encounter an invalid or duplicate test case ID error in the Event Log.

Azure DevOps enforces unique test case IDs across all test plans and suites, ensuring consistent identification and management of test cases throughout your projects. For more information, see the following ADO documentation.

To solve this issue, you can check the test case ID in question in the Event Log. To differentiate tests with similar objectives across different plans or releases, we recommend creating new test cases rather than reusing the same ID.

Cannot create test results for Azure DevOps Test Case ID due to multiple Test Points returned​

When you cannot upload test results to Azure DevOps, you might encounter the following error in the Event Log: Cannot create test results for Azure DevOps Test Case ID=<test-case-ID> due to multiple Test Points returned.

To solve this issue, you can check the IDs of the returned test points in the Event Log to readjust the test configuration, or allow sending test results anyway in project settings.

Learn more with Katalon Academy​

Go to the following course to learn how to integrate ADO with Katalon Studio: Integrating Katalon Studio with Azure DevOps.

Was this page helpful?