Skip to main content

Configure Azure DevOps Test Plans integration in Katalon Studio

Katalon Studio can natively integrate with the Azure Test Plans service of Azure DevOps (ADO). This integration helps you automatically submit test runs and results to ADO with release information (release stage and release ID), execution logs, reports, and images for analysis.

Before you proceed with the integration, below are key concepts you need to learn:

What is a test point? A test point is a unique combination of a test case, test suite, configuration, and tester. Test cases by themselves are not executable. A test point is generated when you add a test case to a test suite. To learn more about test points, see the Microsoft document: Execute tab.

What is a test configuration? A test configuration combines configuration variable values containing operating system information, browser, CPU type, database. For example, Windows 8 + 32-bit CPU or Windows 10 + 64-bit CPU. To learn more about the test configuration, see the Microsoft document: Test different configurations.

Requirements​

Before you get started, ensure that you have the following requirements:

  • An active Katalon Studio Enterprise license.
  • Azure DevOps Server 2022 installed.
  • A team collection set up.
  • A Personal Access Token with full access permissions.

Enable Azure DevOps Integration in Katalon Studio​

To retrieve your test artifacts and create new test results directly on Azure DevOps, you need to integrate and authenticate your project with Azure Server first.

Follow the steps to configure and authenticate your project with Azure Server in Katalon Studio:

  1. In Katalon Studio, go to Project > Settings > Integrations > Azure DevOps.

    1. In the dialog, check the Enable Integration box. The Authentication area can now be edited.
    2. Enter your credentials. Your credentials are encrypted by default.
    • Server URL

      • For Azure DevOps Services (cloud): Use the format https://dev.azure.com/{'{'}yourorganization{'}'}.

      • For Azure DevOps Server 2022 (on-premises): Use the format http(s)://{'{'}instance{'}'}/{'{'}collection{'}'}.

        Examples:

        • http://10.3.6.153/DefaultCollection/
        • https://10.3.6.153/DefaultCollection/
        • https://ec2amaz-q46tu5e/DefaultCollection/
    • Personal Access Token: Your Personal Access Token. We recommend you create a Personal Access Token with full-access scopes. See Microsoft document: Use personal access tokens and Scopes.
    1. Click Connect. If the connection to the Azure server is successful, the Test Connection Succeeded message appears. Authenticate Azure DevOps successfully

  2. After successfully authenticating your project with the Azure Server, navigate to Project and select from the dropdown an ADO project that you have access to.

    To retrieve the latest projects list, click Fetch Project.

    After you select a project, the Test Artifacts Mapping and Submission Option fieldsets automatically expand.

  3. Map test artifacts between Katalon Studio and ADO.

    1. In the Execution Status Mapping, match test results in Katalon Studio with test outcomes in ADO.

    2. In the Test Configuration Mapping, you need to pair Execution OS/Device, Execution Browser/App, and Execution Profile in Katalon Studio with Test Configurations retrieved from Azure Test Plans. This step is to map test cases with test points in ADO for result submission.

      You can click Add or Remove to add or remove one or more items in each section at your convenience.

      Test Artifacts Mapping

  4. To submit test results to ADO, follow these steps:

    1. Select a test plan for the test run to be submitted. To retrieve the latest test plans list, click Fetch Test Plans. By default, the Automatically submit test run option is checked.

    2. Name the test run.

    3. [Optional] If you want to add build and release information to test runs, specify Build Definition ID and Release Definition ID respectively (Release Definition ID was introduced in 8.1.0).

      During runtime, Katalon Studio uses these pipeline definition IDs to get and pass the latest Build and Release to the corresponding properties of a test run.

    4. [Optional] Decide what attachments to be sent together with a test run.

    5. [Optional] With the associated test case ID and test configurations, more than one test point might be returned. These test points share the same test case ID and test configurations, but can still differ depending on the test suite and Tester. In this case, to decide whether Katalon Studio submits test results regardless of the number of test points returned, select Submit test results for multiple test points with the same test case ID.

      Select submission options

      If multiple test configurations are assigned, multiple test points will be created. Select Submit test results for multiple test points with the same test case ID to send test results to Azure DevOps Server 2022.

  5. To save your settings, click Apply and Close.

Map test cases between Katalon Studio and Azure DevOps​

To map test cases between Katalon Studio and Azure DevOps, do as follows:

  1. In Azure Test Plans, open a test suite to view your test case IDs.

    View test case IDs in Azure Test Plans

  2. In Katalon Studio, open a test case.

  3. Select the Integration tab.

  4. Input one or more test case IDs of ADO. You can map one test case ID in Katalon Studio with many test case IDs on ADO. IDs are separate by commas.

  5. To check whether the test case ID is valid, click Verify.

  6. Save your test case. Save your ADO test case

You have successfully integrated Azure Test Plans with Katalon Studio. After a test suite execution finishes, Katalon Studio automatically adds a new test run and test results to the specified test plan.

ADO test results

You can also dynamically change the test plan ID, test run name, build and release definition IDs of a test run by using the command-line option. See Azure DevOps integration arguments.

Troubleshooting​

When you cannot upload test results to Azure DevOps, you might encounter the following error in the Event Log:

Cannot create test results for Azure DevOps Test Case ID=<test-case-ID> due to multiple Test Points returned.

To solve this issue, you can check the IDs of the returned test points in the Event Log to readjust the test configuration, or allow sending test results anyway in project settings.

Learn more with Katalon Academy​

Go to the following course to learn how to integrate ADO with Katalon Studio: Integrating Katalon Studio with Azure DevOps.

Was this page helpful?