Integration with Azure DevOps Test Plans

From version 8.1.0 onwards, you can submit a test run with Release information to Azure DevOps.

Katalon Studio can natively integrate with the Azure Test Plans service of Azure DevOps (ADO). This integration helps you:

  1. Establish a connection between a Katalon Studio project and an ADO project.
  2. Easily map ADO Test Cases to automated Test Cases in Katalon Studio.
  3. Automatically submit test runs and results to ADO with release information (release stage and release ID), execution logs, reports, and images for analysis.

Requirements

  • Katalon Studio version 8.0.0 onwards.
  • An active Katalon Studio Enterprise license.
  • Azure Test Plans already configured.

Enable the Integration and Perform Authentication

To retrieve your test artifacts and create new test results directly on Azure DevOps, you need to integrate and authenticate your project with Azure Server. In Katalon Studio, go to Project > Settings > Integrations > Azure DevOps. The Azure DevOps dialog appears.

  1. Check the Enable Integration box. The Authentication area can now be edited.

  2. Enter your credentials. Your credentials are encrypted by default.

    • Server URL: https://dev.azure.com/{yourorganization}
    • Personal Access Token: your Personal Access Token. We recommend you create a Personal Access Token with full-access scopes. See Microsoft document: Use personal access tokens and Scopes.
  3. Click Connect. If the connection to the Azure server is successful, the Test Connection Succeeded line appears.

    Azure DevOps dialog

Configure the Integration

Step 1: Select a Project for submitting test runs and results

After successfully authenticating your project with the Azure Server, in the dropdown list of Project, select an ADO project that you have access to.

To retrieve the latest projects list, click Fetch Project.

After you select a project, the Test Artifacts Mapping and Submission Option fieldsets automatically expand.

test artifacts mapping

Step 2: Map Test Artifacts between two systems

You can Add or Remove one or more items in each section at your convenience.

In the Execution Status Mapping: Match test results in Katalon Studio with test outcomes in ADO.

In the Test Configuration Mapping: This step reduces the number of test results created for each mapped test case. You need to pair Execution OS/Device and Execution Browser/App in Katalon Studio with Test Configurations retrieved from Azure Test Plans.

Test configuration mapping

What is a Test Point?

A Test Point is a unique combination of a test case, test suite, configuration, and tester. Test cases by themselves are not executable. When you add a test case to a test suite, a test point is generated. To learn more about Test Points, see Microsoft document: Execute tab.

What is a Test Configuration?

A Test Configuration is a combination of configuration variable values containing operating system information, browser, CPU type, database. For example, Windows 8 + 32-bit CPU or Windows 10 + 64-bit CPU. To learn more about the Test Configuration, see Microsoft document: Test different configurations.

Step 3. Configure Submission Options

  1. Select a test plan for the test run to be submitted. To retrieve the latest test plans list, click Fetch Test Plans.

    Submission Options 8.1.0
  2. Name the test run.

  3. To add Build and Release Information to test runs, specify Build Definition ID and Release Definition ID respectively (Release Definition ID was introduced in 8.1.0).

    During runtime, Katalon Studio uses these pipeline definition IDs to get and pass the latest Build and Release to the corresponding properties of a test run.

    Fill Pipeline Defition ID
  4. Decide what attachments to be sent together with a test run.

  5. With the associated Test Case ID and Test Configurations, more than one Test Point might be returned. These Test Points share the same Test Case ID and Test Configurations, but can still differ depending on the Test Suite and Tester. In this case, to decide whether Katalon Studio submits test results regardless of the number of Test Points or not, select Submit test results for multiple test points with the same test case ID.

  6. To save your settings, click Apply and Close.

Map test cases between Katalon Studio and Azure DevOps

In Azure Test Plans:

View Test Case ID on its URL.

In Katalon Studio:

  1. Open a test case.

  2. Select the Integrations tab.

  3. Input one or more Test Case IDs of ADO. You can map one test case ID in Katalon Studio with many test case IDs on ADO. IDs are separate by commas.

  4. To check whether the test case ID is valid, click Verify.

  5. Save your setting.

    verify ADO test case

Auto-Submit test run and test results after execution

After a test suite execution finishes, Katalon Studio automatically adds a new test run and test results to the specified test plan.

new item

Dynamically change the information of a test run in Command-line Option

You can change the test plan ID, test run name, build and release definition IDs of a test run by using the following command-line options:

Requirements

  • An active Katalon Runtime Engine license.
  • Katalon Runtime Engine version 8.0.0 onwards.
  • To use -adoReleaseDefID, Katalon Runtime Engine version 8.1.0 onwards is required.

Katalonc Command-line Option

Description

Mandatory?

-adoPlanId=<testplan ID>

ID of the test plan used for submitting test run(s).

N

-adoTestRunName="text"

Create test run(s) on ADO with the specified name.

N

--info -adoBuildDefId =<Definition ID of Build Pipeline>

Get the latest completed Build ID of the specified Build Definition ID and pass it to the corresponding Test Run property on ADO.

N

--info -adoReleaseDefID=<Definition ID of Release Pipeline>

Get the latest Release ID and its stage based on the specified Definition ID of Release Pipeline and pass them to the corresponding Test Run properties on ADO. (Runtime Engine 8.1.0+ is required)

N

Troubleshoot common issues

Error Solution
Cannot create Test Results for Azure DevOps Test Case ID= due to multiple Test Points returned. Check Test Points with ID = ; or allow sending Test Results anyway in Project Settings.

Feedback