Integration with Azure DevOps Test Plans (Pre-release)

Important

This is just a pre-release version that is not ready for production use. You're highly recommended to test its usability only.

In the release of version 8.0.0, expected to be shipped by the end of April, Katalon Studio can be natively integrated with Azure DevOps (ADO) - Test Plans.

Download Katalon Studio v8.0.0.rc3.

This integration will support you to do the following jobs:

  • When you have manual Test Cases in ADO, you want to map them to a corresponding automated Test Case in Katalon Studio to know which Test Cases are automated.
  • When you execute the integrated Test Cases in Katalon Studio, you want the Test Run to be created accordingly with the test execution reports and results uploaded to its corresponding Test Run in ADO so that you can get the test status and have sufficient materials for debugging.

In terms of features, you can:

  • Enable, Authenticate and Configure the integration in Project Settings.
  • Associate Test Cases between two systems.
  • When a Test Suite/ Test Suite Collection execution finishes, Katalon will create Test Run and submit Test Results to ADO automatically.
  • Dynamically change test plan ID, test run name, and build number of a test run in CLI.

This document introduces how the built-in Azure DevOps Integration feature looks like and how to use it in Katalon Studio.

Requirements

  • Katalon Studio version 8.0.
  • An active Katalon Studio Enterprise license.
  • Set up Azure DevOps.

Enable the Integration and Authenticate Azure DevOps Organization

You need to enable ADO integration and authenticate your ADO to retrieve and map test artifacts between two systems and submit test results to ADO. Do as follows:

In Katalon Studion, go to Project > Settings > Integrations > Azure DevOps:

  1. Select Enable Intergration to enable Authentication section for editting.

  2. Enter the required credentials for Authentication. Your credentials are encrypted by default for security.

    • Server URL: https://dev.azure.com/{yourorganization}
    • Personal Access Token: your Personal Access Token. We recommend you to create a Personal Access Token with full-access scopes.
  3. Select Encrypt authentication data for security assurance.

  4. Click Connect to verify whether Azure DevOps is connected successfully.

Configure the Integration

After successfully authenticating with ADO, you can select an ADO project among those you have access to in the drop-down list of Project.

To configure the integration, do as follows:

  1. Select a Project for submitting the test run.

    • Select a fetched project in the Project drop-down list.

    • The Test Artifacts Mapping and Submission Option fieldsets are expanded automatically. You can customize the settings in each section.

      Click Fetch Project to fetch the latest projects list.

  2. Conduct Test Artifacts Mapping

    • In the Execution Status Mapping, map Katalon Studio's status with Azure DevOps's status to match the test results in Katalon Studio with the test outcomes in ADO.

    • In the Test Configuration Mapping, map the Execution OS/Device and Execution Browser/App configured to run the test in Katalon Studio with the Test Configurations in Azure DevOps.

      Depending on what OS and platform using to execute the test, Katalon Studio will get the corresponding Azure Test Configuration and use it as a filter for its configured test points for the test run submission.

      You can also Add or Remove item(s) to customize the settings.

      What is Test Configuration?

      A Test Configuration is a combination of configuration variable values. Your configuration variables could be, for example, operating system, browser, CPU type, database. A configuration might be "Windows 8 + 32-bit CPU" or "Windows 10 + 64-bit CPU." Learn more

      What is Test Point?

      A test point is a unique combination of a test case, test suite, configuration, and tester. Test cases by themselves are not executable. When you add a test case to a test suite, test point(s) are generated. Learn more

  3. Configure Submission Options

    • Select a fetched test plan in the drop-down list, the test run is submitted to ADO automatically.

      Click Fetch Test Plans to fetch the latest test plans list.

    • If you want to submit test results for ADO test case ID when there are multiple test points returned, select Send test results when ... to enable test run details for editing > enter the required Test Run Name.

    • Decide when and what to submit test results.

  4. Click Apply and Close to save your settings.

Map test cases between Katalon Studio and Azure DevOps

In Katalon Studio:

  1. Double-click on a Test Case to open the test case view.

  2. Select Integrations tab > specify the Test Cases ID(s) of ADO (to map to more than one ID, separate them by a comma).

  3. Click Verify to check whether the test case id exists in ADO for mapping the test case(s) > Save.

Submit test run and test results after execution

Ensure that you have already taken the stated steps.

When the execution finishes, the test run is created, and test results are uploaded automatically to ADO in the format specified as below:

Dynamically changing test run’s information in CLI

You can change the test plan ID, test run name, and build number of a test run by using the following command-line.

Requirements

  • An active Katalon Runtime Engine license.
  • Katalon Runtime Engine v8.0.

Katalonc Command-line Option

Description

Mandatory?

-adoPlanId=<testplan id>

Id of the test plan used for submitting test run(s).

N

-adoTestRunName="text"

Create test run(s) on ADO with the specified name.

N

--info -adoBuildNumber="text"

Pass the build number to Test Run properties on ADO.

N

Troubleshoot common issues

Error Solution
Cannot create Test Results for Azure DevOps Test Case ID= due to multiple Test Points returned. Please check Test Points with Id = ; or allow sending Test Results anyway in Project Settings.

Feedback