Create an automated test run

Requirements

• Make sure you have the Test Lead or Tester role. Go to roles or permissions for more information or learn how to assign a Project Admin here.

Automated test runs let you validate your application continuously and consistently without manual intervention. Instead of running test suites one by one from your local machine, you can define and execute test runs in TestOps, configure environments, schedule executions, and track results over time. This helps teams achieve faster feedback loops, reduce human error, and ensure product quality at scale.

Key benefits:

• Centralized execution: Run tests from TestOps rather than relying on local environments.
• Scalable environments: Choose between cloud-hosted or self-hosted agents depending on your team’s infrastructure.
• Flexible triggers: Run tests instantly or schedule them to align with CI/CD pipelines, sprint cycles, or release dates.
• Deeper insights: Link test runs with releases, sprints, and builds to track coverage and quality trends.

This document shows you how to create an automated test run.

Set up an automated test run

Create an automated test run by going to Execution > + Create > Create Automated Test Run.

1. Go to Executions. The Executions list appears by default.
2. Click + Create > Create Automated Test Run. The Create Automated Test Run page appears.

3. Input the following details:

• Test Run Name: The name of the test run
• Creator: This is set to the current user by default
• Release Version (optional): The associated release
• Sprint (optional): The associated sprint

4. Select a Test Run Type: Desktop & Mobile Browser, Mobile Native App or Web Services.

5. Click Select Tests to select from the list of test suites, then click Save when you’re done.

   

   Alternatively, you can use Command Line Mode to trigger test execution from TestOps test schedules:

   • Select your repository.
   • Katalon Commands: Select this option to execute tests with Katalon Studio. To generate a command, see: Command Syntax and Command Builder.
   • Generic Commands: Select this option to execute tests with other frameworks outside of Katalon Studio (e.g., Pytest).
   

6. Click the settings icon of each test suite to configure the environment you want to run your tests with.

   

note

• For standalone test suites, you can configure multiple environments. However, if a test suite is part of a test suite collection, you can assign only one environment per test suite.

7. In the Select Configurations dialog, select Cloud Hosted and specify the environment as follows:

Desktop & Mobile Browsers

For desktop browser environment, specify the OS, browser and browser version.

Mobile Native App

For mobile native app environment, select a mobile app first, then specify the device configuration.

Web Services

For Web Services environment, choose the local test environment where you can run or execute your tests from the list of available Agents.

Schedule

Schedule an automated test run

Click on Schedule to delay the test run to a later date or to create a recurring schedule for it.

The Schedule Test Run dialog box appears. Click Schedule.

Result

A notification confirms that you have successfully scheduled an automated test run.

Run Now

Run Now

Click on Run Now to instantly execute the automated test run.

tip

• You can terminate a running execution by clicking on the Terminate button on the upper right corner of its overview page.

Result

A notification confirms that you have successfully started an automated test run.

Advanced settings

After setting up your test run configurations, you can further customize how your tests behave through Advanced Settings. These options give you greater control over how tests are executed, including environment preferences, visual testing, and secure access to private or local applications.

• Visual Testing: Switch the Using Visual Testing toggle on to generate a baseline collection to compare with the captured images. For more details on Visual Testing, refer to Visual Testing.

• Katalon Runtime Engine Version: select a Katalon Runtime Engine version to run your test.

• Private/Local Testing: To customize how your tests connect to private or local environments, switch the Private/Local Testing toggle on.

  To run with TestCloud Tunnel, select Private/Local Testing, then choose a specific tunnel to execute the test or use Auto-selected tunnel. With the auto-select option, TestCloud will use the most available tunnel, prioritizing private tunnels over shared tunnels.

  In case your tests also perform API calls, check the Also Include API Calls option.

  Alternatively, you can opt for the TestCloud IPs Whitelisting option. This allows TestCloud to directly access your app without a tunnel connection.