Automated Test Run

Last updated: October 2026

This document shows you how to create, configure, and execute an Automated Test Run in Katalon True Platform.

requirements

• Test artifacts are available in Katalon True Platform. Choose one of these methods:
  • Upload a Katalon Studio project or ZIP file to a local repository: Upload test cases through ZIP upload
  • Sync a connected repository through Git integration: GitHub integration, GitLab integration, Bitbucket integration, or Azure Repos integration
• After the sync is complete, verify that your test suites or test suite collections appear in the project repository before creating the run.

After syncing your projects and test artifacts to Katalon True Platform, you can execute and orchestrate automated test runs there, then review results over time for high-level insights into quality trends, releases, and execution health.

Try this with Katalon AI

You can also ask Katalon AI Assistant to create an automated test run. Include the test run name, selected tests, profile, the environment configuration for each test suite, and whether the run should start now or follow a schedule, then review the generated setup before execution.

See Katalon AI Assistant.

Create an automated test run

By default, you execute an automated test in UI mode.

Follow these steps to create and execute a run in UI mode:

1. Go to Executions > +Create > Create Automated Test Run.

2. Fill in details:

   • Test Run Name
   • Creator (default: current user)
   • Release/Sprint (data from this run will be included in relevant analytics).
   • Additional information: add more information in existing fields and custom fields, for later analytics purpose. See Customizable Fields to learn how to create these fields and assign tags for later analytics.

3. Click Select Tests → pick test suites and test suite collections from the repository.

4. Select a Profile.

5. You can mix environments within a single run. In the test suite configuration table, click the gear icon to specify the environment for each suite. For details, see Test suite configurations.

6. Click Run Now to create and execute the test. You are redirected to the Test Run Details page, where you can monitor the execution.

Test suite configurations

Each test suite in a run can use a different environment. For example, one suite in a Test Suite Collection can create orders in a desktop browser, while another verifies them in a mobile app. You can run these suites sequentially or in parallel as part of the same run.

To configure a suite, click the gear icon in its row to open the Select Configurations dialog. Both standalone test suites and each member suite inside a Test Suite Collection are configured individually this way. The dialog has three environment options:

• Desktop & Mobile Browser
  • For functional tests on web applications.
  • Choose a Cloud-hosted or Self-hosted agent, then pick OS + browser combinations (e.g., Chrome on Windows, Safari on macOS).
  • Mobile browsers offer the platform browser only: Chrome when you select an Android device, Safari when you select an iOS device.
• Mobile Native App
  • For testing Android/iOS apps.
  • Select or upload the Mobile Application first — the app drives the device list. An .apk file filters the list to Android devices; an .ipa file filters it to iOS devices. Then choose the OS version, device type, and device.
• Web Services
  • For testing APIs and backend services.
  • Uses the fixed default configuration TestCloud Linux; it is not editable.

API and CLI compatibility

Payloads that set a testRunType value keep working: the value is applied as the default configuration for test suites that do not have an explicit configuration of their own.

Execution agent

A test run in Katalon True Platform requires you to configure how, where, and with what parameters the tests will execute. These configurations ensure that your automation adapts to your infrastructure, scales effectively, and produces reliable results.

Katalon True Platform test runs always rely on an execution agent—the runtime environment responsible for running your tests. The right choice depends on convenience, compliance requirements, and the infrastructure available to your team.

Self-hosted Agents	Cloud-hosted Agents (Test Execution - Cloud)
Installed on infrastructure you manage, such as a VM or private cloud. Best for secure or restricted environments, or when you need tests to run inside your own network. Supports both UI Mode (Studio artifacts) and Command Line Mode (Katalon Test Execution - Local/other frameworks).	Managed by Katalon. Scales instantly across browsers, operating systems, and devices. Best for fast setup and zero maintenance.

Rule of Thumb

• Use Cloud-hosted for convenience and scalability.
• Use Self-hosted/Katalon Runtime Engine for compliance, local dependencies, or CI/CD integrations.

Auto-select an available agent

For Self-hosted runs, Katalon True Platform can automatically select an available agent each time the run triggers.

1. In Select Configurations, select Self-hosted, then choose Auto-select available agent from the Agent list.
2. Optionally, click + Add condition to filter by OS, Agent name, or Hostname, then click Save. An agent must match all conditions.

Each occurrence uses one matching agent. If none is active at trigger time, that occurrence is cancelled with a reason. The next scheduled occurrence checks for matching agents again.

Execution profile

Profiles (from Katalon Studio Execution Profiles) let you parameterize test data and environments:

• Store variables such as URLs, credentials, API keys.
• Allow the same test suite to run across Dev, QA, Staging, or Prod-like setups.
• Selected in UI Mode via dropdown, or passed in CLI with executionProfile.

When prompting Katalon AI Assistant to create the run, include the exact profile name you want to use.

Environment

For standalone test suites, you can configure multiple environments. For a Test Suite Collection (TSC), Katalon True Platform uses the profiles, environments, and execution modes configured in Katalon Studio. If you change a TSC configuration in Katalon True Platform, it no longer synchronizes with the configuration in Katalon Studio.

Parallel execution

Parallel execution runs independent test cases concurrently across available Test Execution - Cloud sessions, helping reduce the total execution time for large test suites. Enable Parallelize Test Cases when your test cases do not depend on one another. For setup instructions and limitations, see Parallel test case execution.

Advanced settings

Use Advanced Settings for optional capabilities such as visual testing, selecting a Git branch or runtime version, accessing private applications, and pushing results to Azure DevOps.

Setting	When to use it
Visual Testing	Detect visual changes by comparing screenshots against a baseline.
Branch	Run tests on a specific Git branch with a Cloud-hosted Agent.
Katalon Runtime Engine Version	Select a runtime version that is compatible with your tests.
Private/Local Testing	Access applications in a private or local network.
Azure DevOps result push	Send test execution results to Azure DevOps.

• Visual Testing: Enable this option to verify that your application's UI looks correct in addition to functioning correctly. On the first run, Katalon creates a baseline screenshot collection. Future runs compare screenshots against the baseline and flag visual differences in Katalon True Platform, where you can approve a change to update the baseline or reject it as a regression. For details, see Visual Testing.

• Branch: This setting applies only to Cloud-hosted Agents (Test Execution - Cloud). If your project is connected to a Git repository, select the branch to use for this run. The setting defaults to the integrated branch. For details, see Run tests on a specific Git branch.

• Katalon Runtime Engine Version: Select the version required for compatibility with your tests.

• Private/Local Testing: Enable this option to connect tests to private or local environments. Then:

  • Select a specific Cloud Execution Tunnel, or use Auto-selected tunnel to select an available tunnel automatically. Private tunnels take priority over shared tunnels.

  • If your tests make API calls, select Also Include API Calls.

  • Alternatively, select Cloud Execution IPs Whitelisting to allow Cloud Execution to access your application directly without a tunnel.

    For details, see Local testing.

• Azure DevOps result push: Use this setting to send completed test run results to an Azure DevOps Test Plan. In Advanced Settings > ALM, configure the following fields:

1. ALM integration: Select a configured Azure DevOps integration.
2. ALM test plan: Select the destination test plan. This field is optional when you select Manual push only.
3. ALM push mechanism: Choose when Katalon sends the results:
   • Manual push only: Review a completed run and push its results from the Test Run Details page.
   • Automatically push all runs: Push every completed run automatically.
   • Automatically push passed runs only: Push a run only when every test in the run passes.

For integration requirements, manual push instructions, and result verification, see Push test results to Azure DevOps.

Create a command-line test run

Create automated tests in Command Line Mode when:

• You need more flexibility, e.g., passing custom arguments.

• You run non-Katalon frameworks (e.g., Pytest, JUnit).

• You want consistency with existing CI/CD scripts.

This mode supports any framework and is highly customizable. You should be familiar with command-line interfaces and execution syntax.

You can also ask Katalon AI Assistant to help draft the command configuration, especially when you already know the repository, framework, execution profile, and custom arguments you need.

Rule of thumb

Start with UI mode for Katalon-managed tests, use Command Line mode when integrating with external tools or advanced CI/CD workflows.

To execute tests in this mode, instead of selecting tests in the UI, you configure commands:

1. On the Create Automated Test Run screen → choose Command Line Mode.

2. Select repository.

3. Provide commands:

   • Katalon Commands: For Katalon Studio/Katalon Runtime Engine tests. Use Command Builder for syntax help.
   • Generic Commands: For non-Katalon frameworks (e.g., Pytest).

Schedule and manage test runs

Run now or schedule

After setting up a test run, you can execute it immediately or schedule daily or weekly runs to identify patterns and potential anomalies.

If you want Katalon AI Assistant to create a scheduled automated run, mention the start time, recurrence, and selected tests directly in your prompt.

Run Now	Schedule
- Immediately triggers execution. - Best for ad-hoc validation or smoke checks.	- Delays or repeats the execution. - Best for regression, nightly runs, or aligning with sprint/release cadence.

Manage scheduled runs

After you schedule automated runs, use the Manage Automated Schedule screen to monitor them and perform bulk actions such as activating, deactivating, or deleting schedules.

1. Click Manage Automated Schedule to edit schedules.

2. In this view, you can deactivate/activate, or delete the run. Editing in bulk is supported.

3. To view and adjust details of a scheduled test run, select the test run. You'll see a test run details page with three tabs:

   • The Test Runs tab is opened by default, showing past runs of this test and their separate results.

   • The Overview tab displays details of the test's past performance.

   • The Configurations tab shows you how this test was configured for the recent runs.

4. In here, you can click Edit to re-configure profile, environment, schedule, and some additional settings for this test to execute in the next run.

When you save schedule edits, Katalon True Platform applies the updated values to future runs and marks the schedule as changed.