Automated Test Run

Last updated: August 2026

This document shows you how to create, configure, and execute an Automated Test Run in Katalon True Platform.

requirements

• Test artifacts are available in Katalon True Platform. Choose one of these methods:
  • Upload a Katalon Studio project or ZIP file to a local repository: Upload test cases through ZIP upload
  • Sync a connected repository through Git integration: GitHub integration, GitLab integration, Bitbucket integration, or Azure Repos integration
• After the sync is complete, verify that your test suites or test suite collections appear in the project repository before creating the run.

After syncing your projects and test artifacts to Katalon True Platform, you can execute and orchestrate automated test runs there, then review results over time for high-level insights into quality trends, releases, and execution health.

Get started: Configure and execute an automated test in UI mode

By default, you execute an automated test in UI mode.

Follow these steps to create a run with UI Mode:

1. Go to Executions > +Create > Create Automated Test Run.

2. Fill in details:

   • Test Run Name
   • Creator (default: current user)
   • Release/Sprint (data from this run will be included in relevant analytics).
   • Additional information: add more information in existing fields and custom fields, for later analytics purpose. See Customizable Fields to learn how to create these fields and assign tags for later analytics.

3. Select Test Run Type: Desktop & Mobile Browser, Mobile Native App, or Web Services.

4. Click Select Tests → pick test suites and test suite collections from the repository.

5. Select a Profile.

6. Configure environment configurations depending on the type:

   • Standalone suites: You can configure multiple environments.
   • Test Suite Collection (TSC):

     • TSC configurations are defined by the user in Katalon Studio.

     • When you select a TSC and schedule it in Katalon True Platform, the system will automatically apply the configurations that were previously set in Katalon Studio (including test suites, profiles, environments, and execution modes).

     • However, if you manually update any configuration of the TSC in Katalon True Platform, that TSC’s configurations will no longer sync with the one in Katalon Studio.

7. Click Run Now to create and execute the test. You will be redirected to the Test Run Details page, for monitoring purpose (learn more about managing test runs at ...).

Create and Execute an automated test run with Katalon AI

You can also ask Katalon AI Assistant to help create an automated test run. This is useful when you already know the run requirements and want the assistant to prepare the setup faster in Katalon True Platform.

To get the best result, include the key run requirements in your prompt, such as the test run name, test run type, selected tests, execution profile, environment, release or sprint, and whether the run should start immediately or follow a schedule.

1. Open Katalon AI Assistant in Katalon True Platform.
2. Prompt the assistant to create an automated test run and include the required details.
3. Open the Test Run Details page to monitor execution and review results over time.

Basic configurations for automated tests

This section walks you through basic configurations for an automated test run. See Advanced configurations section below to explore more configurations you may find useful for your project.

Select test run types

When creating a test run, you must choose a Test Run Type. Choosing the correct type ensures that your tests are executed in the right runtime with the right dependencies. Each type corresponds to a different kind of testing environment:

If you ask Katalon AI Assistant to create the run, specify the Test Run Type clearly in your prompt so the assistant can guide you to the right configuration path.

• Desktop & Mobile Browser

  • For functional tests on web applications.
  • Runs across OS + browser combinations (e.g., Chrome on Windows, Safari on macOS).

• Mobile Native App

  • For testing Android/iOS apps.
  • Requires uploading the app (APK/IPA) and configuring devices/emulators.

• Web Services

  • For testing APIs and backend services.
  • Runs in your chosen local/cloud agent environment, validating requests and responses.

Self-hosted vs Cloud-hosted agents

A test run in Katalon True Platform requires you to configure how, where, and with what parameters the tests will execute. These configurations ensure that your automation adapts to your infrastructure, scales effectively, and produces reliable results.

Katalon True Platform test runs always rely on an execution agent—the runtime environment responsible for running your tests. The right choice depends on convenience, compliance requirements, and the infrastructure available to your team.

Self-hosted Agents	Cloud-hosted Agents (Test Execution - Cloud)
- Installed on infrastructure you manage, such as a VM or private cloud. - Best for secure or restricted environments, or when you need tests to run inside your own network. - Supports both UI Mode (Studio artifacts) and Command Line Mode (Katalon Test Execution - Local/other frameworks).	- Managed by Katalon. - Scales instantly across browsers, OS, and devices. - Best for fast setup and zero maintenance.

Rule of Thumb:

• Use Cloud-hosted for convenience and scalability.

• Use Self-hosted/Katalon Runtime Engine for compliance, local dependencies, or CI/CD integrations.

Profiles

Profiles (from Katalon Studio Execution Profiles) let you parameterize test data and environments:

• Store variables such as URLs, credentials, API keys.

• Allow the same test suite to run across Dev, QA, Staging, or Prod-like setups.

• Selected in UI Mode via dropdown, or passed in CLI with executionProfile.

When prompting Katalon AI Assistant to create the run, include the exact profile name you want to use.

Parallelize Test Cases

For Test Cases:

This toggle enables execution of individual test cases in parallel within a test suite or across multiple suites, maximizing usage of available sessions for speed and efficiency. Unlike Test Suite Collection (TSC) execution modes, this feature works at the test case level.

Test Execution - Cloud optimizes resource allocation while maintaining a buffer of available sessions. For example, if you have 10 test suites each containing 10 test cases, Test Execution - Cloud continuously analyzes the queue and distributes test cases across your available licenses until all are used.

Note: Your test cases must be independent to avoid unpredictable failures. To learn more, refer to this document.

For Test Suite Collections:

When Parallelize Test Cases is toggled on, the collection will default to Parallelize Test Cases. To choose the two run mode below, you have to untoggle Parallelize Test Cases.

• Parallel mode: Runs multiple test suites in parallel, limited by your available Test Execution - Cloud session quota.

• Sequential mode: Runs one test suite after another.

Command Line Mode

Execute Automated tests in Command Line Mode when:

• You need more flexibility, e.g., passing custom arguments.

• You run non-Katalon frameworks (e.g., Pytest, JUnit).

• You want consistency with existing CI/CD scripts.

This mode supports any framework, and is highly customizable. You should be familiar to CLI and execution syntax.

You can also ask Katalon AI Assistant to help draft the command configuration, especially when you already know the repository, framework, execution profile, and custom arguments you need.

Rule of thumb

Start with UI mode for Katalon-managed tests, use Command Line mode when integrating with external tools or advanced CI/CD workflows.

To execute tests in this mode, instead of selecting tests in the UI, you configure commands:

1. On the Create Automated Test Run screen → choose Command Line Mode.

2. Select repository.

3. Provide commands:

   • Katalon Commands: For Katalon Studio/Katalon Runtime Engine tests. Use Command Builder for syntax help.
   • Generic Commands: For non-Katalon frameworks (e.g., Pytest).

Schedule Automated Tests

As you finish setting up a test case, you can choose to execute them immediately, or schedule them for daily/weekly runs to see patterns and detect potential anomalies.

If you want Katalon AI Assistant to create a scheduled automated run, mention the start time, recurrence, and selected tests directly in your prompt.

Run Now	Schedule
- Immediately triggers execution. - Best for ad-hoc validation or smoke checks.	- Delays or repeats the execution. - Best for regression, nightly runs, or aligning with sprint/release cadence.

After you schedule automated runs, use the Manage Automated Schedule screen to monitor and update and manage them in bulk (deactivate, delete,…).

1. Click the Manage Automated Schedule button to edit tests.

2. In this view, you can deactivate/activate, or delete the run. Editing in bulk is supported.

3. To view and adjust details of a scheduled test run, select the test run. You'll see a test run details page with three tabs:

   • The Test Runs tab is opened by default, showing past runs of this test and their separate results.

   • The Overview tab displays details of the test's past performance.

   • The Configurations tab shows you how this test was configured for the recent runs.

4. In here, you can click Edit to re-configure profile, environment, schedule, and some additional settings for this test to execute in the next run.

When you save schedule edits, Katalon True Platform applies the updated values to future runs and marks the schedule as changed.

Advanced Configurations

After setting up your test run configurations, you can further customize how your tests behave through Advanced Settings. These options give you greater control over how tests are executed, including environment preferences, visual testing, and secure access to private or local applications.

• Visual Testing: Helps your application’s UI looks correct, not just functions correctly.
  When enabled in a test run. Enable this when you need to catch UI layout or design issues across browsers, devices, and environments in addition to functional checks.

  • A baseline screenshot collection is created (first run).
  • Future runs compare screenshots against the baseline to detect visual changes.
  • Differences are flagged in Katalon True Platform, where you can approve changes (update baseline) or reject them (log as regressions).
  • For more information, please refer to the Visual Testing document for more information.

• Branch: Applies to Cloud-hosted Agents (Test Execution - Cloud) only. If your project is connected to a Git repository, choose the branch to run this test against. The dropdown defaults to your integrated branch, and your selection applies to this run only. For more information, please refer to the Run tests on a specific Git branch document.

• Katalon Runtime Engine Version: Choose the version to ensure compatibility.

• Private/Local Testing: To customize how your tests connect to private or local environments, switch the Private/Local Testing toggle on.

  To run with Cloud Execution Tunnel, select Private/Local Testing, then choose a specific tunnel to execute the test or use Auto-selected tunnel. With the auto-select option, Cloud Execution will use the most available tunnel, prioritizing private tunnels over shared tunnels. For more information, please refer to the Cloud Execution document.

  In case your tests also perform API calls, check the Also Include API Calls option.

  Alternatively, you can opt for the Cloud Execution IPs Whitelisting option. This allows Cloud Execution to directly access your app without a tunnel connection.