Execute test suites in Katalon Studio

This guide shows you how to configure and perform test suite execution in Katalon Studio.

Modify execution information of a test suite

Before executing a dynamic test suite, you might want to modify the execution information, including timeout, retry, and email recipient list for test execution reports.

You can specify additional configurations for the test suite execution by expanding the Execution Information section as below:

execution information

Implicit Timeout

In Implicit timeout, you can decide the timeout period that Katalon Studio waits for a page to be loaded by choosing one of the following options:

Use default: Use the predefined default value
User define (in seconds): Set a custom waiting time. Input a value in seconds.

implicit timeout

Retry after executing all

retry after executing all

In Retry after executing all:

Retry All executions: Retry all executions when the test suite fails.
Retry Failed executions only: Retry only failed executions when the test suite fails.

Retry Failed Execution Immediately

By default, each test case is run only one time in a test suite execution. Suppose you want to rerun failed test cases several times to identify flaky tests before executing the next ones. You can set the desired number of retry times in Retry Failed Execution Immediately. The failed test case will be rerun immediately until they pass or all retries are used up. If all rerun attempts fail, Katalon Studio marks that test case as Failed and proceeds with the rest in the test suite.

Important:

You need an active Katalon Studio Enterprise license.

Consolidate Reports

From version 8.1.0 onwards, Katalon Studio automatically consolidates reports in JUnit, HTML, PDF, and CSV format with one final test result for a test case. Browser-based or window-based videos are recorded for runs and reruns of a test case.

Usage example

In this example, we have a test suite with five test cases. We set Retry Failed Execution Immediately for one and two times. When we run the test suite and a test fails, Katalon Studio immediately reruns the problematic test case until it passes or the maximum number of retries is reached.

Example 1: Execute the test suite with a Retry limit of 1:

You can see the test suite status once the test suite finishes executing. Because test case 3 is marked as Failed, we open the Result tab and expand test case 3 to investigate.
test case 3 failed twice, once during the main run and once as a retried run. Only one retried run was allowed.
At this point, Katalon Studio logged its final result as Failed and continued to execute test case 4 and 5 with the same logic.
You can view how many test cases in the test suite were executed and their final status in the Summary tab. In this use case, the test suite has five total test cases; test cases 1, 2, and 4 pass while 3 and 5 fail.
Example 2: Execute the test suite with a Retry limit of 2:
When we set the retry limit to 2, five test cases passed. Test cases 3 and 5 failed at the main run and first rerun but passed at the second try.
Since test cases 3 and 5 did not succeed on the first try but still succeed after two tries, their result pattern is intermittent. This is a clear sign of test flakiness and can be further investigated.

Execute a test suite

To execute a test suite, follow these steps:

Open a test suite.
Select an environment to run the test case from Run command of the main toolbar.
If you click on the Run button, the test case is executed using the default browser defined in Execution Settings.
- Browsers: Select one of the supported browsers, including Chrome, Firefox, IE (for Windows only), Safari, or Edge Chromium, Chrome (headless), and Firefox (headless).
- Mobile Devices: Select one of the listed devices (Android or iOS).
  
  Before executing your test, check if you have set up the environment for mobile testing for Windows or for macOS.
  
  Note:
  If there is no device listed, please make sure the Developer Mode on the phone is turned on, try to unplug, and reconnect several times until you are prompted to accept/trust this device.
- Windows: Select this option if you are executing tests on a desktop application.
- Remote: Make sure you have set up the default configuration for the remote environment in project settings. Refer to Introduction to Desired Capabilities for more details.
- Custom: Make sure you have set up the default configuration for the remote environment in project settings. Refer to Custom Execution for more details. When you have set up your custom environment, select it from the drop-down list.