This guide shows you how to configure and perform test suite execution in Katalon Studio.
Modify execution information of a test suite
Before executing a dynamic test suite, you might want to modify the execution information, including timeout, retry, and email recipient list for test execution reports.
You can specify additional configurations for the test suite execution by expanding the Execution Information section as below:
In Implicit timeout, you can decide the timeout period that Katalon Studio waits for a page to be loaded by choosing one of the following options:
- Use default: Use the predefined default value
- User define (in seconds): Set a custom waiting time. Input a value in seconds.
Retry after executing all
In Retry after executing all:
- Retry All executions: Retry all executions when the test suite fails.
- Retry Failed executions only: Retry only failed executions when the test suite fails.
Retry Failed Execution Immediately
By default, each test case is run only one time in a test suite execution. Suppose you want to rerun failed test cases several times to identify flaky tests before executing the next ones. You can set the desired number of retry times in Retry Failed Execution Immediately. The failed test case will be rerun immediately until they pass or all retries are used up. If all rerun attempts fail, Katalon Studio marks that test case as Failed and proceeds with the rest in the test suite.
You need an active Katalon Studio Enterprise license.
From version 8.1.0 onwards, Katalon Studio automatically consolidates reports in JUnit, HTML, PDF, and CSV format with one final test result for a test case. Browser-based or window-based videos are recorded for runs and reruns of a test case.
In this example, we have a test suite with five test cases. We set Retry Failed Execution Immediately for one and two times. When we run the test suite and a test fails, Katalon Studio immediately reruns the problematic test case until it passes or the maximum number of retries is reached.
Example 1: Execute the test suite with a Retry limit of 1:
You can see the test suite status once the test suite finishes executing. Because test case 3 is marked as Failed, we open the Result tab and expand test case 3 to investigate.
test case 3 failed twice, once during the main run and once as a retried run. Only one retried run was allowed.
At this point, Katalon Studio logged its final result as Failed and continued to execute test case 4 and 5 with the same logic.
You can view how many test cases in the test suite were executed and their final status in the Summary tab. In this use case, the test suite has five total test cases; test cases 1, 2, and 4 pass while 3 and 5 fail.
Example 2: Execute the test suite with a Retry limit of 2:
When we set the retry limit to 2, five test cases passed. Test cases 3 and 5 failed at the main run and first rerun but passed at the second try.
Since test cases 3 and 5 did not succeed on the first try but still succeed after two tries, their result pattern is intermittent. This is a clear sign of test flakiness and can be further investigated.
Execute a test suite
- Open a test suite.
- Select an environment to run the test case from Run command of the main toolbar.
If you click on the Run button, the test case is executed using the default browser defined in Execution Settings.
- Browsers: Select one of the supported browsers, including Chrome, Firefox, IE (for Windows only), Safari, or Edge Chromium, Chrome (headless), and Firefox (headless).
Mobile Devices: Select one of the listed devices (Android or iOS).Note:
If there is no device listed, please make sure the Developer Mode on the phone is turned on, try to unplug, and reconnect several times until you are prompted to accept/trust this device.
Windows: Select this option if you are executing tests on a desktop application.
- Remote: Make sure you have set up the default configuration for the remote environment in project settings. Refer to Introduction to Desired Capabilities for more details.
- Custom: Make sure you have set up the default configuration for the remote environment in project settings. Refer to Custom Execution for more details. When you have set up your custom environment, select it from the drop-down list.
The Job Progress is triggered automatically to show the progress while your test case/test suite is being executed.