Test Suite

A Test Suite is a collection of multiple different or duplicate test cases.

Create a new Test Suite

To create a new Test Suite, do as follows:

  1. From the menu bar, select File > New > Test Suite

  2. Fill in the name of the test suite and the description (optional).

  3. Click OK when you are done.

Alternatively, you can create a new Test Suite after creating a Test Case. In a Test Case, click the Add to test suite button.

You can choose to add that test case to an existing or a new test suite.

Modify Execution Information

You can specify additional configurations for test suite execution by expanding the Execution Information section, as below:

Implicit timeout

In Implicit timeout, you can decide the timeout period that Katalon Studio waits for a page to be loaded by choosing one of the following options:

  • Use default: Use the predefined default value
  • User define (in seconds): Set a custom waiting time. Input a value in seconds.

Retry

Using the Retry feature, you can configure when and how many times Katalon retries an execution of a Test Suite before the Test Suite finishes executing.

retry after executing all

Retry Failed Execution Immediately

By default, each Test Case is run only one time in a Test Suite execution. Suppose you want to rerun failed Test Cases several times to identify flaky tests before executing the next ones. You can set the desired number of retry times in Retry Failed Execution Immediately. The failed test case will be rerun immediately until they pass or all retries are used up. If all rerun attempts fail, Katalon Studio marks that Test Case as Failed and proceeds with the rest in the Test Suite.

Requirement

  • An active Katalon Studio Enterprise license

Consolidate Reports

From version 8.1.0 onwards, Katalon Studio automatically consolidates reports in JUnit, HTML, PDF, and CSV format with one final test result for a Test Case. Browser-based or window-based videos are recorded for runs and reruns of a Test Case.

Usage example

In this example, we have a Test Suite with five Test Cases. We set Retry Failed Execution Immediately for one and two times. When we run the Test Suite and a test fails, Katalon Studio immediately reruns the problematic test case until it passes or the maximum number of retries is reached.

  • Example 1: Execute the Test Suite with a Retry limit of 1:

    You can see the Test Suite status once the Test Suite finishes executing. Because Test Case 3 is marked as Failed, we open the Result tab and expand Test Case 3 to investigate.

    Test Case 3 failed twice, once during the main run and once as a retried run. Only one retried run was allowed.

    At this point, Katalon Studio logged its final result as Failed and continued to execute Test Case 4 and 5 with the same logic.

    You can view how many Test Cases in the Test Suite were executed and their final status in the Summary tab. In this use case, the Test Suite has five total Test Cases, Test Cases 1, 2 and 4 pass while 3 and 5 fail.

  • Example 2: Execute the Test Suite with a Retry limit of 2:

    When we set the retry limit to 2, five Test Cases passed. Test Cases 3 and 5 failed at the main run and first rerun but passed at the second try.

    Since Test Cases 3 and 5 did not succeed on the first try but still succeed after two tries, their result pattern is intermittent. This is a clear sign of test flakiness and can be further investigated.

Retry After Executing All

In Retry after executing all:

  • Retry All executions: Retry all executions when the Test Suite fails.
  • Retry Failed executions only: Retry only failed executions when the Test Suite fails.

Mail Recipients

You can add a list of recipients who will receive execution reports via email once the test suite finishes its execution.

email recipients

There's also another list of recipients who will receive all the reports from Katalon by default. Refer to Emails Settings for more details.

Manage Test Case List

There are several ways to add Test Cases into Test Suites. You can drag and drop the Test Case into Test Suites or using the Test Suite editor to manually add the test case.

Test Cases can be duplicated in the same Test Suite.

Open a test suite, then select the option to add Add Test Case from the command toolbar.

All test cases in Katalon Studio are displayed in the Test Case Browser dialog for you to select your preferred options. The selected test cases are added to the test case list accordingly.

The checkbox at the end of the test case row is checked by default. It means that the test case is executed when you run a test suite.

Dynamic Test Suite (Dynamic Test Cases List)

Dynamic Test Suite is a test suite in which a collection of multiple test cases are added to NOT in a manual way but via a search query. This feature only works when you have already installed the plugin that defines the querying syntax. In case there is no installed plugin, the Query Provider is set to “No query provider available.” by default.

dynamic test suite

Currently, there are three plugins from Katalon Store, which enable this feature by providing query search function and returning any matched test cases or test suites of the query statement.

After installing one of those plugins, go to Katalon Studio and click Reload Plugins. Learn more about how to reload plugins.

Query Provider: The query syntax standard of a plugin that is currently applied. For example, when you successfully install the Basic Search For Dynamic Test Suite plugin, the query syntax standard becomes "Built-in".

Query: This search box allows you to input the query syntax manually. For example, id=(Test Cases/Simple Examples/Katalon Shop/Custom-keyword samples/Order and check out a single product).

Query Builder provides a convenient way to create and run a query in Katalon Studio.

  • Id: to search by the exact IDs of a test artifact.
  • Name: to search by the name of a test artifact.
  • Tag: to search by the tag linked to test artifacts.
  • Comment: to search by the comments attached to test artifacts.
  • Description: to search by the description associated with test artifacts.

Note: Search criteria are applied to all test artifacts, including test case, test suite, folder, and etc.

Preview: View the results after having the searching query.

search querry

Execute a Test Suite

Open a test case/test suite, then select an environment to run the test case from Run command of the main toolbar.

If you click on the Run button, the test case is executed using the default browser defined in Execution Settings.

  • Browsers: Select one of the supported browsers, including Chrome, Firefox, IE (for Windows only), Safari, or Edge Chromium, Chrome (headless), and Firefox (headless).

  • Mobile Devices: Select one of the listed devices (Android or iOS).

    Before executing your test, check if you have set up the environment for mobile testing for Windows or for macOS.

    Troubleshooting

    If there is no device listed, please make sure the Developer Mode on the phone is turned on, try to unplug, and reconnect several times until you are prompted to accept/trust this device.

  • Windows: Select this option if you are executing tests on a desktop application.

  • Remote: Make sure you have set up the default configuration for the remote environment in project settings. Refer to Introduction to Desired Capabilities for more details.

  • Custom: Make sure you have set up the default configuration for the remote environment in project settings. Refer to Custom Execution for more details. When you have set up your custom environment, select it from the drop-down list.

The Job Progress is triggered automatically to show the progress while your test case/test suite is being executed.

See also Execute the test case using console mode.

Submit and view test results on Katalon TestOps

You can centralize test results including logs and attachments on Katalon TestOps. Besides test results management, Katalon TestOps will also provide analytics to improve the quality of your test cases.

Feedback