Dynamic test suite in Katalon Studio

A dynamic test suite is a test suite with multiple test cases added via a search query. You can dynamically add additional test cases while running the test suite. This article shows you how to configure and create a dynamic test suite.

Requirements:

  • An active Katalon Studio Enterprise license. To learn more about activating your license, you can refer to this document here: Activate Katalon license.

Create a dynamic test suite

To create a new Test Suite, do as follows:

  1. In the Test Explorer panel, right-click on the Test Suites folder > New > Dynamic Test Suite.

    Create a dynamic test suite
  2. Fill in the name and the description (optional). Here, we name it Dynamic Test Suite 1 and leave the description blank.

    Create a dynamic test suite
  3. Click OK.

Manage test case list

Enable the query provider

When you implement the dynamic test suite for the first time, the Query Provider is set to No query provider available by default.

dynamic test suite

To enable the query provider for the dynamic test suite, install one of the following plugins from the Katalon Store:

To activate the installed plugin, navigate to Account Settings in Katalon Studio and click Reload Plugin.

After you successfully activate the plugin, the Query Provider automatically applies the query syntax of the installed plugin. For example, after activating the Basic search for dynamic test suite plugin, the Query Provider is set to Built-in. You can now add test cases for dynamic test suite execution.

Results after installing plugins

Add test cases to a dynamic test suite

There are two main ways to add test cases into a dynamic test suite. You can add test cases via a search query or the Query Builder function.

  1. Add test cases via the search query.

    To add test cases via the search query, manually input the search condition into the Query box. Then hit Preview to query out the matching test cases.

    For example, to add the DDT at TC level test case into this dynamic test suite, you can input id=(Test Cases/DDT at TC level) into the Query box. The matched test case appears in the test suite.

    To learn more about the search query syntax, you can refer to this document: Search test cases.

    Results after searching query

    Click the image to enlarge it.

  2. Add test cases via the Query Builder function.

    Query Builder provides a convenient way to create and run a query in Katalon Studio. To open the Query Builder function, in the dynamic test suite view, click Query Builder. The Query Builder dialog appears. Input your search criteria, then click Search.

    query dialog

    The Query Builder dialog includes:

    • Id: to search by the exact IDs of a test artifact.
    • Name: to search by the name of a test artifact.
    • Tag: to search by the tag linked to test artifacts.
    • Comment: to search by the comments attached to test artifacts.
    • Description: to search by the description associated with test artifacts.

    Notes:

    • Every field in this Query Builder mode can be applied to search for all types of test artifacts such as test cases, test suites, folders, etc.
    • You can only search for one keyword at a time when searching by tag, description, or comment.

    For example, we want to filter for all test cases that have Chrome in their name, and add them into the dynamic test suite. After opening the Query Builder dialog, input Chrome into the Name text field. Then hit Search.

    Input Chrome into the Query Builder dialog

    Katalon finds three test cases that have Chrome in their name.

    search query

    Click the image to enlarge it.

Modify Execution Information

You can specify additional configurations for test suite execution by expanding the Execution Information section.

Open the execution information

Implicit timeout

In the Implicit timeout section, you can decide the timeout period. This is how long Katalon Studio waits for a page to load. Choose one of the following options:

Set implicit timeout
Options Description
Use default Use the predefined value set in Project > Settings > Execution > Default wait for element timeout (in seconds).
User define (in seconds) Set a custom waiting time. Input a value in seconds.

Retry after executing all

Katalon allows you to rerun test cases in a failed test suite execution to identify flaky tests.

To do so, in the Retry after executing all text field, set the desired number of retry times. By default, this is set to 0.

Choose one of the following options:

  • Retry all executions: Retry all test cases when the test suite fails.
  • Retry failed executions only: Retry only failed test cases when the test suite fails.
retry after executing all

For example, when a test suite execution fails, we want to rerun failed test cases in that test suite twice.

In the Retry after executing all checkbox, we input 2 in the text field, then we choose the Retry failed executions only option.

retry after executing all

In this case, the test suite is executed three times in total. The first one is the main execution; the latter two are the retried executions when the first execution fails.

retry after executing all

Click the image to enlarge it.

Katalon automatically generates reports after each test execution.

retry after executing all

Mail Recipients

Requirements:

  • You have configured a mail server, an email template, and a default recipient list in Project > Settings > Email. To learn more about configuring email settings, you can refer to this document: Email settings.

Katalon allows you to add an additional list of recipients to receive reports in a specific dynamic test suite. To do so, in the Mail Recipients text field, input the email that you want to send the reports to.

email recipients

Notes:

The default recipient list in email settings and the additional list in a dynamic test suite can receive the same test reports after the test suite execution.

Execute a dynamic test suite

After adding test cases to the dynamic test suite, from the main toolbar, click Run. The test case is executed with the default browser defined in Project > Settings > Execution > Default execution.

The Run dropdown list

Alternatively, you can choose the environment in the dropdown list next to Run.

The Run dropdown list

The Job Progress bar is triggered automatically to show the progress while your test case/test suite is being executed.

The Run dropdown list

You can also run the dynamic test suite in console mode. For detailed instructions on test execution in console mode, you can refer to this document: Command Builder.

Test Reports

After the test suite execution, to view your test reports, go to the Reports folder in the Test Explorer panel.

Reports

Click the image to enlarge it.

Alternatively, you can also view your reports and details in <your-project-folder>/Reports. Katalon Studio supports exporting test reports into different formats, such as HTML, CSV, PDF, and JUnit. You can learn more about exporting reports here: Generate reports.

Notes:

Perform data-driven testing in a dynamic test suite

You can perform data-driven testing by running a dynamic test suite with multiple data points.

Requirements:

  • Katalon Studio Enterprise version 8.3.0 onwards.

Follow these steps:

  1. Configure data binding for the test cases. See: Data-driven testing at the test case level.

  2. Add the associated test cases to a dynamic test suite via a search query. See above: Add test cases to a dynamic test suite.

    Test cases added by query

    Click the image to enlarge it.

  3. Hit Run to execute the dynamic test suite. Test cases in the dynamic test suite are then executed with the configured test data.

  4. Verify the data binding and test execution results in Log Viewer.

    Test cases added by query

    Click the image to enlarge it.

See also

Feedback