A dynamic test suite is a test suite with multiple test cases added via a search query. You can dynamically add additional test cases while running the test suite. This article shows you how to configure and create a dynamic test suite.
- An active Katalon Studio Enterprise license. To learn more about activating your license, you can refer to this document here:Activate Katalon license.
Create a dynamic test suite
To create a new Test Suite, do as follows:
In the Test Explorer panel, right-click on the Test Suites folder > New > Dynamic Test Suite.
Fill in the name and the description (optional). Here, we name it Dynamic Test Suite 1 and leave the description blank.
Manage test case list
Enable the query provider
When you implement the dynamic test suite for the first time, the Query Provider is set to No query provider available by default.
To enable the query provider for the dynamic test suite, install one of the following plugins from the Katalon Store:
To activate the installed plugin, navigate to Account Settings in Katalon Studio and click Reload Plugin.
After you successfully activate the plugin, the Query Provider automatically applies the query syntax of the installed plugin. For example, after activating the Basic search for dynamic test suite plugin, the Query Provider is set to Built-in. You can now add test cases for dynamic test suite execution.
Add test cases to a dynamic test suite
There are two main ways to add test cases into a dynamic test suite. You can add test cases via a search query or the Query Builder function.
Add test cases via the search query.
To add test cases via the search query, manually input the search condition into the Query box. Then hit Preview to query out the matching test cases.
For example, to add the DDT at TC level test case into this dynamic test suite, you can input
id=(Test Cases/DDT at TC level)into the Query box. The matched test case appears in the test suite.
To learn more about the search query syntax, you can refer to this document: Search test cases.
Add test cases via the Query Builder function.
Query Builder provides a convenient way to create and run a query in Katalon Studio. To open the Query Builder function, in the dynamic test suite view, click Query Builder. The Query Builder dialog appears. Input your search criteria, then click Search.
The Query Builder dialog includes:
- Id: to search by the exact IDs of a test artifact.
- Name: to search by the name of a test artifact.
- Tag: to search by the tag linked to test artifacts.
- Comment: to search by the comments attached to test artifacts.
- Description: to search by the description associated with test artifacts.
- Every field in this Query Builder mode can be applied to search for all types of test artifacts such as test cases, test suites, folders, etc.
- You can only search for one keyword at a time when searching by tag, description, or comment.
For example, we want to filter for all test cases that have
Chromein their name, and add them into the dynamic test suite. After opening the Query Builder dialog, input
Chromeinto the Name text field. Then hit Search.
Katalon finds three test cases that have
Chromein their name.
Modify Execution Information
You can specify additional configurations for test suite execution by expanding the Execution Information section.
In the Implicit timeout section, you can decide the timeout period. This is how long Katalon Studio waits for a page to load. Choose one of the following options:
|Use default||Use the predefined value set in Project > Settings > Execution > Default wait for element timeout (in seconds).|
|User define (in seconds)||Set a custom waiting time. Input a value in seconds.|
Retry after executing all
Katalon allows you to rerun test cases in a failed test suite execution to identify flaky tests.
To do so, in the Retry after executing all text field, set the desired number of retry times. By default, this is set to
Choose one of the following options:
- Retry all executions: Retry all test cases when the test suite fails.
Retry failed executions only: Retry only failed test cases when the test suite fails.
For example, when a test suite execution fails, we want to rerun failed test cases in that test suite twice.
In the Retry after executing all checkbox, we input
2 in the text field, then we choose the Retry failed executions only option.
In this case, the test suite is executed three times in total. The first one is the main execution; the latter two are the retried executions when the first execution fails.
Katalon automatically generates reports after each test execution.
You have configured a mail server, an email template, and a default recipient list in Project > Settings > Email. To learn more about configuring email settings, you can refer to this document: Email settings.
Katalon allows you to add an additional list of recipients to receive reports in a specific dynamic test suite. To do so, in the Mail Recipients text field, input the email that you want to send the reports to.
The default recipient list in email settings and the additional list in a dynamic test suite can receive the same test reports after the test suite execution.
Execute a dynamic test suite
After adding test cases to the dynamic test suite, from the main toolbar, click Run. The test case is executed with the default browser defined in Project > Settings > Execution > Default execution.
Alternatively, you can choose the environment in the dropdown list next to Run.
The Job Progress bar is triggered automatically to show the progress while your test case/test suite is being executed.
You can also run the dynamic test suite in console mode. For detailed instructions on test execution in console mode, you can refer to this document: Command Builder.
After the test suite execution, to view your test reports, go to the Reports folder in the Test Explorer panel.
Alternatively, you can also view your reports and details in
<your-project-folder>/Reports. Katalon Studio supports exporting test reports into different formats, such as HTML, CSV, PDF, and JUnit. You can learn more about exporting reports here: Generate reports.
For real-time monitoring and better reporting capabilities, consider integrating your project with Katalon TestOps. Learn more about test result reports here: Upload Test Results to Katalon TestOps from Katalon Studio.
Perform data-driven testing in a dynamic test suite
You can perform data-driven testing by running a dynamic test suite with multiple data points.
- Katalon Studio Enterprise version 8.3.0 onwards.
Follow these steps:
Configure data binding for the test cases. See: Data-driven testing at the test case level.
Add the associated test cases to a dynamic test suite via a search query. See above: Add test cases to a dynamic test suite.
Hit Run to execute the dynamic test suite. Test cases in the dynamic test suite are then executed with the configured test data.
Verify the data binding and test execution results in Log Viewer.