Terminate execution conditionally

In this article, you will learn how to terminate a test execution automatically by configuring a maximum number of test failures in manual view or via the command-line option, with examples using common use cases.

Important:

• A Katalon Runtime Engine License. See Katalon licensing.

Consider using the condition to terminate execution when a set of tests is mature and takes hours to finish. Once a significant number of tests fail, they may fail for the same cause. This feature helps you save time, provide early feedback, and avoid the execution of unnecessary test cases.

Set a maximum number of test failures

Note:

What is a test failure?

A test failure is a failed test case or test iteration in an execution.

• 1 test case fails = 1 test failure
• 1 retried test case fails = 1 test failure
• 1 test iteration (test case + data row) fails = 1 test failure
• 1 retried test iteration fails = 1 test failure

With Command Builder

Note:

For detailed instruction on how to run a test execution in Console mode, see Command Builder.

Go to Katalon Studio and set a maximum number of test failures.

1. In the toolbar, select Build CMD (Build Command).

2. In the Generate Command for Console Mode > Execution Configurations, check the box Terminate the execution once the total number of test failures reaches this threshold. In the text field, input your desired maximum number of test failures.

   
   
   

   Note:

   The maximum test cases failure must be greater than 0 and a natural number (1, 2, 3...). Otherwise, Katalon Studio will not start the execution.

3. After you are done with the configuration, click Generate command. A command containing the -maxFailedTests option is generated.

   Copy the generated command to use in Console mode.

   
   
   

   You can also execute the test with the property file. To save the property file in the Execution Configurations dialog, click Generate property file > Save.

With Command-line option

Use the command-line option: -maxFailedTests=T, where T is the maximum number of test failures.

See Katalonc command-line option for the list of common command-line options supported.

Report Incomplete Test Suite

When a test suite has at least 1 test case Not Run, the final status of that test suite is marked Incomplete.

For test suites marked Incomplete or Not started yet, Katalon Studio automatically sends attachments to TestOps after execution ends. The reports include the execution logs and JUnit Reports.

Test suites marked Incomplete or Not started yet and the attachments of the test suite execution cannot be uploaded to 3rd-party tools (qTest, JIRA, Slack, Azure DevOps Test Plans, TestRail).

Troubleshoot

When a test suite or test suite collection is terminated, you can find the test suite or test suite collection reference ID and the maximum number of test failures in the execution log.

This applies to either sequential or parallel mode execution. You can check which test execution was terminated and why. You can also see the report in the manual view of Katalon Studio. See also View and Customize Execution Log.

Common Use Cases

Below are three example use cases to illustrate how the parameter works:

• In the case of a test suite execution.
• In the case of a test suite collection executed in sequential mode.
• In the case of a test suite collection executed in parallel mode.

Terminate Test Suite Execution

In this section, we outline an example execution of a test suite with this parameter.

• Given that you have a test suite that has 10 test cases.

  
  
  

• You want to stop the test suite after 4 test failures. Set T = 4.

  Configure in Command Builder

  
  
  

  
  
  

  Execute in Console Mode

• The test suite is terminated once the number of failures becomes 4. The execution ends, and the rest test cases do not run.

  
  
  

• Katalon Studio generates a report in JUnit, HTML, PDF, and CSV format. The report does not show information for test cases that were not run.

  
  
  

  Note:

  See also Test Suite and Test Suite Collection Reports.

Terminate Test Suite Collection Executed In Sequential Mode

• Given that you have 5 test suites, each test suite has 10 test cases. You want to stop the test suite collection after 20 test failures. Set T = 20.

• The test suite collection executes in sequential mode and is terminated once the number of failures becomes 20. The rest test cases in the terminated test suite and the rest test suite do not run.

  
  
  

• Katalon Studio generates a report in JUnit, HTML, PDF, and CSV format. The report does not show information for test cases and test suites that were not run.

  Note:

  Learn more about how to create and execute a Manage test suite collections in Katalon Studio.

Terminate Test Suite Collection Executed In Parallel Mode

• Given that you have a test suite collection that has 5 test suites with 3 parallel instances. You want to stop the test suite collection after 20 test failures. Set T = 20.

  Note:

  • Instances is the number of test suites you set to run at the same time.

  • T is the maximum number of test case failures (threshold).

• When the test suite begins execution, the first 3 test suites run at the same time with T = 20 in each test suite.

• The test suite collection is terminated once the total number of failures becomes 20. The rest test cases in the terminated test suite and the rest test suite do not run. However, the currently running test suites will not be terminated.

  
  
  

• Katalon Studio generates a report in JUnit, HTML, PDF, and CSV format. The report does not show information for test cases and test suites that were not run.