Configure Smart Tags

This page explains how you can configure and use smart tags to quickly gain insights on automated test runs in TestOps.

requirements

• Only Project Admins can configure Smart Tag settings, including Time Window, Minimum Executions, and thresholds for Flaky and Slow tags.
• Once it is configured, all project members can view Smart Tags.

Overview

Smart tags are how TestOps flag troubling test cases based on their past run data. Smart tags include:

• Flaky
• New Failure
• Always Fail
• Slow

All Smart Tags are automatically labeled on executed test cases, once they fit criteria for each. You can also customize configurations for Smart Tags, such as Time Windows, Minimum Executions, Flaky tag, and Slow tag.

Flaky Tag

This tag appears on tests with unreliable results.

TestOps calculates Probabilistic Flakiness Score (PFS) based on test run data, and compares the consistency of test results (passed/failed) against this score to determine a test case as “stable”, “potentially flaky”, or “highly flaky”.

For more details on calculations for the PFS, see Probabilistic Flakiness Score

You have two options, which should be selected depending on the stages of your project:

• Label both “potentially flaky” and “highly flaky” tests as Flaky.

• Label only “highly flaky” tests as Flaky.

Always Fail tag

This tag appears on tests that repeatedly failed, within the configured Time Window. This helps detect persistent issues that require immediate attention.

By default, the pre-configured Time Window is 30 days. You can set the value for this Time Window to fit the stages of your project.

New Failure tag

This tag appears on tests that just had a new failed result. This helps flag newly broken tests for immediate troubleshooting and root cause analysis.

Test cases will be tagged New Failure if the following conditions are met:

• Tests that fail for the first time in the configured Time Window. Default Time Window is 30 days.

• The test case ran at least a minimum number of time configured in Minimum Executions in the Time Window. Default Minimum Executions is 5.

Slow tag

This tag appears on test cases with latest performance slower than most past runs. This helps detect regressions or environmental issues.

Test cases will be tagged Slow if the following conditions are met:

• The test case ran at least a minimum number of time configured in Minimum Executions in the Time Window. Default Time Window is 30 days, and default Minimum Executions is 5.

• Most recent run has the run duration above the 99th or 95th percentile, compared to durations of all executions in the Time Window.