Skip to main content

Defects Report automations

What is a Defects Report automation?

After executing a test run in TestOps, you can link test results to an issue in Jira manually by using Report Defect. See: Link test runs to Jira defects in Katalon TestOps. This makes the process of tracing defects easier.

Setting up a Defect report automation is the option to automate that linking process especially when there are multiple test results with the same issue that can be linked to a particular defect.

Create automation rules

Test results with similar failure conditions can be linked to a defect automatically by creating an automation through Report Defect. To do so, follow these steps:
  1. Log in to Katalon TestOps and go to your project.
  2. Go to Reports > Test Runs > Test Runs.
  3. Select a failed test run by clicking on its ID. Then click on the Test Results tab.
    A failed test run report in the test run page.
    1. The Test Results page appears as below.
      A list of a test run's test results.
  4. Click on the Report Defect button at the rightmost column. It has the bug icon.The report defect bug icon in Katalon TestOps
  5. Link or create your issue. To learn how, see: Link test runs to Jira defects in Katalon TestOps.
  6. Click on Create Automation Rule to automatically link this defect to future test results.
    The Report Defect dialog in Katalon TestOps
  7. Input the parameters for your automation rule. You can define the patterns using either exact text or regular expression syntax. Learn more about POSIX regular expressions here.
    • Only upcoming test results will be automatically linked to this defect when an error message or stack trace matches your pattern.

    • Failed tests with matching error or stack trace values will not automatically link to resolved defects. You need to manually link them.

    The Automation Rule settings in the Report Defect dialog
    1. To view the full Error message or Stack trace, click on the Expand button at the upper right corner. The expand button for error message and stack trace.
    2. A pop-up shows more details.Error message details when creating automation rules in Katalon TestOps.
  8. Optionally, you can click on View matched results to see if there are other test results made within the last 30 days that equal your defined pattern. This is disabled by default and will only enable once valid values are input for both patterns.
    View matched results when creating an automation rule in Katalon TestOps.
    1. A pop-up shows a list of matching results. List of matched results when an automation rule is created in Katalon TestOps.
  9. Click Save.
    1. You can also re-edit your automation by clicking on the Edit Automation Rule button. The Edit Automation Rule button in Report Defect.
You have successfully saved the settings for an automation rule. It applies once you have created and linked your test results to a defect.

A pop up notification will confirm this.

To view all your automation rules or delete them, see: View and delete automation rules.

You can automatically be notified in Slack when this action is performed by setting up an integration. See: Defects Report Slack notifications.

View and delete automation rules

Automation rules linked to a particular defect can be viewed or deleted by checking the Automation Rule tab within the Defects page in Reports. Here's how to do it step by step.
  1. Log in to Katalon TestOps and go to your project.
  2. Click on Reports > Defects.
    1. In the displayed Defects page, apply any search filters you may need in order to view all defects.The Defects page within the Reports section of Katalon TestOps
  3. Scroll down to your defects list and click on the arrow > icon to open a side panel that shows two tabs:
    The side panel within the Defects page.
    • Overview: Shows all the test cases linked to the defect.

    • Automation Rule: Shows all the Automation Rules you've made linked to the defect.

  4. Click on Automation Rule to view your automation rules list.
Delete automation rules
  1. Click on the Trash icon to the right of the Automation Rule you'd like to delete.
    The trash icon for deleting an automation rule in Katalon TestOps.
  2. A pop-up confirmation will ask you to confirm the deletion. Click Confirm.
    The delete automation rule confirmation box in Katalon TestOps.
You have successfully viewed or deleted your Automation Rule. A notification confirms this.

Notification confirming the deletion of an Automation Rule in Katalon TestOps.

Troubleshoot automation rules

The Save button won't allow you to continue if there are unacceptable parameters within your automation rule.

All form fields must pass validation before you can Save.

  1. If your Rule Name is blank or duplicated, you have to input a unique name with a maximum length of 255 characters.
  2. If your Error Message or Stack Trace patterns are blank, you have to enter a definition for your patterns using either exact text or regular expression syntax. At least one of them must have a value with a minimum of 2 characters and a maximum of 2048.
  3. If your Regex pattern for Error Message or Stack Trace is invalid, double-check your input and refer to this document to learn more about it: Learn more about POSIX regular expressions here.