Common Automation Errors Report
This document explains how to use the Common Automation Errors Report to identify recurring failure patterns across automated test runs.
Instead of analyzing each failed test in isolation, the report automatically groups similar error messages, providing a consolidated view of your most frequent and impactful issues. This report aims to assist you make a large-scale analysis on automation failure much clearer and more actionable.
Why Use This Report
- Distinguish systemic issues from one-off failures.
- Quantify the frequency and breadth of impact of each error type.
- Prioritize debugging efforts using data-driven insights.
- Track whether recurring issues are being resolved over time.
Explore the Report
Explore the report with our interactive guide:
- Open the report: Go to Reports > Common Automation Errors.
- Select filters: Choose a Time Range / Release or Sprint Scope.
- Review visuals:
- Use the pie chart to identify leading failure types.
- View aggregates for an overall understanding of common errors and impacted test cases.
- Use the data table to find recurring patterns in automation error. Click on test cases to investigate further if needed.
- Refine Your Focus:
- Toggle Latest Errors Only to view recent unresolved issues.
- Sort data table by Occurrences or Impacted Test Cases for prioritization.
- Take action:
- Click an error group to view related test cases and logs.
- Assign debugging tasks based on frequency and severity.
Report Features
The Common Automation Errors Report consists of two main visualizations and several supporting metrics.
Error Distribution by Failure Category
This pie chart shows the breakdown of failure types (Environment, AUT Issue, Test Script, etc.) across all test results. It helps you quickly spot dominant failure patterns (e.g., environment errors vs. script issues).
Common Error Data Table
This detailed, sortable table listing recurring errors with occurrence counts, number of affected test cases, and sample messages. It enables you to drill into critical error patterns. You can click on the impacted test cases to view details.
Impact metrics
The aggregated totals for each error type show how many tests and runs are affected. This reveals the severity and scale of each issue.
Advanced Filtering
The report includes several filters that let you isolate specific problem areas or release cycles.
Available filters:
- Failure Category
- Environment (Execution Profile)
- Test Suite / Test Suite Collection
- Time Range
The report uses TestOps’ internal error-pattern recognition system to automatically cluster similar failure messages into common error types.
Data Table Field Reference
| Field | Description | Example |
|---|---|---|
| Error | Name of the error. | com.kms.katalon.core.exception.StepErrorException: Error |
| Failure Category | AI-assisted classification of similar automation failures grouped by error message patterns. | AssertionError, TimeoutException, ElementNotFoundException |
| Impacted Test Cases | Total number of unique test cases impacted by this error. | 36 |
| Error Count | Number of times this error type appeared across all test runs in the selected time range. | 245 |
| First/Last Occur | Date on which this error is first/last found. | 29 Oct 2025 |