Analyze automation error patterns
This document explains how to detect failure patterns due to automation errors, to identify deep root causes behind test failures.
Ensure you have at least 10-20 failures for meaningful pattern detection as too few failures may not reveal clear patterns.
Overview​
Automation error messages are often inconsistent and don’t clearly reveal root causes. TestOps' Common Automation Errors report groups similar failures into patterns so you can spot recurring issues and prioritize fixes that resolve many failures at once. Use this view to move from investigating individual errors to addressing high-impact problems more efficiently.
Steps to analyze automation error patterns​
Analyze failure patterns is one step in our recommended failure investigation workflow. See Investigate failures to understand how you can use TestOps capabilities to optimize your failure investigation process.
In TestOps, entry to analyze automation error patterns is via Analytics > Reports > Defects Activity Analysis Report.
Once you've accessed the report, follow these steps to analyze defect trends.
Step 1: Apply scopes and filters​
Before analyzing patterns, scope your data to a relevant subset:
Filter by time range: Select the period you want to analyze
- Last 7 days: Recent failures and current sprint issues
- Last 30 days: Broader pattern detection across multiple sprints
- Custom date range: Target specific release cycles or deployment windows
Filter by status: Focus on failure types
- Failed: Assertion failures or test logic issues
- Error: Execution errors, exceptions, or framework problems
- Both Failed and Error: Comprehensive failure analysis
Filter by test suite (optional): Narrow analysis to specific test suites if investigating targeted areas
- Regression suite: Critical path failures
- Smoke tests: High-priority basic functionality issues
- Feature-specific suites: Isolated feature investigation
Step 2: Select a view mode​
The report has two view modes for different purposes:
Latest Error Only (Default)​
This view focuses on the most recent occurrence of each error. It helps teams quickly understand what is actively failing at the moment:
- Shows one latest instance per grouped error.
- Charts reflect only the latest failed result of each test case.
- The “Impacted Test Cases” count accounts for tests affected by the latest instance only.
- Selecting an error opens the most recent failed run and its details.
All Errors​
This view displays all occurances of each error in test cases/runs, across the selected date range or sprint/release. It's ideal to understand recurring issues, their scope of impact, or just to assess stability of the project's automation progress.
- Shows both current and historical occurrences per grouped error.
- Charts reflect all occurences.
- “Impacted Test Cases” count accounts for all affected tests.
- Selecting an error opens Test Result Analysis report filtered by the error itself. This report provides a high-level view of all impacted test cases, and a data table with link to each test case/run if you want to investigate further (learn more about this report at Test Result Analysis report).
Step 3: View grouped error message​
The error data table shows:
- Error message text: The actual error message that's common across tests
- Occurrence count: How many test results share this exact error
- Affected tests: Which specific test cases experienced this error
- Time distribution: When these errors occurred (recent vs. historical)
Our AI capabilities group identical/similar errors and display an example the represents the whole group. Clicking on each error reveals the individual errors.
- If you see an identical error repeated across many test results, those failures are likely caused by the same root issue (perhaps the submit button's visibility logic changed or a CSS rule now hides it).
- If you see multiple similar errors under the representative, it suggests a broader issue: perhaps an entire button container or stylesheet is broken, affecting multiple buttons. You'll need to manually recognize these similar patterns by scanning group names.
Step 4: Prioritize by pattern occurrences​
For each error group, note:
- Total occurrences: High occurrence counts (10+ failures) indicate high-impact issues worth prioritizing
- Affected test count: Different from occurrences if tests run multiple times—5 tests each failing twice = 10 occurrences, 5 unique tests
- Percentage of total failures: If one error pattern represents 40% of all failures, fixing that single issue resolves nearly half your failure volume
Example prioritization by occurrence:
- Error A: 45 occurrences (40% of failures) → High priority: Single fix resolves most failures
- Error B: 18 occurrences (16% of failures) → Medium priority: Significant but not dominant
- Error C: 3 occurrences (3% of failures) → Lower priority: Limited impact, investigate after high-volume patterns
Step 5: Apply additional filters to reveal underlying patterns​
Beyond identical error messages, patterns emerge when you group failures by other characteristics. This reveals systemic issues that may not share the same error text but correlate with specific environments, test structures, or timeframes.
You can save a custom view for each filter, then switch between custom views to understand these underlying patterns, then share findings with your team to validate. For example:
Compare patterns between test suites​
| Scenario | Potential cause |
|---|---|
| "Checkout Flow" suite has 60% failure rate while other suites have 10% | Recent deployments |
| "User Login", "Session Management", and "Authentication" suites all fail | Broken authentication service or shared login utilities |
| New suite with high failed rate | Incomplete test, missing test data setup, or environment configuration issues |
Compare patterns between execution environments​
Available environments:
- Browser: Chrome, Firefox, Safari, Edge
- Operating System: Windows, macOS, Linux
| Scenario | Potential cause |
|---|---|
| All Firefox failures show "WebDriver connection refused" | Firefox WebDriver version incompatible |
| MacOS tests fail with file path errors | Path separators or case sensitivity issues in test scripts |
| Scheduled runs at 2am fail, but manual runs pass | Overnight database maintenance or service restarts interfere with test execution |
Compare patterns between time periods​
Time-based patterns reveal when failures started and whether they correlate with deployments, configuration changes, or external factors.
| Scenario | Potential cause |
|---|---|
| Failures jump from 5% to 40% after a specific date | Recent deployment |
| Failure rate slowly climbing from 8% to 25% over two weeks | Test environment degradation or test data staleness |
| Tests fail during business hours but pass overnight | Production load impacts test environment, or shared test data is modified by concurrent users |
Cross-reference with deployment history. For example:
-
If failure spike occurred on March 15th and application version 2.4.0 deployed March 14th, strong correlation suggests version 2.4.0 introduced regressions.
-
If no deployments preceded failure spike, investigate environment changes, third-party service outages, or expired certificates/credentials.
Combine findings from multiple views​
The most valuable insights come from combining multiple characteristics:
| Scenario | Potential cause |
|---|---|
| “Connection timeout” only on Firefox | Firefox driver networking issue |
| Payment Tests starts failing after a specific execution date | Likely regression, introduced by a recent app deployment or configuration change |
| File Management Tests fail only on Linux | Differences in filesystem behavior, permissions, or case sensitivity |
Configure customizable fields to add custom tags to executions, leave notes, and compare data between these tags too. E.g. by profiles, execution type (local, ci-cd, scheduled, ...), execution time (AM, PM, peak hours, ...)