AI Failure Analysis in Reports
This document introduces AI Failure Analysis feature in Katalon AI Assistant.
- You are using Katalon Studio Enterprise (KSE).
- You have configured an AI service provider. See Configure Katalon AI Assistant and AI providers.
Overview
AI Failure Analysis reads execution logs of failed test cases, and summarizes the failure and identifies root causes in plain language.
In addition to the summary, it assigns a Failure Category to each analyzed failure so you can spot error patterns and prioritize fixes without reviewing every result by hand. Categories are generated by the AI, are read-only, and match those used in Katalon True Platform.
The analysis will be provided via HTML Reports, and Email Reports. Check the below section to learn how to enable and trigger for each method.
Failure categories are available from Katalon Studio 11.2.0 onwards.
Failure categories
| Failure category | Description |
|---|---|
| Application Under Test (AUT) Issue | Functional or UI issues in the application under test. |
| Test Script Issue | Runtime errors, element-not-found errors, and assertion failures. Tests authored in Katalon Studio commonly fall under the Test Script Issue category. |
| Test Data Issue | Incorrect, missing, or invalid test data. |
| Automation Issue | Framework, library, or tooling problems. |
| Integration Issue | Components or systems that do not work together. |
| Configuration Issue | Incorrect tool setup or parameters. |
| Environment Issue | Test environment instability or unavailability. |
| Others | Unspecified issues. This is the default when the AI cannot confidently classify the failure. |
Enable and use AI Failure Analysis
All AI-generated explanations are marked with an AI tag to distinguish them from non-AI failure messages.
In HTML reports
Enable AI analysis in your Project Settings.
-
Go to Project > Settings > Report.
-
Select the checkbox Analyze failed reason using AI, then click Apply and Close.
Result
After you run a test suite in Katalon Studio, the AI-generated explanations appear in the test suite summary and individual test case details. You can view the reports using one of the following ways:
- Auto-generate report: If automatic HTML report generation is configured, open the report that is generated immediately after execution is complete. See Automatically generate reports for setup details.
- Manual export: Go to the Results tab, use the Export report dropdown to export the test results in HTML, then open the exported report. See Manually export reports for step-by-step guide.
In both cases, you will see AI-generated explanations at both the test suite summary and test case details levels.
Click here to view a sample test suite report with AI analysis: Sample test suite report (HTML) with AI faliure analysis.
The HTML report also shows Failure Category cards at the test suite level (after the summary statistics, before the Fail Reason section) and a read-only category tag inside each analyzed test case. Click a card to filter the Test Cases tab by that category. The following demo shows how:
In email reports
You have configured your mail server and email settings to receive email reports after test execution. See Share test reports via email in Katalon Studio.
-
Go to Project > Settings > Email.
-
Select the checkbox Analyze failed reason using AI, then click Apply and Close.
- When selected: The test suite email includes an AI-generated explanation of the failure.
- When unselected: The email displays the original failure message.
The test suite email will display a brief AI-generated summary of why the test suite failed. Below the Fail reason section, it also lists aggregated Failure Categories counts, with Others always last. For example:
Failure Categories:
- Test Script Issue: 3
- Environment Issue: 2
- AUT Issue: 1
- Others: 1
To learn about triggering AI failure analysis in HTML/email reports via Katalon Test Execution - Local executions, see AI execution arguments.
Known limitations
- Katalon AI Assistant may take longer time to generate HTML report, and may not process every result. If it failed to generate the AI analysis, it will revert to displaying the original failure reason.
- The AI badge might still show up even if the failure reasons aren’t AI-generated. This can happen if AI analysis fails for email reports and the system falls back to the original message. You can verify this in the Event Log, which records that the failure reasons were not created by AI.
- Failure categories are read-only; you cannot edit them manually.
- In the HTML report, the Failure Category filter and the test result filter (Passed, Failed, Error, Incomplete, Skipped) are mutually exclusive, and the selection resets when you refresh or reload the report.
- The email report shows aggregated category counts only — no interactive filtering and no per-test-case categories.
- Failure categories do not appear when AI failure analysis is disabled or when all test cases pass.
- Only test cases that the AI successfully analyzes are counted; a test case the AI cannot analyze has no failure category and is excluded from the counts.
See also
- Common automation error analysis report — Failure categories: These failure categories are also available in Katalon True Platform reports.
- Enhance your data privacy by masking any sensitive data (like PII, password) in your HTML or email reports: