AI Failure Analysis in Reports
This document introduces AI Failure Analysis feature in Katalon AI Assistant.
- You have configured an AI service provider. See Configure Katalon AI Assistant and AI providers.
Overview
AI Failure Analysis reads execution logs of failed test cases, and summarizes the failure and identifies root causes in plain language.
The analysis will be provided via HTML Reports, and Email Reports. Check the below section to learn how to enable and trigger for each method.
Enable and use AI Failure Analysis
All AI-generated explanations are marked with an AI tag to distinguish them from non-AI failure messages.
In HTML reports
Enable AI analysis in your Project Settings.
-
Go to Project > Settings > Report.
-
Select the checkbox Analyze failed reason using AI, then click Apply and Close.
- When selected: AI-generated explanations appear in the test suite summary and individual test case details.
- When unselected: Reports display the original failure message.
-
After you run a test suite in Katalon Studio, you can view the AI-analyzed report using one of the following two ways:
- Auto-generate report: If automatic HTML report generation is configured, open the report that is generated immediately after execution is complete. See Automatically generate reports for setup details.
- Manual export: Go to the Results tab, use the Export report dropdown to export the test results in HTML, then open the exported report. See Manually export reports for step-by-step guide.
In both cases, you will see AI-generated explanations at both the test suite summary and test case details levels.
Click here to view a sample test suite report with AI analysis: Sample test suite report (HTML) with AI faliure analysis.
In email reports
You have configured your mail server and email settings to receive email reports after test execution. See Share test reports via email in Katalon Studio.
-
Go to Project > Settings > Email.
-
Select the checkbox Analyze failed reason using AI, then click Apply and Close.
- When selected: The test suite email includes an AI-generated explanation of the failure.
- When unselected: The email displays the original failure message.
The test suite email will display a brief AI-generated summary of why the test suite failed.
To learn about triggering AI failure analysis in HTML/email reports via Katalon Test Execution - Local executions, see AI execution arguments.
Known limitations
- Katalon AI Assistant may take longer time to generate HTML report, and may not process every result. If it failed to generate the AI analysis, it will revert to displaying the original failure reason.
- The AI badge might still show up even if the failure reasons aren’t AI-generated. This can happen if AI analysis fails for email reports and the system falls back to the original message. You can verify this in the Event Log, which records that the failure reasons were not created by AI.
Learn more
Additionally, you can enhance your data privacy by masking any sensitive data (like PII, password) in your HTML or email reports. Refer to the following guides: