StudioAssist AI Failure Analysis
This document introduces AI Failure Analysis, available in HTML and email reports. The feature provides AI-generated explanations for test suite failure, helping you save time analyzing lengthy and technical log errors. It also explains how to enable the feature so you can use it in your HTML and email test reports.
- You have enabled StudioAssist and configure it with an AI service provider. See Configure StudioAssist and AI providers.
Scope​
- Execution: This feature is only available when executing tests directly from Katalon Studio. It is not yet supported for tests executed via Katalon Runtime Engine (KRE).
- Analysis Target: It only analyzes test cases that have a result of Failed or Error.
- Report Level: The analysis is provided at the test suite level. Detailed analysis for individual test cases is included within the generated test suite HTML report.
Overview​
Starting from Katalon Studio 10.4.0, AI-generated failure analysis, powered by StudioAssist, is available in your HTML and Email test reports. This feature automatically reads complex stack traces, identifies the root cause of failed test cases, and summarizes them in plain language. This helps you focus on fixing issues, not reading lengthy logs.
With AI insights in your reports, you can:
- Quickly diagnose failures in large executions with many failed tests.
- Prioritize the most critical issues affecting your test results.
- Streamline communication of test failures across your team (e.g., between QA and developers).
- Accelerate debugging and reduce reliance on developer assistance.
Refer to this table to understand how AI Failure Analysis works in HTML and email reports.
| Report Type | Scope | What AI generates |
|---|---|---|
| HTML Reports | Test Suite & Test Case | Summarized suite-level failure causes and detailed analysis for each failed test case, including root cause, impact, and recommended actions. |
| Email Reports | Test Suite | Concise summary of why the suite failed, ideal for quick reviews and status updates. |
Enable AI Failure Analysis​
You can enable AI failure analysis separately for HTML and Email reports.
- All AI-generated explanations are marked with an AI tag to distinguish them from original failure messages (non-AI).
HTML reports​
Enable AI analysis in your Project Settings. The analysis can then be viewed in either automatically generated or manually exported HTML reports.
-
Go to Project > Settings > Report.
-
Select the checkbox Analyze failed reason using AI, then click Apply Close.
- When selected: AI-generated explanations appear in the test suite summary and individual test case details.
- When unselected: Reports display the original failure message.
-
After you run a test suite in Katalon Studio, you can view the AI-analyzed report using one of the following two ways:
- Auto-generate report: If automatic HTML report generation is configured, open the report that is generated immediately after execution is complete. See Automatically generate reports for setup details.
- Manual export: Go to the Results tab, use the Export report dropdown to export the test results in HTML, then open the exported report. See Manually export reports for step-by-step guide.
In both cases, you will see AI-generated explanations at both the test suite summary and test case details levels.
Click here to view a sample test suite report with AI analysis: Sample test suite report (HTML) with AI faliure analysis.
Email reports​
You have configured your mail server and email settings to receive email reports after test execution. See Share test reports via email in Katalon Studio.
-
Go to Project > Settings > Email.
-
Select the checkbox Analyze failed reason using AI, then click Apply and Close.
- When selected: The test suite email includes an AI-generated explanation of the failure.
- When unselected: The email displays the original failure message.
The test suite email will display a brief AI-generated summary of why the test suite failed.
Known limitations​
- StudioAssist may take longer time to generate HTML report, and may not process every result. If it failed to generate the AI analysis, it will revert to displaying the original failure reason.
- The AI badge might still show up even if the failure reasons aren’t AI-generated. This can happen if AI analysis fails for email reports and the system falls back to the original message. You can verify this in the Event Log, which records that the failure reasons were not created by AI.