Katalon TestOps allows you to centralize and track execution data including test results, execution logs, and test report files. This article will shows you different ways to view test results and execution logs in Katalon TestOps.
View test runs reports in Katalon TestOps
To view test reports in Katalon TestOps, go to Reports.
Switch to the Test Runs tab. In this tab, there are two sections:
Test Run by Status: This diagram shows the number of test runs with Passed or Failed status within 7/14/28 days.
All Test Runs: This section shows all test runs status, test results with its corresponding test environment and test configuration. The following table explains the data represented in each column.
Column Name Description Status This column shows the status of the test runs. There are four statuses as follows:
ID This column shows the hyperlinked test run ID. This hyperlink leads you to the corresponding test run page. Name This column shows the test run name and the corresponding test suite/test suite collection name. Platform - Profile This column shows a test run operating system, browser and execution profile. Time This column shows the test run duration and executed date. Total This column shows the total test results of a test run. There are 5 test results as follows:
Configuration This column contains the test configuration including:
If you have run with an agent, you can see the hyperlinked session ID (e.g, #2288). This hyperlink leads you to the session log.
If you have linked the test run to a release/build, you can see the hyperlinked release/build name (e.g, Release 1). This hyperlink leads you to the corresponding release page.
If you have specified the build label and URL when running with Katalon Runtime Engine, you can see the hyperlinked build name. To learn more about specifying build label with Katalon command line, see: TestOps integration arguments.
By This column shows the avatar of the test executor.
View test run results and execution logs in Katalon TestOps
To view details of a test run result, click the test run ID. Here, we click ID 3020.
A Test Run: #<ID> page appears (e.g, Test Run: #3020). In the Summary tab, you can see the following information:
This bar shows the total number of test results including: Passed, Failed, Error, Imcomplete, Skipped status in the test run.
This section gives you a brief look of how your test scripts perform overtime. In the following example, a scheduled test run AWS - SSRF - Local Agent - Test Suite - Flag ON was triggered on 29/08/2022 with a duration of 250ms. On 13/09/2022, the test run was triggered again but hanged on and finally timed out. This might indicate flakiness in your test scripts.
If you have run with an agent, you can see the sessions results in this section. To view the session logs, click the session ID. Here, we click session ID 2285.
The Session: #<Session-ID> page opens. You can see the full session log, along with the run command in this page. If you want to re-run the test run, click Run at the right upper corner.
If the test run has failed test cases, you can see the failed test cases and the corresponding error in this section. To view the error and further analysis, click the test result ID. Here we click test result ID 102282108.
Error message: This returns the specific reason why the execution fails.
Attachments: This attaches the captured screenshots and/or recorded videos to see what went wrong with failed tests.
Failed Assertions: When you run tests with Katalon Studio, if you have used built-in assertion keywords with the
WebUI.verifyElementPresent) or custom assertion keywords such as
KeywordUtil.markFailedAndStop(message), you can see the failed assertions in this section.
Errors: This shows the full error log. You can download this file.
Logs: This shows the execution logs in each test step and the step status. This allows you to quickly troubleshoot which steps cause test failure. You can view this log in text view or tree view.
Similar Failures: TestOps analyzes all existing failed test results and identifies similar failures. By having failed test results centralized in one place, you can quickly compare and pinpoint the root cause of any issues.
Failure Reason: TestOps automatically assigns failure reason to the corresponding top-level exception of failed test results. Here, you click on the drop-down and change the suggested reason to a different category. You can also click on the Suggested Solution link to open the document for troubleshooting the failure.Note:
The Failure Reason capability is limited to the Premium and Ultimate plans of Katalon Platform. See: Katalon Platform Plans.
Jira Defects: You can link the test failure to Jira defects for test management. See: Link test runs to Jira defects in Katalon TestOps.
This section contains the execution information including the test suite name, execution profile, operating system and browser, duration and executed date. When you click the hyperlinked test suite name, you are navigated to its information page where you can see:
The test cases of the test suite
Execution history of the test suite
View test report files
To view uploaded test report files of a test run, go to Reports > Test Runs > click the test run ID you want to view the uploaded files > Files.
You can also download the test reports files.
View statistics of failed test results
When a test failure occurs, it will be associated with a reason.
To help you better monitor the quality of a Project via test failures, TestOps provides the Failed Test Results chart that groups test failures by reasons.
Failed Test Results is limited to the Premium and Ultimate plans of Katalon Platform. See: Katalon Platform Plans.
To view the chart, go to Reports > Test Runs > Failed Test Results.
All Results: In case a test case is configured to be re-run and has returned multiple results, the chart shows statistics of all failed test results.
Latest Results: In case a test case is configured to be re-run and has returned multiple results, the chart only shows statistics of the latest test result (if the latest one failed).
The failure reasons in use are:
|Network Issues / Third Parties Issues|
|Uncategorized||For other unspecified failure reasons.|
TestOps also provides a list of top 10 exceptions of all failed test results in a specific period of time. This list is not affected by the All Results and Latest Result filters.
To see the list of all related test failures, click on the number of test results. In the opened Test Results list, you can view the detailed error log of a test result by clicking View Errors.
Add custom fields and tags to test results
- Navigate to a test run result by clicking its ID number.
- Navigate to the Custom Fields and Tags section, you can select from the dropdown list the custom fields and tags to assign to your test result. You can also see Add Custom Fields and Tags to test cases for references on how to add or remove custom fields and tags.
Search a test result by custom fields and tags
- Go to Reports > Test Runs.
- Navigate All Test Runs list, right below the search bar, you can see the filters you can use to query from the test run results list.
- Select from the Custom Fields or Tags dropdown lists the custom fields and tags based on which you want to query your test run results. Click Update to add them to your filter.