View test results and execution logs in Katalon TestOps

Katalon TestOps allows you to centralize and track execution data including test results, execution logs, and test report files. This article will shows you different ways to view test results and execution logs in Katalon TestOps.

View test runs reports in Katalon TestOps

To view test reports in Katalon TestOps, go to Reports.

Switch to the Test Runs tab. In this tab, there are two sections:

• Test Run by Status: This diagram shows the number of test runs with Passed or Failed status within 7/14/28 days.

• All Test Runs: This section shows all test runs status, test results with its corresponding test environment and test configuration. The following table explains the data represented in each column.

  Column Name	Description
  Status	This column shows the status of the test runs. There are four statuses as follows: Passed Failed Terminated In Progress
  ID	This column shows the hyperlinked test run ID. This hyperlink leads you to the corresponding test run page.
  Name	This column shows the test run name and the corresponding test suite/test suite collection name.
  Platform - Profile	This column shows a test run operating system, browser and execution profile.
  Time	This column shows the test run duration and executed date.
  Total	This column shows the total test results of a test run. There are 5 test results as follows: Passed Failed Error Incomplete Skipped Besides the Passed result, other test results are highlighted.
  Configuration	This column contains the test configuration including: If you have run with an agent, you can see the hyperlinked session ID (e.g, #2288). This hyperlink leads you to the session log. If you have linked the test run to a release/build, you can see the hyperlinked release/build name (e.g, Release 1). This hyperlink leads you to the corresponding release page. If you have specified the build label and URL when running with Katalon Runtime Engine, you can see the hyperlinked build name. To learn more about specifying build label with Katalon command line, see: TestOps integration arguments.
  By	This column shows the avatar of the test executor.

View test run results and execution logs in Katalon TestOps

To view details of a test run result, click the test run ID. Here, we click ID 3020.

A Test Run: #<ID> page appears (e.g, Test Run: #3020). In the Summary tab, you can see the following information:

The summary bar

This bar shows the total number of test results including: Passed, Failed, Error, Imcomplete, Skipped status in the test run.

Previous Test Runs

This section gives you a brief look of how your test scripts perform overtime. In the following example, a scheduled test run AWS - SSRF - Local Agent - Test Suite - Flag ON was triggered on 29/08/2022 with a duration of 250ms. On 13/09/2022, the test run was triggered again but hanged on and finally timed out. This might indicate flakiness in your test scripts.

Sessions

If you have run with an agent, you can see the sessions results in this section. To view the session logs, click the session ID. Here, we click session ID 2285.

The Session: #<Session-ID> page opens. You can see the full session log, along with the run command in this page. If you want to re-run the test run, click Run at the right upper corner.

Failed Test Results

If the test run has failed test cases, you can see the failed test cases and the corresponding error in this section. To view the error and further analysis, click the test result ID. Here we click test result ID 102282108.

The Test Result page opens. In the Result page, you can see:

• Error message: This returns the specific reason why the execution fails.

• Attachments: This attaches the captured screenshots and/or recorded videos to see what went wrong with failed tests.

• Failed Assertions: When you run tests with Katalon Studio, if you have used built-in assertion keywords with the verify prefix (e.g, WebUI.verifyElementPresent) or custom assertion keywords such as KeywordUtil.markPassed(message), KeywordUtil.markFailed(message), KeywordUtil.markFailedAndStop(message), you can see the failed assertions in this section.

• Errors: This shows the full error log. You can download this file.

• Logs: This shows the execution logs in each test step and the step status. This allows you to quickly troubleshoot which steps cause test failure. You can view this log in text view or tree view.

Besides the execution log, TestOps also provides additional analysis, including:

• Similar Failures: TestOps analyzes all existing failed test results and identifies similar failures. By having failed test results centralized in one place, you can quickly compare and pinpoint the root cause of any issues.

• Failure Reason: TestOps automatically assigns failure reason to the corresponding top-level exception of failed test results. Here, you click on the drop-down and change the suggested reason to a different category. You can also click on the Suggested Solution link to open the document for troubleshooting the failure.

  Note:

  • The Failure Reason capability is limited to the Premium and Ultimate plans of Katalon Platform. See: Katalon Platform Plans.

• Jira Defects: You can link the test failure to Jira defects for test management. See: Link test runs to Jira defects in Katalon TestOps.

Test Suites

This section contains the execution information including the test suite name, execution profile, operating system and browser, duration and executed date. When you click the hyperlinked test suite name, you are navigated to its information page where you can see:

• The test cases of the test suite

• Execution history of the test suite

View test report files

To view uploaded test report files of a test run, go to Reports > Test Runs > click the test run ID you want to view the uploaded files > Files.

You can also download the test reports files.

View statistics of failed test results

When a test failure occurs, it will be associated with a reason.

To help you better monitor the quality of a Project via test failures, TestOps provides the Failed Test Results chart that groups test failures by reasons.

Important:

• Failed Test Results is limited to the Premium and Ultimate plans of Katalon Platform. See: Katalon Platform Plans.

To view the chart, go to Reports > Test Runs > Failed Test Results.

The Failed Test Results chart provides two data filter options:

• All Results: In case a test case is configured to be re-run and has returned multiple results, the chart shows statistics of all failed test results.

• Latest Results: In case a test case is configured to be re-run and has returned multiple results, the chart only shows statistics of the latest test result (if the latest one failed).

The failure reasons in use are:

Failure reasons	Examples
Automation Bug	Test script cannot be compiled (syntax errors) or executed with runtime errors. Test script cannot locate web elements. Insufficient wait time for test objects to load.
Application Bug	Application logic has changed. Unexpected behaviours of web elements. Web elements no longer exist. Use of new /unresolved libraries.
Network Issues / Third Parties Issues	Network connection is down. Authentication issues.
Uncategorized	For other unspecified failure reasons.

TestOps also provides a list of top 10 exceptions of all failed test results in a specific period of time. This list is not affected by the All Results and Latest Result filters.

To see the list of all related test failures, click on the number of test results. In the opened Test Results list, you can view the detailed error log of a test result by clicking View Errors.

Add custom fields and tags to test results

You can add custom fields and tags to test results. To understand more about custom fields and tags or how to create them, see Custom Fields and Tags.

1. Navigate to a test run result by clicking its ID number.
2. Navigate to the Custom Fields and Tags section, you can select from the dropdown list the custom fields and tags to assign to your test result. You can also see Add Custom Fields and Tags to test cases for references on how to add or remove custom fields and tags.

Your test result is successfully assigned with custom fields and tags.

Search a test result by custom fields and tags

After assigning custom fields or tags to test results, you can query test results by custom fields and tags.

1. Go to Reports > Test Runs.
2. Navigate All Test Runs list, right below the search bar, you can see the filters you can use to query from the test run results list.
   
3. Select from the Custom Fields or Tags dropdown lists the custom fields and tags based on which you want to query your test run results. Click Update to add them to your filter.

Your test run result is queried successfully using custom fields and tags filter.