Terminology and Formulas

Katalon TestOps provides you with a specific view of assertions in each test case to evaluate the quality of test cases and ensure your tested application/software is working correctly.

Assertions help to check whether a condition is true (e.g., whether labels, data, API responses are rendered correctly).

In Katalon TestOps, the Features of BDD tests are displayed as Requirements, while Scenarios are displayed as Test Cases.

In Katalon TestOps, a build can consist of one or more test cases.

When users map a test result to a build, the test result is automatically mapped to the corresponding release. However, if users map a test result to a release, the test result is not mapped to any specific build.

In Katalon TestOps, the Interval Unit values are Minute, Hour, Day, Week. For example, if you set Interval Unit as Hour, Interval as 5, it means the execution is scheduled for every 5 hours.

In Katalon TestOps, you link a test case to Jira Requirements and link a test result (if the test run fails) to a Jira Defect.

In Katalon TestOps, within an organization, you can assign a person to maintain a test case and the history of its test results.

In Katalon TestOps, the organization level is the highest level of management, followed by the team level and the project level.

In Katalon TestOps, each test case has a path if it is uploaded from Katalon Studio.

In Katalon TestOps, the KS's execution profiles are displayed as Profiles. A test run can have multiple profiles. This helps cover multiple and different environments to execute automated test scripts with ease.

In Katalon TestOps, a release consists of one or more test cases.

In Katalon TestOps, you can create a script repository to manage all test scripts and decide which one to be executed along with the given test environment. You can upload files to a script repository from an external source or as a .zip file.

In Katalon TestOps, each test run can be executed as one or many test sessions depending on whether the tests run in parallel or not.

A session can contain:

• one or more test cases
• a test environment
• test scripts (in a script repository) used for the execution of each test case.

In Katalon TestOps, you can see detailed information of test cases to see if your software is free of bugs and whether it is working as expected. You can link test cases to Jira Requirements to ensure good test coverage. A test case can be linked to one or more Requirements.

In Katalon TestOps, you can use your local machine as a test environment. This test environment must be controlled by an agent.

Notes:

Not including reports uploaded from Katalon Studio or other tools.

In Katalon TestOps, a test run keeps track of all test results, status, duration, assignee, etc.

In Katalon TestOps, the maximum number of assigned sessions is also called Threshold.

In Katalon TestOps, this function helps leverage remote execution for complete control of the testing plan.

In Katalon TestOps, this feature allows you to:

• invite users to your organization (you can also set up permission access for other Katalon products).
• delete pending invitations.
• set and edit a user's role.
• remove users from your organization.

Notes:

Total number of test results = 30 latest test results (sorted by execution start time).

In Katalon TestOps, you can also archive a release to stop tracking it.

Notes:

The calculation is based on the last 100 test results.

Notes:

Start time is when an agent starts receiving a job.

End time is when uploading reports to TestOps is done.

Notes:

This shows the actual time of running tests, including test runs via Scheduler and Upload Reports.

Notes:

By default, the active time is 5 minutes.

The color of the dot indicates that the test has passed or failed.

• Red dot = failed test
• Green dot = passed test

The size of the dot represents the number of tests (e.g., the bigger the dot is, the more tests are).

The color of the dot indicates that the test has passed or failed.

The size of the dot represents the number of test results.

Katalon TestOps ranks active test cases by their average duration. The shorter an average duration is, the more active/faster a test execution is.

In real-time, the 2 month-point is calculated everytime you click to view the report.

• Ready: all test cases have passed.
• Not Ready: at least one test case has failed.
• Empty: there's no test case linked to the release.

• Queued: session has been created, waiting to be executed.
• Running: session is in progress.
• Failed: session has failed.
• Success: session has succeeded.
• Canceled: session is canceled manually or session timeout.

• Passed: all test results have passed.
• Failed: one of the test results has not passed.

Notes:

The status of a test case is defined by its latest execution.

• Passed: test case runs successfully.
• Failed: test case runs unsuccessfully.
• Error: an error occurs during the execution.
• Incomplete: predefined in Katalon Studio.

• Passed: all test results have passed.
• Failed: one of the test results has failed.

• Passed: all test cases have passed.
• Failed: one of the test cases has failed.

• Pass: checkpoint image has matched the baseline, or it's manually marked as Pass.
• Fail: checkpoint image is marked manually as Fail.
• Unresolved: checkpoint image has mismatched the baseline. You can compare with the baseline image, then mark it as Pass or Fail.

• Pass: all new checkpoint images match the baseline image.
• Fail: one or more checkpoints have failed but no unresolved checkpoints.
• Unresolved: one or more checkpoints are unresolved.

• The size of each block indicates the number of test runs associated with each requirement.

• The color of each block indicates the overall status of the total test cases in each requirement.