Terminology and Formulas

TestOps Terminology

Agent Connects a local machine to the TestOps servers for test executions.

API performance anomalies Identify web services that take more/less time to respond than usual by applying the local outlier factor (LOF) on 30 latest execution requests.

Assertions Determine whether the automated test case succeeds or not.

Katalon TestOps provides you with a specific view of assertions in each test case to evaluate the quality of test cases and ensure your tested application/software is working correctly.

Assertions help to check whether a condition is true (e.g., whether labels, data, API responses are rendered correctly).

BDD Test Result Represents the result of a test following the Behavior Driven Development (BDD) Framework.

In Katalon TestOps, the Features of BDD tests are displayed as Requirements, while Scenarios are displayed as Test Cases.

Build Represents a group of features/tasks in a release, used for management or testing purposes.

In Katalon TestOps, a build can consist of one or more test cases.

When users map a test result to a build, the test result is automatically mapped to the corresponding release. However, if users map a test result to a release, the test result is not mapped to any specific build.

Build Label Represents a piece of code committed to the source code

Defect Identifies a failed test run to link the test result to a Jira issue (Jira bug).

Interval (Trigger) A time period between each scheduled execution.

In Katalon TestOps, the Interval Unit values are Minute, Hour, Day, Week. For example, if you set Interval Unit as Hour, Interval as 5, it means the execution is scheduled for every 5 hours.

Key (Jira) Represents the ID of a Jira issue you have linked a test case/test result to.

In Katalon TestOps, you link a test case to Jira Requirements and link a test result (if the test run fails) to a Jira Defect.

Last Ping (Agent) The last time an Agent connects to a server to ask for a job to execute.

Maintainer A role that is responsible for all test results of a test case.

In Katalon TestOps, within an organization, you can assign a person to maintain a test case and the history of its test results.

Organization Created to brings together all users, teams, and projects of a company.

In Katalon TestOps, the organization level is the highest level of management, followed by the team level and the project level.

Path The location of a test in Katalon Studio.

In Katalon TestOps, each test case has a path if it is uploaded from Katalon Studio.

Profile An execution profile that is created in Katalon Studio (KS).

In Katalon TestOps, the KS's execution profiles are displayed as Profiles. A test run can have multiple profiles. This helps cover multiple and different environments to execute automated test scripts with ease.

Properties (Visual Testing) Represents the ID of a baseline image.

Release Represents a version or a milestone of your tested software.

In Katalon TestOps, a release consists of one or more test cases.

Request (API testing) Represents the action of calling a server and asking it to perform a task.

Requirement (Jira) Represents the Jira issue that is linked to a test case.

Script Repository Stores all the test assets for a test automation project (tests, data files, scripts files, macros, etc.)

In Katalon TestOps, you can create a script repository to manage all test scripts and decide which one to be executed along with the given test environment. You can upload files to a script repository from an external source or as a .zip file.

Session A session that is created once a test run is executed.

In Katalon TestOps, each test run can be executed as one or many test sessions depending on whether the tests run in parallel or not.

A session can contain:

  • one or more test cases
  • a test environment
  • test scripts (in a script repository) used for the execution of each test case.

Test Case Consists of multiple test results once a test is executed.

In Katalon TestOps, you can see detailed information of test cases to see if your software is free of bugs and whether it is working as expected. You can link test cases to Jira Requirements to ensure good test coverage. A test case can be linked to one or more Requirements.

Test Environment A place where a test run could be executed with the command-line interface.

In Katalon TestOps, you can use your local machine as a test environment. This test environment must be controlled by an agent.

Test Execution The results of test cases executed by TestOps scheduler.

Notes:

Not including reports uploaded from Katalon Studio or other tools.

Test Run Consists of one or more test cases to be executed (execution of more test cases depends on whether a test run is conducted in parallel).

In Katalon TestOps, a test run keeps track of all test results, status, duration, assignee, etc.

Test Run Type A collection of all test runs that have the same configurations.

Test Suite A collection of test cases.

Test Suite Collection A list of test suites that allows users to manage and plan test executions better.

Threshold (Agent) The maximum number of sessions that an agent can execute at the same time.

Total Assigned Sessions (Agent) The total number of sessions assigned to an agent to execute tests.

In Katalon TestOps, the maximum number of assigned sessions is also called Threshold.

Total Executing Sessions (Agent) The number of sessions an agent executes in real time.

Traceability Matrix Manages the relationships across requirements, test cases and defects.

Trigger Determines when a test run is executed.

In Katalon TestOps, this function helps leverage remote execution for complete control of the testing plan.

User Management An administration tool to manage users.

In Katalon TestOps, this feature allows you to:

  • invite users to your organization (you can also set up permission access for other Katalon products).
  • delete pending invitations.
  • set and edit a user's role.
  • remove users from your organization.

UUID (Agent) An identifier that is generated when an agent is set up successfully.

Visual Baseline (Visual Testing) A standard reference image used to compare with the Checkpoint screenshot(s) during a test execution.

TestOps Formulas

% change of Test Case (Dashboard) = (Total test cases this week/Total test cases last week)*100

% change of Test Result (Dashboard) = (Total test results this week/Total test results last week)*100

% fail (Build) = (Total failed test cases/Total test cases)*100

% flakiness = (Number of times the status of test results changes/Total number of test results)*100

Notes:

Total number of test results = 30 latest test results (sorted by execution start time).

% pass (Build) = (Total passed test cases/Total test cases)*100

Active/Archived Release A new release is active by default. You can track it when it's active.

In Katalon TestOps, you can also archive a release to stop tracking it.

Active Test Case A test case that has been run in the last 2 months, based on the start time of the latest execution.

Average Duration (Test Case) = (Total duration of all test results/Total number of test results)*100

Notes:

The calculation is based on the last 100 test results.

Average Duration (Web Services) The average amount of time based on the last 30 execution requests (sorted by the most recent execution start time.).

Development Progress (Dashboard) = (Resolved Jira issues/Total Jira issues)*100

Duration (Session) = (End time - Start time)

Notes:

Start time is when an agent starts receiving a job.

End time is when uploading reports to TestOps is done.

Duration (Test Result) = (End time - Start time)

Duration (Test Run) = (End time - Start time)

Notes:

This shows the actual time of running tests, including test runs via Scheduler and Upload Reports.

Execution Time (Dashboard) The total duration of test results.

Flaky Test Case Represents test cases that fail to produce the same results each time the same analytics is run.

Max/Min Duration (Web Services) The max/min duration that is based on the last 30 execution requests.

Offline Agent An agent is offline when the time of the last Ping is longer than the active time

Notes:

By default, the active time is 5 minutes.

Platform Coverage (Test Case) Shows the quality of test cases by operating system (OS) and browser-basis.

The color of the dot indicates that the test has passed or failed.

  • Red dot = failed test
  • Green dot = passed test

The size of the dot represents the number of tests (e.g., the bigger the dot is, the more tests are).

Run Frequency (Test Run) A statistic of the scheduled test runs in a day.

The color of the dot indicates that the test has passed or failed.

The size of the dot represents the number of test results.

Similar Failures The test runs that have at least 70% of error similarities.

Slowest Test Case A test case that has the longest average duration.

Katalon TestOps ranks active test cases by their average duration. The shorter an average duration is, the more active/faster a test execution is.

Stale Test Case A test case that has not been run for at least 2 months.

In real-time, the 2 month-point is calculated everytime you click to view the report.

Status (Release) The status of a release, including:

  • Ready: all test cases have passed.
  • Not Ready: at least one test case has failed.
  • Empty: there's no test case linked to the release.

Status (Session) The status of a session, including:

  • Queued: session has been created, waiting to be executed.
  • Running: session is in progress.
  • Failed: session has failed.
  • Success: session has succeeded.
  • Canceled: session is canceled manually or session timeout.

Status (Test Case) The status of a test case, including:

  • Passed: all test results have passed.
  • Failed: one of the test results has not passed.

Notes:

The status of a test case is defined by its latest execution.

Status (Test Result) The status of a test result, including:

  • Passed: test case runs successfully.
  • Failed: test case runs unsuccessfully.
  • Error: an error occurs during the execution.
  • Incomplete: predefined in Katalon Studio.

Status (Test Run) The status of a test run, including:

  • Passed: all test results have passed.
  • Failed: one of the test results has failed.

Status (Test Suite) The status of a test suite, including:

  • Passed: all test cases have passed.
  • Failed: one of the test cases has failed.

Status (Visual Checkpoint) The status of a visual checkpoint, including:

  • Pass: checkpoint image has matched the baseline, or it's manually marked as Pass.
  • Fail: checkpoint image is marked manually as Fail.
  • Unresolved: checkpoint image has mismatched the baseline. You can compare with the baseline image, then mark it as Pass or Fail.

Status (Visual Test Run) The status of a visual test run, including:

  • Pass: all new checkpoint images match the baseline image.
  • Fail: one or more checkpoints have failed but no unresolved checkpoints.
  • Unresolved: one or more checkpoints are unresolved.

Test Progress (Dashboard) = (Total passed test results in release/Total test results in release)*100

Test Run Coverage The quality of each requirement based on the status of the corresponding test result.

Time (Test Suite) The duration of the last run of a test suite. Also, the start time of the last run of a test suite.

Time (Visual Test Run) = (End time - Start time of a visual test run)

Total Duration in Summary (Test Results) A sum of all test result durations in a day.

Total Duration (Release) The total duration of all test runs in a release.

Feedback