Skip to main content
Version: legacy

Use TestOps Visual Testing

You can compare images captured during test executions with TestOps Visual Testing.

Important:

Set up Visual Testing

To set up Visual Testing, follows these steps:
  1. Enable screenshot capture in Katalon Studio. See Capture Screenshots.
    Note:
  2. Run a test suite in Katalon Studio.
    Note:
    • If you have enabled Katalon Studio Integration, Katalon Studio automatically uploads Test Results to Katalon TestOps.

  3. Sign in to Katalon TestOps and go to your project.
  4. Go to Visual Testing > Visual Test Runs.
  5. Click on the ID of a Visual Test Run.
    visual test run results page
    The Results page appears. You can see the screenshots (known as Visual Checkpoints) captured during a test execution.
  6. Select one of the screenshots to see the details.
    • If you run a Test Suite for the first time, there is no baseline image.

    • If you have existing visual data, it is stored in the System-generated baseline collection folder.

You have successfully set up Visual Testing in Katalon TestOps.

Configure a baseline image

You can set up a baseline image to compare it with the screenshot of the next Test Run. Follow these steps:
  1. Select a screenshot to see the details.
    visual checkpoint passed status
  2. Select Mark as Passed at the top right corner of a screenshot, then close the screenshot.

    An Unsaved label now appears on the Results page.

    unsaved label on results page

  3. Click Save to baseline.
    Note:
    • If you run a Test Suite for the first time and the Test Suite has passed, the screenshots are automatically marked as Passed and there is no Save to baseline button. You can open a screenshot and click Failed, then switch back to Passed so that the Save to baseline button appears.

Unresolved images

If you run the Test Suite again and the new screenshots of this test execution are different from the baseline images, the status of this Visual Test Run is then marked as Unresolved. To resolve these images, follow these steps:

  1. Click on the ID of the unresolved Visual Test Run.
  2. Select the unresolved checkpoint.
    mismatch image comparison method box

    You see a drop-down list in the Comparison Method section. The three options include: Pixel, Layout, Text Content. For detailed information of each comparison method, see: TestOps Visual Testing.

  3. Select one of the three options to start comparing the new screenshot (on the right) with the baseline image (on the left).
    Note:
    • Comparison Method is enabled only if the status of the screenshot is mismatched.
    • If you select Pixel, check the Diffs between two images (the differences are highlighted in red). See: Pixel-based comparison.
    • If you select Layout, check the Identical, Distinguishable and Missing/New boxes to see the differences highlighted in green/red/pink color respectively. See: Layout-based comparison.
    • If you select Text Content, check the Identical, Shifted and Missing/New boxes to see the differences highlighted in green/blue/pink color respectively. See: Content-based comparison.
  4. Click Mark as Passed or Mark as Failed to resolve.
    Note:
    • The Unsaved label appears whenever you change the status (Passed or Failed) of a screenshot. If you click Save to baseline, the new Passed screenshot replace the older baseline image. Katalon TestOps then compares between the new baseline image and screenshots of the next Visual Test Run.

Baseline collection

A baseline collection is a collection of all baseline images in one and/or several related visual test run(s).

In TestOps, the visual baseline collection stores every version of all baseline images that you have saved each time. Each baseline collection has an ID (Baseline Collection ID).

There are two ways a baseline collection is created:
  • If tests are executed in Katalon Studio or run with the CLI as a part of the CI/CD pipeline and their results are uploaded to TestOps, the System-generated Common Baseline Collection folder is automatically created to store the captured images.

  • If you schedule test runs in TestOps, you can select Visual Testing and choose Generate a new Baseline Collection. See: Schedule Test Runs.

To access a baseline collection, follow these steps:

  1. Sign in to Katalon TestOps and go to your project.
  2. Go to Visual Testing > Visual Baseline Collections.
    You will see all baseline collection IDs and their details (latest version, number of the baseline images in each baseline collection, and the test schedule that links to them).visual baseline collections page

  3. Click on the Baseline Collection ID you want to check.
    Layer 1
  4. Click History to see the older versions of this baseline collection. Click on the Pencilicon to edit the name of the baseline collection. You can also edit or delete each baseline image of this baseline collection.

Ignored zones

Specifying ignored zones is especially useful when testing a website with some dynamic and changing content (such as pop-up ads on specific areas). Visual testing can ignore such content while comparing captured images with the baseline one.

In TestOps Visual Testing, you can configure an ignore zone in a baseline image and apply this ignored zone to all baseline images in the baseline collection. You can also delete the ignored zone(s) in all baseline images.

Add an ignored zone to a baseline image

To configure an ignored zone, follow these steps:
  1. Sign in to Katalon TestOps and go to your project.
  2. Go to Visual Testing > Visual Baseline Collections.
  3. Select the Baseline Collection ID you want to check.
    The list of baseline images appears.

    Layer 1

  4. Click on the Edit image icon of the baseline image you want to configure (e.g., the baseline image image10).
    The page displays as below.configure ignored zone in image10

  5. Click on the Configure Ignored Zones button.
    The button is then highlighted in blue, which indicates the edit mode.
    configure ignored zones button highlighted

  6. Draw (in rectangles) to specify the area you want to ignore.
    The ignored zones are highlighted in purple. You can draw up to 20 ignored zones (i.e. 20 rectangles).
    Note:
  7. Click Save.
    You have temporarily saved the ignored zones on this baseline image.
    save baseline image10

    Note:
    • You have not permanently saved the configuration of ignored zones at this stage.
  8. Click Save Changes on the top right corner to permanently save the ignored zones you have configured.
    save changes button for baseline collection

The baseline image has now had ignored zones. Visual testing then ignores these zones when comparing captured images with this baseline image.

When comparing the checkpoint image with the baseline image, you can show or hide the ignored zones by ticking/un-ticking the Ignored Zones checkbox.

Layer 1

Apply an ignored zone to all baseline images

Adding an ignored zone to the same area in multiple baseline images could become time-consuming if there are more than one baseline image. With TestOps Visual Testing, you can apply an ignored zone to multiple baseline images in a baseline collection.

To apply an ignored zone to all baseline images within one baseline collection, follow these steps:

  1. Configure an ignored zone. See: Add an ignored zone to a baseline image.
    The ignored zone is displayed in purple.
  2. Click on the ignored zone, then select the Apply the zone to all images icon.
    apply ignored zone to all images icon
    You then see the Acknowledgement pop-up box.ignored-zone-acknowledgement-popup
  3. Click Acknowledge.
    The ignored zone is now displayed in yellow because it has been applied to all other baseline images in the baseline collection.ignored zone applied to all images
  4. Click Save to temporarily save the configuration.
  5. Click Save Changes on the top right corner to permanently save the configuration.
You have applied an ignored zone to all baseline images within the selected baseline collection.

Delete ignored zones

You can delete an ignored zone in one baseline image and/or all baseline images in the baseline collection.

An ignored zone applied to one baseline image only is displayed in purple. To delete it, click on the ignored zone and select the Delete ignored zone icon.

An ignored zone applied to all baseline images is displayed in yellow. To delete it, follow these steps:

  1. Click on the ignored zone and select the Delete ignored zone icon.
    delete ignored zone icon
    The Delete all ignored zones box pops up.delete all ignored zones popup box
  2. Click Delete all zones to confirm your action.
  3. Click Save to temporarily save the configuration.
  4. Click Save Changes to permanently save this configuration.
    The ignored zone is now deleted in all baseline images of the baseline collection.