Use TestOps Visual Testing

You can compare images captured during test executions with TestOps Visual Testing.

TestOps

Set up a new baseline

You can set up a new baseline image to compare it with the screenshot of the next Test Run:

notes

If you run a Test Suite for the first time and the Test Suite has passed, the screenshots are automatically marked as Passed and there is no Save to baseline button. You can open a screenshot and click Failed, then switch back to Passed so that the Save to baseline button appears.

1. Navigate to your specific project's UI > Executions. Select the test suite that has Visual Testing.

2. Click the Visual Testing tab. Then, select a checkpoint you need to configure.

3. Click Mark as Passed.
4. Then, click Save to baseline.

Unresolved images

If you run the Test Suite again and the new screenshots from this test execution differ from the baseline images, the status of this Visual Test Run is marked as Unresolved. To resolve these images:

1. Navigate to your specific project's UI > Executions. Select the test suite containing the Visual Testing results you need to resolve.

2. Click the Visual Testing tab, then select an Unresolved checkpoint.

• If there is only a Baseline and no Checkpoint, it means the test suite did not capture a screenshot for that checkpoint.

• If there is a Checkpoint and no Baseline, it means the test suite captured a screenshot, but there is no baseline to compare it to. You can make that Checkpoint the new Baseline.

• If there is a mismatch, it means the screenshot captured differ from the baseline, you need to compare the images:

  • Select one of the three comparison options to compare the new screenshot (on the right) with the baseline image (on the left). The Comparison Method is enabled only if the screenshot status is Mismatch.

    • Pixel: This method compares the pixel resolution of two images to figure out the pixel-by-pixel differences between them. See: Pixel-based comparison.
    • Layout: This method identifies and maps the similar zones between the baseline and checkpoint images, highlighting the layout differences between the two images. See: Layout-based comparison.
    • Text Content: This method identifies the text content differences between two images. See: Content-based comparison.
    

3. Click Mark as Passed or Mark as Failed to resolve the checkpoint.

Ignored zones

Specifying ignored zones is especially useful when testing a website with dynamic content, such as pop-up ads, that changes between runs. By defining ignored zones, Visual Testing will skip these areas when comparing captured screenshots to their baselines.

Add an ignored zone to a single baseline image

To configure an ignored zone, follow these steps:

1. In your project’s UI, navigate to Assets > Visual Baseline Collections and open the desired collection.

2. Click on any images you want to set ignored zone for.

3. Click Configure Ignored Zones. The button is then highlighted in purple, which indicates the edit mode.

4. Draw (in rectangles) to specify the area you want to ignore. The ignored zones are highlighted in purple. You can draw up to 20 ignored zones (i.e. 20 rectangles). If you want to apply an ignored zone to all baseline images, see: Apply an ignored zone to all baseline images.

5. Click Save. You have temporarily saved the ignored zones on this baseline image.

6. This update is currently in a draft state (e.g., Version 19 (Draft)). To permanently save the ignored zones under a new version, click Save Version at the top right corner.

Result

The baseline image now contains the defined ignored zones. During visual comparisons, these zones will be excluded, allowing for more accurate test results by ignoring dynamic or irrelevant content.

This configuration is now saved under a new version, for example, Version 19 (Latest Version):

Apply an ignored zone to all baseline images

Manually adding the same ignored zone to multiple baseline images can be time-consuming, especially in large collections. With TestOps Visual Testing, you can apply an ignored zone across all baseline images.

Note

The Apply to all feature only works if all baseline images in the collection have the same resolution (e.g., 400×500). Zones cannot be applied across images of different sizes.

To apply an ignored zone to all baseline images in a collection:

1. In your project’s UI, navigate to Assets > Visual Baseline Collections and open the desired collection.

2. Click on any images you want to set ignored zone for.

3. Click Configure Ignored Zones. The button is then highlighted in purple, which indicates the edit mode.

4. Draw (in rectangles) to specify the area you want to ignore. These ignored zones will appear in purple. You can create up to 20 ignored zones per image.

5. Click on the drawn zone, then click the checkmark icon to confirm that you want to apply this zone to all baseline images in the collection.

6. An Acknowledge popup appears. Click Acknowledge to confirm. The ignored zone now turns green, indicating that it has been applied to all baseline images in the collection.

7. Click Save. You have temporarily saved the ignored zones on this baseline image.

8. This update is currently in a draft state (e.g., Version 19 (Draft)). To permanently save the ignored zones under a new version, click Save Version at the top right corner.

Delete ignored zones

You can delete an ignored zone from a single baseline image or from all baseline images in the baseline collection.

• Purple zones indicate ignored zones applied to a single baseline image.
• Green zones indicate ignored zones applied to all baseline images in the collection.

1. Click Configure Ignored Zones. The button will be highlighted in purple, indicating that you are in edit mode.

2. Click on the drawn zone, then click the Delete (trash can) icon to remove it.

   • If the ignored zone is green (indicating it's applied to all images with the same resolution), a Delete All Ignored Zones confirmation dialog appears. Click Delete to confirm.

3. Click Save to temporarily save the configuration.

4. This change is currently in a draft state (e.g., Version 19 (Draft)). To permanently save the ignored zones under a new version, click Save Version at the top right corner.

Hide or unhide ignored zones when comparing checkpoint

When comparing the checkpoint image with the baseline image, you have the option to show or hide Ignored Zones in the comparison interface.

TestOps Legacy

Legacy Version

Configure a baseline image

You can set up a baseline image to compare it with the screenshot of the next Test Run. Follow these steps:

1. Select a screenshot to see the details.
   
2. Select Mark as Passed at the top right corner of a screenshot, then close the screenshot.

   An Unsaved label now appears on the Results page.

   

3. Click Save to baseline.
   Note:

   • If you run a Test Suite for the first time and the Test Suite has passed, the screenshots are automatically marked as Passed and there is no Save to baseline button. You can open a screenshot and click Failed, then switch back to Passed so that the Save to baseline button appears.

Unresolved images

If you run the Test Suite again and the new screenshots of this test execution are different from the baseline images, the status of this Visual Test Run is then marked as Unresolved. To resolve these images, follow these steps:

1. Click on the ID of the unresolved Visual Test Run.
2. Select the unresolved checkpoint.
   
   
   

   You see a drop-down list in the Comparison Method section. The three options include: Pixel, Layout, Text Content. For detailed information of each comparison method, see: TestOps Visual Testing.

3. Select one of the three options to start comparing the new screenshot (on the right) with the baseline image (on the left).
   Note:

   • Comparison Method is enabled only if the status of the screenshot is mismatched.
   • If you select Pixel, check the Diffs between two images (the differences are highlighted in red). See: Pixel-based comparison.
   • If you select Layout, check the Identical, Distinguishable and Missing/New boxes to see the differences highlighted in green/red/pink color respectively. See: Layout-based comparison.
   • If you select Text Content, check the Identical, Shifted and Missing/New boxes to see the differences highlighted in green/blue/pink color respectively. See: Content-based comparison.
4. Click Mark as Passed or Mark as Failed to resolve.
   Note:

   • The Unsaved label appears whenever you change the status (Passed or Failed) of a screenshot. If you click Save to baseline, the new Passed screenshot replace the older baseline image. Katalon TestOps then compares between the new baseline image and screenshots of the next Visual Test Run.

Ignored zones

Specifying ignored zones is especially useful when testing a website with some dynamic and changing content (such as pop-up ads on specific areas). Visual testing can ignore such content while comparing captured images with the baseline one.

In TestOps Visual Testing, you can configure an ignore zone in a baseline image and apply this ignored zone to all baseline images in the baseline collection. You can also delete the ignored zone(s) in all baseline images.

Add an ignored zone to a baseline image

To configure an ignored zone, follow these steps:

1. Sign in to Katalon TestOps and go to your project.
2. Go to Visual Testing > Visual Baseline Collections.
3. Select the Baseline Collection ID you want to check.
   The list of baseline images appears.

4. Click on the Edit image icon of the baseline image you want to configure (e.g., the baseline image image10).
   The page displays as below.
   
   
5. Click on the Configure Ignored Zones button.
   The button is then highlighted in blue, which indicates the edit mode.
   
   
   
6. Draw (in rectangles) to specify the area you want to ignore.
   The ignored zones are highlighted in purple. You can draw up to 20 ignored zones (i.e. 20 rectangles).

   Note:

   • If you want to apply an ignored zone to all baseline images, see: Apply an ignored zone to all baseline images.

7. Click Save.
   You have temporarily saved the ignored zones on this baseline image.
   
   
   

   Note:

   • You have not permanently saved the configuration of ignored zones at this stage.
8. Click Save Changes on the top right corner to permanently save the ignored zones you have configured.
   
   
   

The baseline image has now had ignored zones. Visual testing then ignores these zones when comparing captured images with this baseline image.

When comparing the checkpoint image with the baseline image, you can show or hide the ignored zones by ticking/un-ticking the Ignored Zones checkbox.

Apply an ignored zone to all baseline images

Adding an ignored zone to the same area in multiple baseline images could become time-consuming if there are more than one baseline image. With TestOps Visual Testing, you can apply an ignored zone to multiple baseline images in a baseline collection.

To apply an ignored zone to all baseline images within one baseline collection, follow these steps:

1. Configure an ignored zone. See: Add an ignored zone to a baseline image.
   The ignored zone is displayed in purple.
2. Click on the ignored zone, then select the Apply the zone to all images icon.
   
   You then see the Acknowledgement pop-up box.
3. Click Acknowledge.
   The ignored zone is now displayed in yellow because it has been applied to all other baseline images in the baseline collection.
4. Click Save to temporarily save the configuration.
5. Click Save Changes on the top right corner to permanently save the configuration.

You have applied an ignored zone to all baseline images within the selected baseline collection.

Delete ignored zones

You can delete an ignored zone in one baseline image and/or all baseline images in the baseline collection.

An ignored zone applied to one baseline image only is displayed in purple. To delete it, click on the ignored zone and select the Delete ignored zone icon.

An ignored zone applied to all baseline images is displayed in yellow. To delete it, follow these steps:

1. Click on the ignored zone and select the Delete ignored zone icon.
   
   The Delete all ignored zones box pops up.
2. Click Delete all zones to confirm your action.
3. Click Save to temporarily save the configuration.
4. Click Save Changes to permanently save this configuration.
   The ignored zone is now deleted in all baseline images of the baseline collection.