Schema compliance testing in Katalon Studio

Schema compliance testing is to assert whether a request or a response follows the associated schema definition. Validating API requests and responses against schemas is an efficient and effective way to make sure that APIs are working as expected.

This document shows you how to validate a response body, request body, or string against schemas for API testing in Katalon Studio. You can find a sample project for this feature on our GitHub repository: Validate against schemas sample project.

With this feature, you can:

• Use JSON schema to validate RESTful request and response body content.
• Use XML schema to validate RESTful and SOAP request and response body content.

• Use OpenAPI/Swagger specification to validate RESTful request and response body content.

• Use GraphQL schema to validate GraphQL request body content. To learn more about GraphQL, you can refer to this document: GraphQL.

What is a schema?

A schema concisely describes the structure of other instances, which can be used to require that an instance satisfies a certain number of criteria. The document being validated or described is called the instance, and the document containing the description is called the schema. A schema validation asserts constraints on the structure of the instance data. Recently, there has been a lot of interest in using schemas, such as a JSON or XML schema, as a basis for contract testing. You can learn more about the reason why we need to maintain the schema during the testing process in this document: Update the Schema of API testing.

The infographic below demonstrates the role of JSON/XML schema in API testing:

Infographic based on the Automation Step By Step course: What is JSON Schema

Learn more from the JSON Schema documentation: What is a schema?

Validate against a schema in the web service request

This section walks you through adding validation to a web service request and viewing results.

Note:

• Make sure that you declare the JSON schema used in that URL or file. For more information, you can refer to JSON schema document: $schema.

1. In an API tests project, open or create a new Web Service Request. To create a new Web Service Request, go to Test Explorer > Object Repository. Right-click and choose New > Web Service Request.

   
   
   

2. In the object editor view, switch to the Validation tab.

   

3. To add new validation, click on Add and input the below information:

   Option	Description
   Name	Name of the validation.
   Target	Response: check the response body. Request: check the request body.
   Schema Type	JSON XML OpenAPI/Swagger GraphQL Auto Detect
   Input Type	URL File Auto detect
   Location/Value	Input an URL to a schema, or file path to a schema. If the Input Type is a file, you can click on Browse on the Location/Value cell to browse to the schema file location.
   Validate	Select which schemas to validate against.

4. Validate against the schema:

   When you are done adding validation, click Save. Then, click on the dropdown menu of the Test Request and choose Test Request And Verify.

   
   
   

5. View validation results:

   In the Response section, switch to the Validation Log tab. The validation results appear:

   

   Result:

   • PASSED: All the requests/ responses passed the validation.
   • FAILED: At least one of the request/ response failed the validation.

   Click on each line of the Result table to view Problem details. The Problem details section appears with a list of issues. For example: $[0].password: string found, integer expected, or $[0].username: string found, integer expected.

You have successfully validated requests/ responses against schemas using Katalon Studio.

Validate against schemas in a test case

To validate against schemas in a specific test case, you can add these keywords directly:

• [WS] Validate a JSON string against a schema
• [WS] Validate an XML string against a schema
• [WS] Validate OpenAPI against Specification
• [WS] Validate GraphQL request against a GraphQL schema

View the test case in script view:

res = WS.sendRequest(findTestObject('XML'))

String xml = '''<?xml version="1.0" encoding="utf-8"?>
<List>
    <item>
        <id>3</id>
        <username>James Johnson</username>
        <password>789</password>
        <gender>FEMALE</gender>
        <age>75</age>
        <avatar/>
    </item>
</List>'''

String xmlFile = FileUtils.readFileToString(new File("example/xml/person.xml"), "utf-16");

WS.validateXmlAgainstSchema(res, "example/xml/person.xsd");
WS.validateXmlAgainstSchema(xml, "example/xml/person.xsd");
WS.validateXmlAgainstSchema(xmlFile, "http://localhost:8080/api/users/xsd", FailureHandling.STOP_ON_FAILURE);

Note:

• You can also add the sendRequestAndVerify keyword as a test step. This action also sends the current request and executes verification snippets. Learn more about this keyword at our javadoc Send Request And Verify and Verification Snippets.

When you execute your test, in the Log Viewer, you can see the result of those validation steps and a list of issues (if any).

• Validation passed:

  

• Validation failed:

  

• Send request failed with root causes:

  

Troubleshooting: Failed to validate OpenAPI/Swagger specification error

When validating a schema using a string (e.g., from Swagger UI), you may encounter the following error message:

Failed to validate OpenAPI/Swagger specification (Root cause: java.lang.NullPointerException: An API is required at com.atlassian.oai.validator.OpenApiInteractionValidator$Builder.withApi(OpenApiInteractionValidator.java:466)

Incorrectly formatted escape characters, such as \n, cause Java to misinterpret the string, resulting in validation errors.

Ensure that escape characters are correctly formatted. For example, convert \n to \\n. For more details on escape characters, see this Java documentation: Characters.