Generate API tests with Katalon Studio's AI

This document introduces two AI-powered workflows with OpenAPI (CATS) specification files

• Generate API tests with AI (beta): Import your OpenAPI file and create/save test cases/test objects with the help of AI, significantly reducing time and effort. You can customize, maintain, and execute them as part of your CI/CD workflows.
• API test generation and report viewing with AI: Import your OpenAPI file to have Katalon’s AI agent automatically generates tests, executes them, and provides a results report without creating or storing any test artifacts in your project.

Together, these features give you flexibility: you can either establish a maintainable API testing framework or perform quick, on-demand API validation depending on your testing needs.

Generate API tests with AI (beta)

This feature imports your OpenAPI file and creates saved, editable test objects and test cases in your project with the help of AI, significantly reducing time and effort. With saved test cases/objects, you can customize, maintain, and execute them as part of your CI/CD workflows.

The test case generator currently supports OpenAPI 2 (Swagger) and OpenAPI 3.

• Katalon Studio Enterprise version 9.6.0 onwards.

• The Account Admin has enabled AI Control for your Account. See: AI Control.

This guide shows you how to use the API test case generator feature (beta) Katalon Studio. Follow these steps:

1. In Katalon Studio, create or open an API/Web Service project.
2. In the toolbar, click Import from OpenAPI / Swagger button. You can also right-click Object Repository and select Import > From OpenAPI.
   
3. In the import dialog, provide the file location or URL of the API specification. Tick Auto-generate test cases for endpoints with AI button, then click OK
   
   The imported endpoints are automatically stored in Object Repository.
4. All imported endpoints are then listed for test generation. Select the endpoints to generate test cases, then click Generate tests.
   

   The test case generation process may take a few minutes depending on the number of selected endpoints. You can click hide to let the process run in background while doing other tasks.

5. Once the test cases are generated, you can select which test cases to save and click Add.
   

The generated API test cases are organized in a single folder of Test Cases.

Known limitations

The beta release of API test case generator comes with some limitations:

• Accuracy: generated test cases might contains made-up code snippet. You need to revise the test cases before executing.
• Generation time: Test case generation may take a long time. You should click hide to let the generation process in background.
• Limited test case types: Currently the feature only generates status code test cases.

• Error: The error message No test case is generated may occur when the request to OpenAI exceeds the token limit of 16,385 tokens in the GPT-3.5-turbo model. This happens when using large test objects, leading to failed endpoint test case generation.

Once the test cases are generated, you need to revise the test scripts to ensure that they can run as expected. For example, the script may use undefined global variable, such as global.baseUrl, you need to create these variables in Global variables and Execution profile.

API test generation and report viewing with AI

By uploading an OpenAPI specification file, Katalon’s AI agent automatically generates tests, executes them immediately, and provides a results report—without creating or storing any test artifacts in your project.

requirement

• This feature is currently available as a beta feature in the RC build of Katalon Studio 11.0.0 and will be included in the official 11.0.0 release.

1. Import your OpenAPI spec file.

   Import your OpenAPI document to create an API Collection folder in the Object Repository.

   This feature currently supports header-based authorization only. If your project uses other authentication types (NTLM, Digest, etc.), those mechanisms will not be included in the generate commands.

2. Double-click on this API collection folder, and select Generate Test tab.

3. Select test types (positive, negative, edge and security testing), API paths, and HTTP methods to test. The options are retrieved from your specification file. If you don't select any API path before generating, KS automatically use all paths to generate the tests.

4. Click Generate Test. Katalon Studio will generate tests using AI for the OpenAPI Spec file, and then execute the tests right after. You can see the progress bar, and click Hide to let KS work in the background.

5. Once done, the report summary pops up for your viewing:

You can then also view the full HTML report, with more details on scores, errors, logs...

After reviewing everything, you can save the test generation configuration to rerun with same configurations later:

This is useful for validating fixes after API updates and comparing results across multiple iterations without reconfiguring the test setup each time.