Use Katalon AI across the testing life cycle

Last updated: November 2026

Use Katalon AI Assistant in Katalon True Platform to move from requirements to reviewed tests, execution evidence, and quality reports. Follow the Demo Shop example to learn what to ask, what to check, and what to carry into the next stage of the testing life cycle.

Treat AI output as a draft

Review generated expectations before saving tests. Confirm changes and outcomes in the project: open saved items, read recorded results, and check cited evidence before using an AI explanation or reporting a pass.

To use an external AI agent, see True Skills and the Katalon True Platform MCP Server for setup, workflows, and tool requirements.

Overview

Stage	Ask Katalon AI Assistant to…	You verify…
Clarify requirements	Identify ambiguities, boundaries, and coverage priorities.	Every accepted rule has a defined outcome.
Generate test cases	Draft cases for the clarified rule and identify overlap with supplied cases.	Assertions match the rule; saved cases and links exist.
Write executable steps	Rewrite vague steps as observable actions and checks.	Steps use real UI labels and saved credential references.
Run tests	Create a run and help execute reviewed cases.	AUT, data, and secrets are ready; recorded results show what happened.
Analyze failures	Separate observations from possible causes and draft supported defects.	Evidence comes from the correct run and step.
Select regression tests	Propose cases for a change and explain the selection.	Critical flows remain covered and gaps are visible.
Report quality	Summarize release evidence with counts and denominators.	Blocked, incomplete, and unexecuted work is included.
Maintain tests	Compare saved cases with changed requirements.	Valid assertions are preserved and modified cases are rerun.

Before you start

Prepare these inputs before asking Katalon AI Assistant to work on your project. The examples use fictional Demo Shop data and illustrative IDs; replace them with your own values.

• Project access: Use a project where you have access to the required AI and testing features.
• Source material: Provide a requirement, acceptance criteria, or an existing test artifact. Configure the relevant external app connection before asking Katalon AI Assistant to read linked content or file a defect.

Before execution, also prepare:

• Test account and data: Use an application under test (AUT) and dedicated test accounts you are authorized to exercise. Prepare synthetic records and the required starting state; assign responsibility for creating and resetting them. A written precondition does not create data.
• Secrets: Store credentials in Secrets & Variables and reference saved keys, such as {demoPassword}. These values require a cloud-hosted execution environment. Keep credentials out of prompts, attachments, and defect reports.
• AUT access: Confirm that the runner can reach the AUT. Configure supported private-network access for private applications. Signing in to Katalon does not sign the runner in to your AUT.
• Login and browser state: Include login steps or call a reviewed login case. If you reuse a browser profile, check the authenticated state, session expiry, and reset needs; shared state can affect later cases.
• OTP and MFA: For configured MailSlurp inboxes, follow Resolve email OTP in AI runs, including inbox naming, concurrency, and timeout requirements. CAPTCHA, hardware prompts, and interactive approvals may need a person or a different test strategy. Keep production authentication controls intact.

Write effective prompts

Give Katalon AI Assistant an identifiable target, relevant evidence, and the output you want. Include requirement text or IDs, case or run IDs, environment, and constraints where they affect the task.

Vague request	Request with actionable context
Test checkout.	Review the acceptance criteria below for Demo Shop cart quantities. Identify missing decisions before generating tests.
Run my tests.	Create a manual run from suite TS-123 against our staging AUT. Use the saved test account. Confirm the selected environment before starting Run with AI.
Explain this failure.	Analyze result TR-456 in run 789. Separate observed behavior from possible causes. Cite the failed step and evidence for each conclusion.
Is the release ready?	Summarize release R-12 using completed runs from October 1 through October 5 in Asia/Ho_Chi_Minh. Include failed, blocked and unexecuted tests, coverage gaps, and the evidence needed for a release decision.

• Supply relevant source material, such as both the old and new rule when requesting test updates. Remove customer data and credentials from attachments.
• Check shared page context and name the target IDs. Restate the target after navigating to another run or requirement.
• Answer clarification questions before continuing. If linked content is unavailable, check access or paste the relevant text; ask Katalon AI Assistant to label unknowns.
• Inspect proposed changes and their targets before approving them. See Tool permissions and approvals.

Continue or start a conversation

Continue the same conversation when you…	Start a new conversation when you…
Clarify a requirement, revise drafts, or answer a question.	Move to an unrelated feature or investigation.
Interpret the same run or need its earlier discussion.	Start a task whose earlier assumptions no longer apply.

Save reviewed drafts before leaving a conversation; reopen the original conversation for unsaved drafts. In a new conversation, provide accepted decisions, saved IDs, the environment, and the open question rather than relying on conversation memory. Indexed project documents can provide additional context through the AI knowledge base.

Examples: revise a draft and start a new conversation

Example — revise one draft in the same conversation

Keep the valid-quantity cases unchanged. Revise only the case for quantity 6. The expected behavior is an inline error and no change to the cart subtotal. Show the revised steps before saving.

Example — hand off saved cases to a new conversation

Continue work on Demo Shop quantity validation using saved cases TC-101 to TC-104 in staging. The accepted range is 1 to 5 inclusive. Quantity 0 or 6 must show an inline error and leave the subtotal unchanged. Review these saved cases for missing assertions. Do not change or run them yet.

Plan and design tests

Use Katalon AI Assistant to clarify expected behavior, prioritize coverage, and turn accepted rules into executable test cases.

Clarify requirements

Start with the business rule and ask for missing decisions. A quantity range alone does not specify whether the application clamps a value, rejects it, or leaves the previous value unchanged.

Example: review a requirement before generating tests

Review this Demo Shop requirement before generating tests: "Customers may buy 1 to 5 units of an item. Values outside the range are rejected." Identify ambiguous outcomes, boundary values, and data or environment prerequisites. Separate questions for the product owner from facts stated in the requirement. Do not save test cases yet.

Answer the questions, then request risk-based coverage. Supply business impact and the time budget when the source does not contain them.

Example: prioritize coverage after clarifying the rule

The highest risk is charging for a quantity different from the cart quantity. We have 30 minutes for this change. Propose smoke and regression coverage, including 0, 1, 5 and 6, and state what you would defer and the remaining risk. Use these priorities as team inputs. Do not invent production incident history.

If you use TrueTest, consider observed user journeys alongside business requirements; an unobserved path can still be critical. See Refine requirements and generate test cases for the product workflow.

Check before you continue

• Missing expectations have accepted answers, and the supplied rule is preserved.
• Coverage reflects release risks, with owners and prerequisites identified for the planned work.

Generate test cases

1. Provide the clarified requirement and existing cases that may already cover it.
2. Ask for missing scenarios, including boundaries and negative outcomes.
3. Review the generated preconditions, actions, data, and expected results.
4. Save selected cases through the available save action or an approved Katalon AI Assistant request.
5. Open the saved cases and verify their IDs, requirement links, and steps.

Example: generate drafts without duplicating supplied cases

Generate manual test drafts for the clarified quantity rule above. Cover valid boundaries 1 and 5, and rejected boundaries 0 and 6. For rejected values, verify the inline error and unchanged subtotal. Identify overlap with existing cases TC-101 and TC-102 before proposing additions. Do not assume a particular error message unless it appears in the requirement.

See Generate test cases with AI for saving and editing options.

Check before you continue

• Every accepted rule has an assertion, and every input has a defined expected result.
• Drafts account for existing coverage and contain no invented labels, permissions, or business behavior.
• Saved cases retain the reviewed content and requirement links. If saving is unclear, check for existing cases before retrying.

Write executable steps

• Give each step an observable action or check, using real control labels and explicit data. Split actions whose failures need separate evidence.
• State exact expected text only when the requirement defines it; describe relationships for dynamic values, such as subtotal calculations.
• Define the starting state and reset needs. Configure and check any deliberate dependencies between cases.

Ambiguous step	Executable rewrite
Log in and test the cart.	Open the staging sign-in URL. Enter the saved username and password in separate steps. Select Sign in. Verify the account page appears. Open the seeded cart.
Use invalid data.	Enter 6 in Quantity. Select Update. Verify the validation error appears and the subtotal remains the value recorded before the update.
Check everything works.	Verify Quantity displays 5, the unit price is the seeded price, and the subtotal equals five times that price.
Wait until processing is done.	Wait for the order status to become Confirmed within the timeout your test environment supports. Record a failure if the expected state does not appear.
Wait 24 hours and check the email.	Create the event in one test. Use an approved scheduler or prepared aged record for the later check. Run a separate test against that known record.

Example: executable test case for quantity validation

Preconditions: a synthetic customer account exists, its cart contains one Demo Mug priced at 10.00 USD with quantity 1, and demoUsername and demoPassword are saved secrets. Replace the fictional URL and seed data before execution.

Step	Action	Expected result
1	Open https://shop.example.com/sign-in.	The sign-in form appears.
2	Enter {demoUsername} in Username.	Username is populated.
3	Enter {demoPassword} in Password.	Password is masked.
4	Select Sign in.	The customer account page appears.
5	Open Cart.	Demo Mug has quantity 1 and subtotal 10.00 USD.
6	Replace Quantity with 5.	Quantity displays 5.
7	Select Update.	Quantity is 5 and subtotal is 50.00 USD.
8	Replace Quantity with 6.	Quantity displays 6 before submission.
9	Select Update.	A quantity validation error appears and subtotal remains 50.00 USD.

This example assumes an editable quantity field that validates on Update. Confirm that interaction with your requirement before adopting the steps.

Check before you continue

• Each action identifies its control and data, and each expected result is observable.
• Credentials use saved references; the case has a reproducible starting state.

Run and analyze tests

Use Katalon AI Assistant to execute reviewed cases and investigate their recorded results. Complete the execution prerequisites in Before you start first.

Run tests with AI

1. Create a manual test run from the reviewed cases or suite.
2. Verify the target AUT, test data, secrets, and execution configuration.
3. Select Run with AI, review the execution environment, and select Start AI Execution.
4. Monitor AI Sessions and respond if execution asks for input.
5. Inspect final case and step results, logs, screenshots, and video where available. Resolve any pending result review.
6. Record failed, blocked, skipped, and incomplete work separately from passed assertions.

Example: inspect the recorded outcome of a run

Read the results of run 789. List each case ID and its recorded status. For failures, cite the failed step and evidence. Identify cases that never finished.

Browser and mobile coverage

One browser run does not establish cross-browser coverage. Use automated test runs for existing executable suites and their required configurations. For mobile targets, follow the separate mobile browser, native app, or live testing procedures. A web Run with AI configuration does not establish native-app AI support.

Check before you continue

• Results correspond to the intended cases, environment, and configuration.
• Each reported pass has recorded assertions; unfinished work remains visible.

Analyze failures and file defects

For AI-executed manual cases, inspect the session and step evidence. Automated-result failure analysis is a separate workflow; use the controls available for that result type.

Example: investigate a failed assertion before drafting a defect

Review the failed quantity-6 assertion in run 789. Report the expected state, observed state, and evidence for the difference. Separate a product defect, incorrect test expectation, missing seed data, authentication problem, and execution-environment failure as hypotheses. State what evidence would distinguish them. Draft a defect only if the evidence supports a product problem. Do not submit it yet.

Prepare the build and environment, prerequisites, reproduction steps, expected and actual results, frequency, impact, and evidence links for the defect draft. Jira or Azure DevOps filing requires the relevant integration and permissions; see File bug tickets.

Check before you continue

• The explanation cites the correct run and step; reproduce the behavior if needed to distinguish possible causes.
• Search for an existing defect and review the draft and target tracker before submission.
• Verify the resulting issue link and its association with the failed result.

Report and maintain

Use Katalon AI Assistant to select regression coverage, summarize quality evidence, and revise tests after changes.

Select regression tests

Provide changed requirements, affected components, and known risks. Ask for the reason behind each selected case and the gaps left by the selection.

Example: select regression coverage for a quantity change

Using the supplied change description and suite TS-123, propose regression cases for quantity validation, cart totals and checkout totals. Keep critical purchase coverage. Show the case IDs and selection reason. List uncovered risks and assumptions. Do not delete cases or start execution yet.

Check before you continue

• Critical flows remain covered and uncovered risks are explicit.
• Review and execute the selection before using its results in a report.

Report release quality

Specify the release or sprint, environment, date range, and timezone. Request counts and denominators so that percentages do not hide unexecuted work.

Example: summarize evidence for a release review

Summarize run 789 for the release review. Include total selected cases, passed, failed, blocked and unexecuted cases, requirement coverage gaps, open defects and untested configurations. Link to the evidence. Separate recorded facts from your release recommendation.

Historical execution results and current coverage answer different questions. Save dated coverage snapshots for comparisons over time; see Reports and analytics.

Check before you continue

• Counts reconcile with the selected scope, including incomplete and unexecuted tests.
• Missing data is identified as unknown, and readiness recommendations account for coverage gaps and open defects.

Update tests after changes

Compare accepted behavior with the saved test when a requirement, UI label, or environment changes. Use test case versions to inspect saved revisions where available.

Example: revise a saved test and verify the change

Compare TC-104 with the revised quantity requirement below. Show the steps and expected results that must change and explain each change. Preserve unaffected checks. After I review the edit, rerun the affected case and report the new result.

Check before you continue

• Preserve valid assertions and review proposed deletions.
• Rerun modified cases and record new executions separately from the old failures. Review affected automation scripts and locators separately from manual-step edits.
• Return newly discovered gaps to the coverage plan.

Troubleshoot

Use these checks when a Katalon AI Assistant workflow is incomplete or its output does not match the project evidence.

Symptom	Next action
Generic or irrelevant tests	Supply the exact requirement and expected outcomes. Ask which assumptions produced the irrelevant cases.
The next chat cannot find a draft	Reopen the original conversation, save the reviewed draft, then provide the saved IDs in the new chat.
A linked requirement cannot be read	Check the connector, project scope and permissions. Supply relevant text when access is unavailable.
Save fails or the result is unclear	Check the project for persisted cases before retrying. Retain the error and conversation details for support.
Execution stops at login	Check AUT access, saved credential keys, explicit login steps, OTP requirements and session expiry.
A run stays queued or asks for input	Inspect the session and available execution capacity. Resolve the prompt or prerequisite before treating the test as failed.
A step times out waiting for business processing	Use bounded checks, a prepared record or a separate scheduled follow-up appropriate to the business delay.
Mobile target or app is unavailable	Verify feature access and app/device prerequisites. Use the supported manual or automated mobile path.
An AI summary disagrees with the run	Open the cited result and evidence. Correct the scope or expectation before using the summary in a report.