Call AI from test scripts with LLM keywords

Last updated: October 2026

Requirements

• Katalon Studio version 11.5.0 or later.
• AI services are enabled for your account. See Configure AI Services.
• An AI provider is selected in Katalon Studio. LLM keywords follow the same license requirements as Katalon AI Assistant. See Access and licensing.

LLM keywords call the AI service configured in Katalon Studio while a test runs. Ask keywords return a response you can use in later steps. Assert keywords judge whether text or files meet an expectation and pass or fail the step.

Choose the right keyword

What you need to check	Use
An exact value or pattern, such as an ID, amount, or status code	A standard assertion, such as WebUI.verifyMatch or a Groovy assert
Meaning that can be expressed in different words	LLM.assertText for text, or LLM.assertFile for files
A value to extract or content to summarize for a later step	LLM.ask, then a standard assertion if the result must match an exact value or pattern

An AI judgment can vary between runs. Use standard assertions when the expected value is exact.

Use LLM keywords in existing scripts

If your script currently...	Consider...
Calls an AI provider API with a custom HTTP request and API key	LLM.ask with the provider configured in Katalon Studio
Uses a long regular expression to accept many wordings of the same message	LLM.assertText with a plain-language expectation
Extracts text from a PDF with a third-party library to check its meaning	LLM.assertFile, or LLM.ask to extract a value before an exact assertion
Saves a screenshot for someone to review later	LLM.assertFile with the expected visual result

Keep exact checks that already work. Replace only checks that benefit from interpreting variable content.

Available keywords

Keyword	Use it to	Reference
LLM.ask	Return a response to a prompt, optionally using attached files	[LLM] Ask
LLM.assertText	Judge a text value against an expectation	[LLM] Assert Text
LLM.assertFile	Judge one or more files against an expectation	[LLM] Assert File

Add and run LLM keywords

Select an AI provider

LLM keywords use the AI provider selected in Katalon Studio. They do not have a separate configuration.

1. Go to Katalon Studio > Preferences > AI configurations.
2. Select Katalon AI Service or Personal API key for a supported provider.
3. Click Apply and Close.

For provider details and supported models, see AI service configuration.

Add a keyword to a test case

In the Manual view

1. Open a test case and select the Manual tab.
2. Click Add > AI Keyword.
3. In the Item column, select ask, assertText, or assertFile.
4. Double-click the Input cell, enter the prompt or expectation and any other required values, then click OK.
5. For ask, enter an Output variable name to use the response in later steps.
6. Save the test case.

For each keyword's inputs, see Ask, Assert Text, or Assert File.

In the Script view

Call the keywords with the LLM alias. New test cases import the alias automatically.

String invoiceNumber = LLM.ask('Return only the invoice number in this PDF.',
    'Data Files/invoice.pdf')

LLM.assertText(confirmationMessage,
    'The message confirms that the order was completed.')

LLM.assertFile('Data Files/checkout.png',
    'The image shows a successful checkout confirmation.')

Run the test and review the result

1. Run the test case or test suite as usual. LLM keywords also run in Katalon Runtime Engine (KRE) and CI/CD pipelines.
2. Open the Log Viewer. An assert step logs its pass or fail verdict and the AI's reason. An ask step returns text that you can log or pass to another keyword.

How LLM keywords work

Results and errors

LLM.assertText and LLM.assertFile report Pass when the AI judges that the expectation is met and Fail when it is not. The Log Viewer shows the reason for the verdict. Ambiguous or partial matches fail.

Invalid input, an unavailable provider, or an empty or unreadable AI response causes an Error. Failures and errors follow the test step's failure handling setting. If execution continues after an error, LLM.ask returns null.

Files and provider

LLM.ask attachments and LLM.assertFile files support png, jpg, jpeg, webp, gif, heic, heif, txt, csv, json, log, and pdf. Each file must be 10 MB or smaller. Paths can be absolute or relative to the project folder. Invalid files cause an error before the request is sent.

LLM.assertFile judges multiple files together against one expectation. The selected AI provider and model apply to all LLM keyword steps; you cannot choose a model for an individual step. Prompts and files are sent to that provider, so remove or mask sensitive data before using them.

Example use cases

Check a message whose wording can vary:

String reply = WebUI.getText(findTestObject('Chat/lbl_LastReply'))
LLM.assertText(reply, 'The reply explains how to reset a password without asking for the current password.')

Extract a value from a PDF, then check its exact format:

String invoiceNumber = LLM.ask('Return only the invoice number in this PDF.',
    'Downloads/invoice-1042.pdf')
WebUI.verifyMatch(invoiceNumber.trim(), 'INV-\\d{5}', true)

For examples with screenshots, multiple files, and failure handling, see the Assert File, Ask, and Assert Text references.

Good practices

• Check one outcome per assertion. A focused expectation makes a failed step easier to understand. State the meaning to check and include any facts that must be present.
• Use assertText or assertFile to judge content. Use ask when a later step needs the returned text. If that text feeds an exact check, request a strict format and validate it with a standard assertion.
• Choose failure handling deliberately. OPTIONAL can let a test case pass after an AI assertion fails. Use another setting when the verdict must affect the test result.
• Limit AI requests. Send only the files needed for the check.

Troubleshooting

Symptom	What to check
No AI provider is available	Select a provider in Katalon Studio > Preferences > AI configurations, and confirm that AI services are enabled for your account. See Select an AI provider.
assertText fails although the text looks right	Read the reason in the log. Narrow or split an expectation that combines several outcomes; partial matches fail.
assertFile reports an error before the check	Check that each path exists, its type is supported, and its size is at most 10 MB. Relative paths start from the project folder.
LLM.ask returns null	Check the error in the log. Failure handling may have allowed execution to continue.
LLM.ask returns different answers between runs	Request a strict output format. Use an assert keyword if you need a verdict rather than returned text.
A failed AI check did not fail the test case	Check whether the step uses FailureHandling.OPTIONAL.