Introducing Simulations in Serval: Test your agent like you test your code

IT teams are trusting AI agents with more of their work, from answering questions to granting access and onboarding employees. But each edit to an agent's skills can affect how it handles requests, and teams need a way to know that a change works as expected without breaking anything that used to work.

Software teams solve this with tests that run after every change, but IT teams haven't had an equivalent for AI agents. We built Simulations so teams can test Serval's Help Desk Agent, the AI agent that resolves employee requests end to end, reliably and at scale.

Test an automation while you build it

Take a laptop request form that IT is building in Catalyst. The form should check each requester's employment type in the HR system. If they're a contractor, it should ask for their sponsor and contract end date, then send the request to that sponsor for approval instead of a manager. Partway through building, the team asks Catalyst to test how the Help Desk Agent handles the form. It can now run a Simulation on a test ticket.

In a Simulation, AI models stand in for the employee and your systems, while the real Help Desk Agent handles the request.

1. 1.Catalyst drafts a scenario: a full-time employee asks for a new laptop because theirs keeps crashing. The expected result is that the request goes to their manager, with no sponsor questions.
2. 2.An AI model plays the employee as the simulated requester, and sends the request the way the employee would.
3. 3.The Help Desk Agent works the ticket and fills in the form. Your systems are simulated too: a model called the mocker replies the way your workflow would, like "Order submitted for approval," so no approval or order goes out.
4. 4.Another model, the judge, compares what the agent said and submitted with the expected result. This run passes.

Build a Simulations suite from past tickets

One scenario shows the form works for one kind of employee. To cover the requests that actually come up, the team asks Catalyst to look at past laptop tickets, whether they live in Serval, a connected ITSM like ServiceNow, or a spreadsheet. Catalyst drafts a scenario for each kind of request and groups them into a suite. Because the ticket history includes contractors, the suite covers them too.

Outside of a suite, you can also add scenarios one at a time. Describe what you want to test and Catalyst drafts it, or create one from a ticket that didn't go as expected. You review each scenario before you publish it.

Fix failures before they reach employees

When the team runs the suite, the scenario for a contractor listed as "Contingent worker" fails. The judge expected the form to ask for a sponsor and send the request to that sponsor. Instead, the form treats the contractor as a full-time employee and sends the request to a manager.

From the failed run, the team selects Investigate with Catalyst, which reads the judge's reasoning and the conversation to find the cause. The form's employment type check only looks for "Contractor," the label HR uses today, so it misses older records labeled "Contingent worker." Once the team adds that label to the check, the suite passes, and they publish the form.

Ship every change safely with Simulations

As more people build on Serval, one person's edit to a skill or workflow can break a request someone else relies on. Once the form is live, each edit can run against the laptop request suite as a draft, while employees keep using the published version.

Why you can trust the results

The Help Desk Agent works the same way in a Simulations run as it does in production. When you test a draft, the agent has to find it just as it would any published skill or workflow. If it can't find the draft in a Simulations run, it wouldn't in production either, so you know to improve it before you publish. The judge checks whether the agent reached the right outcome, and it doesn't require one exact conversation.

Access is never granted during a Simulations run, and workflows only run against your real systems if you set them to live and confirm it.

We also use Simulations ourselves. Before we ship a new version of the Help Desk Agent, we run it against our own internal test suites to check for regressions.

Get started

Simulations are available in Serval today. Read our docs to create your first scenario, or book a demo to see Simulations in action.

Frequently asked questions

How do you test an AI help desk agent before it reaches production?

In Serval, you create scenarios that describe an employee's request and the result you expect. The simulated requester plays the employee, and the Help Desk Agent handles the request on a test ticket, separate from your real help desk. A judge then decides whether the agent reached the expected result.

Do Simulations change anything in my real systems?

Not unless you choose to. Access requests are always simulated, so no access is granted and no approver is notified. Workflows are simulated by default, and one only runs against your real systems if you set it to live and confirm it.

Can I turn past support tickets into test cases?

Yes. You can create a scenario directly from a ticket, or have Catalyst draft scenarios from your ticket history. You can then group scenarios into suites and rerun them after every change.