Reinforcement learning environments

Simulated-user environments for multi-turn agents

Last updated /Reinforcement learning environments

A simulated-user RL environment is a multi-turn task. One side is the agent, with tools and a policy. The other side is a simulated customer with a goal the agent does not see at the start. The episode ends when the conversation stops. The grade is not whether the replies sounded kind. In the setup published as tau-bench, the grade compares the database at the end of the conversation with an annotated goal state. The paper is https://arxiv.org/abs/2406.12045 . That citation is their method. It is not a score, and it is not our benchmark.

The type overview is types of RL environments, and what each one trains. The pillar is reinforcement learning environments: what they are and what real work adds. Tools without a second speaker are tool-use RL environments for agents that call APIs. Screens are computer-use RL environments: training agents on screens. App stand-ins are enterprise software replicas as RL environments.

Conversation between a simulated customer and an agent, with a policy page and an end-state check, labeled Buyer guide
A conversation, a policy, and a check on the ending. Buyer guide is not a benchmark score.

Why multi-turn tasks need a simulated user

A single prompt can ask for a refund. A real ticket unfolds: the customer adds a detail, changes the order, or asks a second question. If the only user is a frozen paragraph, the agent never has to recover. A simulated user carries the hidden instruction and answers from the conversation so far. The tau-bench paper says that simulated user does not see the agent's tool calls. The agent can still tell the user something false. You have to grade both.

Conversation between a simulated customer and an agent, with a policy page and an end-state check, labeled Buyer guide
A conversation, a policy, and a check on the ending. Buyer guide is not a benchmark score.

Policy documents as the rules of the job

The policy is the rulebook: what may be refunded, what must be verified, what the agent may promise. Put it in the environment as text the attempt is scored against. Do not hope the model remembers a help center. If your real rule is not written down, do not invent one for the demo and then treat the demo as your process.

Support history that might later supply scenarios, with the licensing questions still open, is how support tickets and chat logs can be licensed. A license to discuss tickets is not this environment.

Hidden user goals with one correct outcome

Each scenario needs a goal the user is pursuing and one outcome the database should reach, or a small set of outcomes your policy treats as equivalent. The goal stays out of the agent's first view. Many wordings can lead to the same database ending. That is a feature of an end-state check: the conversation can vary and the record can still be right. It is a bug if you also needed a required sentence and never graded the words.

Grading the end state and what was said

Compare the database with the goal state, as tau-bench describes. Separately, check claims the agent made to the user against the policy. A correct database with a promise you forbid is a fail. A polite transcript with the wrong refund is a fail. Publish both checks. A single preference score from a judge, with no record of the database, is not this type.

Building user scenarios from real support history

Real tickets can make the hidden goal specific: the order state, the request, the constraint. They also contain names, addresses, and payment details. Remove personal details before a ticket becomes a scenario. If the correct outcome depended on a fact you had to delete, drop that ticket. The simulated user is not the customer who wrote the original thread, and the prompt should not make them that person.

We do not turn a help desk into scenarios on request. An owner decides whether any slice can be discussed. This page does not move tickets.

Limits of simulated users

A simulated user is a model with an instruction. It can drift, agree too easily, or reveal the goal. It does not get tired, and it does not match every customer you have. A benchmark built on one simulator can look stable and still miss the calls your desk actually gets. Keep a set of scenarios you wrote from policy, and do not treat the simulator's chatter as evidence the policy was followed.

  • The user's goal is hidden from the agent.
  • The user cannot see tool traffic, if that is the setup you bought.
  • The database goal is written down, not implied by a friendly ending.
  • Policy breaks are graded even when the database matches.

Questions buyers ask

Is tau-bench a product you sell?

No. It is a published benchmark. We cite how it grades the final database. We do not report a score and we do not resell the benchmark.

Does the paper's user see tool calls?

The tau-bench paper says the simulated user does not see the agent's tool interactions. Confirm that split in any environment you buy. Do not assume it.

Can we grade only the transcript?

Then you will miss a wrong write. Grade the database and the claims.

Are real tickets required?

No. You can write scenarios from the policy alone. Real tickets are optional and only with the owner's permission and personal details removed.

Will a simulated user stay on script?

Not reliably. That is a limit of the method. Read a sample of simulated turns before you trust the reward.

Is this legal advice about customer chats?

No. Whether you may reuse a thread is a rights question. This page only describes the shape of the task.

Describe the data you need

Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.

Tell us what data you need