Reinforcement learning environments
State-based graders: checking what changed, not how
Last updated /Reinforcement learning environments
A state-based grader compares the world after the attempt with a target. The world might be a database, a file, or the settings of an app. Many sequences of actions can be valid if the ending matches. That is the point: you grade what changed, not the particular path, unless a path was required. The tau-bench paper uses this idea for conversations: it compares the database at the end with an annotated goal state, so different wordings can still pass when the record is right (https://arxiv.org/abs/2406.12045). OSWorld's paper does the computer version: example-specific scripts read the final machine state, such as files or the accessibility tree, and then decide success (https://arxiv.org/abs/2404.07972).
Those citations are methods. They are not scores. The verifier hub is verifiers and rewards: how an RL environment decides success. The pillar is reinforcement learning environments: what they are and what real work adds. Siblings are reward hacking in RL environments, LLM judge rubrics for tasks without one right answer, and calibrating RL task difficulty.

Comparing the end state to a target
Write the target as fields, files, or settings a program can read. The attempt passes when those match and fails when they do not. The drawing shows a before table and an after table to make that comparison visible. It is not a schema from a delivery and not a customer's database. If you cannot name the target without looking at the agent's trace, you do not have a state check yet.

Reference paths vs required actions
A reference path is one way a person solved the task. It is useful as a hint and dangerous as the only passing trace. If any other legal sequence fails, you are grading style, not outcome. Required actions are different: a disclosure that must be said, an approval that must be recorded, a test that must be run. Put required actions in the spec. Do not smuggle them in by demanding the reference path.
Checking what the agent told the user
State checks miss sentences. An agent can set the refund flag and also promise a remedy the policy forbids. The database matches. The customer was misled. If the words matter, grade them separately, with a rubric or a list of forbidden claims. The rubric page is the tool for that residue. A state check that pretends to cover the conversation will pass lies that never touched a column.
Getter and evaluator functions
OSWorld's writeup of execution-based evaluation, in the paper linked above, separates gathering the final facts from deciding whether those facts pass. One function reads the file or the interface. Another applies the rule. That split is worth copying even when you are not using their benchmark: the getter should not contain the pass rule, and the evaluator should say what it compared. A single opaque script that returns true is harder to red-team.
Where state checks miss policy breaks
Anything that is not in the state will not fail the check. A skipped warning, a rude sentence, a tool call that was forbidden but left no row: all of those need their own clause. Also watch leaks. If the target row sits in the same database the agent can read, the agent can copy the target instead of doing the work. That is reward hacking, covered on the sibling page. Soundness and end-state comparison are not the same property.
Using real outcomes as targets
Past business outcomes can be the target: the ticket state after a real fix, the match a clerk posted, the resolution a policy allowed. Hold that target out of the starting view. Outcomes used later as tests of an agent, which is a different project from grading one attempt, are described in how past business outcomes become agent test cases. Do not mix a state check with a leaderboard.
- The target fields, files, or settings, written down.
- Which paths may vary, and which actions are required.
- A separate check for claims to the user, if those claims matter.
- The target hidden from the agent.
We do not run these graders. A diagram of two tables is not a database you can train on.
Questions buyers ask
Does the final state have to match one exact trace?
No. Match the target outcome. Require an action only when the job requires that action, and say so.
Is tau-bench's database check the only pattern?
No. It is one published example, cited above. OSWorld reads files and interface state instead. Your target should match your system of record.
Can a state check see that the agent lied?
Only if the lie is in the state. Otherwise add a check on what was said.
What if the agent can read the goal row?
Then the check can be copied. Hide the target. That leak is reward hacking.
Are past outcomes the same as this grader?
They can supply the target. Using them as a held-out test of an agent is the other page linked above. Do not treat the two as one review.
Will you write the getter?
No. Ask the vendor for the function that reads the ending and the function that decides the pass. We do not supply them.
Describe the data you need
Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.
Tell us what data you need