Reinforcement learning environments

Verifiers and rewards: how an RL environment decides success

Last updated /Reinforcement learning environments

An RL verifier is the function that reads an attempt and returns a reward. The attempt is whatever the agent did inside the environment. The reward is the number, the pass, or the partial credit that training will treat as success. If that function can be satisfied without the task being solved, the environment is teaching the shortcut. Grader quality is the product. The container is the stage.

This guide sits under reinforcement learning environments: what they are and what real work adds. It is about the score, not about a file review and not about a model scorecard. Held-out cases used later to test an agent are how past business outcomes become agent test cases. Do not mix a grader audit with a model leaderboard.

Flow from an agent attempt through verifier checks to a reward score, labeled Buyer guide
Attempt, then checks, then a score. The Buyer guide chip is not a certification of any grader.

What a verifier is

A verifier is code or a rubric that runs after the attempt, or sometimes during it. It can read the final database, a file the agent saved, the tool log, or the words that were said. It should not read a solution that was sitting in the prompt. The cleanest verifiers are boring: they compare an ending to a condition a second person can restate.

People also say reward function and grader. Use the words your contract uses, and make sure they point at the same function. A narrative that says the environment is verifiable, without showing the check, is not a verifier.

Flow from an agent attempt through verifier checks to a reward score, labeled Buyer guide
Attempt, then checks, then a score. The Buyer guide chip is not a certification of any grader.

Deterministic checks vs LLM judges

A deterministic check has a rule a program can apply the same way twice. Tests passed. The invoice matches the purchase order. The ticket status is the required value. An LLM judge is a model scoring an answer against written criteria. Use a judge when the outcome is prose and there is no single correct string. Do not use a judge to avoid writing a check that the database could have answered. How to write the rubric is LLM judge rubrics for tasks without one right answer.

Judges drift. Pin the judge model, keep the rubric text, and spot-check scores. A deterministic check can be wrong too, if the rule is loose. The choice is which failure you can detect.

Binary, partial and combined rewards

Binary rewards are pass or fail. They are easy to explain and harsh on almost-right attempts. Partial rewards give credit for pieces of the job. They can also pay the agent for a fragment that should not count. Combined rewards add a state check to a rubric, or a format check to an outcome check. Write the weights down. A combined score with a secret formula is not something you can accept.

Checking the final state

State-based grading compares the world after the attempt to a target. Many paths can be valid if the ending matches. That is a feature when the job has more than one legal sequence. It is a bug when a required action, such as a disclosure to the user, never touches the database. If the words matter, grade the words separately from the records. The longer account is state-based graders: checking what changed, not how.

When you receive the package, the check belongs in the same review as the files. A receiving team already has a pattern for schemas and integrity in what a receiving team checks before accepting a dataset. An environment adds the grader to that list. Arrival of a container is not acceptance of the score.

Reward hacking and soundness

Reward hacking is any way to earn the reward without doing the job. The answer was in the prompt. A future state was readable. The unit test checked a string the agent could echo. The judge liked a fluent refusal. Soundness means a high reward implies the task was solved. Epoch AI's FAQ, from interviews with environment builders and lab staff, treats that robustness as a top quality bar: https://epoch.ai/gradient-updates/state-of-rl-envs . The buyer page is reward hacking in RL environments.

Ask for the leaks they already closed, and for a chance to try a cheat yourself. A polished interface does not answer this. Red-team notes do.

Difficulty and pass rates

A task nobody can solve, and a task everybody solves, both fail as training signal. The same Epoch FAQ reports interviewees wanting tasks that are hard but feasible, including a minimum pass rate around 2 to 3 percent, described as at least one success in 64 or 128 rollouts. Those figures are their interviewees' targets, not a standard this site enforces, and not a score you should invent for a sample you have not run. How to ask for the measurement is calibrating RL task difficulty.

Ask how the pass rate was measured, on which model, and over how many attempts. A single lucky solve is an anecdote. Also ask what they do when a task becomes easy after training. Retiring it is a process, not a slogan.

Grader evidence buyers should receive

Before you rely on a reward, ask for the verifier source or a readable spec, a few attempts that should pass, a few that should fail, and the reason each one does. Ask what the grader is forbidden to see. Ask whether a second person solved the task blind. We will not certify that evidence. Your team compares it with the brief.

  • The check is re-runnable on a fresh reset.
  • Failed attempts fail for the reason the spec names.
  • A cheat you can think of in ten minutes does not pass.
  • The reported difficulty comes with the measurement, not a single percent.

Describe the training need if you want profiles compared. A request does not mean a sound grader is waiting.

Questions buyers ask

Is a unit test always enough?

Only when the job is exactly what the test measures. A test that checks a filename can pass while the work inside is wrong.

Should we ban LLM judges?

No. Ban them as a substitute for a check you could have written against the records. Use them when the outcome is judgment, and keep the rubric.

Is a 2 to 3 percent pass rate a requirement you impose?

No. That range is reported by Epoch AI's interviewees at the URL above. Your spec should name the band you want and how it will be measured.

Does a high reward mean the agent is good?

It means the grader returned a high score. Soundness is the claim that those two things match. Ask for evidence.

Can we accept the environment and audit the grader later?

You can, and you will then train on a score you have not checked. Acceptance of the bytes is not acceptance of the reward.

Do you grade vendors for us?

No. This page lists the questions. It is not a scorecard and not a certification.

Describe the data you need

Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.

Tell us what data you need