Reinforcement learning environments
Calibrating RL task difficulty
Last updated /Reinforcement learning environments
A task that every attempt fails, and a task that every attempt passes, both fail as training signal. Epoch AI's FAQ reports that point from interviews: if the pass rate is 0 percent or 100 percent, the model will not learn from the task. The same FAQ says interviewees wanted a minimum pass rate around 2 to 3 percent, described as at least one success in 64 or 128 rollouts, so the task is hard and still solvable. One person described wanting a mix, including tasks around 0 percent, 5 percent, and 30 percent, and retiring a task once it reached about 70 percent. Those figures are interviewee comments in https://epoch.ai/gradient-updates/state-of-rl-envs . They are not a standard this site enforces.
The verifier overview is verifiers and rewards: how an RL environment decides success. The pillar is reinforcement learning environments: what they are and what real work adds. Siblings are reward hacking in RL environments, LLM judge rubrics for tasks without one right answer, and state-based graders: checking what changed, not how.

Why 0% and 100% tasks teach nothing
If nothing ever succeeds, the update has no example of the job going right, and you cannot tell a hard task from an impossible one. If everything succeeds, the update has no example of the job going wrong. The Epoch comment cited above is the interview version of that point. The histogram on this page flags the two ends as a picture of the idea. The bars are not data from a task set we ran.

Pass rates over many rollouts
A pass rate is a count of successes over attempts, on a named model, with a named grader. One lucky solve is an anecdote. The 64 and 128 rollout figures above are the interviewees' way of saying the denominator has to be large enough to see a rare success. Ask which model, which grader version, and how many attempts. A percent without those three is not a measurement.
A spread of difficulty
A set that is all one difficulty teaches one narrow band. The mix quoted above, tasks near 0, 5, and 30 percent, is one interviewee's picture of a spread, not a recipe. What you should require is a written spread: some tasks the current model rarely solves, some it sometimes solves, and none that are secretly impossible. The drawing shows that shape. It does not set your bands.
Proving a task can be solved
A zero on the model is not proof the task cannot be done. The Epoch FAQ notes the worry that a zero might mean the task is infeasible, unless another person has solved it blind. Ask for that blind solve, by someone who did not write the grader, before you keep a task the model never passes. A task the author can only solve by looking at the answer is not a task.
Retiring tasks that get easy
The same interview notes that a task around 70 percent might be discarded so training moves to harder work. That is a process comment, not a cutoff you must copy. Decide in advance what you will do when a task becomes easy, and write the rule down. Leaving easy tasks in the set forever lets the score rise without new skill. Deleting them in silence makes yesterday's pass rate incomparable to today's.
Task sets that share skills
Calibrate inside a family: the same tools, the same policy, varied situations. A set that jumps from a refund to a ledger close is not one difficulty curve. It is several products. Say which skill the set is for. When the files arrive, treat the pass-rate table as part of acceptance, next to schema and version. The receiving habit for datasets is what a receiving team checks before accepting a dataset. An environment adds the measurement method to that list.
- Model, grader version, and number of attempts behind every rate.
- A blind human solve for tasks the model never passes.
- A written rule for retiring tasks that have become easy.
- The illustrative histogram on this page is not that table.
We will not compute your pass rates. Quoting the Epoch figures back to us does not make them your result.
Questions buyers ask
Is 2 to 3 percent the pass rate we require?
No. That range is what Epoch's interviewees described, at the URL above. Name the band you want and how you will measure it.
Are the bars on the image real rates?
No. They illustrate impossible and too-easy ends. They are not a histogram from a delivery.
What if the only solve is the author's?
Then you do not yet know the task is fair. Ask for a blind solve before you train on it.
Should we keep tasks forever so scores stay comparable?
Keep a record of what you retired and when. Silent deletion makes the rates incomparable. Keeping only easy tasks inflates the score.
Does a high pass rate mean the grader is sound?
No. A hacked grader can pass everything. Difficulty and soundness are different reviews.
Will you calibrate a vendor's set?
No. Ask them for the measurement. This page does not run rollouts.
Describe the data you need
Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.
Tell us what data you need