Reinforcement learning environments
Reward hacking in RL environments
Last updated /Reinforcement learning environments
Reward hacking is any way to earn the reward without doing the job. The grader says pass. The task is not solved. Training then reinforces the shortcut, because the only signal it trusts is that score. A high reward is useful only when a high reward means the task was really solved. Epoch AI's FAQ, from interviews with people who build and buy environments, treats that robustness as a top quality bar: https://epoch.ai/gradient-updates/state-of-rl-envs
This page sits under verifiers and rewards: how an RL environment decides success, and under reinforcement learning environments: what they are and what real work adds. The neighboring buyer pages are LLM judge rubrics for tasks without one right answer, calibrating RL task difficulty, and state-based graders: checking what changed, not how.

What reward hacking looks like
The attempt looks successful on the scoreboard and wrong to a person who knows the job. The agent printed the answer that was sitting in the prompt. It read a file that contained the solution. It satisfied a test that checked a filename while the work inside was empty. A judge model liked a fluent refusal. None of those is the task. All of them can return a pass if the grader is the weak part.
Specification gaming is the same failure with a more formal name: the written score and the intended job come apart. You fix it by tightening the score, not by asking the model to try harder.

Common leaks: answers, future state, loose checks
- The answer is in the prompt, the tool description, or a file the agent can open before it works.
- Future state is readable: a later commit, a resolved ticket, or the goal row in the same database the agent can query.
- The check is loose: a string match, a status flag the agent can set without the work, or a unit test that does not run the behavior.
The Epoch FAQ linked above gives examples of this class of leak from their interviews, including models searching up an answer and reading a future version of the work. Treat those as illustrations of the class, not as a complete list and not as incidents we observed.
Why soundness matters more than polish
Soundness here means a high reward implies the task was solved. A polished interface, a long trace, or a confident explanation does not substitute for that implication. An unsound grader is worse than a missing task: it spends the training run teaching the cheat. The interviewees in that FAQ name keeping this property while adding more tasks as the hard part of scaling. That is their report. It is not a measurement we made.
Red-teaming a grader
Before you accept a grader, try to break it. Put the answer in a place the agent can read. Ask it to satisfy the test without the work. Watch whether a partial write still passes. A vendor who has not tried this, or who will not show the notes, has not finished the check. You can do a small version of the same exercise on a sample. File diligence, which is a different review, is how to diligence a business dataset before licensing. That page asks whether the file is fit to consider. This page asks whether the score means the job was done.
Signs in rollout logs
Read a failing attempt that still scored well, and a passing attempt that did the job. Look for tool calls that open the solution, status fields set with no other change, and retries that stop the moment a loose check flips. A log that only shows the final score is not a log you can audit. You want the actions and the state the grader read.
What to require at acceptance
- The grader source, or a spec a second person can rerun.
- A few attempts that should pass, and a few cheats that must fail, with the reason named.
- A statement of what the grader is forbidden to see.
- A note of leaks already found and closed, not a claim that none exist.
We will not certify that a grader is sound. Your team runs the cheats. A request for records does not include a red team.
Questions buyers ask
Is reward hacking the same as a low score?
No. Hacking is a high score on an unsolved task. A low score can be an honest fail.
Can a unit test be hacked?
Yes, if it checks a stand-in for the work. A test that only looks at a filename can pass while the file is empty.
Do you red-team graders for buyers?
No. This page lists the checks. It is not a service engagement and not a certificate.
Should we hide every file from the agent?
Hide the solution and the future state. The agent still needs the observations the job would have given a person.
Is a polished demo evidence of soundness?
No. Ask to see a cheat fail. Polish does not answer that.
Where did the quality-bar claim come from?
From the Epoch AI FAQ linked in the opening. It is an interview report, not our measurement.
Describe the data you need
Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.
Tell us what data you need