Reinforcement learning environments
LLM judge rubrics for tasks without one right answer
Last updated /Reinforcement learning environments
An LLM judge is a model that scores an attempt against written criteria. Use one when the outcome is prose or a judgment and there is no single correct string a program can compare. Do not use one to avoid a check the database, the file, or a test could have answered. The parent page for that choice is verifiers and rewards: how an RL environment decides success. The pillar is reinforcement learning environments: what they are and what real work adds.
Siblings are reward hacking in RL environments, calibrating RL task difficulty, and state-based graders: checking what changed, not how. Naming the criteria in a brief, which is a habit and not this rubric, is what belongs in an AI dataset requirements brief.

When a deterministic check is not enough
A unit test can say whether the code ran. It cannot say whether a support reply followed the tone and the remedy rules when both matter and the wording can vary. A state check can say whether the refund flag was set. It cannot say whether the agent told the customer something the policy forbids if that sentence never touched the database. Those leftovers are where a rubric earns its place. If you can settle the question with the records, do that instead. A judge is another model, and it can be gamed.

Writing rubric criteria
Each criterion should be something a second reader could apply: the reply states the decision, the reply does not promise a remedy the policy forbids, the steps named match the tools that were actually called. Vague criteria such as helpful or high quality produce a score you cannot audit. Write the fail case next to the criterion. A criterion without a fail case is a wish.
Keep the criteria few enough that a person can score a sample against them in one sitting. A rubric of dozens of overlapping lines will not be applied the same way twice.
Weights and combined scores
If some criteria matter more, write the weights down. The drawing on this page shows a card with weights as an illustration of that idea. Those weights are not a scale we measured, and they are not a recommendation. A secret blend of criteria is not a rubric you can accept. A combined score that adds a judge to a state check should say which part can fail the attempt on its own. Otherwise a fluent answer can carry a wrong database.
Judge models and drift
Name the judge model and pin it. A later model, or the same name with a silent update, can score the same answer differently. Keep the rubric text under version control next to the tasks. Spot-check a fixed set of attempts whenever the judge changes. If the scores move and you did not mean them to, do not keep training on the new numbers.
We do not publish a stability threshold. Ask the vendor how they noticed drift, and read the spot-check. A claim of stability without the sample is a slogan.
Dropping defective tasks
Some tasks cannot be rubric-scored cleanly: the policy is missing, the criteria contradict, or raters disagree on the fail case for reasons you cannot write down. Drop those tasks. Training on a noisy judge teaches the noise. A smaller set with criteria a second person can apply is worth more than a large set with a moody score.
Combining judges with state checks
Prefer a state check for anything the system of record can answer, and a rubric only for the residue. Say so in the spec. A judge that is allowed to override a failed database check will be the path the agent learns. The state-check page is the other half of this choice.
- Criteria a second person can apply, each with a fail case.
- Weights written down, or an explicit statement that every criterion can fail the attempt.
- The judge model pinned, plus a spot-check set.
- Tasks you dropped, and why.
We will not score your answers. A rubric on this website is an explanation, not a service.
Questions buyers ask
Should every RL task use a judge?
No. Use a judge when no program can compare the outcome to one correct string. If the database can answer, use the database.
Are the weights on the diagram a standard?
No. They show that weights belong on the card. They are not a scale we measured.
Can the same model be the agent and the judge?
You can, and you should assume it will be generous to itself. Prefer a pinned judge you can spot-check, and do not hide which model it is.
What if two reviewers disagree?
The criterion is not ready. Rewrite the fail case or drop the task. Do not average the disagreement and call it a reward.
Do you write rubrics for clients?
No. This page says what a rubric has to contain. Your task owners write the criteria.
Is a rubric a requirements brief?
No. A brief names what you need. A rubric scores an attempt. They can share criteria. They are not the same document.
Describe the data you need
Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.
Tell us what data you need