Reinforcement learning environments

Gymnasium-style APIs for agent environments

Last updated /Reinforcement learning environments

A Gymnasium-style API is a small contract. The caller resets the world and receives an observation. The caller steps with an action and receives the next observation, a reward, and whether the episode is done. OpenEnv carries that idea into agent environments that run as a service: the trainer is a client, the environment is a server, and the step result includes the reward. This page is that contract and what it implies for delivery. It is not a library. Where packaged environments are listed is open RL environment hubs.

The field guide that points here is RL training environments: a field guide. What the world inside the API is for is reinforcement learning environments: what they are and what real work adds. Checking a file delivery, which is a cousin of checking an environment delivery, is what a receiving team checks before accepting a dataset.

Client and environment server exchanging reset, step and reward messages, labeled Buyer guide
Reset, step, and reward between a client and a server. Buyer guide is the chip, not a runtime we host.

reset, step and state

Gymnasium's documentation is the public reference for reset and step: https://gymnasium.farama.org/index.html . You call reset to start an episode. You call step with an action. You read back what the world did. OpenEnv's concepts guide describes the same loop between a client and an environment server: https://huggingface.co/docs/openenv/en/guides/concepts . Its core reference names reset, step, and state: https://huggingface.co/docs/openenv/main/en/reference/core . State is how you inspect the environment, not how you sneak a look at the answer during the attempt.

If a package cannot show you those calls, it is not yet on this interface, whatever the pitch says. A notebook that mutates global variables is not a server.

Client and environment server exchanging reset, step and reward messages, labeled Buyer guide
Reset, step, and reward between a client and a server. Buyer guide is the chip, not a runtime we host.

Observations, actions and rewards

The observation is what the agent is allowed to see after reset or step. The action is what it is allowed to send: a tool call, a click, a message. The reward is the environment's score for that step, computed by the environment rather than by the trainer's private notes. OpenEnv's training guide says the environment computes its own reward so a trainer that can call the client can train on it: https://huggingface.co/docs/openenv/guides/training . That is their split. Confirm it in the package you are buying. Do not assume a JSON field named reward is the check you specified.

Containers and HTTP servers

A container or an HTTP service is how you get a reset that does not depend on one person's laptop. The client posts reset and step. The server owns the world. You can start a second copy for a parallel attempt. You can pin an image. A delivery that is only a folder of scripts, with no way to run reset twice and get the same start, will fail the first training job.

The network boundary is also a control. The server should not reach production systems unless the spec says so. A container that holds production credentials is not a sandbox.

MCP tools inside an environment

Tools can sit inside the environment and be exposed to the agent, including through MCP. The Model Context Protocol site introduces that style of tool exposure: https://modelcontextprotocol.io/ . The tools are part of the action space. They are not a second, undocumented environment. List them. Say which ones write. If a tool can see the goal state, the interface is leaking, no matter how clean reset and step look.

Delivery formats buyers expect

Expect a versioned runtime you can start, a client or a documented HTTP contract for reset and step, the observation and action schemas, and the code or spec for the reward. Expect a way to run two episodes that do not collide. Expect the version to be nameable so a later delivery is not a silent replacement.

  • Reset returns a start you can describe.
  • Step accepts only the actions in the schema.
  • The reward is computed where the spec says, and you can rerun it.
  • A second copy does not see the first copy's writes.
  • Secrets for production systems are absent.

Receiving a dataset has a similar habit of checking format, schema, and version before you call it accepted. Use that habit here. We do not operate the runtime, and a profile is not a container.

Questions builders ask

Must we adopt OpenEnv to buy an environment?

No. OpenEnv is one documented client and server. Gymnasium is the older reset-and-step reference. Ask which contract the package actually implements.

Is state the same as the observation?

No. The observation is what the agent is given. State can be broader. If state includes the goal, do not hand it to the agent as the observation.

Do you host these servers?

No. This page describes the interface. It is not a runtime and not a hub.

Where are the benchmark and hub pages?

The hub page is live and linked above. This page stays on reset, step, reward, and delivery.

Can a notebook be the delivery?

Only if someone else can reset it twice and get the same start without the author. Otherwise it is a demo.

Does a reward field prove the grader?

No. Ask to see how the number was computed, and rerun it. A field name is not a check.

Describe the data you need

Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.

Tell us what data you need