Reinforcement learning environments

Computer-use RL environments: training agents on screens

Last updated /Reinforcement learning environments

A computer-use RL environment gives the agent a screen instead of a function list. The agent clicks, types, and reads pixels, an accessibility tree, or the structure of a page. The world is a browser or a desktop that can be put back to a known start. The grade should be the file, the setting, or the account the clicks were supposed to change. A screenshot that looks finished is a weak grade, because a picture can match while the file is wrong.

This type sits under types of RL environments, and what each one trains, and under reinforcement learning environments: what they are and what real work adds. The other types are tool-use RL environments for agents that call APIs, enterprise software replicas as RL environments, and simulated-user environments for multi-turn agents.

Virtual desktop with a spreadsheet task and a checker reading the saved file, labeled Buyer guide
A desktop task and a checker on the saved file. The chip says Buyer guide. It is not a score.

Screenshots, accessibility trees and DOM actions

The observation can be a screenshot, a text description of the accessible controls, or the document structure of a page. Actions are clicks, keystrokes, and scrolls, sometimes named as operations on those controls. A screenshot shows what a person sees and hides the identity of the control. An accessibility tree names the control and can miss what the pixels imply. Say which one the agent receives. A vendor who says vision without saying which signal is graded has not specified the task.

Virtual desktop with a spreadsheet task and a checker reading the saved file, labeled Buyer guide
A desktop task and a checker on the saved file. The chip says Buyer guide. It is not a score.

Resettable virtual machines and browsers

Each attempt needs the same start. That means a virtual machine or a browser profile you can restore: the same files, the same login session, the same open documents. If the machine keeps yesterday's spreadsheet, the next attempt is not the task you bought. Parallel attempts need separate machines, or they will edit one file together.

Reset is part of the product. A script the vendor runs by hand is not a reset you can train on. Ask to see the restore, then run a second episode without them.

Execution-based checks on the final state

OSWorld is a public computer environment whose paper grades tasks by execution rather than by a single screenshot score. Custom scripts read the final state of the machine, including files and the accessibility tree, and then apply a check written for that task. The paper is https://arxiv.org/abs/2404.07972 . That is a description of their method. It is not a score for any model, and it is not a result we measured.

Copy the method, not a leaderboard. Your checker should open the saved file, the settings document, or the account and compare it with the outcome you named. A checker that only looks at the last frame will pass a screen that was painted to look done.

Why GUI tasks are still hard for models

Small interface changes move buttons. Pop-ups cover the control the task needs. Text in a screenshot is easy to misread. A task that was reliable on one build of an application fails on the next build for reasons that have nothing to do with the decision you meant to teach. That brittleness is a reason to prefer a tool-use environment when a stable API exists, and a reason to pin the application version when the screen is the job.

We are not quoting a pass rate. The OSWorld paper reports model results. Those numbers belong to that paper and to the version they ran. Do not paste them onto a vendor's demo.

Business apps with no API

Some line-of-business tools only offer a screen. Computer use is then the type that matches the work: a desktop app, an old web form, a spreadsheet the team actually uses. The task still needs a file or a setting the checker can read. If the only record of success is a person's opinion of the screen, you do not have a check yet.

What buyers should specify

Write the observation, the allowed applications, the start state, the reset, and the artifact the checker opens. That list belongs in the same kind of brief you would write for records. The brief format, for data rather than for a desktop, is what belongs in an AI dataset requirements brief. Use it as a habit of naming exclusions. It does not replace a desktop spec.

  • Which signal the agent sees: pixels, accessibility tree, page structure, or a mix.
  • The application version, and what happens when it changes.
  • How the machine is restored, and that a second run does not see the first run.
  • The file or setting the checker reads. Not a screenshot alone.

Nothing here is a desktop we host. A request does not create a virtual machine.

Questions buyers ask

Is a screenshot grade enough?

Usually no. OSWorld's paper, linked above, checks the machine's final state. A matching picture can hide a wrong file.

Do you publish OSWorld scores?

No. We cite their method. We do not rerun their benchmark and we do not adopt their numbers as ours.

When should we buy tools instead of a screen?

When the decision is a function call and the screen is only a view of it. A screen adds ways for the check to be fooled.

Can the agent use our production desktop?

No. A training environment resets. A production desktop does not, and it holds data you did not mean to include.

What if the app has no export the checker can read?

Then you do not yet have a grade. Find a file, a setting, or a log. Do not accept a glance at the window.

Is this a catalog of GUI environments?

No. It is the spec for the type. We are not listing desktops for sale.

Describe the data you need

Tell us the use, the records, and the limits. A request is not a promise that a matching dataset is available.

Tell us what data you need