AI at the computer is a model that looks at the screen, moves the mouse and types, like a person at a PC. It does not need the program’s API: it works in any window — from a browser to an old accounting system. The agent already does short tasks in office software reliably; long chains of hundreds of steps — not yet.
How it works
The loop is simple: a harness program takes a screenshot and sends it to the model along with the task. The model finds the right field, button or menu item in the image and replies with an action: “click at these coordinates”, “type the text”, “press Enter”. The harness performs the action, takes a new screenshot, and the model looks at what changed. If the program provides a text description of its interface, it is sent along with the image, and there are fewer misses.
The main difference from the old RPA script robots is flexibility. A script remembers that the “Save” button sits at a particular spot and breaks when an update moves it. The model reads the screen afresh every time, like a person: the button has moved — the model finds it. So you can put the agent on programs without integrations: old accounting systems, supplier websites, internal portals.
The weak spot is length. Every step carries a small chance of error: missing a field, not waiting for a page to load, misreading a window. Over ten steps this is barely noticeable; over hundreds the errors add up: in the OSWorld 2.0 test, on tasks longer than 300 steps, the best agent succeeds 20.6% of the time. So a long process is cut into short pieces with a check after each one.
Where it already works
- Moving data between programs without an integration: from emails and PDFs into the accounting system.
- Filling in repetitive forms on supplier websites and internal portals.
- Interface testing: the agent goes through scenarios like a user and flags bugs.
- Short routine tasks in office software: spreadsheets, documents, email.
What to consider
- Long chains are not reliable yet: split the process into short steps and check the result.
- Run the agent in a separate account or a virtual machine, with minimal permissions.
- Payments, sending emails and deleting — only with a person’s confirmation.
- In Russia it is complicated: Western cloud models are unavailable, so you need an option on your own server.