Vercel Labs agent-desktop 1.0: Accessibility-Tree Computer Use, Snapshot Refs and the Retry Contract

agent-desktop is an Apache-2.0 CLI and C library from Vercel Labs that lets an agent read and operate desktop apps through the OS accessibility tree instead of screenshots. GitHub shows v1.0.0 published on October 28, 2026, and v1.0.2 on October 30. The repository itself was created on February 19, so 1.0 is a milestone for an older project, not a new one. The interesting part is the contract it gives the agent: refs tied to snapshots, errors that say whether a retry is safe, and an explicit warning about private desktop content.

I read the repository README and the JSON output doc, and checked the releases and repository metadata through the GitHub API. I did not install or run it. Behavior below is what the project documents say.

What it does, per the README

• It ships as a CLI and C library for macOS, Windows and Linux, installed with npm install -g agent-desktop, which includes native binaries for x64 and ARM64.
• A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check. The README says the loop stays in your agent, with "no pixel matching".
• It lists 60 command names, 56 of them operational. Four (key-down, key-up, mouse-down, mouse-up) "refuse to run until a daemon can release a held key or button".
• Command groups: observe, act on a ref, keyboard and mouse, apps and windows, notifications, clipboard, wait, sessions, and system.
• Ref actions are "headless by default": they go through the accessibility API without moving the pointer or taking focus. A global --headed flag allows physical input.

Refs belong to their snapshot

The quick start is three commands: snapshot an app with interactive elements, click a ref, snapshot again. The README states the rules:

• A ref is scoped to its snapshot. A bare @e8 is rejected with INVALID_ARGS.
• When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent should take a new snapshot.
• Several windows give AMBIGUOUS_TARGET, with candidate window ids in the error details.
• For dense apps, map first with a skeleton snapshot, then drill into one region.

On macOS, the README says CLI calls reuse a private command host that retains native accessibility objects, so a removed row cannot silently resolve to a surviving row with the same identifier. The host exits after 15 idle minutes.

The retry contract

This is the most useful design detail. The JSON output doc says every command returns an envelope with ok and either data or error. An error always carries code, message and disposition. The disposition records whether the action was delivered and whether retry is safe or unsafe. The README says to retry only when error.disposition.retry is safe. In the doc's example, a successful click is marked delivered_unverified with retry unsafe, and a STALE_REF error is not_delivered with retry safe plus a recovery strategy of refreshing the snapshot.

For agent builders this matters because a blind retry of a click or a typed message can do the action twice. The tool tells the agent which case it is in. The doc also says a snapshot that runs out of budget returns a successful but explicitly partial response, with complete set to false.

Platform support is uneven

The README's table shows the accessibility tree on all three systems, with Linux through AT-SPI. Differences it lists:

• macOS needs Accessibility permission. Shell surfaces are marked planned.
• Windows needs no grant for apps at your terminal's integrity level.
• Linux needs the user's graphical session with at-spi2-core. Mouse input is X11 with --headed. Screenshots work for X11 windows and via the Wayland portal. Notifications are GNOME Shell only.
• Homebrew and Scoop packages are described as "being submitted" and not yet available in default catalogs.

Privacy and the optional model helper

The README warns that snapshots, clipboard reads, screenshots, command output and debug artifacts can contain private desktop content and should stay out of public logs and uploads. It also says an optional JEV helper sends screen descriptions to a configured model provider and is separate from the native core. I did not read the helper's documentation.

What I would check

These are my untested suggestions.

1. Snapshot the three apps your agent touches most and see how many controls get a usable ref.

2. Force a stale ref on purpose and confirm your agent follows the disposition.retry field.

3. Decide where snapshots and clipboard reads are logged, given the privacy warning.

4. Run a headless action and a --headed action on a login-protected app and compare.

5. Compare against a screenshot-based tool on one dense app, using your own pass criteria.

What I would not conclude

• That accessibility trees cover every app. Apps with poor accessibility support may expose little, and I did not test any.
• That it is a Vercel product with a support commitment. The README labels it "Vercel Labs Product" and I found no support terms in what I read.
• That a delivered click was verified. The doc's own example is "delivered_unverified".
• That the ecosystem is settled. The same page says package-manager submissions are still pending.

Frequently asked questions

What is agent-desktop?

A CLI and C library that lets agents operate desktop apps through the OS accessibility tree, per the README.

What license does it use?

Apache-2.0, per the GitHub repository metadata.

Which platforms does it support?

macOS, Windows and Linux, with different capabilities listed in the README table.

Does it use screenshots?

It can take screenshots, but the main loop reads accessibility snapshots and acts on refs, per the README.

How does it handle failed actions?

Errors carry a code and a disposition that says whether retry is safe, per the JSON output doc.

Is it safe for private data?

The README warns that snapshots and related output can contain private desktop content and should be kept out of public logs.

Did you run it?

No. I read the documentation and release data.

Sources

All read on November 1, 2026.

• vercel-labs/agent-desktop README
• JSON output contract
• GitHub repository metadata and releases (GitHub API)

Not read: install.md, FAQ, CONCEPTS.md, the agent skills files, the JEV helper docs, the source code, and press coverage of the release.