Table of Contents
Key Takeaways
- Attach the accessibility tree to the screenshot. On 358 OSWorld tasks, agents scored 7.0% with screenshots alone and 15.6% with a linearized accessibility tree. A compressed tree reached 20.7%.
- The screenshot is the smaller half of the signal. The DailyDroid benchmark ran 300 trials in April 2026 and found structured UI text alone got within 4 to 5 points of text plus screenshot.
- Claude Code and Cursor cannot read your Mac's UI structure. They run against files and a terminal. The Accessibility API is not reachable from there, so a capture tool has to hand the tree over.
- The tree is cheaper than the pixels. A window's tree is 1 to 5 KB of text, roughly 250 to 1,200 tokens. The same window as an image the model must read costs 1,500 to 4,000.
- Trees go blind on canvas, WebGL, video, and layout defects. A misaligned button reports a perfectly normal frame. That is the case the pixels exist for.
- Stash records the accessibility tree of the captured window and serves it to Claude Code or Cursor over a local MCP socket, then purges it 24 hours later by default.
Send the Tree With the Picture
The highest-value thing you can attach to a screenshot is the accessibility tree of the window you captured. It is the same structure VoiceOver reads: every element in the window with its role, its label, its value, whether it is enabled, and where it sits. macOS already builds it. Nobody was sending it to their coding agent.
That is the whole argument, and the benchmark numbers behind it are not subtle.
Two research groups published ablations in April and May 2026 that isolate exactly this variable. Same tasks, same models, one difference: what representation of the screen the agent received. The structured text wins, and it wins by more than double.
What the Benchmarks Measured
A11y-Compressor, from a team at Hosei University led by Michito Takeshita, ran 358 tasks from the OSWorld benchmark under three observation types. The results are in their May 1, 2026 paper.
| What the agent received | Task success rate |
|---|---|
| Screenshot only | 7.0% |
| Linearized accessibility tree | 15.6% |
| Compressed accessibility tree | 20.7% |
Their conclusion, stated flat:
"Relying solely on screenshots is insufficient for solving general GUI tasks."
The second data point cuts the other way and is worth sitting with. Shiquan Zhang and colleagues at the University of Melbourne and the University of Auckland built DailyDroid, 75 tasks across 25 Android apps, and ran 300 trials with two input modes. Text-only scored 26.7% and 29.3% on the two models they tested in April 2026. Adding the screenshot on top moved those to 32.0% and 33.3%.
Read that as a ratio and it reframes the workflow. The structured text delivers roughly 85% of the result. The pixels add the last 4 to 5 points. Most developers have those two backwards, because pasting an image is one keystroke and there was never a keystroke for the tree.
For scale on how much headroom is still out there: the original OSWorld paper measured humans at 72.36% across 369 tasks against 12.24% for the best agent they tested. Grounding is the named bottleneck. Input representation is the cheapest lever on it.
Where OCR Loses on UI
UI is the worst possible input for optical character recognition, and it fails in ways that are hard to notice.
Small icon-only buttons carry no readable text at all. Table cells in narrow columns get merged or split. Low-contrast placeholder text in a disabled field reads as real content. Truncated labels ending in an ellipsis get transcribed as complete. Accent-colored status pills lose their words. A checkbox that is checked and a checkbox that is focused look nearly identical at 2x.
None of those failures announce themselves. The model produces confident text, you have no marker telling you which fragment was guessed, and the wrong label propagates into the fix your agent writes.
The accessibility tree does not guess, because it is not reading anything. The app declared the values. A disabled button says enabled: false. A truncated label carries its full string. A text field reports its actual contents, not the pixels rendering them. That is why the same model gets more done with fewer tokens: it stops spending capacity on transcription and spends it on the problem.
Your Coding Agent Has No Mac
Here is the split nobody draws clearly, and it is the reason this matters for a coding workflow specifically.
| Computer-use agent | Coding agent (Claude Code, Cursor) | |
|---|---|---|
| What it operates on | Your live desktop | Your repo and your terminal |
| Accessibility API access | Yes, queries it directly | None |
| How it sees UI | Live tree plus live frames | Whatever you pasted |
| Can ask for more detail | Yes, takes another look | Only if you go take another screenshot |
Every benchmark above tested the left column. Those agents drive the machine, so reading the Accessibility framework is a function call away. The published research is about them.
The right column is where most of us actually work. Claude Code runs in a terminal against a filesystem. It cannot call AXUIElementCopyAttributeValue. It has no window server, no display, no idea what your app looks like right now. When you paste a screenshot, that flat image is the entire channel, and every advantage the research just measured is unavailable to it by construction.
Unless something on your Mac captures the tree and hands it over. That is a capture-tool job, not a model job.
What a Captured Tree Contains
Every Stash screenshot records the accessibility tree of the captured app window alongside the image. Each node holds role, label, value, enabled state, position, and children. For an editor or terminal capture, Stash also records the active file path, language, cursor line and column, the visible buffer text, and for terminals the shell, working directory, and recent commands with their output.
That last item solves a problem screenshots cannot. A scrolled terminal shows you one viewport. The tree and the terminal context carry the commands and output that scrolled past. Your agent reads what happened, not what happened to still be on screen.
The tree reaches the agent through the local MCP server. Ask for a capture by its 8-character ID and get_capture returns the dossier:
get_capture("a3f9c2e1")
app Cursor (com.todesktop.230313mzl4w4u92)
window ArticleView.swift
os macOS 26.6
file Sources/Views/ArticleView.swift:214:12
a11y Button "Publish" enabled:false
TextField "Slug" value:"accessibility-tree-vs-screenshot"
StaticText "3 validation errors"
annotations arrow (612,340) to (688,402) color:#EF4444Compare that to what the same moment looks like as a pasted PNG. The agent sees a red arrow it cannot distinguish from the app's own chrome, a grayish button it has to guess is disabled, and a filename it may or may not read correctly. Then it asks you which function you meant.
Under the current MCP specification revision 2026-07-28, published July 28, this is a normal tool call. The spec went stateless and added a 12-month minimum deprecation window, which mostly matters for hosted servers. For a local one the useful part is that the contract you write into your agent instructions stays stable.
The Token Math
The structured tree is the cheaper input, which is the part people get wrong when they assume text costs more than an image.
| Input | Approximate token cost |
|---|---|
| Stash accessibility tree, typical app window | 250 to 1,200 (1 to 5 KB) |
| Same window as a PNG the model must read | 1,500 to 4,000 |
A11y-Compressor measured the same effect from the other direction. Their compressed trees came in at 22% of the original token count and still raised task success by 5.1 percentage points on average, with per-domain input staying under roughly 3,500 tokens. Less input, better results. That combination is rare enough to pay attention to.
It compounds across a session. Ten screenshots into a debugging loop, the pixel-only path has spent 15,000 to 40,000 tokens on transcription that the tree would have handed over correctly for a fraction of that.
Where the Tree Goes Blind
Send the pixels too, because the tree misses real things.
Canvas and WebGL surfaces are a single opaque node. Charts, maps, and game UI are pixels by definition. Video frames and images carry nothing. Custom-drawn controls in a heavily styled app often declare almost nothing useful, and some Electron apps expose a tree so shallow it is barely worth reading.
Layout bugs are the sharper case. A button that has drifted 40 points to the left reports a frame, and that frame is correct. The tree has no opinion about whether it looks wrong. Clipping, overlap, wrong spacing, broken dark mode, a font that failed to load: all of it is invisible in structure and obvious in a picture.
Which is why the benchmark answer is both, not either. Every ablation that tested the combination put it at the top. The tree tells the agent what is there. The image tells it what that looks like.
Doing This on a Mac Today
Press ⌘⌃S. Stash captures the region, window, or screen, and records the accessibility tree with it. The image gets a context banner across the bottom carrying app, window title, URL, OS version, display, timestamp, and an 8-character capture ID, plus the same data structured as XMP inside the PNG.
From there you have two paths. Paste the image straight into Claude Code and the banner alone gives it app, window, and environment. Or name the capture and let the agent pull the full dossier over MCP, tree included. The installer at yourstash.ai/install-claude.sh wires up Claude Code, Claude Desktop, and Cursor in one command.
On privacy, an accessibility tree holds text field contents, so treat it as sensitive. Stash purges trees, selected text, file paths, git branches, and terminal working directories 24 hours after capture by default, adjustable from one hour to never. Copied text is scanned for secrets before it is written, so sk-, AKIA, ghp_, JWTs, Bearer tokens, and PEM blocks are replaced with [redacted] before touching disk. The MCP server listens on a Unix domain socket at ~/Library/Application Support/Stash/mcp.sock. There is no network path, which means the agents you connect are talking to your Mac and not to us.
Download Stash for free and give your agent the tree.
The Bottom Line
Screenshots alone scored 7.0% on 358 OSWorld tasks. The accessibility tree scored 15.6%, a compressed tree 20.7%, and every study that tested the combination ranked it first. Your coding agent cannot get the tree on its own, because it runs in a terminal with no access to the window server. So the capture tool has to carry it. That is a five-minute setup that costs fewer tokens than what you are doing now and stops your agent guessing at button labels it was never able to read.
Frequently Asked Questions
What is the accessibility tree on macOS?
It is a structured description of every UI element in a window, maintained by macOS itself through the Accessibility framework. Each node carries a role, a label, a value, an enabled state, a screen position, and its children. VoiceOver reads it. So can a capture tool, and so can an AI agent once the tree is attached to a capture.
Is the accessibility tree better than OCR for AI agents?
Yes, on every published ablation I have found. A11y-Compressor ran 358 OSWorld tasks and measured 7.0% success on screenshots alone against 15.6% on a linearized accessibility tree and 20.7% on a compressed one. Their stated conclusion is that relying solely on screenshots is insufficient for general GUI tasks. The best result comes from sending both.
Does Claude Code read the accessibility tree by itself?
No. Claude Code and Cursor run against your files and your terminal, not against your Mac's window server, so the Accessibility API is not reachable from there. A computer-use agent that drives the machine can query the tree live. A coding agent cannot. Something has to capture the tree and hand it over, which is what get_capture on the Stash MCP server does.
What does the accessibility tree miss?
Anything drawn as pixels rather than declared as an element. Canvas and WebGL surfaces, chart plots, video frames, images, and custom-drawn game UI show up as an opaque node or nothing. Layout defects are invisible too, because a misaligned button reports a perfectly normal frame. Those are the cases the screenshot exists for.
How many tokens does an accessibility tree cost compared to a screenshot?
Fewer, usually by a factor of three to five. A Stash tree for a typical app window is 1 to 5 KB of structured text, roughly 250 to 1,200 tokens. The same window as an image the model has to read costs 1,500 to 4,000 tokens. A11y-Compressor measured compressed trees at 22% of the original token count while raising success rates 5.1 percentage points on average.
Is capturing an accessibility tree a privacy risk?
It can be, which is why Stash expires it. A tree contains text field contents, so a capture can hold whatever was on screen. Stash purges accessibility trees, selected text, file paths, git branches, and terminal working directories 24 hours after capture by default, adjustable from one hour to never. Nothing leaves your Mac either way, since the MCP server runs on a local Unix domain socket.
References
- Takeshita, M., Kawada, T., Ohashi, T., Kitada, S., Iyatomi, H. "A11y-Compressor: Enhancing the Efficiency of GUI Agent Observations." arXiv:2605.00551, May 1, 2026. https://arxiv.org/abs/2605.00551
- Zhang, S., Zhang, T., Fang, L., D'Alfonso, S., Jia, H., Kostakos, V. "Do LLMs Need to See Everything? Screentext vs. Screenshots." arXiv:2604.17817, April 20, 2026. https://arxiv.org/abs/2604.17817
- Xie, T. et al. "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments." NeurIPS 2024. https://arxiv.org/abs/2404.07972
- Model Context Protocol specification, revision 2026-07-28. https://modelcontextprotocol.io/specification/2026-07-28
- Apple Developer Documentation, Accessibility framework. https://developer.apple.com/documentation/accessibility