awlevin/typesafe-computer-use: Pc use for about $0.0002 a step: OCR the display, classify the following motion with TypeSafe, click on. macOS. · GitHub

typesafe-computer-use drives a Mac towards a purpose you kind in plain English, for a few
fiftieth of a cent per step. It by no means sends a screenshot to an enormous mannequin. Instead it
reads the display deterministically, asks a small classifier which motion comes subsequent,
and solely calls a writing mannequin when a textual content discipline genuinely wants free textual content.

clicker "go to techcrunch and take me to the checkout web page for the most affordable tickets to their subsequent upcoming occasion" --act

Frontier-model laptop use is succesful and costly: each step ships a screenshot and
waits a number of seconds for a plan. Most steps don’t want a plan. They want one selection
from a brief record, made rapidly and cheaply, with a confidence quantity you may gate on.

TypeSafe sells precisely that: a choice mannequin that solutions
a Choice over as much as 255 choices with a full likelihood distribution and a calibrated
confidence, in a number of hundred milliseconds, with free output tokens. This mission is a
computer-use loop constructed round it.

Measured on the identical screenshot and purpose, one choice every:

The trustworthy caveat: the large mannequin learn the occasion dates off the pixels and in contrast them
unaided. The classifier wanted the date parsing described beneath. Every piece of
reasoning the frontier mannequin does totally free must be rebuilt right here as deterministic state.

macOS 14 or newer, Python 3.12 or newer, uv.

git clone https://github.com/awlevin/typesafe-computer-use
cd typesafe-computer-use
uv sync
cp .env.instance .env     # fill within the keys

Grant your terminal Screen Recording and Accessibility in System Settings >
Privacy & Security. Without the primary, captures are wallpaper. Without the second,
artificial clicks are silently dropped, and --act refuses to start out.

uv run clicker "open the Playground"                 # dry run: one step, prints what it might do
uv run clicker "open the Playground" --act           # drives the machine, as much as 100 steps
uv run clicker "log in" --act --steps 20 --delay 3   # longer and slower
uv run clicker-inspect "any purpose"                    # 3-2-1, seize, open the annotated display + payload

Clear the terminal first. It is on display, so its textual content is OCR enter.

Stopping a dwell run. Ctrl-C when the terminal has focus, or slam the mouse into the
top-left nook of the display from any app. The loop additionally stops itself on achieved or
none, on confidence below --min-confidence (0.4), after two consecutive no-ops, or
at --steps.

The reply. When the loop stops itself, the author reads the display it stopped on
and prints the outcome: the data the purpose requested for, or the place issues stand and
the following step when the display doesn’t maintain it. A dry run that might have acted, and
an aborted run, print no reply.

screencapture ─► Vision OCR ─► merge strains into blocks ─► drop strains echoing the purpose
accessibility ─► actionable parts (function, label, body), pruned to the show,
                 the labelled pressable ones it pruned stored as off-screen controls
                     │
                     └─► one numbered record of things, every carrying its supply
                     │
accessibility ─► centered discipline (function, label, placeholder, worth, body)
AppleScript   ─► frontmost app and pid, energetic tab URL
clock         ─► native date and time
dates.py      ─► "dated 2026-10-13 (in 27 days)" on any block containing a date,
                 "close to a line dated ..." on its neighbours
                     │
                     ▼
        one TypeSafe request, three Choices, 4 with off-screen controls
        ┌────────────────────────────────────────────────────────────┐
        │ form      : click_item | use_browser | type_text | scroll… │
        │ merchandise      : which merchandise (used just for click_item)          │
        │ web site      : which web site (used just for use_browser)      │
        │ offscreen : which hidden management (just for press_offscreen)│
        └────────────────────────────────────────────────────────────┘
                     │
                     ▼
        deterministic motion ─► wait ─► subsequent step

Items carry the place they got here from: ocr for a textual content block, ax for a management the app
declared, ax+ocr when each discovered the identical factor. An ax merchandise reads as
button 'Share' (top-right) within the standards, so the classifier can inform an actual management
from a line of textual content.

Splitting the choice into three questions retains display noise out of the motion
selection. Every stall discovered whereas constructing this got here from two choices that meant the
similar factor. Confidence measures focus, so overlapping choices all the time learn as
doubt. Keep the motion set mutually unique.

Vision is about two thirds of a step, and it expenses by the quantity of textual content reasonably than
the variety of pixels, so the one actual saving is studying much less of the display.

• Crop. Each step reads the frontmost window with an 8 pt margin, plus the menu bar
  strip over the identical columns, clamped to the show. Text on the desktop and in
  background home windows is noise to the choice. Clipping the strip to the window’s width is
  what makes the crop pay on a full-height window. The price: the clock and the menu extras
  to the best of the window go unread. They keep clickable by means of the accessibility tree.
• Reuse. The seize is in contrast with the earlier one at 1/8 scale, in 256 px tiles.
  Unchanged tiles hold the strains they produced final step. The modified tiles are clustered
  into blobs, sides and corners counting as touching, and every blob turns into a rectangle
  learn by itself. Scattered change is the atypical case, a clock digit plus one repaint,
  and one rectangle round each would span the show. Each rectangle grows till no recognized
  line straddles its edge, as a result of a crop by means of a line returns the half it may well see; ones
  that meet after rising merge, and greater than 4 merge by closest pair all the way down to 4.
  Past 60% modified tiles, previous 60% of the area in summed rectangle space, or on an app
  swap or a window transfer, the entire area is learn as a substitute.

The timing line says how a lot was learn, and in what number of items: ocr 0.31s (22% of display, 2 rects). A replay (--image) all the time reads the entire picture and by no means reuses, so an offline
repro matches the unique run.

OCR can not see an icon. The accessibility tree can, so every step additionally walks the frontmost
course of for labelled, on-screen controls. Coverage is uneven, measured on ten apps on one
Mac: Finder 100% of on-screen controls labelled, Chrome 88%, Slack 85%, Notion 68%,
Spotify 0 (its CEF shell exposes three window buttons and nothing else). Terminals expose
the grid as one textual content space. So AX is a bonus supply, by no means a substitute.

Labels dwell in AXDescription for internet and Electron, AXTitle for AppKit, and a brief
AXValue in any other case. An ornamental picture takes the label of the management round it; an inventory
row takes it from a shallow AXStaticText.

Frames lie, so the stroll prunes exhausting:

• skip any subtree whose actual body misses the show (Notes reviews rows 200 screens
  down, Chrome parks scrolled-out nodes above the viewport)
• skip any node below 4 pt large or tall (Chromium clamps scrolled-out internet nodes to slivers)
• skip AXMenu subtrees, that are 1000’s of zero-sized objects behind a closed menu
• skip anonymous AXGroup structure bins, even pressable ones
• cease at 4000 nodes or 0.6 s and say so

Walks measured right here: Finder 152 controls in 0.08 s, Chrome 172 in 0.59 s. The assistive
handshake attributes (AXManualAccessibility, AXEnhancedUserInterface) are unsupported
on this macOS, so nothing depends on them.

AXPress doesn’t want a component to be seen. Notes selects a row parked 1000’s of
factors beneath the show, Chromium delivers a click on to a hyperlink it clamped to a 1 px sliver
as a result of the web page is scrolled previous it, and an auto-hidden Dock arms over all 37 of its
objects from 5 pt beneath the underside edge. So the identical stroll retains the labelled, pressable nodes
it pruned, and provides them as a separate capped record reasonably than mixing them into the objects:
nothing on the seize factors at them, and a mouse click on would land some other place solely.

The record is deduplicated by function and label, drops any label the seen objects already carry,
and stops at 120 controls, after which these subtrees are pruned as earlier than, so the stroll prices
what it all the time did. It is obtainable solely when it isn’t empty, as a press_offscreen motion
plus an offscreen query, and the step log counts it subsequent to ax=. A refusal is the top
of it: there is no such thing as a pixel to fall again on, so it reads as a no-op. What a stroll finds relies upon
on the app, and the node and time caps bind first on an enormous tree: Notes and Chrome spend all
4000 nodes on what’s already on display and report nothing hidden.

The classifier by no means generates textual content. The author mannequin runs in three locations, every with a
small packet and a structured reply:

• type_text receives the purpose, latest actions, the centered discipline’s label and
  placeholder, and the OCR strains close to the sector. It returns {fill, textual content}. Credential
  fields come again fill: false and nothing is typed. After typing, a Noul scores
  whether or not the sector now holds a wise worth. Under 0.5 the sector is cleared.
• use_browser with web site: different receives the purpose and returns {okay, url}.
  Code rejects something that isn’t a clear https URL with a hostname.
• The reply, as soon as, when the loop stops itself. It receives the purpose, each motion
  taken, why the run stopped, the textual content of the final display, and the seize itself,
  as a result of OCR misreads a letter right here and there and drops structure. It returns
  {achieved, reply}, and is instructed to take the reply from the display alone. When an
  motion ran after the final seize, the display is captured once more first. This one
  name makes use of CLICKER_ANSWER_MODEL, a stronger reader than the per-step author.

Passwords are by no means typed. Rely on the browser’s password supervisor or an SSO button
the OCR can learn.

Every run writes runs// so a stall might be replayed and glued offline:

Each step additionally logs what it price, so a sluggish part is clear:

  timing: seize 0.31s  screenshot 0.28s  app 0.01s  window 0.02s  discipline 0.01s  url 0.01s  ocr 0.31s (22% of display)  ax 0.06s  resolve 0.21s  act 0.05s  complete 0.95s

seize covers the 4 spherical journeys below it; act is overlooked when the step didn’t act.

Replay a saved seize as if it had been dwell, with out touching the display:

uv run clicker "similar purpose" --image runs//step-003-raw.png --app "Google Chrome" --url "https://example.com/"

typesafe_computer_use/
  macos.py        the one module that touches Quartz, AX, AppleScript   (platform adapter)
                  together with the bounded stroll for actionable parts
  notion.py   seize, OCR, the learn area and the changed-tile cache,
                  block merging, goal-echo filter, the accessibility merchandise
                  supply, and the merge of the 2
  dates.py        date parsing and "in N days" hints
  resolve.py       state, standards, the three-Choice request, the Noul verify
  author.py       the author mannequin, structured replies, URL validation, the ultimate reply
  actions.py      one handler per motion, every returning a historical past line
  runner.py       the step loop, run folder, cease guidelines, the hand-off for the reply
  report.py       logging, annotated screenshots, payload dump
  timing.py       part stopwatches, the timing line, run abstract
  cli.py          `clicker` and `clicker-inspect`
checks/            pure logic: dates, merging, studying order, echo filter, config,
                  choices, the tree stroll in opposition to a faux tree

A Linux port replaces macos.py with xdotool and AT-SPI, and swaps Vision OCR for
PaddleOCR or RapidOCR. The tree stroll itself takes its kids, attributes, and actions
as callables, so solely these three bindings change. Nothing else is aware of the platform.

• OCR solely sees textual content, and the accessibility tree solely covers apps that publish one.
  In a terminal, a canvas, or Spotify, an icon-only button reaches neither supply.
• Two an identical labels get solely a rough area trace and break up the vote.
• Only the principle show is captured.
• Using the machine throughout an --act run fights it for focus and the cursor.
• The web site catalog is small on function; the author covers the remaining.

uv run ruff verify . && uv run ruff format --check .
uv run pytest -q

CI runs the identical on macOS. See CONTRIBUTING.md.

MIT