Unreal Agent — Unreal Labs

If you’re enthusiastic about frontier cost-efficiency to your AI brokers, we’d like to work collectively! Get in contact: [email protected].

---

While deploying brokers within the wild, we wished them to reply to customers shortly and be cost-effective to run. We’ve seen that brokers spend a variety of time and tokens managing device calls, which motivated us to construct Unreal Agent with a harness that would cut back the mannequin’s tool-management overhead.

The Unreal Agent harness manages device calls in a totally asynchronous method, relieving the underlying mannequin of the necessity to handle waits, polls, and heartbeats for instruments.

This method drives two main advantages. First, it at all times permits customers to steer the agent with out the necessity to anticipate device calls to complete. Second, it permits the agent to schedule extra great tool name work between mannequin calls, driving frontier value effectivity. The present model achieves as much as 40% value financial savings in comparison with Codex and as much as 20% in comparison with Pi in actual workloads and on agentic benchmarks, which we share right here.

We consider harness design is a analysis space in its personal proper, with many promising concepts nonetheless to be researched and applied.¹

If you attempt to construct an agent-first product, you’ll shortly notice that there’s no golden path for implementing one. Big-brand distributors provide totally different SDKs to construct brokers, every with a special set of trade-offs that may not be instantly obvious.

At Unreal Labs, we have now constructed quite a few agentic merchandise and discovered just a few issues about common SDKs alongside the way in which.

For instance, CLI-oriented SDKs equivalent to Claude’s Agent SDK carry assumptions about native classes, subprocesses, and useful resource limits that don’t translate neatly into manufacturing use. Handling completion, cancellation, and background duties reliably usually means constructing your individual lifecycle administration round them.

Supporting different suppliers provides compatibility work: switching API modes can break instruments or compaction, whereas SDK upgrades can change message codecs and power integration rewrites. Heavy dependency bushes add upkeep and supply-chain danger to a runtime we already want to grasp and patch ourselves.

Security and approvals that depend on harness hooks and specialised instruments, in our expertise, are inclined to require extra upkeep and be much less sturdy than deterministic setting or sandbox constraints, exterior the harness: allowed/disallowed hosts, granular entry tokens, proxies with approval gates.

Along with these technical motivations, we additionally wished to construct a harness that might at all times settle for person steering messages immediately and juggle heterogeneous device calls with out additional cognitive load for the mannequin. For instance, we wished the agent to have the ability to kick off a dev setting setup that may take minutes, whereas exploring the codebase and looking the online in parallel, all with out additional token tax.

Every time Unreal Agent points a device name, we instantly append an event-log report that the device has returned within the “in-progress” state, whereas persevering with its execution within the background. Once a device truly finishes, we append the consequence into the session log and name an LLM. Making this work with out breaking cache was an attention-grabbing engineering problem in itself.²

On the floor, Unreal Agent achieves the identical outcomes with fewer mannequin turns and fewer enter tokens.

We attribute value financial savings to 2 elements:

1. Minimal harness footprint and cautious engineering of device output utilization. Unreal Agent has easy prompts, token-optimized device outcomes, and no sub-agents or workflows.³

2. More device work per mannequin flip. Unreal Agent has a simple asynchronous tool-calling mannequin that’s clearly defined to an LLM. This permits it to challenge extra heavy device calls per mannequin flip with out losing tokens on polling or ready for them.

We’ve constructed Unreal Agent to ship actual manufacturing workflows for us, nevertheless it appears good within the benchmarks too. We examined it with GPT-6 Astra xhigh and in contrast it with Codex and Pi. Here are among the outcomes.

There are marginal variations in go price, which we attribute to benchmark variance.

#Terminal-Bench 4.0

GPT-6 Astra · xhigh. Codex (lb) is the leaderboard baseline; Unreal Agent and Pi runs are linked beneath.

Agent	Rate	Total $	In/trial	Out/trial	Turns	Tools	Harbor
unreal-agent	57.9%	1428	1.73M	32k	28	37	27133053
Codex (leaderboard)	57.9%	2350	—	—	—	—	—
Pi	55.0%	1827	2.83M	35k	44	57	6ccd097a

#SWE-Atlas Codebase QnA

Agent	Rate	Total $	In/trial	Out/trial	Turns	Tools	Harbor
unreal-agent	65.8%	936	898k	15k	16	27	3d2fa057
Codex	63.3%	1303	1.69M	17k	22	21	11a440fb
Pi	64.0%	1033	1.29M	16k	24	60	da4ac972

#DeepSWE 1.1

Agent	Rate	Total $	In/trial	Out/trial	Turns	Tools	Harbor
unreal-agent	72.4%	1367	1.60M	28k	26	38	2311ca63
Codex	69.0%	1633	2.19M	30k	30	29	e20ecafd
Pi	69.6%	1584	2.21M	30k	40	75	cd7d8de6

#Agents’ Last Exam · ALE-CLI

Full go charges and imply scores are listed beneath. These runs usually are not on Harbor.

Agent	Full go	Mean rating	Total $	In/activity	Out/activity	Turns	Tools
unreal-agent	30.0%	59.7	217	0.76M	18k	18	23
Codex	29.0%	58.1	292	1.59M	15k	—	21
Pi	29.0%	59.2	262	1.19M	19k	27	37

We run principally coding benchmarks as a result of they’re out there on Harbor, which makes copy and verification simpler, however the harness is domain-agnostic.

The Unreal Agent SDK at the moment provides:

• Go library you could combine instantly into your codebase
• Runner executable just like claude -p / codex exec
• Benchmark runner appropriate with Harbor

Check out our github repo if you wish to attempt it for your self