Coverage-Cat/consumer-agent-testing-tool: Consumer-agent surrogate testing for MCP servers, apps, browsers and APIs, with service-side consequence checks and sealed proof · GitHub

Consumer Agent Testing Tool (CATT) exams how an assistant makes use of your service.
Run a scripted or model-driven agent towards an MCP server, an MCP app, a
web site or an HTTP API. Check what the service did, examine the proof, and
evaluate outcomes throughout builds.

Muse and Instinct are supported surrogate emulation targets. Muse stays
the default. Both use the shared connector testing contract,
together with routing, reminiscence, funds, browser/HTTP duties and scheduling.
See the Instinct research for its sources and
limits. The mannequin that drives a trial is separate from the assistant being emulated.

Contributing | Security | Changelog | MIT license

A check host. CATT is a surrogate: it approximates an assistant’s
host, not its non-public implementation. A passing check doesn’t certify
compatibility with Muse or any hosted assistant. Each Muse-specific conduct
has a recorded foundation: documented, unofficial, assumed or unknown.
This mission shouldn’t be affiliated with or endorsed by Meta, Instinct, OpenAI or Anthropic.

The connector lane data the premise for every Muse-specific conduct. Tool lessons
and approval prompts comply with documented tips. The muse-cli projection and
template shopper comply with unofficial mirrors; the native projection is an assumed
generic baseline. The impartial cli projection helps each targets as a CATT
remedy. Instinct runs have a separate supply report and label the
shared host algorithms as assumptions. Native/CLI MCP is CATT’s check adapter;
no public Instinct MCP integration contract was discovered.

Run the native reserving instance underneath Instinct and not using a mannequin or account:

uv run consumer-agent-testing-tool connector-run examples/connector/instinct-booking.yaml 
  --driver scripted --script examples/connector/fixture-booking.calls.json 
  --start "python -m consumer_agent_testing_tool.connector_fixture --port {port} --variant good --auth none"

Set emulation_target: intuition in your state of affairs or move
--emulation-target intuition. Both targets use native or cli projection and the generic
MCP shopper. Instinct defaults to plain-text replies and doesn’t run Muse guideline
checks. Its native pockets fixture covers cost conduct; reside cost rails,
trusted-network protocol, Vault and Files stay exterior the
shared contract.

Three quickstarts comply with. The connector lane and
the popular MCP server quickstarts run on
loopback with no account; the Coverage Cat sandbox
quickstart calls a hosted sandbox. Test your own service
turns them right into a workflow, and the documentation index lists
each web page.

You want Python 3.12 or newer (CI exams 3.12 on Linux and three.14 on macOS),
uv, and Node.js 20 or newer with npm (the
put in packages declare >=20; examined with 25.9). The recorded launch
proof covers macOS. From a clone of this repository, set up the Python and
Node dependencies and the Chromium construct the browser lanes use:

uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" node tooling/payment-ui/node_modules/playwright/cli.js set up chromium

The connector lane, the readiness checks and the scripted runs want solely
uv sync. Node and Chromium are for the MCP Apps and browser lanes.

On Linux (Debian or Ubuntu), set up Chromium with its system libraries
as a substitute. Linux shouldn’t be coated by the recorded launch proof: the instructions
under match the CI workflow however haven’t been
certified right here.

sudo apt-get replace && sudo apt-get set up -y nodejs npm   # or Node 20+ from nodesource or nvm
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra openrouter --extra dev
npm --prefix tooling ci --ignore-scripts
npm --prefix tooling/payment-ui ci --ignore-scripts
PLAYWRIGHT_BROWSERS_PATH="$PWD/.cache/ms-playwright" 
  node tooling/payment-ui/node_modules/playwright/cli.js set up --with-deps chromium

Three options depend on macOS Seatbelt (sandbox-exec) and report unsupported
or blocked on Linux relatively than operating unconfined: the --offline readiness
mode, the network-denied validator employees behind run --surface api-tools, and
the local-model reminiscence profile. The connector lane, the scripted runs, the
readiness checks and the Docker-backed stdio targets don’t rely upon it.

Check the setup with physician:

uv run consumer-agent-testing-tool physician

It checks Node towards the flooring the put in packages declare, the
tooling/payment-ui packages towards their pinned variations, and Chromium and
its headless shell underneath .cache/ms-playwright. It reviews the non-obligatory
protocol tooling, and checks Docker just for growth lanes or configured
stdio MCP and CLI targets. It makes no mannequin requests, even when an current
configuration or --model-profile selects a mannequin. Each failed or blocked test
prints one line on stderr with its motive and, for a setup downside, the command
that fixes it.

To explicitly run the supplier preflight, which sends two brief mannequin requests
and will incur prices, add --model-preflight:

uv run consumer-agent-testing-tool physician --config agent.yaml --model-preflight

Scripted runs want no mannequin account, and the scripted instructions in all three
quickstarts want no configuration file. Agent runs want a mannequin profile and an
API key; Model providers reveals methods to create one. Supply
credentials by atmosphere variables, and preserve them out of situations and
shared proof. consumer-agent-testing-tool.yaml, the default settings file,
is native and git-ignored, and no profile is shipped.

This quickstart stays on loopback and wishes no account. Complete
Setup, then begin the Cedar Stays-like check service in a second
terminal. It prints one JSON line with its URLs when it’s prepared; Ctrl-C stops
it:

uv run python -m consumer_agent_testing_tool.connector_fixture --port 8765 --variant good --auth none

Replay the instance’s scripted calls, with no mannequin:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml 
  --driver scripted --script examples/connector/fixture-booking.calls.json

The script searches rooms, will get a quote, books and provides a closing reply. The
host fills the visitor’s e mail from the state of affairs’s non-public: values, which the
mannequin by no means sees, and the scripted person solutions every approval card from the
state of affairs’s permissions: guidelines. After the run, the host checks the fixture’s
non-public oracle (one confirmed reserving, no duplicate prices) and the reserving’s
standing by get_booking. It additionally checks that every despatched name matched its
approval card (approval.dispatches_match, approval.disclosure_bound) and
that the ultimate reply made no false declare and relayed the full and the
affirmation (reply.no_false_claims, reply.relay_complete).

The command prints the run listing and Connector report: .../report.html,
and exits 0 when each required test handed. Runs are sealed underneath
runs/connector-/. Share the output of export, not the run listing:
its non-public/ folder holds host-private values.

For an agent run, create agent.yaml with the init command in
Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool connector-run examples/connector/fixture-booking.yaml --config agent.yaml

The mannequin preflight runs first and prints its personal proof path. The agent then
sees the 4 allowed instruments as native capabilities, with the host-private e mail
faraway from their schemas, and writes the opposite arguments itself. This makes use of
paid mannequin inference. To change the remedy with out modifying the state of affairs,
--projection cli affords the identical instruments by one exec command for both goal,
--exposure search-only hundreds their parameters by tool_search,
--output-schema strict refuses outcomes that break a device’s outputSchema as
the official MCP SDK does, and --trials N runs unbiased trials and prints
move^okay. The run lock data the projection, publicity, client-profile and
output-schema overrides underneath overrides.

connector-run exits 0 when each required test handed; 1 when a test
failed, the reply made a false declare or an agent restrict was reached; 2 for a
setup, harness or reporting error; and three when the trial is blocked or
unsupported, for instance by an unsure write to reconcile, an unobservable
test or a client-profile restrict. To begin from your individual server, run
scaffold; to replay a recorded, shareable agent trial and not using a mannequin, flip
it right into a calls file with scaffold --from-run. The
connector lane reference covers situations, flags and
proof. The examples
record every shipped state of affairs with its fixture command, and
lane extensions covers routes, customized
connectors, the browser sub-agent, wallets, schedules and the opposite
extensions.

The connector lane works with any MCP server over Streamable HTTP, not solely the
bundled check service. This quickstart factors it at “Everything”
(@modelcontextprotocol/server-everything,
MIT), the reference server the MCP mission maintains for shopper authors. Its
climate device returns simulated knowledge, so it wants no account or key. Start it in
a second terminal; the primary run downloads the pinned launch from npm:

PORT=3001 npx -y @modelcontextprotocol/[email protected] streamableHttp

See what the host infers from the reside catalog. scaffold runs the handshake
and instruments/record, calls no device, and writes a state of affairs skeleton with recommended
device lessons from the server’s annotations:

uv run consumer-agent-testing-tool scaffold http://127.0.0.1:3001/mcp --output my-everything.yaml
uv run consumer-agent-testing-tool test http://127.0.0.1:3001/mcp --suite muse-guidelines

The second command opinions the catalog statically towards the pinned Muse
connector tips. Then replay the shipped instance, which asks for the
climate in Chicago, first with native instruments after which by the Muse-style
command-line projection over the unofficial Muse template shopper:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml 
  --driver scripted --script examples/connector/everything-weather.calls.json
uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml 
  --projection muse-cli --client-profile muse-template-unofficial-2026-10 
  --driver scripted --script examples/connector/everything-weather-cli.calls.json

Both exit 0. The state of affairs affords the mannequin three of the server’s instruments; a
reviewed allowlist retains the remaining off its record, together with get-env, which
returns the server course of’s atmosphere variables, and
gzip-file-as-resource, which fetches any URL it’s given. The server has no
non-public oracle, so the result test calls get-structured-content itself
after the agent stops, and the reply checks search for the temperature and
circumstances the server returned. With a mannequin profile, let an agent do the identical
job 3 times and print move^okay (paid inference). The state of affairs declares no
reset URL, so connector-run warns that the trials share the server’s state;
this climate device retains none:

uv run consumer-agent-testing-tool connector-run examples/connector/everything-weather.yaml 
  --config agent.yaml --trials 3

test additionally runs readiness suites that learn catalogs and paperwork as a substitute of
operating an agent. With the connector-lane quickstart’s fixture nonetheless operating,
overview its catalog and the instance manifest towards the pinned Muse connector
tips:

uv run consumer-agent-testing-tool test http://127.0.0.1:8765/mcp --suite muse-guidelines 
  --manifest examples/connector/readiness.yaml

The tips overview lists the server’s instruments and calls none; --drift-check
fetches the reside tips, and solely when given. Poppy discovery solely reads.
Its probes run solely with --manifest and act on the corporate’s personal hosts: they
begin classes, signal within the declared check account and ensure one declared
operation. The gadget suite reads one native file. The contract suite checks a
goal named within the configuration file (--config). For a sort: mcp goal
with out declared contract_cases, its operation-contracts test is
unsupported and the run exits 3.

A clear readiness end result shouldn’t be an admission. Meta opinions connectors by hand,
Poppy is draft 0.1 with no manufacturing Muse proof, and the Gadget SDK’s phrases
say it isn’t a developer platform. Readiness checks
describes the foundations, manifests, --offline conduct and the
mcp-contract-2026-07-28 profile.

From a supply checkout or unpacked supply distribution, full
Setup, then run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml --driver scripted

The instance creates a fictional shopper’s draft in Coverage Cat’s hosted
sandbox at https://www.coveragecat.com/api/consumer/mcp. It wants no
Coverage Cat account, operator credentials or mannequin API key.

The first umbrella_consumer_prefill name units sandbox: true, as laid out in
the consumer MCP manifest.
That argument prompts the server’s sandbox. The state of affairs’s test_only: true
flag merely data the check creator’s intent.

The host fixes the device arguments and retains the returned entry token non-public.
It permits two instruments, umbrella_consumer_prefill and consumer_intake_issues,
and checks that:

1. Prefill returns sandbox: true.
2. Follow-up retains sandbox: true.
3. Follow-up returns the unique software ID.
4. The server reviews application_submitted: false.

Each run creates a brand new draft and stops earlier than submission. A failed or lacking
sandbox test stops the workflow. These checks cowl draft creation and
follow-up, not cost widgets, browser conduct or unbiased backend checks.

To let a mannequin select the steps, create agent.yaml with the init command in
Model providers, set OPENROUTER_API_KEY in your shell, and run:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox.yaml 
  --driver agent --config agent.yaml

The agent driver is a scripted motion menu: Spark Open picks one prepared
state of affairs step per flip, and the host runs it with the state of affairs’s arguments. The
mannequin by no means sees the server’s instruments, schemas or uncooked outcomes, solely the end result
fields the state of affairs exposes, and the proof is labeled scripted motion menu.
The host provides the check inputs and checks the outcomes. This model makes use of
paid mannequin inference by Together, with ZDR enabled and fallback disabled
by default.

Verified 15 October 2026: each scripted and Spark Open agent runs handed all
4 sandbox checks. Both mannequin responses recognized Together; each requests
used the configured ZDR coverage. See the
recorded quickstart results.

Each command prints an consequence line, equivalent to Surface run accomplished (exit 0),
then Surface report: .../report.html. Open the report to examine the workflow
and its checks. The similar listing comprises end result.json for the structured
end result and occasions.jsonl for the redacted occasion timeline. Agent runs additionally retailer
mannequin and power data underneath agent/.

Replace the placeholders under with the run directories printed by your instructions:

uv run consumer-agent-testing-tool report runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool export runs/surface-YOUR_RUN_ID
uv run consumer-agent-testing-tool diff runs/surface-BASELINE_ID runs/surface-CANDIDATE_ID

Share the generated export. Keep the total run folder non-public. While a run can
resume, checkpoint.non-public.json holds the credentials, device outcomes and host
inputs it wants; a accomplished run, or one which failed with exit 1 and left no
write unconfirmed, retains solely a scrubbed report.

To check your individual MCP service, copy the instance and substitute its endpoint, allowed
origins, device insurance policies, check knowledge and response checks.
Surface workflows explains UI steps, the agent
driver and resuming runs.

The checkout example provides overview,
submission, Monoline supply choice, consent, bind follow-up, doc add
and simulated payment-method setup. Its 14 steps comprise 62 assertions. All
seven instruments use the buyer umbrella MCP, and each service step should return
the identical sandbox consumption.

Current integration standing, 15 October 2026: blocked at submission.
Coverage Cat returns quote fields that its MCP output schema excludes. It additionally
returns minimum_auto_limits as a string the place the schema requires an object.
The host rejects the response. The draft quickstart passes, however the full checkout
check has not handed finish to finish. See the
failure report and separate diagnostics.

Run this command to breed the failure or retest after the service contract
is corrected:

uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml 
  --driver scripted 
  --replies examples/coveragecat-sandbox-checkout-replies.json

The reply information provide a fictional shopper’s solutions: approval to submit,
credit score consent, the Monoline $2M supply at $389, ample auto limits, declined
extra UM/UIM, and “No” to each provider questions. These fixtures don’t authorize
transactions for an actual shopper. A modified quote or lacking or unfavorable approval
stops the subsequent write. The two generated declarations PDFs are labelled
SYNTHETIC TEST ONLY.

Once submission works, the instance pauses with exit code 3 at
sandbox_document_review_poll. Only at that checkpoint, wait a minimum of the
reported 60 seconds and resume the printed run:

sleep 60
uv run consumer-agent-testing-tool surface-run examples/coveragecat-sandbox-checkout.yaml 
  --driver scripted --resume runs/surface-YOUR_RUN_ID 
  --replies examples/coveragecat-sandbox-checkout-after-review.json

The host pauses in your reply however doesn’t measure how lengthy you wait. The
after-review fixture approves the simulated cost callback.

For an agent-driven run, substitute --driver scripted with
--driver agent --config agent.yaml on each instructions. A resumed run should
preserve its authentic state of affairs, driver and mannequin configuration.

The sandbox provides mock cost credentials. The instance sends the returned
seti_sandbox_... with the fixture pm_sandbox_consumer_agent_testing_tool, then requires
payment_method_on_file, waiting_on_carrier, and no cost affirmation.
Billing checks examine setup metadata relatively than rendered fields. No Stripe
kind opens, no card particulars are collected, and no fallback URL is used. This
check establishes neither premium assortment nor energetic insurance coverage. Test the
embedded cost UI and Stripe test-mode setup individually.

The quickstarts above are the items of 1 workflow. For an MCP server you
personal:

1. Write a state of affairs. scaffold URL --output my-connector.yaml reads the
   reside catalog (--auth-env VAR for an API key, or --auth-from FILE with a
   state of affairs that holds an OAuth block) and writes a skeleton. Replace each
   TODO(scaffold): the person’s aim in their very own phrases, the allowlist of instruments a
   reviewed connector would expose, device lessons your documentation settles,
   approval guidelines for the scripted person, host-private values such because the
   person’s e mail, and checks that learn the true consequence, from a personal state URL
   (an oracle) or one in every of your learn instruments.
2. Check readiness. test URL --suite muse-guidelines --manifest FILE
   opinions the catalog and your declarations towards the pinned connector
   tips. A clear result’s a documented constraint, not an admission.
3. Pin the glad path. Replay a scripted calls file with --driver scripted; it wants no mannequin and belongs in CI. After an agent trial you
   belief, scaffold --from-run RUN --output FILE turns it into one.
4. Run agent trials. --trials N with a mannequin profile provides move^okay. Run
   the identical state of affairs underneath every remedy a Muse-like host could use:
   --projection muse-cli, --exposure search-only,
   --client-profile muse-template-unofficial-2026-10 and
   --output-schema strict. The run lock data each.
5. Add realism. Lane extensions add
   competing connectors and connect-on-request, cost with single-use playing cards,
   background jobs, untrusted content material, channel limits and a simulated person.
6. Compare builds. diff compares two sealed runs and exits 1 on an
   consequence regression. Share the output of export; the run listing’s
   non-public/ folder holds host-private values.

For an MCP app with its personal UI, begin from the Coverage Cat state of affairs and
surface-run. For a web site, use browse with a browser coverage, or the
connector lane’s browser extension. For an HTTP API, write a mapping for
run --surface api-tools, or check whether or not an agent can construct its personal shopper
out of your docs with the custom_connector extension.
Run a consumer test covers the opposite lanes.

Create a profile with init. This one makes use of Muse Spark Open
(meta/muse-spark-open) by OpenRouter, routed solely to Together, with zero
knowledge retention (ZDR) required and fallback disabled; set OPENROUTER_API_KEY in
your shell earlier than an agent run:

uv run consumer-agent-testing-tool init --provider openrouter --model meta/muse-spark-open 
  --routing-provider collectively --config agent.yaml

init additionally phases, beside the configuration, the API situations, the browser
situations and the mapped-API instance. It phases solely situations the supported
CLI can run, and refuses suppliers which have solely a configuration template
(openai, anthropic, gemini, azure-openai, vertex-gemini,
openai-compatible) until --development-extensions comes earlier than init.
init --provider none --config scripted.yaml writes a configuration for
scripted instructions that want one. init refuses to overwrite an current file.

Profiles that init writes are labeled specified/unverified. physician --model-preflight, run
and the agent drivers of connector-run, surface-run and browse print a
warning for any profile not labeled supported, and the run lock data the
label. Model IDs that comprise contributor choose a mode by which the supplier
could practice in your prompts; such a profile wants
allow_training_on_prompts: true (init --allow-training-on-prompts). See
model providers for each adapter, its checks and
its failure labels.

Consumer Agent Testing Tool requests zero knowledge retention (ZDR) by default, even when
routing.zdr is omitted. To change this, edit the chosen profile in
consumer-agent-testing-tool.yaml or the file handed to --config:

model_profiles:
  openrouter-dev:
    supplier: openrouter
    api: chat-completions
    model_id: meta/muse-spark-open
    auth:
      mode: api_key
      env: OPENROUTER_API_KEY
    routing:
      solely: [together]
      allow_fallbacks: false
      require_parameters: true
      data_collection: deny
      zdr: true  # Set to false to disable this device's per-request ZDR requirement.
defaults:
  model_profile: openrouter-dev

Use unquoted YAML booleans: true or false. Consumer Agent Testing Tool applies the setting to
each OpenRouter request, together with requests that proceed after a device end result,
and data it within the routing proof.

Setting zdr: false removes this request-level requirement. It leaves
data_collection: deny, the pinned supplier and the ban on fallbacks in place.
Your OpenRouter account or guardrails should implement ZDR. Request logs present
the coverage despatched; they can not independently confirm the supplier’s retention
practices. See OpenRouter’s ZDR documentation.

Checked 15 October 2026: OpenRouter listed Together for Spark Open in each
its mannequin and ZDR endpoint catalogs. A reside tool-call check handed with the
routing settings above. Both responses recognized Together, and the mannequin used
the device end result appropriately. See the
test record.

To select one other supplier, set routing.solely or move --routing-provider to
init. Requests by no means swap
suppliers or fashions silently.

Runs use Inspect’s devoted openrouter supplier, which replays earlier
reasoning in structured kind relatively than as textual content. Each response should title the
pinned supplier. Responses don’t title endpoint variants, so a pin equivalent to
nebius/fp4 is recorded as variant unverified. Streaming shouldn’t be supported.
See model providers.

With zdr: true, solely ZDR-listed endpoints are eligible. On 15 October 2026,
Muse Spark 1.3 and Muse Wonder had been served solely by Meta on OpenRouter and had been
not ZDR-listed, so no ZDR-compliant OpenRouter route existed for them. A
request for them is predicted to fail, however no reside run has recorded the error.
When OpenRouter’s error cites the info coverage, it’s labeled
no endpoint satisfies the routing/ZDR coverage. With routing.solely: [together],
the init default, it could as a substitute be labeled
routing coverage: no allowed supplier.

Sources: Meta announcement,
OpenRouter model,
provider endpoints,
and ZDR endpoints.

The standalone smoke check and older shopper helper learn the identical --config
and --model-profile, together with privateness settings. Profiles that init writes
go away reasoning effort unset.

.venv/bin/python scripts/openrouter_smoke.py --config agent.yaml 
  --model-profile openrouter-dev

This command exams the mannequin reference to artificial device knowledge. It wants an
API key and an accessible path to the pinned supplier. It makes no requests to
the service you propose to check.

init --provider meta writes a profile for Meta’s Model API:

uv run consumer-agent-testing-tool init --provider meta --model muse-spark-1.3 --config meta.yaml

The profile makes use of the Responses API at https://api.meta.ai/v1 and reads the important thing
from MODEL_API_KEY or META_API_KEY; if each are set, they have to agree. Add
--api chat-completions for Chat Completions, or --key-env NAME to make use of one other
variable, which is then the one one learn. The adapter makes use of the OpenAI SDK from
the openrouter further.

The host refuses requests that Meta rejects, equivalent to a compelled device selection,
earlier than sending them (exit 3). Meta provides steering context to each immediate on its
servers, so captured requests are incomplete, and each Meta run data this
as an commentary hole. Streaming shouldn’t be supported, and the API is
pay-as-you-go. See Meta Model API.
No reside Meta Model API run has been recorded; the adapter is examined towards
native fixtures.

This part covers the embedded MCP apps (surface-run), browser (browse)
and HTTP API lanes. To let an agent use your MCP server’s personal instruments, use the
connector lane.

A state of affairs defines the endpoint, allowed origins, instruments, non-public arguments, UI
actions, consent necessities and end result checks, and optionally an API key
for the MCP connection (auth). Use endpoints and artificial transactions you
have permission to check.

consumer-agent-testing-tool surface-run path/to/state of affairs.yaml --driver scripted
consumer-agent-testing-tool surface-run path/to/state of affairs.yaml --driver agent --config agent.yaml
consumer-agent-testing-tool surface-run path/to/state of affairs.yaml --resume runs/existing-run
consumer-agent-testing-tool browse path/to/browser-policy.json --actions path/to/actions.json

The host declares MCP Apps assist as a shopper, finds every device’s UI by its
instruments/record definition and renders it in a local browser. With
--driver agent, the mannequin picks among the many state of affairs’s steps (a scripted motion
menu) and by no means sees the server’s instruments, uncooked outcomes or non-public inputs. Service
checks decide the end result; a rendered kind or a mannequin’s declare of success
can’t show {that a} transaction accomplished.

browse drives a restricted browser by accessibility snapshots and aspect
references, with no selectors, scripts or CDP. Its coverage file units the beginning
url, the permitted origins and request guidelines, snapshot_max_chars
(default 20,000), history_snapshots (2) and dialogs (agent, settle for or
dismiss), as in examples/browser-policy.json.
Long snapshots are paged with {"sort": "snapshot", "offset": N}. Refused or
failed actions come again to the agent as observations with a standing, and
dialogs and file choosers are answered with sort: dialog or sort: add.
With --driver agent, the supplier preflight runs earlier than the browser opens.
The end result’s observations block counts truncated snapshots and refused
actions. When the agent requests a takeover, the run pauses (exit 3). The CLI
has no handback, so management doesn’t return to the agent in that run. In
native mode, WebSockets opened by devoted employees and WebRTC TCP relays are
not enforced. The coverage area webmcp: true, off by default, lists
page-registered WebMCP instruments within the snapshot and lets the agent name them with
sort: webmcp; that lane is labeled “generic WebMCP web page instruments, not Muse”
(foundation: assumed), and its proof notes that Meta paperwork WebMCP for its
AI glasses, a Meta product that isn’t Muse. See
restricted browser behavior.

HTTP API exams use specific mappings with the state of affairs runner. Mappings can
ship declared request headers and put the goal credential in a customized API-key
header. In API and browser trials, invalid device proposals are returned to the
mannequin as device errors, and exhausted limits fail the trial; see
API scenarios. To check
whether or not an agent can construct its personal shopper from an API’s documentation as a substitute,
use the connector lane’s custom_connector extension. Prism, Schemathesis and
generated-OpenAPI instruments can be found solely as growth extensions. Their
5 packages with unresolved authentic license notices are excluded from the
supported launch, not handled as cleared.

surface-run follows the exit codes in spec §18:

Before Surface report: ... it prints one line:
Surface run (exit N) at []: . end result.json
data the identical failure_class and failure_stage, and whether or not the run is
resumable. A accomplished run, or one which failed with exit 1, is closing until
the failure left a write unconfirmed: that write could have taken impact, so the
run stays resumable to reconcile it. Blocked runs and setup failures can resume.

run and browse use the identical 4 codes, besides that exhausting a flip,
tool-call or time restrict is a failure (exit 1). Only Inspect operator and customized
limits, and in run a deadline that handed earlier than an try’s first
era, block as a substitute (exit 3). run prints every proof path on stdout.
browse prints the ultimate commentary and its proof path when the run
completes or pauses for takeover, and nothing on stdout in any other case. For any
consequence aside from accomplished, each additionally print one line on stderr:
() at : . Evidence: . The label
names the step, for instance Model preflight, Trial 2/10 of stale_total,
Resumed trial of stale_total or Browse (physician makes use of Doctor test). A
run with a failed or blocked trial, or with --trials above 1, additionally prints a
abstract line with the counts. With --trials above 1 it provides the suite
move^okay, the imply over situations, for instance
Run accomplished: 0 failed, 0 blocked and 4 accomplished of 4 trials; move^1 1.00, move^2 1.00. Evidence: ....
The suite’s metrics additionally break the trials down by state of affairs
(statistical decisions). run DIR refuses
(exit 2), earlier than the preflight, a listing whose state of affairs information share an id.

connector-run prints the identical stderr line, labeled for instance
Connector trial 1/1 of fixture-booking-happy-path; its codes are listed within the
connector-lane quickstart. With --trials N, one
code stands for all trials: 2 when any trial’s proof shouldn’t be shareable, else 1
when any trial failed, else 2 when any hit a setup, harness or verification
error, else 3 when any was blocked, else 0. The Connector trials: line on
stderr provides move^okay and names the trials that aren’t shareable or led to an
error. The readiness suites’ codes are in Readiness checks.

The embedded-app lane is generic MCP Apps host proof (MCP Apps 2026-01-26),
for hosts equivalent to ChatGPT or Claude. It shouldn’t be Muse proof: Muse paperwork no
third-party UI. Each end result.json says so underneath mcp_apps_host.

The host implements:

• the shopper functionality (io.modelcontextprotocol/ui with
  textual content/html;profile=mcp-app), UI discovery from the device’s instruments/record
  definition, and HTML sources as textual content or base64 blobs;
• tool-input with the decision’s arguments, tool-result, and a hostContext
  (theme, show mode, locale, time zone, platform, container width and
  most peak, and power data);
• a content material safety coverage constructed from the useful resource’s _meta.ui.csp, or the
  specification’s restrictive default when the useful resource declares none, and the
  declared digicam, microphone, geolocation and clipboard-write permissions;
• within the native renderer, a sandbox proxy on an origin separate from the host
  web page. With browser_origins, the host web page additionally has Chromium’s
  local-network-access permission, each redirect hop is checked once more (GET
  or HEAD to an allowlisted origin solely), and every direct write is recorded as
  an unconsented host_action earlier than it’s despatched;
• view instruments/name requests for the step’s allowlisted instruments, and
  sources/learn and sources/record for the step’s callback_resources;
• ui/message, delivered as soon as to agent runs within the subsequent commentary a mannequin
  reads, as app_messages; the view is informed when the host doesn’t ship
  it. ui/update-model-context, ui/open-link, display-mode, log and
  teardown requests are recorded, and the most recent model-context replace reaches
  agent runs as app_context;
• dimension modifications, ui/resource-teardown earlier than the view closes, and
  tool-cancelled when the view by no means obtained the decision’s end result;
• _meta.ui.visibility, for view calls and for device steps;
• MCP 2026-07-28 servers, when the state of affairs units version_negotiation: auto.
  The default is the 2025 initialize handshake;
• an API key on the MCP connection (auth), as a bearer token, a customized
  header or a question parameter;
• a legacy window.openai bridge.

It doesn’t implement a mannequin flip for a view’s message, useful resource templates,
_meta.ui.area, show modes aside from inline, partial device enter,
host-context change notifications, OAuth on the MCP connection, or consent for
a view’s direct writes within the native renderer. The legacy bridge and the
remoted renderer haven’t any sandbox proxy. See
MCP Apps host.

consumer-agent-testing-tool report runs/existing-run
consumer-agent-testing-tool export runs/existing-run
consumer-agent-testing-tool replay runs/existing-run
consumer-agent-testing-tool diff runs/baseline runs/candidate

Replay checks the proof seals and stops on a mismatch. It by no means substitutes
a reside request. To check the service once more, begin a brand new run. Reports mark lacking
measurements as unknown and preserve partial totals distinct from full totals.

Reports open with the decision, which names the failing step and test, then
the checks and host capabilities, and a per-step timeline that reveals what the
mannequin obtained in full. Then come a connectors desk for runs with a number of
connectors, the floor views (what the service printed, what the mannequin was
supplied, what it obtained at every step, and what solely the developer sees),
the metrics, and the viewers and linked proof: agent and preflight runs,
their Inspect logs and the examine view command, and, domestically, a captured
Playwright hint. Gaps and limits comply with, then the run lock and the total
recorded end result, every collapsed; see
report layout. report on an agent
sub-run of a sealed run returns its sealed report unchanged.

Evidence information and exports conceal credentials, registered secrets and techniques, contact
particulars and different private fields equivalent to names and addresses. Model views preserve
the person’s personal knowledge seen: they conceal solely credentials and declared secrets and techniques,
plus browser password fields and, in a resumed browser state of affairs, textual content typed in
an earlier try. The supplier boundary refuses any mannequin request that also
comprises a registered credential. Export additionally applies the run’s personal non-public
values, that are by no means exported, and refuses (exit 2) when a staged file, or
a member of an Inspect log or different archive, nonetheless holds one. See
redaction.

diff compares two sealed runs, equivalent to floor, browser or connector runs. A
change to the bundle supply counts as a run-lock change, so evaluate runs made
from the identical checkout. It exits 0 when they’re
comparable with no consequence regression, and 1 when an consequence regressed: a worse
standing, or a passing test that now not passes or is lacking. It exits 3 when
they aren’t comparable, due to a run-lock change exterior --treatment or
an unsealed or unverifiable run. Generated IDs and repair timestamps are
normalized; run locks are in contrast precisely. --ignore POINTER (repeatable)
excludes a JSON pointer underneath /outcomes, /capabilities or /milestones,
and --treatment /server declares an intentional server construct change. See
replay and diff.

For extra element, see the documentation index,
connector lane,
lane extensions,
readiness checks,
consumer tests,
browser diagnostics,
measurement coverage,
model timing, and
release extensions, which incorporates growth historical past.

Scripted connector runs want no mannequin and no key, they usually replay the identical calls each time, in order that they
work as a pull-request test. connector-run can begin the service itself and write reviews a CI
system reads:

uv run consumer-agent-testing-tool connector-run my-connector.yaml 
  --driver scripted --script my-connector.calls.json 
  --start "python -m my_service --port {port}" 
  --junit reviews/connector.xml --summary-json reviews/connector.json

• --start COMMAND runs the service on a free loopback port. The command is break up like a shell
  command however not run in a shell.
  • {port} within the command is that port. The service additionally will get it as PORT and the state of affairs as
    ${PORT}.
  • The device waits till --ready-url, or the state of affairs’s first connector endpoint, solutions HTTP. It
    waits at most --ready-timeout seconds (default 30).
  • When the run ends, the service’s course of group will get SIGTERM, then SIGKILL after 5 seconds.
  • Its output goes to runs/service-.log, exterior the sealed run. That log is not redacted, so
    move secrets and techniques to the service by the atmosphere. The run lock data the command with
    credentials redacted.
  • A service that exits early or by no means turns into prepared exits 2.
• --junit FILE writes one check suite per trial and one check case per test. Blocked checks are
  . test --suite muse-guidelines accepts it too.
• --summary-json FILE writes the decision, exit code, checks, trials, move^okay and run paths. Its format
  is schemas/summary-v1.schema.json. --summary-json - prints solely
  that JSON on stdout and strikes the opposite strains to stderr.
• Both reviews are additionally written when the command stops with an error. See
  CI outputs and services.

The exit codes in Outcomes and exit codes resolve the step. A 3 (blocked
or unsupported) shouldn’t be a move.
examples/ci/connector-lane.yml is a GitHub Actions template with
two jobs. On pull requests, it runs the scripted calls and the readiness overview. Nightly or on demand,
it runs agent trials underneath every Muse-like remedy, and solely when a mannequin key secret is ready.
.github/workflows/ci.yml runs this repository’s exams and shipped
examples on macOS and Linux. Coding brokers ought to begin with AGENTS.md.
All matrix jobs feed the required CI test; Linux failures will not be allowed to move silently.
Workflow validation, tracked-file hygiene, secret scanning and schema freshness are checked too.

Plain pytest runs the supported assortment:

exams/conftest.py provides every session a short lived listing, dwelling and cache
underneath .cache/pytest/, removes them after a passing session, and removes the
floor and browser run directories its exams create underneath runs/. The full assortment takes about
twenty minutes and makes no community request past loopback and no mannequin
request. pytest --development-extensions additionally collects the experimental
exams, which can want different runtimes, instruments and exterior proof.
Async cleanup and unhandled thread exceptions fail exams as a substitute of remaining warnings.
See CONTRIBUTING.md for the overview workflow and
the launch checklist for the private-to-public transition.

The supported launch is local-consumer-v1: the connector lane and its
extensions, the readiness checks, shopper MCP apps, the restricted browser,
mapped HTTP APIs, transactional consent, continuation, proof and replay,
measurements, the distribution, and the OpenRouter and Meta Model API
adapters. Real-Muse calibration, cloud provisioning, native Glimmer, browser OS
isolation and reside Bedrock qualification are exterior it; see
release scope for the total boundary.

Release proof is generated per launch and isn’t dedicated.
docs/release-gates.json is the dedicated coverage:
the gates, the scope and the exclusions. The
release workflow runs on a v* tag. It data
each gate on macOS, requires the audit to move, and attaches three issues to
the GitHub launch:

• the wheel and the supply archive, with signed construct provenance;
• release-evidence-.tar.gz, which holds launch/release-gates.json
  (the recorded ledger) and launch/proof/.

Use consumer-agent-testing-tool release-audit to test launch readiness. It
reads launch/release-gates.json when proof has been recorded or extracted
there. Without it, it reads the dedicated coverage and reviews each gate
unverified. Exit code 0 means release-ready for local-consumer-v1; exit
code 3 means a minimum of one required gate wants consideration. The proof binds a
digest of the supply, so a supply change or an expired gate (seven days after
recording) can’t inherit an earlier passing end result. To audit a printed
launch, try its tag, extract its proof archive on the repository root
and run the audit earlier than the gates expire.

Day to day, you do not want any of this. Run the exams for what you modify, and
let CI run the total assortment on every pull
request.

A separate reside Spark Open check handed by Together with ZDR enabled. It
verified native device calls, continuation after a device end result, and the requested
routing coverage. It didn’t check streaming, independently confirm retention
practices, or set up compatibility with each shopper workflow. The Coverage
Cat checkout failure described above stays an open integration subject.

The audit requires 5 gates:

To report proof domestically as a substitute, for instance earlier than tagging, run the identical
steps because the workflow, on this order. verify_distribution.py regenerates the
dependency notices, that are a part of the supply fingerprint, so the
distribution comes earlier than the regression. The regression takes about twenty
minutes. Everything is written underneath launch/, which git ignores:

.venv/bin/python scripts/verify_distribution.py
.venv/bin/python scripts/record_distribution.py
.venv/bin/python scripts/run_regression.py launch/proof/regression-run.xml
.venv/bin/python scripts/record_regression.py launch/proof/regression-run.xml
.venv/bin/python scripts/record_release_tests.py launch/proof/regression-run.xml
consumer-agent-testing-tool release-audit

run_regression.py refuses an current report path, so take away launch/ or
choose a brand new title to report once more. It data workspace-relative paths and writes
the house listing as ~ within the log.

Distribution checks use caches within the workspace and the put in browser.
They want no Docker VM, cloud employee or mannequin credentials. If a dependency cache
is lacking, populate it individually with
scripts/verify_distribution.py --prepare-cache; that setup step makes use of the community.
The set up proof covers macOS solely.

The audit verifies supply fingerprints, artifact hashes, required checks and
expiry dates. It returns 0 when each required gate passes, or 3 in any other case.
Excluded capabilities stay labelled as excluded.
Qualification inputs explains what every gate wants.

The repository retains employee, isolation, local-model, calibration and
generated-OpenAPI experiments, and the CLI/ability lane for a developer’s personal CLI
binary (run --surface cli-skill), which wants Docker and an oracle that
returns the reservation fixture’s snapshot schema. The regular CLI rejects these
paths. To use them, put --development-extensions earlier than the subcommand and
set up the separate development-extensions dependency group and any required
tooling. These experiments are exterior the supported launch. The supported method
to provide an agent a CLI over an MCP server is
connector-run --projection muse-cli.

The non-obligatory Node API instruments have known dependency advisories.
Do not use them with untrusted inputs whereas these findings stay open.

The original specification and earlier check data doc
the mission’s historical past; they don’t outline the scope of this launch.

MIT; see LICENSE. The license doesn’t cowl the Coverage Cat and CATT names or
the brand; see TRADEMARKS.md. Third-party parts preserve their very own
licenses; the distribution ships their notices underneath third_party/.