When the machine boots however the reply disappears · Mainbrella
You ask a server to create a Linux machine. The machine boots. The reply disappears. From the consumer’s chair, this appears very like a request that by no means reached the server. Retrying is affordable. Starting a second machine would make the restoration dearer than the failure.
This is a helpful means into Mainbrella’s backend. Follow one machine far sufficient and the identical drawback retains returning: a command can run with out its caller seeing the outcome; a non-public request can outlive a community membership; a disk seize can succeed with out leaving us the deal with wanted to revive it. Each boundary wants a choice about what a retry is allowed to do.
Let’s comply with an illustrative job that runs evaluation.py, reads information from one other container, and writes metrics.csv. Its machine occupies slot c17. The filenames and slot are examples; the mechanisms under come from the open-source backend. We’ll begin earlier than Linux boots, as a result of that’s the place possession and funds should be determined.
One place decides who will get the final slot
Two create requests can arrive whereas an account has one slot left. Reading a counter, booting a machine, and updating the counter afterward provides each requests an opportunity to win. By the time we discover, the costly half has already occurred.
We give every account a Cloudflare Durable Object: a persistent coordinator with its personal storage. The public API resolves the authenticated consumer and paid entitlement, then addresses that coordinator as account:. Each machine slot has a separate Durable Object. For c17, its identify is consumer:. The first slot retains the older identify consumer: and the general public ID small; that ID doesn’t choose the machine’s dimension.
The account coordinator serializes admission. A request should reserve its slot, month-to-month begin, and compute allowance earlier than it will possibly ask the runtime besides. The subsequent request subsequently sees occupied capability even whereas the primary machine remains to be beginning. The account controller implements the queue explicitly with a promise tail; an await inside an admission resolution doesn’t let the subsequent resolution slip previous it.
Holding this lock till each machine was prepared would additionally serialize all their boot instances. We launch it after the sturdy reservation and provision exterior it. One account makes the selections so as; its runtimes can boot collectively.
There’s a very helpful test for this distinction. It sends six concurrent requests on a Builder account, which has 5 slots, and holds each runtime’s readiness verify behind a gate. Five machines attain boot earlier than that gate opens. The sixth request receives a unrest, and the account information 5 begins. It checks each halves of the design: unique admission and parallel provisioning.
Ownership is established earlier than this coordination begins. The authentication adapter accepts an API key or a login session. An explicitly invalid Bearer credential fails authentication even when a legitimate browser cookie accompanies it. The internal request builder provides the account id and entitlement itself. Letting the caller select an x-mainbrella-user header would undo the account boundary we simply constructed.
A receipt exists earlier than the machine does
For our evaluation.py job, the consumer provides an Idempotency-Key with POST /containers. That key means “this specific try and create a machine.” The account information which slot and reservation belong to it, along with a fingerprint of the requested configuration. Changing the picture, dimension, or different fingerprinted choice whereas reusing the important thing produces a unrest.
The essential write is small. Here is the related a part of container-account-core.js, with the encompassing validation omitted:
const reservationId = ++state.subsequentReservationId;
state.reservations[slot] = reservationId;
const creation = {
id: crypto.randomUUID(), slot, reservationId, fingerprint,
expiresAt: this.now() + CREATION_RETENTION_MS,
};
await this.ctx.storage.put({
[KEY]: state,
[CREATION_PREFIX + idempotencyKey]: creation,
});That multi-key write commits the charged reservation and creation receipt atomically. We don’t need a receipt for a slot we by no means reserved, or a charged slot {that a} keyed retry can’t discover. Only afterward does the controller dispatch the boot.
Now lose the reply. An identical retry finds the receipt earlier than making an attempt contemporary admission. While the reservation is pending, it will possibly return beginning. Pending reservations have a ninety-second reconciliation window; after that, the coordinator asks the runtime what truly exists. If it finds the operating machine, it returns that machine with out charging one other begin. The retry by no means dispatches the ambiguous reservation a second time.
The receipt lasts twenty-four hours. If the machine has stopped or its slot has been reused, that very same retained key returns creation_no_longer_running. It doesn’t quietly launch a substitute. A consumer that desires a substitute makes a brand new creation try with a brand new key. This can be why the consumer ought to save its key and request earlier than sending them: server-side deduplication is little assist if the consumer forgets the id it must ask about.
What establishes readiness? The runtime controller begins the chosen picture with sleep infinity as its entrypoint, then executes uname -a. The command should exit efficiently inside sixty seconds. This establishes that the visitor can execute a command. Our Python evaluation nonetheless wants its personal dependencies and checks; an answering kernel can’t certify a monetary calculation.
Yesterday’s message can arrive at tomorrow’s machine
Suppose the dispatch for our first boot is delayed. Meanwhile, the account cancels the reservation, releases c17, and assigns that slot to a substitute. The outdated dispatch lastly arrives. Its goal remains to be the identical runtime object. Looking up the slot by identify can’t inform us whether or not this message is entitled to begin something.
Each admitted begin receives an growing reservation quantity. At the runtime, we persist two high-water marks: the most recent accepted begin and the most recent cancellation. A boot at or under both mark is rejected. Cancellation writes its fence even when there isn’t a operating visitor to destroy, so a later arrival can’t resurrect canceled work.
The reverse race issues too. A delayed cleanup for reservation 41 should not cease the machine from reservation 42. The runtime’s DELETE path advances the cancellation mark however leaves the visitor alone when the cancellation is older than the accepted begin. Back on the account, the completion of a boot should nonetheless match the present slot reservation earlier than it will possibly clear pending state. An outdated profitable reply has no authority over a substitute both.
Reservation numbers shield the interior lifecycle messages. Public instructions and file requests establish a operating era with { id, createdAt }. The slot ID is reusable; the era will not be. Despite its timestamp-shaped illustration, createdAt is computed as max(now, previousCreatedAt + 1). Recreating a machine in the identical millisecond, or shifting the wall clock backward, nonetheless provides the brand new visitor a special id.
Keep each values from the returned operating container. A cleanup request for simply c17 can’t specific which lifetime you imply. The lifecycle assessments intentionally cease and reuse a slot earlier than releasing an outdated boot dispatch, together with with out advancing the clock. This is a extra revealing verify than one other profitable hello-world launch.
The onerous deadline is a part of admission
Our machine additionally wants permission to maintain spending compute. A month-to-month counter checked solely at launch would let many simultaneous machines devour the identical remaining allowance. We reserve the runtime they might use earlier than any of them begins.
Machine sizes have weights in plan-policy.js: Lite makes use of one compute unit, Medium makes use of ten, and XL makes use of twenty-eight. The account reserves unit-milliseconds. Its lease ends on the earliest of 4 boundaries:
onerous deadline = min(
begin time + plan session restrict,
paid entry expiration,
subsequent UTC month boundary,
begin time + remaining unit-ms / machine weight
)For an arithmetic instance, reserving a Medium machine for one hour commits ten compute-unit-hours. Confirm a cease after 5 minutes and the consumed quantity is 10 × 5 / 60, about 0.833 compute-unit-hours; the unused reservation is launched. A failed cease or unreadable runtime retains its reservation. Treating “couldn’t contact it” as “it have to be free” would permit the account to spend that allowance twice. A failed admitted launch nonetheless consumes its month-to-month begin; runtime settlement is a separate calculation.
The runtime persists this tough deadline and an idle deadline. Real exercise can transfer the idle deadline, capped by the onerous deadline. Status polling doesn’t. A Durable Object alarm enforces expiration even when the consumer has gone away. Plan adjustments can shorten an present lifetime, however can’t prolong its unique onerous deadline.
Stopping work ought to stay potential when the billing lookup is unavailable. The public DELETE path authenticates possession with out requiring a contemporary billing decision; the coordinator makes use of its saved entitlement for cleanup when it stays legitimate. Likewise, a more recent unpaid commentary beats an older paid one. Otherwise a delayed verify might reauthorize a machine we had already revoked.
A disconnected viewer shouldn’t personal a course of
With a operating era in hand, we are able to begin evaluation.py. A brief foreground command has a sixty-second most. For work whose outcome we need to discover after a disconnect, we use a managed execution. Its identifier belongs to a specific container era, and its creation requires an idempotency key of its personal.
// machine is the precise { id, createdAt } returned for a operating visitor.
// Persist executionKey and this request earlier than sending them.
const question = new URLSearchParams(machine);
const response = await fetch(`${apiOrigin}/containers/executions?${question}`, {
technique: 'POST',
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'software/json',
'Idempotency-Key': executionKey,
},
physique: JSON.stringify({
argv: ['python3', '/workspace/analysis.py'],
timeoutMs: 120_000,
}),
});
if (!response.okay) throw new Error(`Execution HTTP ${response.standing}`);
const execution = await response.json(); // Save execution.id.The argv type provides literal arguments as a substitute of assembling shell textual content. In executions.js, the execution file is saved earlier than the method begins. An identical retry finds that retained id. An aborted creation request or a disconnected occasion stream doesn’t purchase the correct to kill the managed job; specific cancellation does.
Output occasions have growing sequence numbers, dedicated alongside the file’s up to date cursor. A consumer reconnects to the occasions endpoint with its final processed sequence as cursor. It reads the saved suffix quite than relying on a specific socket having seen each byte. The stream itself is bounded to thirty seconds, so reconnecting is odd operation. The course of can run for as much as fifteen minutes, additional restricted by its requested timeout and the container’s remaining onerous lease.
These information are finite assets. Each runtime retains as much as thirty-two execution information, with a one-hour retention deadline measured from job admission. Managed jobs share a pool of 4 concurrent operations with foreground instructions and file transfers. Output is bounded to 1 MiB and a finite occasion depend. Hitting the output restrict terminates the job and marks it truncated; discovering a number of believable strains in stdout is subsequently inadequate. We examine terminal standing, exit code, timeout, and truncation earlier than trusting the outcome.
Here is the uncomfortable boundary: sturdy information don’t make course of handles sturdy. When a runtime object restarts, restoration marks unfinished executions interrupted. If they belong to its present visitor era, it destroys that visitor quite than depart untracked work operating. It by no means silently reruns the command. For our evaluation this implies an interruption can discard an unsaved CSV; for a command that sends invoices, automated replay might do one thing worse. Restart restoration is intentionally extra disruptive than reconnecting a viewer.
The information request arrives with an id the visitor can’t select
Suppose evaluation.py reads a shard from http://data.internal/shards/west. We register its era as a member of a non-public service community. In that very same community, one other era owns the service identify information and port 8080. The supply could also be a caller-only member, with no listening port.
The visitor’s request names neither an account nor a community. private-services-runtime.js installs an outbound HTTP interceptor for *.inner. Its relay removes reserved id headers and provides trusted account, slot, and era values from the runtime entrypoint’s configured properties. Sending an invented possession header from Python doesn’t choose a special buyer.
The account’s private service registry finds the community containing that precise supply era, then resolves information inside it. Another account—or one other community on this account—can reuse the identify. The vacation spot rechecks its operating era and registration below its lifecycle lock earlier than opening the registered software port. Resolving the identify is simply the primary permission verify.
Why verify once more? Our information service could possibly be indifferent or changed whereas it’s answering. The vacation spot buffers the bounded response and checks its registration once more. The account rechecks supply liveness and each memberships earlier than releasing the response. A request admitted below yesterday’s membership should not ship bytes below at the moment’s association.
Those checks don’t undo an software facet impact that already occurred. If the request modified the information service earlier than its response was denied, the appliance nonetheless wants a approach to reconcile that change. Immutable shard reads are straightforward right here; a job queue or fee service would want its personal operation identities.
This characteristic is bounded personal HTTP routing: one-MiB request and response our bodies, a ten-second timeout, and no WebSocket improve or arbitrary TCP connection. A PostgreSQL consumer received’t change into private-service-aware as a result of its hostname ends in .inner. Deployment enablement and the potential marketed by GET /capabilities should even be current earlier than we construct a workflow round it.
The CSV wants a life exterior the command’s output
Our job writes /workspace/metrics.csv. We retrieve it with a generation-bound file request whereas the visitor remains to be operating. The file endpoint transfers uncooked bytes, with a one-MiB restrict, quite than decoding binary information as textual content or hiding an export inside a truncated stdout stream.
The write path in files.js demonstrates one other helpful ordering selection. It writes a brief file within the vacation spot’s present mother or father listing, then renames it over the goal. Readers shouldn’t observe a half-uploaded common file. The path is handed as a positional argument, not interpolated into shell code. Writes reject symlink targets; reads could comply with hyperlinks contained in the owned visitor.
These ensures are narrower than a transaction over the evaluation. A profitable file switch doesn’t show the job used the correct enter or completed all its rows. The software ought to verify the artifact’s schema and provenance, and replica helpful output exterior the ephemeral machine earlier than teardown. If we need to return to the surroundings itself, we’d like a saved workspace.
A snapshot can exist with out being recoverable
Saving seems like one operation: seize the disk, keep in mind the outcome, cease the machine. It truly crosses three homeowners of state—the account, the runtime, and the supplier—and none of them can atomically commit the opposite two.
In workspaces.js, the account reserves save capability and persists the operation earlier than requesting seize. At the runtime, a receipt containing the save id is continued earlier than calling snapshotContainer(). After seize returns, the runtime saves the supplier deal with in that receipt. The account then commits the deal with to its workspace file. Only after that commit could cease: true destroy the supply.
Lose the reply between the runtime and account after the runtime has saved the deal with, and a retry can learn the receipt. We recuperate the identical seize with out taking one other one. But interrupt the runtime after the supplier accepts seize and earlier than its deal with is saved, and the receipt comprises solely an intent. We know we tried; we don’t know which supplier object to revive.
The runtime refuses to recapture that unresolved operation and returns workspace_save_unavailable. The save path leaves the supply intact, topic to its odd lease. Choosing a brand new key simply to make the error go away can be a brand new seize try, not restoration of the outdated one. That distinction is the restrict of the idempotency promise.
Save quotas account for the price of uncertainty too. Admission reserves the supply dimension’s full disk capability earlier than seize. Deleting a workspace frees its dwell saved-workspace quota, however doesn’t refund the historic seize funds: the supplier could have already got accomplished the work. A failed or ambiguous name can’t change into an affordable approach to repeat captures indefinitely.
Restoring a prepared workspace goes by means of odd container admission and consumes a brand new begin. The restored visitor receives a contemporary era and should use the saved dimension and web coverage. The picture digest should nonetheless match; an incompatible picture or failed supplier restore produces an error quite than silently substituting an empty machine.
The saved state is a filesystem. RAM, operating processes, previews, and private-service memberships don’t return with it. Our Python job wants progress in recordsdata whether it is to renew, and a restored service should begin its course of and register its new era. Quiescing writers earlier than seize is an software accountability too. We can protect a disk filled with mutually inconsistent recordsdata completely.
Make the late message arrive
The quickest approach to examine these decisions is to run the lifecycle assessments from a backend checkout. They delay dispatch, drop replies after negative effects, reconstruct controllers from saved storage, and reuse slots earlier than delivering the outdated messages:
node --test containers/container-account.take a look at.mjs
containers/user-container.take a look at.mjs
containers/workspaces.take a look at.mjs
containers/private-services.take a look at.mjsStart with stopping and reusing a slot fences outdated delayed dispatches even in the identical millisecond. Then learn misplaced snapshot response reconciles receipt with out recapture beside unsure seize can't be repeated in the workspace tests. The first recovers a saved reply. The second preserves the truth that a solution is lacking. That is the choice an agent wants from its compute backend earlier than it will possibly safely resolve what to do subsequent.
Source evaluate: backend commit 6e5bef3, October 7, 2026. Examples and diagrams clarify management move; they aren’t manufacturing traces or latency measurements. The lifecycle assessments use simulated supplier state. Public request shapes and deployment capabilities are documented in API.md and the agent workflow.
