Cross-host work and recovery

A run lives on one host, but its workers do not have to. This page covers what changes when a worker runs somewhere else, and what the engine does on its own when a coordinator, a worker, or the whole process disappears.

Placing a worker on another host

glueprint host list returns the server-authoritative inventory: each host’s id, label, platform, online status, engine version, and whether it accepts delegated workers. Pass one of those ids as --on <HOST_ID> to the session and worker commands that support it (worker start, worker show, worker stop, and session start, send, steer, interrupt, status, stop). From MCP, the same choice is host_id on gp_start_session or gp_create_session. To find a project there, call gp_list_projects (or gp_list_worktrees) with the same host_id. A session’s project_id may also be a project key such as repo:github.com/owner/repo, or your own host’s project id, which selects the project with the same repository key on the target. A project_not_found refusal lists the projects the target does have.

The request is server-mediated: your local engine forwards it over the relay to the target host, which spawns or controls the session and answers back. Naming your own host is identical to omitting --on. Project ids and working directories belong to the target host, not to the machine you typed on.

Two things never cross the wire:

The operator bearer. The CLI authenticates only to the local runtime. A forwarded request carries the local engine’s identity, never your credential.
The run. Runs, tasks, gates, and dispatch rows stay on the coordinator’s host. The target host only knows it is running a delegated child with a run id and task id attached.

Cross-host controls follow the same plan gate as Glueprint’s other cross-host features. A host on a plan without relay access gets PLAN_UPGRADE_REQUIRED.

Inline subagents and remote viewing

An inline subagent created with gp_spawn_subagent always runs on its parent’s host. Here, “local” means local to the parent session, not to the device you are viewing it from. If your parent session runs on a server and you open it from a laptop, the portal, or your phone, its subagents also run on that server.

With full cloud relay access, you can expand their cards to read activity and respond to child questions and permission requests remotely. On desktop and portal, Open shows the child’s full chat in a sheet; on mobile it opens the child’s session detail page. Restricted status-and-decision relay access does not include inline transcripts, and clients without the parent metadata may show children as ordinary session rows.

Tool	Where the child runs	Presentation	Spawn approval
`gp_spawn_subagent`	Same host as the parent	Inline card in the parent’s transcript	Auto-approved
`gp_create_session` (human-initiated caller)	Local host or the requested `host_id`	Full session with its own workspace tab	Required
`gp_start_session` (assistant caller)	Local host or the requested `host_id`	Full session with its own workspace tab	Required

For gp_spawn_subagent, omit host_id or pass the parent’s own host id. Another host returns remote_subagent_not_supported without creating a child. Cross-host inline spawning is not supported yet; use the full-session tool for your caller when work must run on another machine. Remote viewing and control of same-host subagents is supported today.

What is refused remotely, and why

Operation	Behavior
`session list`, `session events`, `worker read`, `worker retry` with `--on`	Rejected by the CLI before any request is made, exit 2. Listings and event pages read the local store, and a retry must preserve local attempt lineage; the engine does not forward them.
A worker for an operator-owned run with `--on` another host	Refused by the engine with `operator_run_remote_unsupported`. Only a session-owned run can place workers remotely, because a remote child reports to its coordinator session. Start the worker locally or create the run with `--owner`.
Inline subagent on another host	`gp_spawn_subagent` refuses a foreign `host_id` with `remote_subagent_not_supported`; use the full-session creation tool for your caller.
Retry of a remote attempt	Not supported; `retry_of_session_id` is local only.
Drafts on a remote host	A remote session must have a prompt; a session created without one is not started, and remote creation does not support that.

Failures the engine identifies as remote — a host that is unknown, offline, too old to accept delegated children, plan-gated, not authorized, timed out, or a target-side refusal — keep their engine error code and exit 7. For those, the JSON envelope carries "remote": true inside error.details, and an unsupported target also reports its host_version there. One refusal is special-cased: operator_run_remote_unsupported arrives as a plain error with no details, and the CLI classifies it as remote by its code alone, so it also exits 7. Every other failure keeps the exit code it would have had locally, so a script can tell “the other machine said no” from “my request was wrong” without parsing messages.

Report-derived status versus live status

For a local worker, the coordinator’s view of a dispatch is live. The engine derives liveness from the process and the session’s activity: active, idle, waiting (with the count of pending approvals or questions), paused, terminal, or stalled with a reason (no_output, idle_unreported, launch_timeout, process_gone). Nothing in the worker has to send a heartbeat.

For a remote worker, the coordinator’s view comes from reports. The dispatch row shows the last delegation status the worker reported and its settlement outcome; liveness reads as unknown, because the coordinator’s engine cannot observe a process on another machine. worker show --on and session status --on ask the target host for the session’s current status when you want a live answer.

From MCP, gp_get_session_status with host_id asks the target host for the session’s live status, its latest delegation report, and any pending permission prompts or questions; when the target cannot answer, it shows the latest inbox report and why. gp_get_session_events with host_id reads a size-bounded page of the session’s transcript from its host. Both find the host on their own when this host has no copy of the session. A target on an older Glueprint answers these reads, and project keys, with target_feature_unsupported: update Glueprint on that host.

Reports are delivered durably. The worker’s host writes each gp_set_delegation_status report to a persistent outbox before sending it, then retries delivery immediately, after 10 seconds, after 60 seconds, and every 30 seconds after that until the coordinator’s host acknowledges it. A relay outage delays a report; it does not lose one. Report sequence numbers keep the order intact, and a report from a superseded attempt is delivered but cannot complete the replacement.

Crash recovery

The engine reconciles every run when it starts and again every 30 seconds. Each pass repairs, in this order:

Abandoned coordinators. If the coordinator session has been archived, the run closes as abandoned with that reason, its open gates are withdrawn and their question cards cancelled, and the remote children recorded on the archived session are finalized so the target host stops expecting a parent. If the coordinator’s record no longer exists at all, the run still closes as abandoned and its gates are withdrawn, but the list of remote registrations lived in that record, so no remote-child finalization is possible.
Lost local workers. For each current local dispatch still marked dispatched, the engine checks the child session. A child whose record is gone settles the dispatch and its task as failed, with an abandoned outcome and the summary “child work item missing”. A child that already carries a terminal report the store never applied is settled from that report. A child whose process has ended in a terminal lifecycle state (completed, failed, stopped, or interrupted) without reporting is finalized as an unreported child, so the dispatch and task settle instead of hanging. A child that is stopped because it has not started yet — a worker whose first turn is still launching, or one created without a first message — is left alone for the 10-minute launch window, and a worker being launched holds its session’s start reservation, so reconciliation never settles a worker that is about to start. Remote dispatches are not probed here; they settle from delivered reports.
Missed readiness notifications. A task that became ready while nothing was listening gets its task_ready record re-queued.
Stuck runs, notified once. A run is stuck when it is open, nothing is ready, nothing is dispatched, and at least one pending task is blocked by a chain that includes a failed or cancelled task. The engine stamps when the episode began and delivers exactly one dag_stuck record for it. If the run later recovers and gets stuck again, that is a new episode and a new record. An open gate on its own never counts as stuck.
Auto-close. A run created with auto-close closes as soon as every task is terminal: completed when all completed, failed when any failed or was cancelled. Closing withdraws open gates. It never stops live workers.

The same reconciliation also watches ordinary delegated children that stop reporting progress. A local worker that produces no output, sits idle without reporting, or never finishes launching within the engine’s thresholds (10 minutes each) is flagged stalled and the coordinator is notified with one inbox record per stall episode. The child is not finalized; it keeps its process and its dispatch, and the coordinator decides whether to retry it (a retry stops a still-live stalled prior by default) or leave it alone. If the retry’s replacement then fails to start, retry from the replacement; see When a retry’s replacement does not start. Only a child whose process is gone enters the unreported-child finalization described in step 2.

Reading runs also reconciles the coordinator’s delegated children first, so a run show right after a restart reflects worker outcomes that settled while nothing was reading.

Retention of closed runs

Closed runs are kept for 30 days from the time they closed, and never more than the 100 most recently closed. Anything older or beyond that count is removed when the engine loads its run catalog at startup. Open runs are never pruned. Inbox records are independent of run retention: unacknowledged records stay until acknowledged.

`session stop --offline`

glueprint session stop <ID> --offline is a last-resort repair for a session whose saved status says it is still running after the runtime itself has gone away — a crashed daemon, a machine that lost power. It rewrites the saved status on disk and nothing else:

It does not terminate any process. Before using it, shut the runtime down and check for orphaned agent processes yourself.
It refuses to run while any listener is reachable, and also when the probe is inconclusive, because a live runtime owns that state. Use a plain session stop in that case.
It cannot be combined with --on, --wait, or --timeout, and it never applies to a session on another host.

When the runtime is simply not running and you have not passed --offline, session stop exits 3 with a message pointing you at the flag.