rvbox/docs/protocol.md

RVBox v1 agent protocol

The authoritative schemas are ../protos/rvbox/v1/common.proto and ../protos/rvbox/v1/agent.proto. This document specifies their use over WSS.

Transport and compatibility

The nginx-terminated wss:// connection carries exactly one serialized rvbox.v1.AgentEnvelope in each binary WebSocket message. There is no extra length prefix. A decoded envelope may not exceed 1 MiB. A chunk's uncompressed payload may not exceed 64 KiB. Both peers validate declared and actual expanded sizes before allocation/decompression.

The protobuf package is rvbox.v1. Registration negotiates a major/minor protocol range: incompatible majors are rejected; the highest shared minor is chosen. New fields are append-only. A peer ignores unknown optional fields. A new required behavior requires a negotiated minor-version change; an envelope with no recognized payload is a PROTOCOL_ERROR rather than an implicit “required unknown field” mechanism that protobuf cannot represent.

Heartbeat and reconnect

Both endpoints use the same algorithm. Any received valid WebSocket frame is inbound activity. After 10 seconds without inbound activity, send a WebSocket Ping. After 30 seconds without inbound activity, close the session and treat it as dead. Pong processing is normal WebSocket behavior; it is not an application message and never queues behind command traffic.

The client reconnects with full-jitter exponential backoff (initial 1 second, cap 60 seconds). A session stable for 60 seconds resets the backoff. The server does not reconnect; it waits for clients. A reconnect always registers again, receives a new fencing generation, resends unacknowledged delivery/output, and reconciles only commands the server still considers non-terminal.

Session fencing

ClientHello starts registration and includes a random client_instance_id generated once in the client state directory and retained across daemon restarts and reconnects. ServerWelcome gives the selected version, random session_id, and session_generation. Except ClientHello, all envelopes carry those values. A new accepted registration from the same instance fences and disconnects its prior session. A different instance claiming the same live client ID is rejected and recorded as the current pending claim unless an operator explicitly authorized that exact instance. The one-shot authorization expires after 5 minutes by default and is consumed by the matching reconnect. When no session for that client ID is live, a new instance is accepted normally. This is collision protection, not peer authentication. The server accepts messages only from the current generation; command dispatches also identify their intended generation.

Reliable work flows

Dispatch and reconciliation

The server persistently creates a command before dispatching CommandDispatch. It redelivers until it receives CommandAccepted. Clients durably deduplicate on issue_uuid. A client whose queue is full sends a transient capacity rejection, which the server requeues with backoff. A permanent validation or unsupported-platform rejection makes the server command terminal rejected; it is not retried or mislabeled as a launched-process failure. An exact UUID/request-hash hit in the compact tombstone ledger returns CODE_ALREADY_EXECUTED; the server suppresses dispatch and reconciles its stale state instead of representing the prior execution as a new rejection.

Following registration the server sends ReconcileRequest containing its non-terminal commands for that client. The client replies once with a complete, bounded ReconcileSnapshot of every command it still retains, including terminal-but-unacknowledged records and matching requested tombstones, then continues normal retransmission from the server's last acknowledged event sequence. The server does not dispatch new work until this snapshot is complete. For healthy client storage, absence means the client never durably accepted the UUID: server queued/dispatched work returns to queued, while absence of an accepted/running record is an invariant failure and becomes interrupted. A matching UUID/request-hash tombstone always suppresses replay. A client-known server-missing active command, or a contradiction with server-confirmed terminal history, is terminated locally after the server returns a durable ReconcileResult and creates a recovery incident rather than inventing server state. That result also identifies client-retained terminal records the server has already stored or deliberately tombstoned, allowing the client to discard them even when its last EventAck was lost. The result is idempotent and is written before any new dispatch on that session. A client with an unresolved essential-store incident must not claim a complete snapshot or accept work until repaired/acknowledged.

Output and events

Client execution entries first have a durable local order. They receive an increasing wire event_seq durably when admitted to the bounded send window; once assigned, the sequence/content is pinned until acknowledged and retries reuse it exactly. The server durably writes an event before sending EventAck. EventAck is cumulative through a sequence number. Output is Zstandard compressed with an explicit original-size field. Server storage can reuse the validated compressed bytes.

When rolling output removes old retained segments, the server writes OutputTruncation metadata containing the removed event/byte ranges. Queries must show that marker rather than silently presenting an apparently complete stream. Assigned but unacknowledged events occupy the pinned 1 MiB per-command send window and are not evicted. An offline client that reaches a cap replaces one or more still-unsequenced output runs in its durable local order with OutputTruncation; when admitted to the send window the marker receives the next normal event_seq. Its event-range fields are absent because the discarded bytes never had wire sequences. The normal cumulative EventAck acknowledges the marker. Server-created retention markers include their removed event range and remain query metadata because the server cannot allocate client sequences.

If a capture pipe cannot be drained to a provably complete EOF, the client emits a sequenced OutputIncomplete event before the terminal lifecycle event. It does not invent a missing sequence or byte count for bytes it never observed.

Stdin and signals

StdinWrite is binary-safe and ordered by write_seq; the client durably deduplicates it and returns a sequenced stdin-acknowledgement command event. append_newline is true by default in the CLI but explicit on the wire. CloseStdin is a separate idempotent action. Signal and cancellation requests carry a command revision to settle start/kill races; the resulting lifecycle or signal-result event echoes that revision. Control-plane request_id values stay at the server and map to the assigned revision rather than crossing the agent protocol.

Scripts

For a script command, dispatch first contains a ScriptDescriptor; then the server sends ScriptChunk messages and a commit. The client checks offset, chunk order, full length, and SHA-256 before it reports upload complete or launches the process. ScriptUploadStatus.received_bytes is a cumulative durable progress acknowledgement: the server sends at most the command's unacknowledged window, waits for progress, and resumes from that offset. A retransmitted chunk is idempotent by offset/content.

Acceptance is durable queue admission, not proof that every later preparation step will succeed. A permanent script checksum/write error or failure to prepare the requested executable emits terminal rejected before any launch authorization. User cancellation in that interval emits cancelled; an uncertain crash window emits interrupted. None is mislabeled as process failed.

Flow control and failure containment

No receive loop runs an executor, database write, decompressor, or slow socket operation inline. Each side has bounded staging queues. Durable spools are the source of truth and are charged to the tiered unified command-storage budgets described in the architecture document. Senders keep a bounded unacknowledged window of encoded wire bytes (defaults: 1 MiB per command and 8 MiB per client session); unsent data waits durably. WebSocket Ping/Pong/Close plus fencing and protocol errors use a reserved priority lane and a dedicated writer with bounded data-frame size and write deadlines. A full essential ingress queue does not stall the sole WebSocket reader indefinitely: the receiver closes without acknowledgement and the durable sender retries after jittered reconnect. Droppable output uses the sequenced overload-loss path instead.

Before compression, raw output uses the architecture's high/low watermarks. Once a client high watermark is crossed, still-unsequenced droppable chunks are summarized in durable local order instead of consuming unbounded compression work. A server at its ingress high watermark does not acknowledge or discard an already sequenced event; it closes without acknowledgement if its bounded queue cannot admit the frame, and the client retries from durable spool. Fair scheduling prevents one verbose command from monopolizing workers. Reserved metadata capacity remains available to close each gap and emit the final lifecycle event.

Malformed protobuf, over-size payload, invalid compressed data, impossible sequence, bad session token, or protocol-version violation yields a structured error where safe and closes that WebSocket session. It does not crash either daemon. Network loss is normal and is handled through idempotent resend.