# RVBox v1 Go implementation plan ## 1. Objective and implementation boundary Implement the v1 contracts in the existing design documents as three Go binaries: - `rvbox-server`: durable server, WebSocket agent endpoint, local gRPC control endpoint, optional JSON-RPC adapter, storage, and audit owner. - `rvbox`: reverse-connecting client daemon, command supervisor, durable active spool, and reconnect/reconciliation owner. - `rvc`: local CLI over the server's Unix-domain gRPC socket. The primary release target is a Linux server and Linux client. The Go client is structured behind OS interfaces from the outset; Windows implementations of process management, diagnostics, and resource controls are completed before a Windows client is declared supported. Windows v1 requires Windows 10 or Windows Server 2016 or newer. Do not claim Windows feature parity while those implementations are absent. This plan implements the already agreed v1 contract. In particular, it does not add authentication, client enrollment, mutual TLS, or a client allowlist. The deployment trust boundary remains nginx TLS termination and restricted network access. The unauthenticated JSON-RPC listener remains disabled by default and loopback-bound when enabled. ## 2. Development rules and definition of done ### 2.1 Local-environment rules `../ENV.md` is binding for this repository: - Do not install Go, Buf, `protoc`, SQLite tooling, or other heavy development dependencies on the host. - Put the toolchain, code generation, unit/integration testing, linting, and local services in Docker/Docker Compose. - Run container commands as UID/GID `1001:1001` so generated Go code, module caches mounted into the project, and test artefacts remain host-owned. - Do not delete Docker volumes, generated artifacts, or state directories as a convenience cleanup action. Tests use dedicated temporary volumes/directories and remove only the exact resources they created. ### 2.2 Completion standard for every phase A phase is complete only when all of the following are true: 1. The public behavior and failure modes are covered by focused tests. 2. The package has context cancellation, bounded channels/queues, and no network or disk operation on the WebSocket receive loop. 3. Structured logs and metrics exist for the new state transitions/failures. 4. Configuration defaults, validation, and errors are documented. 5. `docker compose run --rm toolchain make fmt lint test` passes, with no host installation required. The lint target explicitly snapshots the accepted first-draft Buf enum-prefix/service-suffix findings and fails on any other or newly added finding; application/code lint has no such exception. 6. Any recovery or retention code is tested with a real temporary SQLite database and filesystem rather than mocks alone. ## 3. Repository and build bootstrap (Phase 0) ### 3.1 Establish the repository layout Create the following layout. Generated code is kept separate from handwritten logic and is never manually edited. ```text cmd/ rvbox-server/main.go rvbox/main.go rvc/main.go internal/ agentproto/ # envelope validation, transport-neutral helpers config/ # file/flag parsing and cross-field validation domain/ # command state machine and typed errors server/ control/ # gRPC implementation and JSON-RPC adapter session/ # client registry, fencing, dispatch store/ # SQLite metadata, segments, audit, migrations client/ runtime/ # reconnect loop, transport, dispatcher spool/ # active command/event/output durable spool supervisor/ # OS-neutral interface supervisor/unix/ # process groups, /proc, cgroup v2 supervisor/windows/# Job Objects and Windows diagnostics observability/ # logging, metrics, health/readiness testkit/ # clocks, fake transport, fault helpers gen/go/rvbox/v1/ # generated protobuf/grpc code protos/rvbox/v1/ # existing wire authority docs/examples/ # fully annotated server/client TOML deploy/ Dockerfile.toolchain Dockerfile.runtime compose.yaml nginx/ systemd/ ``` Use a single Go module rooted at the repository. Select and pin exact Go module versions in `go.mod`; record reasons for non-standard dependencies in `docs/dependency-decisions.md`. Prefer a pure-Go SQLite driver to avoid a C toolchain for normal Linux/Windows client builds. Use a maintained, context-aware WebSocket implementation and a Zstandard implementation that supports bounded decompression. Do not rely on an archived WebSocket package. ### 3.2 Containerized developer tooling Add: - `deploy/Dockerfile.toolchain`: pinned Go base image plus Buf, `protoc`, `protoc-gen-go`, `protoc-gen-go-grpc`, format/lint tools, and `make`. - `deploy/Dockerfile.runtime`: minimal non-root image for server/client smoke tests; it contains only built binaries and runtime CA/config assets. - `deploy/compose.yaml`: a `toolchain` service with the repository mounted at `/workspace`, run as `1001:1001`; optional `server`, `client`, `nginx`, and fault-injection test services use isolated named volumes. - `Makefile`: `generate`, `fmt`, `lint`, `test`, `test-race`, `test-integration`, `build`, and `verify`. Each target invokes the container workflow; it must not silently fall back to host tools. - `buf.gen.yaml`: Go protobuf and gRPC generation paths under `gen/go`. Generation is deterministic: `make generate` followed by `git diff --exit-code` must be clean in CI. Update `buf.yaml`/`buf.gen.yaml` only with a matching generation run. ### 3.3 Initial quality gates Add CI (or a repository script ready for CI) that runs, in order: 1. `buf format --diff` and `buf lint`. Snapshot only the already accepted enum-prefix/service-suffix findings so CI still fails if their set changes or another warning appears. This pre-implementation cleanup defines the first v1 compatibility baseline; enable `buf breaking` against that baseline immediately after it merges, not against the obsolete draft. 2. Generation freshness check. 3. `go fmt`, `go vet`, static analysis, and unit tests. 4. Race tests for server/client concurrency packages. 5. Linux integration tests in Compose. 6. Cross-compilation/build verification for Windows client packages; Windows runtime tests run on a Windows runner once available. **Exit criteria:** all three empty `main` packages build in the toolchain container; generated code is checked in; `make verify` works from a fresh clone. ## 4. Shared contracts, configuration, and state machine (Phase 1) ### 4.1 Preserve proto authority Generate `common.proto`, `agent.proto`, and `control.proto` before writing transport code. Do not fork request/response structs by hand. These schemas are an unshipped first draft, so remove obsolete fields and compact their tags now rather than preserving compatibility holes. Once Phase 0 generated APIs merge, freeze field meanings: later additions use new tags, enum zero remains unspecified, and removed tags are then reserved normally. At the application boundary, validate what protobuf cannot express: - `client_id` is configured hostname text, 1–128 ASCII characters. - command and mutation IDs are canonical RFC 9562 UUIDv7 strings; all public UUID parsing rejects empty, non-canonical, or wrong-version input. - decoded envelopes are <= 1 MiB; output/stdin/script data validates every raw, compressed, and declared size before allocation or decompression. - a serialized `ExecutionSpec` is <= 768 KiB, decoded control gRPC requests are <= 16 MiB, JSON-RPC HTTP bodies are <= 24 MiB, and decoded field limits remain identical across the two control transports. - shell type is explicit or assigned only to the documented platform default. - CWD exists, is a directory, and is allowed by local daemon policy. - an execution request has exactly one source: command text or script descriptor. - script descriptors and payloads agree on SHA-256 and <= 10 MiB size. - user-supplied script filenames are display-only basenames and never path components; command text and scripts use generated private wrapper paths. - profile flags are deduplicated and checked for configured supported limits; `LIGHT` is exclusive, otherwise accept at most one CPU, one memory, and one disk tier. ### 4.2 Domain package Implement a pure-Go `internal/domain` package with no database/network imports. It contains: - the allowed command transition table and terminal-state predicate; - `CommandRevision` compare-and-transition helpers for launch/cancel/signal races; - typed domain errors mapped to gRPC status/structured details, JSON-RPC error objects, or agent-protocol `ControlError` as appropriate; - event-sequence validation, duplicate equivalence checks, and declared gap validation through `OutputTruncation` metadata; - default values and hard-limit validation; - stable ordering/pagination token encoders. Model `launch_prepared` and `launch_authorized` as durable internal execution phases without exposing them as additional public lifecycle enum values. No recovery path may redispatch a UUID after `launch_authorized`; an uncertain outcome becomes `interrupted`. The normal legal state transitions are: ```text queued -> dispatched -> accepted -> running -> succeeded|failed|terminated|interrupted queued -> expired dispatched -> rejected (permanent pre-acceptance failure) dispatched -> queued (transient rejection with backoff) accepted -> rejected (permanent upload/preparation failure before launch) queued|dispatched|accepted -> cancelled (subject to revisioned race resolution) dispatched|accepted -> interrupted (reconciliation/recovery invariant failure) ``` An attempted repeat event is harmless only if it has identical immutable content. A conflicting duplicate UUID/sequence is a protocol error and fences the bad session rather than rewriting history. ### 4.3 Configuration model Use strict TOML 1.0 and implement the normative contract in [`configuration.md`](configuration.md). Keep the annotated [`examples/server.toml`](examples/server.toml) and [`examples/client.toml`](examples/client.toml) synchronized with the Go config structs and compiled defaults. Implement configuration in this order: 1. Define separate decode-only TOML structs and immutable validated domain structs. Every public TOML key has an explicit tag and documentation comment; packages outside `internal/config` receive only a validated subsection. 2. Decode exactly one UTF-8 file with unknown-field and duplicate-key rejection. Do not merge include files, expand environment variables, coerce strings to numbers, or accept bare numeric durations. 3. Apply precedence deterministically: compiled defaults, file values, then flags that were explicitly present. Generate flag bindings from one key registry so omitted flags cannot accidentally overwrite TOML values. 4. Parse duration strings with overflow and non-negative checks. Parse byte/count integers without floating point. Enforce v1 hard ceilings even when TOML asks for more; operators may lower them but not create an incompatible wire peer. 5. Canonicalize data/state/socket/CA/shell/CWD-root paths to absolute paths. Reject aliases that collapse distinct server data subdirectories and a control socket outside its intended runtime-directory policy. Check existing path ownership/type without claiming that private modes encrypt stored data. 6. Resolve and validate current-platform shells at client startup. Each nonempty applicable path must be absolute and identify an executable regular file; other-platform strings are syntax-checked but never advertised. Advertise only validated entries and require the current platform default to be one of them. Cache the canonical path and never re-resolve through a command's overridden `PATH`. Revalidate identity before launch and reject rather than fall back if the executable changed incompatibly. 7. Validate resource profiles by dimension. `LIGHT` is exclusive; otherwise allow at most one `CPU_*`, one `MEM_*`, and one `DISK_*`. Each enabled profile must give nonzero values for its `required_controls`; validate Linux device keys as canonical `major:minor` values and Windows Job rate support before advertising the profile. 8. Enforce cross-field invariants: low watermark below high, send window below durable tier, closeout reserve below command quota, command quota below both aggregate tiers, reconnect initial <= cap, heartbeat idle < liveness timeout, positive page/queue limits, per-client server queue <= global server queue, and zero only where the schema documents disable or indefinite behavior. 9. Print the normalized effective non-command configuration once after validation. Log a conspicuous warning for enabled non-loopback JSON-RPC, but retain the accepted behavior. Do not log TLS file contents or profile device discovery details that reveal unrelated host paths. 10. Bind no command listener and launch no child until static configuration is valid. Storage recovery remains asynchronous after this static gate as specified in Phase 2. Server TOML covers routing/listeners, queue/retry bounds, unified storage and audit limits, group-commit behavior, flow windows/watermarks, protocol ceilings, takeover TTL, and observability. Client TOML covers WSS/TLS routing, durable identity/state, exact shell paths, allowed CWD roots, queue/concurrency, reconnect/liveness, spool/flow limits, execution diagnostics/grace periods, resource profiles, and observability. Do not add or claim application-level at-rest encryption in v1. Command-owned payloads are stored in plaintext under the private state directories; document the equivalent sensitivity of backups and defer encryption/key management. Hard defaults are: 10-second heartbeat-idle interval, 30-second liveness timeout, 1–60-second full-jitter reconnect, reset after 60 stable seconds, 16 running/100 queued client commands, 1,000 server-queued commands per client and 10,000 server-wide, 5-minute one-shot live-conflict takeover grant, 15-minute queue TTL, 64 KiB raw stream chunk, 1 MiB decoded agent envelope, 768 KiB serialized execution spec, 16 MiB decoded control request, 24 MiB JSON-RPC HTTP body, 10 MiB output window/raw script cap, 32 MiB total per command, 256 MiB per client/client daemon, 4 GiB server-wide command storage, 30-day terminal retention, 100 MiB audit storage with quota-only rotation by default (zero age retention), one million compact command tombstones, raw-output high/low watermarks of 1 MiB/256 KiB per command, 8 MiB/4 MiB per client, and 64 MiB/32 MiB server-wide, plus unacknowledged send windows of 1 MiB per command and 8 MiB per session. Reserve 64 KiB within every accepted command's quota for terminal/loss closeout metadata and bound protocol detail/reason plus incident-note text to 4 KiB. Use emergency filesystem free-space floors of 256 MiB server-side and 64 MiB client-side by default. Test both annotated example files through the production decoder. Add table tests for every unknown key, invalid duration/size, high/low inversion, tier inversion, zero semantic, noncanonical path/device, unsupported default shell, conflicting profile combination, and flag-precedence case. Add a test proving a request-level `PATH` override cannot change the chosen shell executable. Golden-test the redacted effective configuration and keep its key order stable enough for operators to compare deployments. **Exit criteria:** state-machine and configuration tests cover all legal/illegal transitions and boundary values; both examples parse to the documented defaults; generated protos compile and are the only DTOs crossing process boundaries. ## 5. Server persistence and retention (Phase 2) ### 5.1 Storage ownership and directory safety Implement `internal/server/store` as the sole writer for server state. On first boot, create a private data directory (mode `0700`), SQLite database, segment directory, and audit directory. Validate that paths resolve below the configured data directory; never construct a path directly from unvalidated client IDs or filenames. Use encoded UUID/client-ID path components. Open SQLite with WAL enabled, foreign keys enabled, a busy timeout, and an explicit durable-write policy. Run migrations transactionally. On startup: 1. acquire a single-instance lock; 2. bind liveness and diagnostic/control endpoints with readiness false; 3. run integrity/schema checks and committed-range recovery asynchronously; 4. safely truncate file bytes beyond SQLite's `committed_end_offset`, while a missing/corrupt committed range creates a scoped durable incident; 5. mark prior live sessions disconnected and retain trustworthy non-terminal commands for reconciliation; interrupt only affected commands whose essential state cannot be trusted; 6. enable each healthy scope independently. Mutations against an unrecovered or dirty scope return `UNAVAILABLE`; liveness, incident inspection, and unaffected scopes remain available throughout best-effort recovery. Represent recovery with explicit atomic scope states: `recovering`, `ready`, `dirty_readable`, and `unavailable`. Liveness depends only on the process/event loop; global readiness is true only when every required scope is ready, while an RPC checks the narrower scopes it will touch. The WebSocket endpoint may accept and close with a retryable storage error during recovery, but it must not issue a durable `ServerWelcome`, accept events, or dispatch commands until the relevant client/session scope is ready. Run `PRAGMA quick_check`, migration checksum validation, and segment reconciliation on a bounded recovery pool. Persist a startup incident in the database when it is writable; otherwise expose an in-memory bootstrap incident through health and require the offline repair tool. Never loop forever on one corrupt client or segment. Record progress counters and the last failing path/offset without putting payload bytes in logs. ### 5.2 Metadata schema and indexes Use migrations to create at least the following tables; names may vary but their invariants must not. | Data | Required fields/invariants | | --- | --- | | `clients` | client ID, most-recent platform/capabilities/CWD/version, durable instance ID, current generation, connection/last-seen timestamps, latest rejected live-conflict instance/time, unified charged command-storage total | | `sessions` | opaque session ID, client ID, durable client-instance UUID, generation, opened/fenced/closed times, close reason | | `commands` | UUID, client ID, indexed issue/queue-expiry/terminal times, lifecycle, revision, exit result, last event sequence, retention status, and a Zstandard-compressed immutable execution-spec payload including plaintext environment values | | `command_payloads` | command UUID, payload kind (script or other command-owned blob), Zstandard compression, raw/stored sizes, digest, inline bytes or validated segment reference | | `command_events` | command UUID + event sequence unique key, observed and server receipt times, event type, payload metadata, immutable duplicate checksum | | `output_segments` | command UUID, segment ordinal/path, `committed_end_offset`, min/max event sequence, stream mix, compressed/raw byte totals, checksum, created time | | `output_truncations` | command UUID, removed event range/byte totals, source, reason, recorded time; non-overlapping ranges | | `stdin_writes` | command UUID + write sequence unique key, compressed payload/raw/stored sizes/checksum, newline/close intent, acknowledged state | | `control_mutations` | request UUIDv7 unique key, owner command/incident/client, method/target, immutable request hash, assigned write sequence/revision and stable result; conflicting reuse is rejected | | `takeover_authorizations` | client ID + exact pending instance ID, creation/expiry/consumption times and authorizing request ID; one live, one-shot grant per client | | `command_tombstones` | binary UUIDv7 unique key, immutable request hash, client ID, completion/acknowledgement time; compact global FIFO capped at 1,000,000 rows | | `audit_events` | timestamp, source transport/principal where available, client/command IDs, action, outcome, error code, command text/script digest/env key names | | `storage_incidents` | incident UUID, detected/resolved times, state, affected scope/client/command, summary, repairability and data-loss flags; unresolved rows derive dirty health | | `schema_migrations` | applied version/checksum/time | Index client command lists by `(client_id, issue_time DESC)`, active command dispatch by `(client_id, lifecycle, issue_time)`, and output reads by `(command_uuid, event_seq)`. Use cursor tokens based on the sorted key plus a signature/version, not SQL offsets, so page results remain stable under writes. Add foreign keys with explicit delete behavior and database constraints for terminal timestamps, nonnegative byte counters, unique `(command,event_seq)` and `(command,write_seq)`, one active session per client, and one unconsumed takeover grant per client. Store UUIDs and SHA-256 values as fixed-size blobs internally; canonical strings exist only at protocol/log boundaries. Canonical request hashes use deterministic protobuf encoding of immutable fields after defaults and map ordering are normalized; mutable lifecycle/receipt fields are excluded. Every mutation transaction updates its owner record, all three applicable quota counters, the idempotency row, and audit intent/result consistently. Identical `request_id` plus hash returns the stored result without re-running side effects; same ID with a different method, owner, or hash returns `CONFLICT`. Add migration invariant queries that recompute charged totals and fail readiness if persisted counters disagree until repaired. ### 5.3 Segment format and atomic append Store large command, output, and audit payloads outside SQLite. An active segment is append-only; seal it at the configured 256 KiB target and never append again. Use an encoded UUID/ordinal filename created with exclusive create. Each record contains fixed magic/version/header length/record length, command and event identity, observed/receipt timestamps, payload kind/stream/compression, raw and stored lengths, SHA-256 payload digest, header CRC, payload, and trailing record CRC. All lengths are unsigned and validated against configured maxima before allocation or seeking. The append path is: 1. validate compressed bytes and bounded decompression; 2. choose/append the active command segment; 3. append and `fsync` (possibly as a configured group commit); 4. in one SQLite transaction, insert event/segment metadata, advance that segment's `committed_end_offset`, update command sequence/counts, and record receipt time; 5. only after durable success, return the cumulative `EventAck`. The writer serializes appends per active segment but group-commits independent commands together. Creating/renaming a segment also syncs its containing directory before metadata can commit. A group-commit timer is a maximum batching delay, never permission to acknowledge before file sync. SQLite uses a documented synchronous mode sufficient for the selected durability promise; tests assert the actual PRAGMA values after opening every connection. On recovery, file bytes beyond the committed offset are unacknowledged tail and are truncated. A short file or checksum failure inside the committed range is never “repaired” by silently moving the database offset backward: mark the affected output unavailable/truncated, preserve evidence, and create an incident. Essential-state corruption interrupts only the affected command and gates unsafe mutations. Apply this committed-offset protocol to command blob and audit segments too. For a missing committed output range, preserve all still-valid records, create query-visible truncation/incomplete metadata, and never fabricate byte totals. For a missing immutable execution spec, request hash, revision, or active stdin state, mark the command essential state untrustworthy and interrupt it. Recovery must be idempotent after a crash at every repair step. If database or segment persistence fails, do not acknowledge the event. Surface the server health failure, stop dispatching work as appropriate, and keep the session/control loops responsive. ### 5.4 Retention transaction Implement retention in a single serialized maintenance worker, never on the WebSocket receive loop. Define a versioned deterministic charge formula: compressed/encoded payload and segment lengths plus a conservative fixed charge for each SQLite row/index entry. Do not attempt to attribute shared SQLite pages after the fact. Reconcile the logical counters transactionally and also reject allocations below the configured 256 MiB server filesystem free-space floor, regardless of logical quota headroom. Implement one server `Reserve(owner, kind, chargedBytes)` path used by command creation, script append, stdin, event append, and audit. Reuse the same contract in the client spool, where it also covers raw execution-file preparation. Under the store writer lock/transaction it checks, in order: hard field maximum, per-command total and closeout reserve, per-client total, server/client-daemon total, and filesystem floor. It either reserves all applicable tiers or none. Release uses the same stored charge version; a future estimator change requires a migration, not a silent reinterpretation of old rows. 1. Account all stored command-owned data, including request/script payload, pending stdin, metadata/events, and output. The client additionally charges generated raw execution files while they exist. Reserve non-evictable active state before accepting it, including 64 KiB per-command closeout headroom and the bounded result for each accepted stdin/signal mutation; reject an allocation that cannot fit. 2. Enforce the command's 10 MiB output window and 32 MiB total cap by rotating oldest output segments and recording `OutputTruncation`; never roll essential active state. 3. Enforce the 256 MiB per-client and 4 GiB server-wide caps by evicting terminal commands as whole UUID records in oldest server issue-time/UUIDv7 order. If active data alone creates pressure, rotate/drop output with markers and reject new essential state. 4. Reclaim every whole terminal command 30 days after authoritative terminal time by default, independently of byte pressure; zero explicitly disables age rotation. Preserve compact tombstones and separate audit/incidents. 5. Maintain audit events in compressed segments under their independent 100 MiB default, rotating complete oldest segments. Apply an independently configurable audit age as a second trigger; zero, the default, disables age rotation but not byte-budget rotation. 6. Keep unresolved storage incidents non-evictable. Resolved incident summaries may rotate with their audit history under the separate 100 MiB budget, but resolving dirty health never immediately rewrites or disguises recorded loss. Retention selects candidates deterministically in `(server_issue_time, issue_uuid)` order and rechecks terminal state plus generation inside the delete transaction. Before deleting a command, close follower snapshots over its retained ranges, ensure the compact tombstone exists, mark `evicting`, and move all owned files to a same-filesystem deletion directory using collision-proof names. A crash-safe sweeper rolls forward rows/files in either order. Never use a glob or client-provided path for deletion. For active pressure, first evict server-retained output and record exact `RetentionTruncation`; then reject new unreserved mutations. Do not evict the assigned client send window or essential lifecycle/revision/dedupe state. Age rotation and byte-pressure eviction call the same whole-command primitive so their crash behavior cannot diverge. Deletion is recoverable: mark rows `evicting` transactionally, move segment files to a same-filesystem tombstone location, commit metadata deletion, then remove tombstones asynchronously. Startup completes or rolls forward stale evictions deterministically. Replay tombstones must never have a delete/insert gap. The server inserts the command tombstone transactionally with authoritative terminal state before it can acknowledge or later evict command history. On the client, receipt of an acknowledgement through the terminal event atomically inserts or retains the UUID/request-hash tombstone and removes the full command record. Enforce each one-million-entry FIFO cap in that same transaction. ### 5.5 Incident resolution Run safe, deterministic repairs automatically and through the control API. `RepairStorageIncident` must refuse any action that would knowingly discard committed data. `AcknowledgeStorageIncident` requires an operator note and is the explicit path for accepting irrecoverable loss. Both are idempotent by `request_id`. A repaired or acknowledged incident no longer contributes to the derived dirty flag; resolution itself does not delete history. Bound notes to 4 KiB and retain resolved summaries/audit entries under their separate 100 MiB rotation. Provide the same store library behind offline `rvbox-server repair --data-dir` list/repair/acknowledge modes. Provide equivalent local client-spool handling through `rvbox repair --state-dir` and its local health diagnostics. Never open a live store from the offline tool; acquisition of the same single-instance lock is required. Define stable incident kinds for uncommitted tail, missing committed bytes, checksum mismatch, counter mismatch, SQLite integrity failure, failed eviction, permission error, and disk exhaustion. An incident has immutable evidence plus mutable state `OPEN -> REPAIRED|ACKNOWLEDGED`; terminal states never reopen, so a recurrence creates a new incident ID. Scope keys identify global store, client, command, segment, or audit without embedding raw filesystem paths in the public API. Safe repair may truncate only bytes beyond `committed_end_offset`, rebuild a derivable counter/index, or roll forward a marked eviction. Anything that would discard acknowledged bytes or immutable essential state requires acknowledgement instead. Online repair acquires the same per-scope maintenance lock as recovery and retention; offline repair additionally requires the process-wide instance lock. Persist the repair/acknowledgement mutation and audit record before clearing derived dirty health. ### 5.6 Store tests Test real SQLite/filesystem cases: duplicate event idempotence; conflicting duplicate rejection; crash between segment write and metadata transaction; crash after metadata commit; corrupted final record; command-window rotation; per-client/global whole-command eviction; active-command protection; cursor stability; uncommitted-tail truncation; short/corrupt committed ranges; dirty derivation and repair/acknowledgement; and disk-full/permission failure simulation. Verify recovery endpoints become live promptly while unsafe scoped mutations remain gated. **Exit criteria:** a store restart preserves acknowledged data and never acknowledges uncommitted data; all rolling, tiered unified-quota, 30-day terminal-age, audit, and tombstone rules match the documented semantics. ## 6. Server sessions, WSS endpoint, and dispatch (Phase 3) ### 6.1 Agent endpoint Expose an HTTP endpoint suitable for nginx WebSocket proxying. TLS may terminate at nginx, but the server must verify WebSocket upgrade origin/path/configuration and impose read limits itself. It accepts one binary protobuf envelope per message and refuses text messages, fragmentation misuse, oversized frames, invalid compressed payloads, and malformed/unknown mandatory messages. Use four independently bounded activities per session: - a read loop that validates/fragments/decode-dispatches only; - a prioritized write loop with reserved Ping/Pong/Close/fencing/error capacity, bounded data frames, and write deadlines; - a dispatch worker that reads persisted command work; - a persistence/event worker that writes through `store`. No goroutine may hold a session-registry lock while doing database, filesystem, compression, or WebSocket I/O. Every goroutine receives a session context and must terminate on fencing/close. Keep at most 1 MiB unacknowledged per command and 8 MiB per client session; additional data waits in durable storage. If an essential ingress queue fills, close without acknowledging rather than blocking the sole reader; the durable sender retries after jittered reconnect. Ping/Pong remains sendable even while the data window is full. Implement the write scheduler as two bounded lanes owned by one socket writer: the reserved control lane contains Ping/Pong/Close, welcome/fencing, protocol errors, acknowledgements, and reconciliation control; the data lane contains dispatch, script, stdin/signal, and events. Always drain a bounded burst of control before one data frame so data still progresses. Impose write deadlines and split data below the frame ceiling. A full control lane is a session-fatal internal invariant; a full data lane leaves work durable and wakes the writer later rather than allocating another goroutine. The reader checks WebSocket type/size before protobuf decode, validates envelope session fields before payload allocation, and routes only lightweight immutable work descriptors. Persistence workers preserve per-command ordering while a fair round-robin/deficit scheduler prevents a verbose command from monopolizing decompression or disk. Unit-test queue capacities and goroutine shutdown with a fake connection that never completes writes. Apply the server 64 MiB/32 MiB ingress backlog as admission hysteresis, not as a license to discard client-sequenced events. Above high, stop acknowledging new output and close the producing session if the bounded descriptor queue cannot accept its next frame; the client's durable spool retries it. Resume normal admission only below low. Keep lifecycle/control reserve independent so another client's heartbeats and command closeout remain responsive. ### 6.2 Registration and fencing On `ClientHello`, validate client ID/capabilities/version and the durable random client-instance UUID; choose the highest shared minor under major v1; persist the registration; allocate a cryptographic random opaque session ID and increment the client's generation transactionally. Fence the prior session for the same client ID and instance before making the new one active. Accept a different instance normally if no session for that client ID is live. Otherwise record it as the pending claim and reject it unless an unexpired one-shot grant matches that exact instance; consume the grant transactionally when accepting the reconnect and fencing the old session. The default grant TTL is 5 minutes. Send `ServerWelcome` with the outer envelope's canonical session fields. Persist a rejected live-conflict claim before closing the candidate socket and expose only its exact `client_instance_id` and observation time through `GetClient`. `AuthorizeClientTakeover` verifies that this ID still matches the pending claim, stores an idempotent UUIDv7 mutation and five-minute expiry, and does not fence immediately. On matching reconnect, one transaction increments generation, consumes the grant, closes the old session record, and installs the new session; only then may the writer send welcome. A mismatched/expired grant is never broadened to “the next instance.” If no session is live, accept a different instance without creating or consuming a grant, as agreed. All post-hello envelopes must match both session ID and generation. Stale or unknown session traffic is discarded/audited and its WebSocket is closed. A dispatch includes the intended generation, and the client must reject a mismatch. Unknown client IDs are accepted by design; hostname is display/routing identity, not an authentication assertion. ### 6.3 Heartbeat and liveness Implement the identical policy at both ends: any received valid WebSocket frame updates inbound activity; after 10 seconds silent send Ping; after 30 seconds without inbound frame close the session. Pings/Pongs use WebSocket controls, not protobuf envelopes. Make timers injectable/fakeable for deterministic tests. Use a monotonic clock for elapsed activity/backoff and wall-clock UTC only for persisted observation. Pong/any valid inbound frame updates activity in the read loop without waiting for a database worker. Reset reconnect backoff only after a continuous 60-second stable session. Add boundary tests at exactly 10 and 30 seconds, delayed writer tests proving Ping bypasses a full data lane, and clock jump tests proving wall-clock adjustment cannot spuriously kill a session. ### 6.4 Queueing, dispatch, and reconciliation `RunCommand` persists a `queued` command before returning. The dispatcher sends to the active session while either an immediate running slot or a client queue slot is available: `running < max_running || queued < max_queued`. Dispatch and capacity updates are serialized per session so concurrent sends cannot oversubscribe the advertised slots. It updates to `dispatched` before write and retries after reconnect until a matching `CommandAccepted` arrives. Maintain per-session shadow reservations for in-flight dispatches: reserve a running slot when immediately available, otherwise a queued slot, before enqueuing the frame. Reconcile the shadow with each `ClientCapacity` and `CommandAccepted`; release it on definite rejection/session close. The dispatcher uses one serialized worker per client plus a fair global ready queue, so the `OR` capacity condition cannot oversubscribe and an offline/noisy client cannot starve others. Apply the configurable queue TTL (15 minutes by default; zero means indefinite) until server-confirmed acceptance. A never-dispatched command becomes terminal `expired`. A dispatched command with uncertain acceptance remains protected and is rendered expired pending reconciliation rather than being prematurely made terminal. Late client evidence advances the actual lifecycle, creates a late-after-expiry incident, and triggers best-effort termination while retaining subsequent actual outcome events. Persist an absolute server-clock `queue_expiry_time` when creating the command; never recompute it after configuration changes or retries. Drive expiry from a database-backed ordered index and recheck lifecycle/revision transactionally. Queued commands become `EXPIRED`; dispatched commands set a derived late-pending-reconciliation indicator without writing a false terminal event. On late acceptance, persist the incident and higher termination revision before sending best-effort TERM, then retain every actual subsequent event. Classify a negative `CommandAccepted` before changing lifecycle. Explicitly transient capacity/storage pressure returns `dispatched -> queued` with bounded jittered backoff and remains under the original TTL. Invalid execution data or unsupported required platform controls become terminal `rejected` with the structured reason persisted on `CommandRecord`. Reserve `failed` for a process that reached `running`. On replacement/reconnect, query non-terminal commands for that client and send a `ReconcileRequest` with last durable event sequence/revision plus immutable request hash. Require one complete client snapshot of all locally retained commands, including terminal-but-unacknowledged records and matching requested tombstones, and compare the sets idempotently before enabling dispatch. With healthy client storage, an absent `queued`/`dispatched` command returns to `queued`; an absent `accepted`/`running` command is interrupted as an invariant failure. A matching UUID/request-hash tombstone suppresses replay. The client terminates a local active command absent from the server target set only after receiving the server's durable `ReconcileResult`; either peer records a contradiction with server-confirmed terminal state as a storage/recovery incident, and the server does not invent missing state. The result also lists terminal local records safe to discard when the server has stored or deliberately tombstoned them, covering a lost final `EventAck` followed by server retention. Write this idempotent result before dispatch. A client with unresolved essential-store corruption cannot complete reconciliation or accept work. Implement reconciliation as this explicit matrix: | Server state | Client evidence | Durable result | | --- | --- | --- | | queued/dispatched | absent from healthy complete snapshot | return/remain queued; redelivery permitted | | dispatched/accepted/running | matching retained record | keep highest valid revision and resume event/script delivery | | any non-terminal | matching tombstone | interrupt/suppress replay and record stale-server incident | | accepted/running | absent | interrupt and record client-state-loss incident | | missing or contradictory terminal | client active | `ReconcileResult.terminate_local_issue_uuids` plus incident; invent no server command | | missing/tombstoned or fully stored terminal | client terminal | `discard_local_terminal_issue_uuids` | Validate matching UUID, immutable request hash, revision monotonicity, terminal immutability, and event-sequence bounds for every row before applying any row. Persist all server state/incident decisions in one reconciliation transaction, then send one deterministic sorted `ReconcileResult` through the control lane. The client durably records terminate/discard intent before reporting capacity. If the result frame is lost, the next complete snapshot yields the same result; dispatch is disabled until the current result has been written on the session. Handle pre-start `kill` locally only for work that has never been dispatched. For dispatched or accepted work, persist a higher revision and send the revisioned signal while retaining the existing visible lifecycle until the client resolves the race. Accept a revisioned `cancelled` lifecycle if launch was not authorized; otherwise retain the revisioned signal result and actual terminal outcome. The server maps control `request_id` to this revision/result and never sends that request ID across the agent protocol. ### 6.5 Server session tests Cover version negotiation, same-instance replacement, different-instance pending claim, exact one-shot takeover consumption/expiry, stale-generation late output, Ping/Pong inactivity thresholds, full client queue, dispatch retry after lost acceptance, complete active/terminal-unacknowledged reconciliation, lost/repeated `ReconcileResult`, permanent/transient dispatch rejection, cancellation before launch, launch/cancel race, malformed envelope close, and a slow client that cannot block another client or local control RPC. **Exit criteria:** multiple simulated clients can register, replace each other, receive bounded dispatches, and recover connection faults without duplicate execution or server deadlock. ## 7. Client runtime, spool, and Unix supervisor (Phase 4) ### 7.1 Client state and reconnect runtime Keep client state private (mode `0700`): durable accepted-command records, active process metadata, command event journal, stdout/stderr spool segments, stdin write acknowledgements, and outstanding script upload state. It contains no terminal command history after server acknowledgement and local cleanup. It retains a separate compact FIFO of the most recent 1,000,000 command tombstones (binary UUIDv7, immutable request hash, and acknowledgement time) so stale server state cannot replay recently completed work after full history is removed. The runtime has a long-lived supervisor/dispatcher and replaceable network session. Generate the random client-instance UUID once in the private state directory and retain it across daemon restarts and reconnects. Network loss must not stop running commands or pipe readers. The connection loop uses full-jitter exponential backoff from 1 second to 60 seconds, resetting only after 60 stable seconds. On each connection it sends a hello with the durable client-instance UUID, waits for welcome, replays retained sequenced loss events and other unacknowledged events in order, answers reconciliation, and resumes dispatch. All client queues are bounded. The disk spool, not an in-memory channel, is the source of truth. Give stdout, stderr, status, signal results, and stdin acknowledgements one durable command-local order. Assign and persist wire `event_seq` only when an entry enters the bounded send window; an assigned entry is pinned until acknowledgement and immutable on retry. Create `client_instance_id` with a cryptographic UUID generator, write and fsync a temporary file, atomically rename it, and sync the state directory before first registration. Never regenerate it merely because the network/server rejects a session. If the identity file is corrupt, enter dirty local health and require the repair path rather than silently presenting a new installation. Give the client store the same committed-offset segment discipline as the server. Persist, at minimum, command phase/revision/hash, platform launch identity, local event ordinal, optional assigned wire sequence, payload digest, stdin write state, script received ranges, and last server acknowledgement. Maintain separate monotonic `local_ordinal` and `event_seq` columns: insertion assigns only local order; admission to the send window transactionally assigns the next contiguous wire sequence and pins the row. Cumulative `EventAck` atomically unpins/deletes acknowledged payload and updates the command cursor. Structure reconnect as explicit states: `backoff -> connecting -> hello -> reconciling -> active -> closing`. Only pipe/supervisor workers outlive the replaceable network session. After welcome, send the complete snapshot, apply `ReconcileResult` durably, replay assigned rows first, then assign/send local rows in order. Advertise capacity and accept new dispatch only in `active`. Cancellation of one session context must join all its readers/writers before a new session can use their queues. ### 7.2 Output capture and offline caps Create non-blocking readers for stdout and stderr immediately after process start. Read raw bytes irrespective of newline boundaries, divide them into <= 64 KiB raw chunks, add observed UTC time/local order, Zstandard-compress, append to the command spool, and enqueue only a durable reference to the sender. Use one reader per pipe and a bounded shared raw-chunk scheduler. A reader never waits for network I/O. Under normal load it transfers ownership of a bounded buffer to a fair compressor, which returns buffers to a pool only after durable append. Validate Zstandard frame size/checksum on readback before replay. Do not log raw chunks on compression, checksum, or storage failure. Bound work before compression as well as on disk. Defaults are high/low raw backlogs of 1 MiB/256 KiB per command and 8 MiB/4 MiB for the client daemon; the server compression/persistence tier independently uses 64 MiB/32 MiB globally with the non-dropping admission behavior in Phase 3. Crossing a client high watermark enters loss mode until the matching low watermark. Aggregate discarded, still-unsequenced chunks into local-order/raw-byte loss records and replace them with a `CLIENT_OVERLOAD` truncation marker; event-range and compressed-byte counts are absent when sequencing/compression never occurred. Use fair worker scheduling and reserved metadata capacity so lifecycle, stdin/signal acknowledgements, gap markers, and terminal state remain lossless. Loss mode is hysteretic and scoped: crossing either command or daemon high watermark marks affected raw chunks as dropped until both relevant backlogs fall below their lows. Coalesce only adjacent dropped local ordinals with identical cause/stream set; close a loss run before inserting any essential event. Persist known raw-byte totals and observation interval in local metadata, then emit one bounded `OutputTruncation`; omit event-range/compressed-byte fields if they were never assigned/compressed. If even reserved marker persistence fails, mark the command/store dirty and interrupt rather than silently claiming complete output. Respect the local hard limits at all times: - 10 MiB compressed rolling output window and 32 MiB total per command; - 256 MiB aggregate command-owned storage per client daemon. When connected, discard server-acknowledged oldest segments first. Assigned but unacknowledged entries are pinned and bounded by the 1 MiB per-command send window. When offline or an acknowledgement lags, rotate only still-unsequenced output as needed and replace each removed run in durable local order with `OutputTruncation` using `CLIENT_SPOOL`. The marker later receives a normal wire sequence and is retained until cumulative `EventAck`; no wire sequence is fabricated or skipped. The server's accepted truncation event is the auditably visible explanation for the lost bytes. Continue draining child pipes after caps are reached. A verbose child may lose old or newly produced output, but it must not block the client daemon or deadlock itself. Test a continuously verbose process, alternating stdout/stderr, essential events interleaved with dropped output, reconnect during loss mode, ack loss at every send-window boundary, and terminal exit while above the high watermark. Assert contiguous assigned sequences, exact known raw-byte totals, bounded heap/queue size, and a terminal event after every marker/incomplete event. ### 7.3 Command acceptance and idempotency Persist an accepted dispatch record keyed by `issue_uuid` before responding `CommandAccepted`. Repeat dispatch with identical immutable fields returns the same acceptance/revision and does not start another child; a conflicting repeat is a protocol error. Enforce 16 running/100 queued defaults before acceptance. After full terminal cleanup, also check the compact tombstone ledger: the same UUID/hash is rejected as already executed, while changed immutable content is a conflict. Make acceptance a single durable state transition: 1. Validate the session generation, UUIDv7 syntax, immutable request hash, command revision, shell/profile/CWD policy, script descriptor, and expiry. 2. Look up both the active-command table and tombstone ledger before reserving capacity. An exact active duplicate returns its existing result; an exact tombstone returns `CODE_ALREADY_EXECUTED`; either hash mismatch is a permanent conflict. The server treats the former as stale-state reconciliation/interruption, not as a newly rejected execution. 3. Reserve the running/queued slot and worst-case command metadata allocation in the same local-store transaction that inserts the accepted command. Do not count retransmission of an existing record twice. 4. Commit and fsync the accepted record, immutable hash, revision, script cursors, and capacity counters before sending positive acceptance. A failure before commit returns a transient rejection and leaves no partial command; loss of the response is resolved by exact replay. 5. Classify policy/schema/hash/unsupported-profile failures as permanent and local I/O/temporary-capacity failures as transient. Include a bounded, operator-readable reason without command or environment contents. For script dispatch, persist descriptor and chunks at validated offsets, verify each chunk checksum and final SHA-256/length before launch, and keep the upload restart-safe until terminal cleanup. Create the final temporary file with owner-only permissions under the effective CWD using an atomic write/rename; delete it after the terminal event is durable locally. Emit cumulative durable `ScriptUploadStatus.received_bytes` progress often enough to advance the 1 MiB server send window; the server never streams the full 10 MiB without progress acknowledgement. Accept a chunk only when its offset equals the durable contiguous prefix, or when the entire byte range is an exact replay of already persisted bytes. Reject holes, changed overlaps, arithmetic overflow, data past the declared length, and chunks received after commit. Append and fsync bytes before advancing `received_bytes`; after the final prefix, verify length and SHA-256, reserve the raw-file quota, atomically materialize the wrapper, sync its parent directory, and only then make the command launch-eligible. A permanent upload failure durably ends the command as `REJECTED` before launch authorization and releases every associated reservation. Materialize ordinary `command_text` through the same private generated-file machinery, while retaining its separate command-text metadata. Execute the file with exactly `sh`/`bash`, `cmd.exe /D /S /C`, or `powershell.exe` with `-NoLogo -NoProfile -NonInteractive -File` according to `shell_type`; never infer or fallback to another shell. Build Windows application name/arguments with the platform quoting routine rather than shell string concatenation. Treat `$0`, `%0`, or equivalent exposing the generated wrapper name as documented behavior. Resolve each shell setting once during configuration validation to a canonical absolute executable path. Launch that exact path with an explicit argument vector/application name; never search the daemon's `PATH`, honor a command's environment overrides during resolution, follow a wrapper shebang, or select by file extension. Re-stat the configured executable immediately before launch and reject if it is no longer the validated regular executable. Add tests with an attacker-controlled `PATH` entry and same-named fake shell to prove it cannot be selected. ### 7.4 Unix process supervisor Define a narrow interface used by the runtime: ```go type Supervisor interface { Start(context.Context, StartSpec) (Process, error) Signal(context.Context, Process, SignalKind) (SignalOutcome, error) Snapshot(context.Context, Process) (ResourceSnapshot, error) StopAll(context.Context) error } ``` The Unix implementation starts an RVBox launcher as leader of a new session/process group; after authorization that launcher creates the selected `sh`/`bash` child inside the same group and remains as its watchdog. It rejects unsupported shell values. It applies daemon environment plus persisted overrides, validates the CWD, connects stdin/stdout/stderr pipes, and records launcher/root/group identities. Signal the full process group. On orderly shutdown and unclean-start recovery, terminate surviving managed groups and emit `interrupted` rather than pretending pipe monitoring survived. Treat root exit as the start of a configurable 5-second tree/output drain grace period. Wait for the cgroup/process group and capture readers; terminate residual descendants after the grace period, drain to EOF, and emit incomplete-output metadata as a sequenced `OutputIncomplete` event if handles still cannot be drained. The terminal lifecycle event must be sequenced and spooled only after all retained output/truncation events and is always the final client event. Implement launch through an internal blocked-launcher mode with a private release/watchdog channel. The launcher remains alive as a non-user-code supervisor for the process-tree lifetime. After the launcher reports ready, persist and flush its PID/process-group/platform birth identity as `launch_prepared`; persist and flush `launch_authorized` before releasing requested command code. EOF before release aborts without executing it. On Linux create a per-command cgroup v2 for supervision regardless of resource-profile selection, place the blocked launcher into it before release (`clone3` with `CLONE_INTO_CGROUP` and `CLONE_PIDFD` where available), and persist the cgroup path plus `/proc//stat` start time. Recovery prefers `cgroup.kill`; a process-group fallback is allowed only after positive birth-identity verification. Other Unix platforms use the watchdog/process-group fallback and document that descendants which deliberately create a new session may escape it. Encode launch as `accepted -> launch_prepared -> launch_authorized -> running` and permit no shortcut: 1. Create the private cgroup/process group, pipes, wrapper, and close-on-exec release/watchdog channel without executing user code. 2. Start the blocked launcher and obtain a positive ready message containing the platform process identity; place and verify it in its command cgroup. 3. Commit and fsync `launch_prepared` with that identity. Recheck the latest revision and pending cancellation while the launcher remains blocked. 4. If still executable, commit and fsync `launch_authorized`, then send the one-byte release and keep the watchdog channel open until tree cleanup. Record `running` only after the launcher reports successful requested-shell creation/`exec`. 5. A crash before durable preparation cleans an untrusted orphan and may retry. A prepared-but-unauthorized launcher must exit on channel EOF and may retry only after positive death verification. After release, channel EOF makes the launcher terminate its process group; an authorized but uncertain launch is killed as a tree and ends `interrupted`, never launched again. Make launch-journal and launcher-control records checksum-framed and bounded. Fault-inject process death before and after every fsync, ready/release, and exec acknowledgement; assert that user-visible side effects occur at most once and that recovery cannot confuse a reused PID. Implement the portable Unix set `HUP`, `INT`, `TERM`, `KILL`, `USR1`, and `USR2`, accepting optional `SIG` prefixes in the CLI and mapping only through the `SignalKind` enum. Reject arbitrary native numbers and unsupported names. Never allow signal zero or arbitrary PID targeting. The process's group ID comes only from durable supervisor metadata, never a request field. ### 7.5 Linux diagnostics and resource profiles Poll active processes at a configurable interval outside pipe/network loops. Read readable `/proc` values for CPU time, RSS, I/O, state, CWD, and wait reason; aggregate only clearly associated group/child data. Missing/unreadable values remain absent. Set `suspected_hung` only after default 10 minutes without observable progress and label it diagnostic, not lifecycle. Make requested resource-profile handling explicit. On Linux the supervisor uses a delegated writable cgroup v2 for tree supervision whenever available, even without a profile. Profiles add administrator-defined CPU/memory/disk/ process controls to that cgroup. Without delegation, no-profile execution uses the watchdog/process-group fallback; a profile whose essential control cannot be applied is rejected as unsupported. No profile means no resource restriction, even when a supervisory cgroup exists. Compile TOML resource profiles into immutable validated launch policies during startup. Enforce `LIGHT` as exclusive; otherwise allow at most one CPU, one memory, and one disk profile. On Linux, translate configured policy into the per-command cgroup's `cpu.max`/`cpu.weight`, `memory.max`/`memory.high`, `pids.max`, and per-device `io.max` controls as applicable. Write and read back all essential controls while the launcher is blocked. If any essential control is unsupported or cannot be applied, destroy the empty command cgroup and permanently reject the dispatch; never run partially constrained. Define diagnostic progress as a change in process-tree CPU ticks, cumulative I/O counters, retained output/input activity, or lifecycle state. Persist only the latest sample and no-progress start time, clear `suspected_hung` on the next observed progress, and tolerate counter reset/process exit. Diagnostics must not keep a command alive, change terminal status, or block the supervisor. ### 7.6 Client tests Use fake transport/clock plus real subprocess tests for shell defaults, CWD/env overlays, at-most-once duplicate dispatch, concurrent output with no newlines, stdin ordering/close, process-group termination, reconnect/replay, offline rolling truncation before sequence assignment, assigned-window pinning, script progress/checksum failure/cleanup, atomic tombstone replacement, queue limits, shutdown interruption, and `/proc` absence. Run race tests with multiple commands and forced network churn. Add a table-driven crash suite for every acceptance, script-upload, launch, event-spool, terminal-acknowledgement, and tombstone-rotation commit point. Run the same duplicate dispatch after each restart and assert exactly one of: durable rejection before authorization, one supervised process, or an `interrupted` uncertain launch—never a second execution. **Exit criteria:** a Linux client can stay alive through server loss/restart, execute up to its capacity at most once, preserve/replay bounded history, and cleanly manage full command process trees. ## 8. End-to-end agent protocol (Phase 5) Wire the server session layer and client runtime together before adding the CLI. Use real protobuf bytes through an in-memory WebSocket test server first, then through Docker Compose with nginx proxying a WSS endpoint. Implement flows in this order: 1. hello/welcome/version/fencing and heartbeat; 2. persisted background command dispatch and lifecycle events; 3. stdout/stderr event persistence and cumulative acknowledgements; 4. reconnect replay, duplicate delivery, and non-terminal reconciliation; 5. queued cancellation and revisioned signal delivery; 6. ordered stdin/close acknowledgement; 7. verified chunked scripts; 8. capacity advertisements, diagnostics, and profile results. At each step, add a failure-injection test that drops one frame at every acknowledgement boundary. Required scenarios include lost `CommandAccepted`, lost `EventAck`, stale old-session output after takeover, connection loss during script transfer, server restart before/after event transaction, client restart with managed process cleanup, and offline spool overflow. Verify command UUIDs never execute twice and output gaps always appear as truncation metadata. Build a deterministic scenario harness with fake wall/monotonic clocks, controllable frame loss/reordering, daemon kill points, and bounded virtual disk capacity. For each flow, run disconnect before delivery, after delivery/before durable commit, after commit/before acknowledgement, and after acknowledgement. Assert database invariants and quota counters after every restart, not merely the visible CLI result. Include these cross-cutting cases: - a queued command expires at the default 15-minute TTL while disconnected, then a late client reports a real terminal result and `stat` displays both expiry history and the updated terminal state; - server per-command, per-client, and global quotas cross high and low watermarks while assigned client events remain pinned; - terminal age reclamation and command-level quota eviction create tombstones, retention metadata, and audit entries without removing active work; - client identity collision, exact-instance takeover expiry/consumption, and an old connection attempting output after the new generation is committed; - storage corruption found during asynchronous recovery, safe repair, explicit acknowledgement of unavoidable loss, and continued service in healthy scopes; - heartbeat control traffic remains serviceable while both data lanes are full over a slow, intermittently writable WebSocket. **Exit criteria:** Compose tests demonstrate a complete background command, foreground follow, stdin interaction, signal, reconnect, and restart recovery through the nginx WebSocket path. ## 9. Server control plane and `rvc` (Phase 6) ### 9.1 gRPC over Unix socket Start a Unix socket only after removing an existing socket only when it is proven to be a stale socket owned by this server account; do not blindly unlink arbitrary paths. Set mode `0600` immediately. Implement the generated `Control` service against domain/store/session interfaces, never directly against socket state. Control response messages represent success only and contain no embedded error alternative. Return canonical non-OK gRPC statuses with structured RVBox details; the JSON-RPC adapter maps the same domain error rather than inspecting response payloads. Agent-protocol rejection envelopes continue using `ControlError`. Implement and test: - `ListClients`, `GetClient`, `ListCommands`, `GetCommand`; - `AuthorizeClientTakeover` for an exact displayed pending instance, with a one-shot 5-minute grant and normal mutation idempotency; - `RunCommand` for durable creation and immediate UUID return, followed by `FollowCommand` for foreground ordered history plus live events; - `FollowCommand` beginning at a supplied event sequence and returning a wrapper around either a client event or non-sequenced server retention metadata; - `AppendStdin`, `CloseStdin`, and `SignalCommand` with persisted idempotency; - `GetOutput` with stream selection, byte-bounded opaque cursor pagination that can resume within an event, distinct client-observed/server-receipt times, and truncation/incomplete metadata; - `ListStorageIncidents`, `RepairStorageIncident`, and `AcknowledgeStorageIncident`; dirty health is derived from unresolved rows, safe repair resolves automatically, and acknowledging loss preserves history. Foreground context cancellation/timeouts stop only the local RPC stream. They must never send a remote kill. The server uses domain/store authorization of state transitions and an explicit command UUID for every mutation. Foreground `rvc run` first completes idempotent unary `RunCommand`, then starts `FollowCommand` from sequence zero with existing history included. Reconnects resume after the last received client sequence; retention markers do not advance that cursor and are emitted before a relevant terminal event. Do not add a combined create-and-follow RPC whose stream can fail before returning the newly created UUID. For every mutation, canonicalize the validated protobuf request excluding transport metadata, compute its immutable hash, and transact an idempotency row keyed globally by `request_id`, with method included in the hashed/stored identity. An exact replay returns the stored domain result or stored error; a different hash returns `ALREADY_EXISTS`/request-conflict and performs no work. Commit the idempotency row atomically with the mutation, retain it at least as long as the referenced command/takeover/incident record, and count its compressed storage against the owning command or audit scope. Reads accept the field at the transport edge but deliberately drop it before domain execution. Define opaque output/page tokens as versioned, authenticated encodings of query filters plus stable position (command, event sequence, byte offset, snapshot boundary). Reject a token reused with changed filters. Pagination reads a consistent upper boundary so concurrent appends do not duplicate/skip retained bytes; if retention removes the resume point, return the next retained position with explicit server-retention metadata. ### 9.2 CLI Implement `rvc` with no direct server database access. Required commands are: - `rvc stat [client-id] [issue-uuid] [--all --page --per-page]`; - `rvc run [--background] [--cwd DIR] [--shell TYPE] [--env K=V]... [--profile FLAG]... [--request-id UUID] [--queue-ttl DURATION] CLIENT COMMAND`; - `rvc run --script PATH [same execution options] CLIENT`; - `rvc append CLIENT UUID TEXT`, `--file PATH`, raw/no-newline mode, and `--attach` stdin streaming; - `rvc close-stdin CLIENT UUID`; - `rvc kill [HUP|INT|TERM|KILL|USR1|USR2] CLIENT UUID` (optional `SIG` prefix; default `TERM`); - `rvc storage incidents`, `rvc storage repair INCIDENT`, and `rvc storage acknowledge INCIDENT --note TEXT`; - `rvc client takeover CLIENT INSTANCE-ID`; - output selection, `--timestamped`, pagination, and `--follow` semantics. Accept `--request-id` globally. Generate a UUIDv7 for every control mutation unless supplied, reuse it across transport retries, and discard it for read-only operations. Create the request ID before dialing the Unix socket and retain it for all automatic retries of that invocation. Validate user-supplied IDs as canonical UUIDv7 strings. For `run`, map the same value directly to `issue_uuid`; for other mutations keep it only as `request_id`. Do not generate a second operation identifier. Print the issue UUID immediately after durable creation, including before entering foreground follow mode. Make `stat` render lifecycle and history independently: an expired queued command is visibly marked `expired` with its expiry time/reason; if a late, previously accepted client result later arrives, the actual terminal lifecycle becomes current while the expiry contradiction remains in metadata/audit. Also show dirty storage scope/incident ID and pending takeover instance, output retained range, and whether terminal output is incomplete. Render output without inventing line boundaries in stored data. Historical pagination uses opaque cursors and byte limits; the display layer may buffer partial lines for presentation and clearly prints retained range/truncation notices. Map structured control errors to stable non-zero exit codes while retaining machine-readable JSON output as a later optional CLI mode. ### 9.3 JSON-RPC adapter Keep this adapter small and disabled by default. Use standard JSON-RPC 2.0 over HTTP and protobuf JSON mapping for unary control request/response bodies. Implement the documented lower-camel method names only, including storage incident list/repair/acknowledge and `authorizeClientTakeover`. Bind default `127.0.0.1:6900`; configuration may bind elsewhere but startup logs a conspicuous unauthenticated-exposure warning. Do not add streaming or a second event model: callers poll `getOutput`/`getCommand` by cursor. Audit every control request with transport/source metadata where available, action, target, result, and error. Audit logs include environment key names but not override values or raw stdin/stdout/stderr. Use the JSON-RPC envelope `id` only for JSON-RPC response correlation. Accept the same optional protobuf `requestId` member as gRPC for domain idempotency; when it is absent, generate a server UUIDv7. Consequently raw JSON-RPC remains easy to use but only callers that persist/reuse `requestId` receive retry idempotency. Map parse/invalid-request/method-not-found to standard JSON-RPC codes and domain failures to one stable server-error code carrying the same structured RVBox detail as gRPC. **Exit criteria:** `rvc` can drive each documented example against the Compose stack; the Unix socket has mode `0600`; JSON-RPC behavior matches gRPC unary semantics and is off unless explicitly enabled. ## 10. Windows client implementation (Phase 7) Keep Windows code in platform-specific files/build tags so Linux builds never import Windows APIs. Implement the same supervisor interface and event/spool runtime; only OS execution/diagnostics/resource enforcement differ. 1. Create a non-inheritable per-command Job Object, enable `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`, and do not enable breakaway. 2. Start an RVBox launcher suspended with `CREATE_NEW_CONSOLE`, `CREATE_UNICODE_ENVIRONMENT`, and `EXTENDED_STARTUPINFO_PRESENT`. Assign it to the Job at creation through `PROC_THREAD_ATTRIBUTE_JOB_LIST`; inherit only an explicit standard-I/O/control handle list and never the Job handle. 3. Persist and flush `launch_prepared` with launcher PID, `GetProcessTimes` creation `FILETIME`, and launch generation; persist and flush `launch_authorized` before `ResumeThread`. The launcher then creates the requested shell suspended in its console with `CREATE_NEW_PROCESS_GROUP`, reports its PID/group, connects the allowlisted pipes, and resumes it. A failure after authorization terminates the Job and becomes interrupted. 4. Keep the launcher as an in-console signal proxy. This separation is mandatory: Windows ignores `CREATE_NEW_PROCESS_GROUP` when combined with `CREATE_NEW_CONSOLE`, and `GenerateConsoleCtrlEvent` reaches only groups sharing the caller's console. On daemon crash, closing the sole Job handle terminates the launcher and complete tree; restart never signals by PID alone. 5. Expose Job Object accounting snapshots. Children normally remain in the Job. 6. Accept only `SIGTERM` and `SIGKILL`. For TERM, have the launcher call `GenerateConsoleCtrlEvent(CTRL_BREAK_EVENT, shell_group_id)`, wait 10 seconds, then terminate the Job. For KILL, terminate the Job immediately. Report graceful-attempt/escalation outcome in the event. 7. Apply requested profile limits through Job Object limits. If the requested control cannot be applied, reject the dispatch with structured `UNSUPPORTED`. 8. Use available process/Job Object telemetry for CPU/RSS/I/O; do not emit a Linux-style wait reason. Implement the Windows launcher as a small mode of the same RVBox binary, not a searchable external helper. Pass it an inherited, random per-launch control pipe and fixed-size launch-generation token. Apply an explicit `PROC_THREAD_ATTRIBUTE_HANDLE_LIST` so only stdin/stdout/stderr and that control pipe cross creation; make every database, log, listener, Job, and unrelated pipe handle non-inheritable. Frame ready/release/exec/error messages with length, type, generation, and checksum, and reject any mismatched generation. Open the configured shell executable by canonical absolute path and pass it as the explicit `lpApplicationName`. Produce the command line with one reviewed Windows argument-quoting routine and construct an explicit Unicode environment block. Do not invoke `%COMSPEC%`, search `PATH`/file associations, or let command environment overrides influence executable selection. Associate the Job with an I/O completion port and use `JOB_OBJECT_MSG_ACTIVE_PROCESS_ZERO` plus pipe EOF for tree/drain completion. Persist the launcher and shell creation `FILETIME` identities before trusting any PID. On recovery, reopen a process only to compare creation time and generation evidence; if ownership cannot be proven, record a dirty incident rather than terminating an unrelated reused PID. Fault-inject daemon/launcher death around Job creation, attribute-list process creation, prepared/authorized fsync, resume, requested-shell creation, and terminal drain. Test both supported shells, paths with spaces/non-ASCII, malicious `PATH`/`COMSPEC`, nested descendants, CTRL_BREAK refusal/escalation, Job limit rejection, and inherited-handle leaks on supported Windows versions. Run build checks on every platform and dedicated Windows integration tests for shell selection, process-tree kill, forced termination, profile rejection, reconnect, and output/spool behavior. No Windows release is supported until these tests run on Windows CI. ## 11. Reliability, observability, and operational delivery (Phase 8) ### 11.1 Health, metrics, and logs Expose separate liveness/readiness for the server. Liveness and incident inspection start before asynchronous recovery. Global readiness requires all required scopes recovered and writable, while scoped operations can proceed on healthy scopes; no health state requires a client connection. Client health reports reconnect state, spool health, unresolved incidents, and supervisor health without leaking command output. Publish counters/histograms/gauges for registrations/takeovers, stale messages, heartbeat timeouts, reconnect duration, dispatch latency, command transitions, queue depth, spool bytes, output compression/rotation/loss, segment eviction, SQLite write latency/failure, script verification failure, and protocol errors. Use stable labels with bounded cardinality—never UUID/client ID as a metric label. Use structured logs and audit records for those identifiers instead. ### 11.2 Deployment assets Provide: - nginx configuration showing WebSocket upgrade proxying and TLS termination; - systemd units for server/client with private state directories, restart policy, working directory, file descriptor limits, and least privilege; - example server/client configuration files with every default and an explicit JSON-RPC exposure warning; - a backup/restore procedure for SQLite plus output/audit segment directories; - an upgrade procedure that stops dispatch safely, snapshots data, migrates, and verifies recovery; - a troubleshooting runbook for no client, stale session, spool full, output truncation, storage full, and daemon restart. Install the annotated TOML examples from `docs/examples/` and keep the most operationally important identity/listener/state/TLS/quota keys first in each section. Add `--check-config` to both daemons: it must run the production strict decoder, defaulting, cross-field/profile/path/shell validation, print a redacted normalized summary, and exit without opening stores/listeners or changing state. CI parses both examples with this path on Linux; Windows CI additionally validates the documented Windows path/shell variant. Use `Delegate=yes` in the Linux client systemd unit when cgroup supervision is enabled and create a private writable cgroup subtree for the service. Refuse a configured mandatory resource profile when delegation/controllers are absent, while still allowing unrestricted execution through the documented fallback. Package upgrades must retain the previous TOML, run `--check-config` before restart, and never silently ignore a newly unknown/removed key. ### 11.3 Security and regression review Before v1 release, review path traversal, script temp-file permissions, command logging, environment override redaction, malformed compression, decompression bombs, oversized frames, Unicode/ASCII ID validation, Unix socket ownership, JSON-RPC external bind warnings, SQL injection (all parameterized), segment record corruption, and process-group PID reuse. Add fuzz tests for envelope decode, compressed output validation, segment-tail recovery, pagination tokens, and JSON-RPC parsing. **Exit criteria:** release Compose/systemd/nginx assets exist; load/fault tests show no unbounded memory/goroutine growth; the operations runbook reproduces recovery and retention behavior. ## 12. Release gates and implementation order The recommended merge order is deliberately vertical: 1. Phase 0: Docker-first repository/toolchain/proto generation. 2. Phase 1: domain/config validation and state machine. 3. Phase 2: SQLite/segments/audit/retention with recovery tests. 4. Phase 3: server registration, fencing, heartbeat, and persisted dispatch. 5. Phase 4: Linux client spool/supervisor and at-most-once execution. 6. Phase 5: full WSS protocol, fault injection, and nginx end-to-end tests. 7. Phase 6: gRPC Unix socket, `rvc`, and optional JSON-RPC. 8. Phase 7: Windows supervisor and Windows CI. 9. Phase 8: operational assets, stress/fuzz/recovery testing, release review. Do not merge a later vertical slice by stubbing a durability/safety invariant. For example: foreground mode may wait on a durable background command, but must not bypass persistence; client output may be truncated under the documented caps, but must never block a child pipe; and a reconnection may replay work, but may never re-execute an already accepted UUID. The v1 release is ready only after all release-gate tests pass on clean Docker environments, Linux server/client end-to-end behavior matches the design docs, Windows support is either fully tested or explicitly not shipped, and every accepted limitation (self-reported identity and unauthenticated optional JSON-RPC) is conspicuous in deployment documentation.