rvbox/docs/operations-runbook.md

RVBox Linux-server operations runbook

This runbook covers the v1 Linux server and native Windows clients. It does not describe a Unix-like client, which is outside v1.

Deploy and verify

Use the checked-in Compose deployment as the sole Linux-server runtime. Start from its Compose-specific server.toml.example and retain json_rpc.enabled = false unless performing loopback-only debugging.

For Compose, follow the setup in deploy/production/README.md, set a pinned RVBOX_SERVER_IMAGE, RVBOX_TLS_CERT, and RVBOX_TLS_KEY, then run:

docker compose -f deploy/production/compose.yaml config
docker compose -f deploy/production/compose.yaml up -d
docker compose -f deploy/production/compose.yaml ps

The first command must succeed before any containers start. /livez shows that the process is up; /readyz becomes successful only after durable recovery. The public endpoint accepts only wss://HOST/v1/agent. Do not publish port 6900 or add a proxy route for JSON-RPC.

Health, metrics, and logs

Keep the configured observability listener private to the host or monitoring network. /metrics exposes only bounded, aggregate Prometheus samples: health state; registration/takeover and protocol failures; session/reconnect and heartbeat timing; dispatch/event/transition counts and latency; and client output accounting. It never includes a command body, stdin, output, client ID, or UUID label. Duration histograms use fixed buckets, so monitoring traffic cannot create unbounded series.

For a Windows client, readiness is false while its durable spool is recovering or it has no reconciled server session; it becomes true only while the active WSS session can exchange command data. Server liveness starts before asynchronous recovery, while server readiness remains false until that recovery completes. The configured rotating JSON/text service logs are diagnostics, not a command-output store; use rvc history and the audited storage for command evidence.

Backup and restore

Stop dispatch before copying data: stop the server gracefully, confirm it is down, then copy the entire configured server.data_dir tree. That tree contains SQLite, WAL/SHM state, retained command segments, audit segments, and incidents; backing up SQLite alone is incomplete.

To restore, keep the server stopped, move the failed directory aside without deleting it, restore the complete backup with ownership restricted to the server account, and run rvbox-server --check-config followed by a normal start. Keep readiness under observation. A failed recovery leaves readiness false and records an incident; do not delete segments to force readiness.

Upgrade and rollback

1. Record the running image/binary digest and rvc stat output.
2. Stop the server gracefully so no new dispatch is accepted.
3. Take a full data-directory backup as above.
4. Install the new image/binary without changing configuration, and run --check-config before starting it.
5. Start, wait for /readyz, and verify rvc stat plus one known client reconnect.
6. If readiness or storage recovery fails, stop, restore the prior binary/image and full data directory, then start the known-good version. Preserve logs and the failed copy for diagnosis.

Release bundle

Build a release candidate only from a clean, committed worktree. The containerized release wrapper embeds the supplied version in all three binaries, creates an immutable dist/rvbox-VERSION directory, and writes SHA256SUMS plus manifest.json only after the Windows executable has optionally been signed:

scripts/release build --version 1.0.0-rc.1
(cd dist/rvbox-1.0.0-rc.1 && sha256sum -c SHA256SUMS)
dist/rvbox-1.0.0-rc.1/rvbox-server-linux-ARCH --version
dist/rvbox-1.0.0-rc.1/rvc-linux-ARCH --version

Replace ARCH with amd64 or arm64 for the Linux host. Both variants are created from the same pinned toolchain and have the same embedded version.

For a Windows-signed release, provide an executable host-side signing hook via --sign-windows-hook /absolute/path/to/hook. The wrapper invokes it with the Windows executable path and version, then records windows_signed: true in the manifest. Without that hook the manifest deliberately declares the artifact unsigned; this is suitable for CI/test evidence but not a signed public release. --output is intentionally constrained beneath this repository's ignored dist/ tree so the pinned build container always sees the exact output mount. The wrapper never overwrites a final bundle, so correcting a failed candidate requires choosing a new version/output or deliberately removing that exact ignored dist/ directory after preserving any evidence.

Common incidents

• No client / stale session: verify nginx has WebSocket 101 entries for /v1/agent, server readiness is true, and the Windows service is running. Use rvc stat CLIENT_ID; do not restart the client merely to clear history.
• Spool or storage full: CAPACITY_EXHAUSTED is intentional admission protection. Inspect command retention and free space, allow terminal-age or quota rotation to reclaim eligible data, or enlarge the owned filesystem. Never manually remove live SQLite/WAL/segment files.
• Output truncated: query command status and output history for its explicit loss/truncation markers. The command may still have completed correctly.
• Server restart / dirty health: wait for readiness and inspect the incident record. Use the documented repair/acknowledgement controls only after preserving evidence; a late client report is authoritative and is not rewritten to match an earlier provisional state.