Files
rvbox/docs/operations-runbook.md
T

9.0 KiB

RVBox Linux-server operations runbook

This runbook covers the v1 Linux server and native Windows clients. It does not describe a Unix-like client, which is outside v1.

Deploy and verify

Use the checked-in Compose deployment as the sole Linux-server runtime. Start from its Compose-specific server.toml.example and retain json_rpc.enabled = false unless performing loopback-only debugging.

For Compose, follow the setup in deploy/production/README.md, set a pinned RVBOX_SERVER_IMAGE, RVBOX_TLS_CERT, and RVBOX_TLS_KEY, then run:

docker compose -f deploy/production/compose.yaml config
docker compose -f deploy/production/compose.yaml up -d
docker compose -f deploy/production/compose.yaml ps

The first command must succeed before any containers start. /livez shows that the process is up; /readyz becomes successful only after durable recovery. The public endpoint accepts only wss://HOST/v1/agent. Do not publish port 6900 or add a proxy route for JSON-RPC.

Health, metrics, and logs

Keep the configured observability listener private to the host or monitoring network. /metrics exposes only bounded, aggregate Prometheus samples: health state; registration/takeover and protocol failures; session/reconnect and heartbeat timing; dispatch/event/transition counts and latency; and client output accounting. It never includes a command body, stdin, output, client ID, or UUID label. Duration histograms use fixed buckets, so monitoring traffic cannot create unbounded series.

For a Windows client, readiness is false while its durable spool is recovering or it has no reconciled server session; it becomes true only while the active WSS session can exchange command data. Server liveness starts before asynchronous recovery, while server readiness remains false until that recovery completes. The configured rotating JSON/text service logs are diagnostics, not a command-output store; use rvc history and the audited storage for command evidence.

Backup and restore

Stop dispatch before copying data: stop the server gracefully, confirm it is down, then copy the entire configured server.data_dir tree. That tree contains SQLite, WAL/SHM state, retained command segments, audit segments, and incidents; backing up SQLite alone is incomplete.

To restore, keep the server stopped, move the failed directory aside without deleting it, restore the complete backup with ownership restricted to the server account, and run rvbox-server --check-config followed by a normal start. Keep readiness under observation. A failed recovery leaves readiness false and records an incident; do not delete segments to force readiness.

Upgrade and rollback

  1. Record the running image/binary digest and rvc stat output.
  2. Stop the server gracefully so no new dispatch is accepted.
  3. Take a full data-directory backup as above.
  4. Install the new image/binary without changing configuration, and run --check-config before starting it.
  5. Start, wait for /readyz, and verify rvc stat plus one known client reconnect.
  6. If readiness or storage recovery fails, stop, restore the prior binary/image and full data directory, then start the known-good version. Preserve logs and the failed copy for diagnosis.

Release bundle

Build a release candidate only from a clean, committed worktree. The containerized release wrapper embeds the supplied version in all three binaries, creates an immutable dist/rvbox-VERSION directory, and writes SHA256SUMS plus manifest.json only after the Windows executable has optionally been signed:

scripts/release build \
  --version 1.0.0-rc.1 \
  --bootstrap-server-url wss://rvbox.example.test/v1/agent
(cd dist/rvbox-1.0.0-rc.1 && sha256sum -c SHA256SUMS)
dist/rvbox-1.0.0-rc.1/rvbox-server-linux-ARCH --version
dist/rvbox-1.0.0-rc.1/rvc-linux-ARCH --version

Replace ARCH with amd64 or arm64 for the Linux host. Both variants are created from the same pinned toolchain and have the same embedded version.

For a compact, directly runnable Windows package, add --windows-compression upx to that command. The default none retains the ordinary executable. Compact builds use checksum-pinned UPX 5.2.1 with LZMA, retain PE relocations, and leave icons/manifest/version resources uncompressed. Packaging checks UPX integrity, compares every extracted Windows resource to the original, and requires a smaller result before publication. The bundle also retains rvbox-windows-amd64-unpacked.exe; both variants are included in SHA256SUMS, and the manifest records windows_compression. Optional signing runs after packing and signs both variants before checksumming. The packer runs only inside Docker; release automatically rebuilds an older toolchain that lacks it. Existing build-image/cache cleanup and recovery still apply.

UPX expands the image in memory at each process start, including service, tray, launcher, and signal-helper modes. This trades smaller files/downloads for startup work and reduced sharing of executable memory between processes. Antivirus products may flag packed executables; use the unpacked fallback if that occurs rather than weakening protection. Resource/integrity checks are not native execution evidence. Before choosing compressed packages for regular distribution, exercise fresh install, upgrade, same-version activation, service restart/reboot, tray, and ordinary/elevated command execution on Windows. The installer still needs no external extractor or separate resident process. See the UPX manual and upstream antivirus issue.

For a Windows-signed release, provide an executable host-side signing hook via --sign-windows-hook /absolute/path/to/hook. The wrapper invokes it with the Windows executable path and version, then records windows_signed: true in the manifest. Without that hook the manifest deliberately declares the artifact unsigned; this is suitable for CI/test evidence but not a signed public release. --output is intentionally constrained beneath this repository's ignored dist/ tree so the pinned build container always sees the exact output mount. The wrapper never overwrites a final bundle, so correcting a failed candidate requires choosing a new version/output or deliberately removing that exact ignored dist/ directory after preserving any evidence.

Common incidents

  • Reconnect loop after a command produces substantial output: inspect C:\ProgramData\RVBox\service-startup.log on Windows and the server log for StatusMessageTooBig / read limited at 32769 bytes. Early v1 transports inherited a 32 KiB WebSocket ceiling despite permitting 64 KiB chunks and 1 MiB envelopes. Both transports now set the configured envelope ceiling before reading any messages. Upgrade the server first to drain pending client output, then the client for large dispatch/reconciliation responses. Preserve spool/history: rebooting cannot remove a replayable oversized event. New server logs include bounded read/reconciliation errors and an agent_message_too_big counter; new clients advertise the actual build version.
  • No client / stale session: verify nginx has WebSocket 101 entries for /v1/agent, server readiness is true, and the Windows service is running. Use rvc stat CLIENT_ID; do not restart the client merely to clear history.
  • Spool or storage full: CAPACITY_EXHAUSTED is intentional admission protection. Inspect command retention and free space, allow terminal-age or quota rotation to reclaim eligible data, or enlarge the owned filesystem. Never manually remove live SQLite/WAL/segment files.
  • Startup remains unready and control requests time out: a corrected server release is required if payload recovery is waiting for SQLite's sole connection while holding a command cursor. Recovery now joins script descriptors and payloads in one query and decompresses stored execution descriptions before validation. Preserve the existing data directory during upgrade; deleting command history is unnecessary.
  • Repeated signal command ...: command is not running: early v1 clients closed their session instead of returning a negative signal result when an old queued/replayed signal reached an already-exited process. Upgrade the Windows client with the corrected installer, preserving TOML/spool/history. The client records accepted=false for that signal; the existing server acknowledges it and stops replaying it. Do not wipe state or resend the original command to clear this loop.
  • Output truncated: query command status and output history for its explicit loss/truncation markers. The command may still have completed correctly.
  • Server restart / dirty health: wait for readiness and inspect the incident record. Use the documented repair/acknowledgement controls only after preserving evidence; a late client report is authoritative and is not rewritten to match an earlier provisional state.