# RVBox Linux-server operations runbook This runbook covers the v1 Linux server and native Windows clients. It does not describe a Unix-like client, which is outside v1. ## Deploy and verify Use the checked-in Compose deployment as the sole Linux-server runtime. Start from its Compose-specific [`server.toml.example`](../deploy/production/server.toml.example) and retain `json_rpc.enabled = false` unless performing loopback-only debugging. For Compose, follow the setup in [`deploy/production/README.md`](../deploy/production/README.md), set a pinned `RVBOX_SERVER_IMAGE`, `RVBOX_TLS_CERT`, and `RVBOX_TLS_KEY`, then run: ```sh docker compose -f deploy/production/compose.yaml config docker compose -f deploy/production/compose.yaml up -d docker compose -f deploy/production/compose.yaml ps ``` The first command must succeed before any containers start. `/livez` shows that the process is up; `/readyz` becomes successful only after durable recovery. The public endpoint accepts only `wss://HOST/v1/agent`. Do not publish port 6900 or add a proxy route for JSON-RPC. ## Health, metrics, and logs Keep the configured observability listener private to the host or monitoring network. `/metrics` exposes only bounded, aggregate Prometheus samples: health state; registration/takeover and protocol failures; session/reconnect and heartbeat timing; dispatch/event/transition counts and latency; and client output accounting. It never includes a command body, stdin, output, client ID, or UUID label. Duration histograms use fixed buckets, so monitoring traffic cannot create unbounded series. For a Windows client, readiness is false while its durable spool is recovering or it has no reconciled server session; it becomes true only while the active WSS session can exchange command data. Server liveness starts before asynchronous recovery, while server readiness remains false until that recovery completes. The configured rotating JSON/text service logs are diagnostics, not a command-output store; use `rvc` history and the audited storage for command evidence. ## Backup and restore Stop dispatch before copying data: stop the server gracefully, confirm it is down, then copy the entire configured `server.data_dir` tree. That tree contains SQLite, WAL/SHM state, retained command segments, audit segments, and incidents; backing up SQLite alone is incomplete. To restore, keep the server stopped, move the failed directory aside without deleting it, restore the complete backup with ownership restricted to the server account, and run `rvbox-server --check-config` followed by a normal start. Keep readiness under observation. A failed recovery leaves readiness false and records an incident; do not delete segments to force readiness. ## Upgrade and rollback 1. Record the running image/binary digest and `rvc stat` output. 2. Stop the server gracefully so no new dispatch is accepted. 3. Take a full data-directory backup as above. 4. Install the new image/binary without changing configuration, and run `--check-config` before starting it. 5. Start, wait for `/readyz`, and verify `rvc stat` plus one known client reconnect. 6. If readiness or storage recovery fails, stop, restore the prior binary/image and full data directory, then start the known-good version. Preserve logs and the failed copy for diagnosis. ## Release bundle Build a release candidate only from a clean, committed worktree. The containerized release wrapper embeds the supplied version in all three binaries, creates an immutable `dist/rvbox-VERSION` directory, and writes `SHA256SUMS` plus `manifest.json` only after the Windows executable has optionally been signed: ```sh scripts/release build \ --version 1.0.0-rc.1 \ --bootstrap-server-url wss://rvbox.example.test/v1/agent (cd dist/rvbox-1.0.0-rc.1 && sha256sum -c SHA256SUMS) dist/rvbox-1.0.0-rc.1/rvbox-server-linux-ARCH --version dist/rvbox-1.0.0-rc.1/rvc-linux-ARCH --version ``` Replace `ARCH` with `amd64` or `arm64` for the Linux host. Both variants are created from the same pinned toolchain and have the same embedded version. For a compact, directly runnable Windows package, add `--windows-compression upx` to that command. The default `none` retains the ordinary executable. Compact builds use checksum-pinned UPX 5.2.1 with LZMA, retain PE relocations, and leave icons/manifest/version resources uncompressed. Packaging checks UPX integrity, compares every extracted Windows resource to the original, and requires a smaller result before publication. The bundle also retains `rvbox-windows-amd64-unpacked.exe`; both variants are included in `SHA256SUMS`, and the manifest records `windows_compression`. Optional signing runs **after** packing and signs both variants before checksumming. The packer runs only inside Docker; release automatically rebuilds an older toolchain that lacks it. Existing build-image/cache cleanup and recovery still apply. UPX expands the image in memory at each process start, including service, tray, launcher, and signal-helper modes. This trades smaller files/downloads for startup work and reduced sharing of executable memory between processes. Antivirus products may flag packed executables; use the unpacked fallback if that occurs rather than weakening protection. Resource/integrity checks are not native execution evidence. Before choosing compressed packages for regular distribution, exercise fresh install, upgrade, same-version activation, service restart/reboot, tray, and ordinary/elevated command execution on Windows. The installer still needs no external extractor or separate resident process. See the [UPX manual](https://github.com/upx/upx/blob/v5.2.1/doc/upx.pod) and [upstream antivirus issue](https://github.com/upx/upx/issues/437). For a Windows-signed release, provide an executable host-side signing hook via `--sign-windows-hook /absolute/path/to/hook`. The wrapper invokes it with the Windows executable path and version, then records `windows_signed: true` in the manifest. Without that hook the manifest deliberately declares the artifact unsigned; this is suitable for CI/test evidence but not a signed public release. `--output` is intentionally constrained beneath this repository's ignored `dist/` tree so the pinned build container always sees the exact output mount. The wrapper never overwrites a final bundle, so correcting a failed candidate requires choosing a new version/output or deliberately removing that exact ignored `dist/` directory after preserving any evidence. ## Common incidents - **No client / stale session:** verify nginx has WebSocket `101` entries for `/v1/agent`, server readiness is true, and the Windows service is running. Use `rvc stat CLIENT_ID`; do not restart the client merely to clear history. - **Spool or storage full:** `CAPACITY_EXHAUSTED` is intentional admission protection. Inspect command retention and free space, allow terminal-age or quota rotation to reclaim eligible data, or enlarge the owned filesystem. Never manually remove live SQLite/WAL/segment files. - **Startup remains unready and control requests time out:** a corrected server release is required if payload recovery is waiting for SQLite's sole connection while holding a command cursor. Recovery now joins script descriptors and payloads in one query and decompresses stored execution descriptions before validation. Preserve the existing data directory during upgrade; deleting command history is unnecessary. - **Repeated `signal command ...: command is not running`:** early v1 clients closed their session instead of returning a negative signal result when an old queued/replayed signal reached an already-exited process. Upgrade the Windows client with the corrected installer, preserving TOML/spool/history. The client records `accepted=false` for that signal; the existing server acknowledges it and stops replaying it. Do not wipe state or resend the original command to clear this loop. - **Output truncated:** query command status and output history for its explicit loss/truncation markers. The command may still have completed correctly. - **Server restart / dirty health:** wait for readiness and inspect the incident record. Use the documented repair/acknowledgement controls only after preserving evidence; a late client report is authoritative and is not rewritten to match an earlier provisional state.