Files
rvbox/docs/operations-runbook.md
T

155 lines
8.2 KiB
Markdown

# RVBox Linux-server operations runbook
This runbook covers the v1 Linux server and native Windows clients. It does not
describe a Unix-like client, which is outside v1.
## Deploy and verify
Use the checked-in Compose deployment as the sole Linux-server runtime. Start
from its Compose-specific
[`server.toml.example`](../deploy/production/server.toml.example) and retain
`json_rpc.enabled = false` unless performing loopback-only debugging.
For Compose, follow the setup in
[`deploy/production/README.md`](../deploy/production/README.md), set a pinned
`RVBOX_SERVER_IMAGE`, `RVBOX_TLS_CERT`, and `RVBOX_TLS_KEY`, then run:
```sh
docker compose -f deploy/production/compose.yaml config
docker compose -f deploy/production/compose.yaml up -d
docker compose -f deploy/production/compose.yaml ps
```
The first command must succeed before any containers start. `/livez` shows that
the process is up; `/readyz` becomes successful only after durable recovery.
The public endpoint accepts only `wss://HOST/v1/agent`. Do not publish port
6900 or add a proxy route for JSON-RPC.
## Health, metrics, and logs
Keep the configured observability listener private to the host or monitoring
network. `/metrics` exposes only bounded, aggregate Prometheus samples: health
state; registration/takeover and protocol failures; session/reconnect and
heartbeat timing; dispatch/event/transition counts and latency; and client
output accounting. It never includes a command body, stdin, output, client ID,
or UUID label. Duration histograms use fixed buckets, so monitoring traffic
cannot create unbounded series.
For a Windows client, readiness is false while its durable spool is recovering
or it has no reconciled server session; it becomes true only while the active
WSS session can exchange command data. Server liveness starts before
asynchronous recovery, while server readiness remains false until that recovery
completes. The configured rotating JSON/text service logs are diagnostics, not
a command-output store; use `rvc` history and the audited storage for command
evidence.
## Backup and restore
Stop dispatch before copying data: stop the server gracefully, confirm it is
down, then copy the entire configured `server.data_dir` tree. That tree contains
SQLite, WAL/SHM state, retained command segments, audit segments, and incidents;
backing up SQLite alone is incomplete.
To restore, keep the server stopped, move the failed directory aside without
deleting it, restore the complete backup with ownership restricted to the
server account, and run `rvbox-server --check-config` followed by a normal
start. Keep readiness under observation. A failed recovery leaves readiness
false and records an incident; do not delete segments to force readiness.
## Upgrade and rollback
1. Record the running image/binary digest and `rvc stat` output.
2. Stop the server gracefully so no new dispatch is accepted.
3. Take a full data-directory backup as above.
4. Install the new image/binary without changing configuration, and run
`--check-config` before starting it.
5. Start, wait for `/readyz`, and verify `rvc stat` plus one known client
reconnect.
6. If readiness or storage recovery fails, stop, restore the prior binary/image
and full data directory, then start the known-good version. Preserve logs
and the failed copy for diagnosis.
## Release bundle
Build a release candidate only from a clean, committed worktree. The
containerized release wrapper embeds the supplied version in all three binaries,
creates an immutable `dist/rvbox-VERSION` directory, and writes `SHA256SUMS`
plus `manifest.json` only after the Windows executable has optionally been
signed:
```sh
scripts/release build \
--version 1.0.0-rc.1 \
--bootstrap-server-url wss://rvbox.example.test/v1/agent
(cd dist/rvbox-1.0.0-rc.1 && sha256sum -c SHA256SUMS)
dist/rvbox-1.0.0-rc.1/rvbox-server-linux-ARCH --version
dist/rvbox-1.0.0-rc.1/rvc-linux-ARCH --version
```
Replace `ARCH` with `amd64` or `arm64` for the Linux host. Both variants are
created from the same pinned toolchain and have the same embedded version.
For a compact, directly runnable Windows package, add
`--windows-compression upx` to that command. The default `none` retains the
ordinary executable. Compact builds use checksum-pinned UPX 5.2.1 with LZMA,
retain PE relocations, and leave icons/manifest/version resources uncompressed.
Packaging checks UPX integrity, compares every extracted Windows resource to
the original, and requires a smaller result before publication. The bundle
also retains `rvbox-windows-amd64-unpacked.exe`; both variants are included in
`SHA256SUMS`, and the manifest records `windows_compression`. Optional signing
runs **after** packing and signs both variants before checksumming. The packer
runs only inside Docker; release automatically rebuilds an older toolchain
that lacks it. Existing build-image/cache cleanup and recovery still apply.
UPX expands the image in memory at each process start, including service, tray,
launcher, and signal-helper modes. This trades smaller files/downloads for
startup work and reduced sharing of executable memory between processes.
Antivirus products may flag packed executables; use the unpacked fallback if
that occurs rather than weakening protection. Resource/integrity checks are
not native execution evidence. Before choosing compressed packages for regular
distribution, exercise fresh install, upgrade, same-version activation, service
restart/reboot, tray, and ordinary/elevated command execution on Windows.
The installer still needs no external extractor or separate resident process.
See the [UPX manual](https://github.com/upx/upx/blob/v5.2.1/doc/upx.pod) and
[upstream antivirus issue](https://github.com/upx/upx/issues/437).
For a Windows-signed release, provide an executable host-side signing hook via
`--sign-windows-hook /absolute/path/to/hook`. The wrapper invokes it with the
Windows executable path and version, then records `windows_signed: true` in the
manifest. Without that hook the manifest deliberately declares the artifact
unsigned; this is suitable for CI/test evidence but not a signed public
release. `--output` is intentionally constrained beneath this repository's
ignored `dist/` tree so the pinned build container always sees the exact output
mount. The wrapper never overwrites a final bundle, so correcting a failed
candidate requires choosing a new version/output or deliberately removing that
exact ignored `dist/` directory after preserving any evidence.
## Common incidents
- **No client / stale session:** verify nginx has WebSocket `101` entries for
`/v1/agent`, server readiness is true, and the Windows service is running.
Use `rvc stat CLIENT_ID`; do not restart the client merely to clear history.
- **Spool or storage full:** `CAPACITY_EXHAUSTED` is intentional admission
protection. Inspect command retention and free space, allow terminal-age or
quota rotation to reclaim eligible data, or enlarge the owned filesystem.
Never manually remove live SQLite/WAL/segment files.
- **Startup remains unready and control requests time out:** a corrected server
release is required if payload recovery is waiting for SQLite's sole
connection while holding a command cursor. Recovery now joins script
descriptors and payloads in one query and decompresses stored execution
descriptions before validation. Preserve the existing data directory during
upgrade; deleting command history is unnecessary.
- **Repeated `signal command ...: command is not running`:** early v1 clients
closed their session instead of returning a negative signal result when an
old queued/replayed signal reached an already-exited process. Upgrade the
Windows client with the corrected installer, preserving TOML/spool/history.
The client records `accepted=false` for that signal; the existing server
acknowledges it and stops replaying it. Do not wipe state or resend the
original command to clear this loop.
- **Output truncated:** query command status and output history for its explicit
loss/truncation markers. The command may still have completed correctly.
- **Server restart / dirty health:** wait for readiness and inspect the
incident record. Use the documented repair/acknowledgement controls only
after preserving evidence; a late client report is authoritative and is not
rewritten to match an earlier provisional state.