rvbox/docs/operations-runbook.md

# RVBox Linux-server operations runbook

This runbook covers the v1 Linux server and native Windows clients. It does not
describe a Unix-like client, which is outside v1.

## Deploy and verify

Use the checked-in Compose deployment as the sole Linux-server runtime. Start
from its Compose-specific
[`server.toml.example`](../deploy/production/server.toml.example) and retain
`json_rpc.enabled = false` unless performing loopback-only debugging.

For Compose, follow the setup in
[`deploy/production/README.md`](../deploy/production/README.md), set a pinned
`RVBOX_SERVER_IMAGE`, `RVBOX_TLS_CERT`, and `RVBOX_TLS_KEY`, then run:

```sh
docker compose -f deploy/production/compose.yaml config
docker compose -f deploy/production/compose.yaml up -d
docker compose -f deploy/production/compose.yaml ps
```

The first command must succeed before any containers start. `/livez` shows that
the process is up; `/readyz` becomes successful only after durable recovery.
The public endpoint accepts only `wss://HOST/v1/agent`. Do not publish port
6900 or add a proxy route for JSON-RPC.

## Health, metrics, and logs

Keep the configured observability listener private to the host or monitoring
network. `/metrics` exposes only bounded, aggregate Prometheus samples: health
state; registration/takeover and protocol failures; session/reconnect and
heartbeat timing; dispatch/event/transition counts and latency; and client
output accounting. It never includes a command body, stdin, output, client ID,
or UUID label. Duration histograms use fixed buckets, so monitoring traffic
cannot create unbounded series.

For a Windows client, readiness is false while its durable spool is recovering
or it has no reconciled server session; it becomes true only while the active
WSS session can exchange command data. Server liveness starts before
asynchronous recovery, while server readiness remains false until that recovery
completes. The configured rotating JSON/text service logs are diagnostics, not
a command-output store; use `rvc` history and the audited storage for command
evidence.

## Backup and restore

Stop dispatch before copying data: stop the server gracefully, confirm it is
down, then copy the entire configured `server.data_dir` tree. That tree contains
SQLite, WAL/SHM state, retained command segments, audit segments, and incidents;
backing up SQLite alone is incomplete.

To restore, keep the server stopped, move the failed directory aside without
deleting it, restore the complete backup with ownership restricted to the
server account, and run `rvbox-server --check-config` followed by a normal
start. Keep readiness under observation. A failed recovery leaves readiness
false and records an incident; do not delete segments to force readiness.

## Upgrade and rollback

1. Record the running image/binary digest and `rvc stat` output.
2. Stop the server gracefully so no new dispatch is accepted.
3. Take a full data-directory backup as above.
4. Install the new image/binary without changing configuration, and run
   `--check-config` before starting it.
5. Start, wait for `/readyz`, and verify `rvc stat` plus one known client
   reconnect.
6. If readiness or storage recovery fails, stop, restore the prior binary/image
   and full data directory, then start the known-good version. Preserve logs
   and the failed copy for diagnosis.

## Release bundle

Build a release candidate only from a clean, committed worktree. The
containerized release wrapper embeds the supplied version in all three binaries,
creates an immutable `dist/rvbox-VERSION` directory, and writes `SHA256SUMS`
plus `manifest.json` only after the Windows executable has optionally been
signed:

```sh
scripts/release build \
  --version 1.0.0-rc.1 \
  --bootstrap-server-url wss://rvbox.example.test/v1/agent
(cd dist/rvbox-1.0.0-rc.1 && sha256sum -c SHA256SUMS)
dist/rvbox-1.0.0-rc.1/rvbox-server-linux-ARCH --version
dist/rvbox-1.0.0-rc.1/rvc-linux-ARCH --version
```

Replace `ARCH` with `amd64` or `arm64` for the Linux host. Both variants are
created from the same pinned toolchain and have the same embedded version.

For a Windows-signed release, provide an executable host-side signing hook via
`--sign-windows-hook /absolute/path/to/hook`. The wrapper invokes it with the
Windows executable path and version, then records `windows_signed: true` in the
manifest. Without that hook the manifest deliberately declares the artifact
unsigned; this is suitable for CI/test evidence but not a signed public
release. `--output` is intentionally constrained beneath this repository's
ignored `dist/` tree so the pinned build container always sees the exact output
mount. The wrapper never overwrites a final bundle, so correcting a failed
candidate requires choosing a new version/output or deliberately removing that
exact ignored `dist/` directory after preserving any evidence.

## Common incidents

- **No client / stale session:** verify nginx has WebSocket `101` entries for
  `/v1/agent`, server readiness is true, and the Windows service is running.
  Use `rvc stat CLIENT_ID`; do not restart the client merely to clear history.
- **Spool or storage full:** `CAPACITY_EXHAUSTED` is intentional admission
  protection. Inspect command retention and free space, allow terminal-age or
  quota rotation to reclaim eligible data, or enlarge the owned filesystem.
  Never manually remove live SQLite/WAL/segment files.
- **Output truncated:** query command status and output history for its explicit
  loss/truncation markers. The command may still have completed correctly.
- **Server restart / dirty health:** wait for readiness and inspect the
  incident record. Use the documented repair/acknowledgement controls only
  after preserving evidence; a late client report is authoritative and is not
  rewritten to match an earlier provisional state.