10 KiB
RVBox Linux-server operations runbook
This runbook covers the v1 Linux server and native Windows clients. It does not describe a Unix-like client, which is outside v1.
Deploy and verify
Use the checked-in Compose deployment as the sole Linux-server runtime. Start
from its Compose-specific
server.toml.example and retain
json_rpc.enabled = false unless performing loopback-only debugging.
For Compose, follow the setup in
deploy/production/README.md, set a pinned
RVBOX_SERVER_IMAGE, RVBOX_TLS_CERT, and RVBOX_TLS_KEY, then run:
docker compose -f deploy/production/compose.yaml config
docker compose -f deploy/production/compose.yaml up -d
docker compose -f deploy/production/compose.yaml ps
The first command must succeed before any containers start. /livez shows that
the process is up; /readyz becomes successful only after durable recovery.
The public endpoint accepts only wss://HOST/v1/agent. Do not publish port
6900 or add a proxy route for JSON-RPC.
Health, metrics, and logs
Keep the configured observability listener private to the host or monitoring
network. /metrics exposes only bounded, aggregate Prometheus samples: health
state; registration/takeover and protocol failures; session/reconnect and
heartbeat timing; dispatch/event/transition counts and latency; and client
output accounting. It never includes a command body, stdin, output, client ID,
or UUID label. Duration histograms use fixed buckets, so monitoring traffic
cannot create unbounded series.
For a Windows client, readiness is false while its durable spool is recovering
or it has no reconciled server session; it becomes true only while the active
WSS session can exchange command data. Server liveness starts before
asynchronous recovery, while server readiness remains false until that recovery
completes. The configured rotating JSON/text service logs are diagnostics, not
a command-output store; use rvc history and the audited storage for command
evidence.
Backup and restore
Stop dispatch before copying data: stop the server gracefully, confirm it is
down, then copy the entire configured server.data_dir tree. That tree contains
SQLite, WAL/SHM state, retained command segments, audit segments, and incidents;
backing up SQLite alone is incomplete.
To restore, keep the server stopped, move the failed directory aside without
deleting it, restore the complete backup with ownership restricted to the
server account, and run rvbox-server --check-config followed by a normal
start. Keep readiness under observation. A failed recovery leaves readiness
false and records an incident; do not delete segments to force readiness.
Upgrade and rollback
- Record the running image/binary digest and
rvc statoutput. - Stop the server gracefully so no new dispatch is accepted.
- Take a full data-directory backup as above.
- Install the new image/binary without changing configuration, and run
--check-configbefore starting it. - Start, wait for
/readyz, and verifyrvc statplus one known client reconnect. - If readiness or storage recovery fails, stop, restore the prior binary/image and full data directory, then start the known-good version. Preserve logs and the failed copy for diagnosis.
Release bundle
Build a release candidate only from a clean, committed worktree. The
containerized release wrapper embeds the supplied version in all three binaries,
creates an immutable dist/rvbox-VERSION directory, and writes SHA256SUMS
plus manifest.json only after the Windows executable has optionally been
signed:
scripts/release build \
--version 1.0.0-rc.1 \
--bootstrap-server-url wss://rvbox.example.test/v1/agent
(cd dist/rvbox-1.0.0-rc.1 && sha256sum -c SHA256SUMS)
dist/rvbox-1.0.0-rc.1/rvbox-server-linux-ARCH --version
dist/rvbox-1.0.0-rc.1/rvc-linux-ARCH --version
Replace ARCH with amd64 or arm64 for the Linux host. Both variants are
created from the same pinned toolchain and have the same embedded version.
For a compact, directly runnable Windows package, add
--windows-compression upx to that command. The default none retains the
ordinary executable. Compact builds use checksum-pinned UPX 5.2.1 with LZMA,
retain PE relocations, and leave icons/manifest/version resources uncompressed.
Packaging checks UPX integrity, compares every extracted Windows resource to
the original, and requires a smaller result before publication. The bundle
also retains rvbox-windows-amd64-unpacked.exe; both variants are included in
SHA256SUMS, and the manifest records windows_compression. Optional signing
runs after packing and signs both variants before checksumming. The packer
runs only inside Docker; release automatically rebuilds an older toolchain
that lacks it. Existing build-image/cache cleanup and recovery still apply.
UPX expands the image in memory at each process start, including service, tray, launcher, and signal-helper modes. This trades smaller files/downloads for startup work and reduced sharing of executable memory between processes. Antivirus products may flag packed executables; use the unpacked fallback if that occurs rather than weakening protection. Resource/integrity checks are not native execution evidence. Before choosing compressed packages for regular distribution, exercise fresh install, upgrade, same-version activation, service restart/reboot, tray, and ordinary/elevated command execution on Windows. The installer still needs no external extractor or separate resident process. See the UPX manual and upstream antivirus issue.
For a Windows-signed release, provide an executable host-side signing hook via
--sign-windows-hook /absolute/path/to/hook. The wrapper invokes it with the
Windows executable path and version, then records windows_signed: true in the
manifest. Without that hook the manifest deliberately declares the artifact
unsigned; this is suitable for CI/test evidence but not a signed public
release. --output is intentionally constrained beneath this repository's
ignored dist/ tree so the pinned build container always sees the exact output
mount. The wrapper never overwrites a final bundle, so correcting a failed
candidate requires choosing a new version/output or deliberately removing that
exact ignored dist/ directory after preserving any evidence.
Common incidents
- Auto-chunking rollout hold: the negotiated transport implementation is intentionally offline at the user's request. Do not upgrade the running deployment or replace published installers to enable it yet. See the wire format and isolated checks. It requires both endpoints to negotiate support; it does not change logical admission limits.
- Reconnect loop after a command produces substantial output: inspect
C:\ProgramData\RVBox\service-startup.logon Windows and the server log forStatusMessageTooBig/read limited at 32769 bytes. Early v1 transports inherited a 32 KiB WebSocket ceiling despite permitting 64 KiB chunks and 1 MiB envelopes. Both transports now set the configured envelope ceiling before reading any messages. Upgrade the server first to drain pending client output, then the client for large dispatch/reconciliation responses. Preserve spool/history: rebooting cannot remove a replayable oversized event. New server logs include bounded read/reconciliation errors and anagent_message_too_bigcounter; new clients advertise the actual build version. If the server reports receiving that close code from an older client, the client rejected a server-to-client frame. Outbound script chunks target 16 KiB (or a lower configured chunk ceiling), preserving offsets/digests and the 1 MiB send window, so pending uploads can reach older clients. This does not lower the server's inbound limits. Large inline command text, stdin, or reconciliation responses still require the corrected client reader. terminal command cannot accept script dataduring reconnect: the server may replay an upload while the client is still delivering that command's terminal events. Corrected clients ignore these frames only after verifying the command is locally terminal; existing event replay reports the outcome. They neither change script bytes nor invoke launch again. The same error for a committed but still-live script, unknown owners, and malformed transfers remains a failure. Upgrade the client while preserving its spool.- No client / stale session: verify nginx has WebSocket
101entries for/v1/agent, server readiness is true, and the Windows service is running. Uservc stat CLIENT_ID; do not restart the client merely to clear history. - Spool or storage full:
CAPACITY_EXHAUSTEDis intentional admission protection. Inspect command retention and free space, allow terminal-age or quota rotation to reclaim eligible data, or enlarge the owned filesystem. Never manually remove live SQLite/WAL/segment files. - Startup remains unready and control requests time out: a corrected server release is required if payload recovery is waiting for SQLite's sole connection while holding a command cursor. Recovery now joins script descriptors and payloads in one query and decompresses stored execution descriptions before validation. Preserve the existing data directory during upgrade; deleting command history is unnecessary.
- Repeated
signal command ...: command is not running: early v1 clients closed their session instead of returning a negative signal result when an old queued/replayed signal reached an already-exited process. Upgrade the Windows client with the corrected installer, preserving TOML/spool/history. The client recordsaccepted=falsefor that signal; the existing server acknowledges it and stops replaying it. Do not wipe state or resend the original command to clear this loop. - Output truncated: query command status and output history for its explicit loss/truncation markers. The command may still have completed correctly.
- Server restart / dirty health: wait for readiness and inspect the incident record. Use the documented repair/acknowledgement controls only after preserving evidence; a late client report is authoritative and is not rewritten to match an earlier provisional state.