193 lines
11 KiB
Markdown
193 lines
11 KiB
Markdown
# RVBox Linux-server operations runbook
|
|
|
|
This runbook covers the v1 Linux server and native Windows clients. It does not
|
|
describe a Unix-like client, which is outside v1.
|
|
|
|
## Deploy and verify
|
|
|
|
Use the checked-in Compose deployment as the sole Linux-server runtime. Start
|
|
from its Compose-specific
|
|
[`server.toml.example`](../deploy/production/server.toml.example) and retain
|
|
`json_rpc.enabled = false` unless performing loopback-only debugging.
|
|
|
|
For Compose, follow the setup in
|
|
[`deploy/production/README.md`](../deploy/production/README.md), set a pinned
|
|
`RVBOX_SERVER_IMAGE`, `RVBOX_TLS_CERT`, and `RVBOX_TLS_KEY`, then run:
|
|
|
|
```sh
|
|
docker compose -f deploy/production/compose.yaml config
|
|
docker compose -f deploy/production/compose.yaml up -d
|
|
docker compose -f deploy/production/compose.yaml ps
|
|
```
|
|
|
|
The first command must succeed before any containers start. `/livez` shows that
|
|
the process is up; `/readyz` becomes successful only after durable recovery.
|
|
The public endpoint accepts only `wss://HOST/v1/agent`. Do not publish port
|
|
6900 or add a proxy route for JSON-RPC.
|
|
|
|
## Health, metrics, and logs
|
|
|
|
Keep the configured observability listener private to the host or monitoring
|
|
network. `/metrics` exposes only bounded, aggregate Prometheus samples: health
|
|
state; registration/takeover and protocol failures; session/reconnect and
|
|
heartbeat timing; dispatch/event/transition counts and latency; and client
|
|
output accounting. It never includes a command body, stdin, output, client ID,
|
|
or UUID label. Duration histograms use fixed buckets, so monitoring traffic
|
|
cannot create unbounded series.
|
|
|
|
For a Windows client, readiness is false while its durable spool is recovering
|
|
or it has no reconciled server session; it becomes true only while the active
|
|
WSS session can exchange command data. Server liveness starts before
|
|
asynchronous recovery, while server readiness remains false until that recovery
|
|
completes. The configured rotating JSON/text service logs are diagnostics, not
|
|
a command-output store; use `rvc` history and the audited storage for command
|
|
evidence.
|
|
|
|
## Backup and restore
|
|
|
|
Stop dispatch before copying data: stop the server gracefully, confirm it is
|
|
down, then copy the entire configured `server.data_dir` tree. That tree contains
|
|
SQLite, WAL/SHM state, retained command segments, audit segments, and incidents;
|
|
backing up SQLite alone is incomplete.
|
|
|
|
To restore, keep the server stopped, move the failed directory aside without
|
|
deleting it, restore the complete backup with ownership restricted to the
|
|
server account, and run `rvbox-server --check-config` followed by a normal
|
|
start. Keep readiness under observation. A failed recovery leaves readiness
|
|
false and records an incident; do not delete segments to force readiness.
|
|
|
|
## Upgrade and rollback
|
|
|
|
1. Record the running image/binary digest and `rvc stat` output.
|
|
2. Stop the server gracefully so no new dispatch is accepted.
|
|
3. Take a full data-directory backup as above.
|
|
4. Install the new image/binary without changing configuration, and run
|
|
`--check-config` before starting it.
|
|
5. Start, wait for `/readyz`, and verify `rvc stat` plus one known client
|
|
reconnect.
|
|
6. If readiness or storage recovery fails, stop, restore the prior binary/image
|
|
and full data directory, then start the known-good version. Preserve logs
|
|
and the failed copy for diagnosis.
|
|
|
|
## Release bundle
|
|
|
|
Build a release candidate only from a clean, committed worktree. The
|
|
containerized release wrapper embeds the supplied version in all three binaries,
|
|
creates an immutable `dist/rvbox-VERSION` directory, and writes `SHA256SUMS`
|
|
plus `manifest.json` only after the Windows executable has optionally been
|
|
signed:
|
|
|
|
```sh
|
|
scripts/release build \
|
|
--version 1.0.0-rc.1 \
|
|
--bootstrap-server-url wss://rvbox.example.test/v1/agent
|
|
(cd dist/rvbox-1.0.0-rc.1 && sha256sum -c SHA256SUMS)
|
|
dist/rvbox-1.0.0-rc.1/rvbox-server-linux-ARCH --version
|
|
dist/rvbox-1.0.0-rc.1/rvc-linux-ARCH --version
|
|
```
|
|
|
|
Replace `ARCH` with `amd64` or `arm64` for the Linux host. Both variants are
|
|
created from the same pinned toolchain and have the same embedded version.
|
|
|
|
For a compact, directly runnable Windows package, add
|
|
`--windows-compression upx` to that command. The default `none` retains the
|
|
ordinary executable. Compact builds use checksum-pinned UPX 5.2.1 with LZMA,
|
|
retain PE relocations, and leave icons/manifest/version resources uncompressed.
|
|
Packaging checks UPX integrity, compares every extracted Windows resource to
|
|
the original, and requires a smaller result before publication. The bundle
|
|
also retains `rvbox-windows-amd64-unpacked.exe`; both variants are included in
|
|
`SHA256SUMS`, and the manifest records `windows_compression`. Optional signing
|
|
runs **after** packing and signs both variants before checksumming. The packer
|
|
runs only inside Docker; release automatically rebuilds an older toolchain
|
|
that lacks it. Existing build-image/cache cleanup and recovery still apply.
|
|
|
|
UPX expands the image in memory at each process start, including service, tray,
|
|
launcher, and signal-helper modes. This trades smaller files/downloads for
|
|
startup work and reduced sharing of executable memory between processes.
|
|
Antivirus products may flag packed executables; use the unpacked fallback if
|
|
that occurs rather than weakening protection. Resource/integrity checks are
|
|
not native execution evidence. Before choosing compressed packages for regular
|
|
distribution, exercise fresh install, upgrade, same-version activation, service
|
|
restart/reboot, tray, and ordinary/elevated command execution on Windows.
|
|
The installer still needs no external extractor or separate resident process.
|
|
See the [UPX manual](https://github.com/upx/upx/blob/v5.2.1/doc/upx.pod) and
|
|
[upstream antivirus issue](https://github.com/upx/upx/issues/437).
|
|
|
|
For a Windows-signed release, provide an executable host-side signing hook via
|
|
`--sign-windows-hook /absolute/path/to/hook`. The wrapper invokes it with the
|
|
Windows executable path and version, then records `windows_signed: true` in the
|
|
manifest. Without that hook the manifest deliberately declares the artifact
|
|
unsigned; this is suitable for CI/test evidence but not a signed public
|
|
release. `--output` is intentionally constrained beneath this repository's
|
|
ignored `dist/` tree so the pinned build container always sees the exact output
|
|
mount. The wrapper never overwrites a final bundle, so correcting a failed
|
|
candidate requires choosing a new version/output or deliberately removing that
|
|
exact ignored `dist/` directory after preserving any evidence.
|
|
|
|
## Common incidents
|
|
|
|
- **Auto-chunking rollout hold:** the negotiated transport implementation is
|
|
intentionally offline at the user's request. Do not upgrade the running
|
|
deployment or replace published installers to enable it yet. See the
|
|
[wire format and isolated checks](transport-chunking.md). It requires both
|
|
endpoints to negotiate support; it does not change logical admission limits.
|
|
- **Reconnect loop after a command produces substantial output:** inspect
|
|
`C:\ProgramData\RVBox\service-startup.log` on Windows and the server log for
|
|
`StatusMessageTooBig` / `read limited at 32769 bytes`. Early v1 transports
|
|
inherited a 32 KiB WebSocket ceiling despite permitting 64 KiB chunks and
|
|
1 MiB envelopes. Both transports now set the configured envelope ceiling
|
|
before reading any messages. Upgrade the server first to drain pending
|
|
client output, then the client for large dispatch/reconciliation responses.
|
|
Preserve spool/history: rebooting cannot remove a replayable oversized event.
|
|
New server logs include bounded read/reconciliation errors and an
|
|
`agent_message_too_big` counter; new clients advertise the actual build version.
|
|
If the server reports receiving that close code from an older client, the
|
|
client rejected a server-to-client frame. Outbound script chunks target
|
|
16 KiB (or a lower configured chunk ceiling), preserving offsets/digests and
|
|
the 1 MiB send window, so pending uploads can reach older clients. This does
|
|
not lower the server's inbound limits. Large inline command text, stdin, or
|
|
reconciliation responses still require the corrected client reader.
|
|
- **Reconnect fails after accumulating command history:** the server used to
|
|
echo the client's entire compact tombstone ledger as terminal-discard UUIDs.
|
|
Roughly 860 UUIDs plus envelope overhead exceed a legacy 32 KiB reader, even
|
|
when the commands produced no large output. The correction validates and
|
|
reconciles tombstones normally, then omits redundant discard instructions;
|
|
it retains history and still authorizes genuinely retained terminal bodies.
|
|
This can be backported independently of held auto-chunking. Keep client
|
|
spool/tombstones intact, back up the full server state before applying the
|
|
isolated fix, and verify both reconnect and a fresh command's terminal result.
|
|
Large genuine action lists still require an updated client reader.
|
|
- **`terminal command cannot accept script data` during reconnect:** the server
|
|
may replay an upload while the client is still delivering that command's
|
|
terminal events. Corrected clients ignore these frames only after verifying
|
|
the command is locally terminal; existing event replay reports the outcome.
|
|
They neither change script bytes nor invoke launch again. The same error for
|
|
a committed but still-live script, unknown owners, and malformed transfers
|
|
remains a failure. Upgrade the client while preserving its spool.
|
|
- **No client / stale session:** verify nginx has WebSocket `101` entries for
|
|
`/v1/agent`, server readiness is true, and the Windows service is running.
|
|
Use `rvc stat CLIENT_ID`; do not restart the client merely to clear history.
|
|
- **Spool or storage full:** `CAPACITY_EXHAUSTED` is intentional admission
|
|
protection. Inspect command retention and free space, allow terminal-age or
|
|
quota rotation to reclaim eligible data, or enlarge the owned filesystem.
|
|
Never manually remove live SQLite/WAL/segment files.
|
|
- **Startup remains unready and control requests time out:** a corrected server
|
|
release is required if payload recovery is waiting for SQLite's sole
|
|
connection while holding a command cursor. Recovery now joins script
|
|
descriptors and payloads in one query and decompresses stored execution
|
|
descriptions before validation. Preserve the existing data directory during
|
|
upgrade; deleting command history is unnecessary.
|
|
- **Repeated `signal command ...: command is not running`:** early v1 clients
|
|
closed their session instead of returning a negative signal result when an
|
|
old queued/replayed signal reached an already-exited process. Upgrade the
|
|
Windows client with the corrected installer, preserving TOML/spool/history.
|
|
The client records `accepted=false` for that signal; the existing server
|
|
acknowledges it and stops replaying it. Do not wipe state or resend the
|
|
original command to clear this loop.
|
|
- **Output truncated:** query command status and output history for its explicit
|
|
loss/truncation markers. The command may still have completed correctly.
|
|
- **Server restart / dirty health:** wait for readiness and inspect the
|
|
incident record. Use the documented repair/acknowledgement controls only
|
|
after preserving evidence; a late client report is authoritative and is not
|
|
rewritten to match an earlier provisional state.
|