12 KiB
Troubleshooting
Contents
- Triage by symptom
- Windows identity, GUI, and shells
- Storage, output loss, and recovery
Triage by symptom
Record the deployment route, versions, client ID, issue/request IDs, exact arguments, timestamps, status/event details, and relevant daemon logs. Keep the initial evidence; do not restart services or resubmit work merely to inspect it.
| Symptom | Check / next action |
|---|---|
| Local sandbox denies Docker/SSH | Use the established authorized execution route. This does not establish a remote outage. Preserve SSH host verification. |
| Socket missing / connection refused | Confirm namespace, server process, explicit socket config, container mounts, and startup logs. A running proxy alone does not prove the server is listening. |
| Socket permission denied | Run as the existing socket owner through the deployment route; inspect directory traversal and mount permissions. Do not chmod the control socket world-accessible. |
| Client missing or NOT_FOUND | List all client pages; verify the exact registered ID and server instance. A hostname assumption or wrong Compose project can select different state. Retention can also remove command history. |
| Disconnected / OFFLINE / UNAVAILABLE | Separate server recovery from client connectivity. Inspect SCM, WSS hostname/path, proxy WebSocket upgrade, certificate/time/CA, DNS, reconnect logs, and pending instance conflicts. Missing optional HTTP health is not proof of a stopped daemon. |
StatusMessageTooBig / read limited at 32769 bytes after output or during reconnect |
Early v1 WebSocket readers kept the library's 32 KiB default despite allowing larger protocol envelopes. Upgrade the server first to accept pending output, then the client for large incoming dispatch/history. Preserve spool/history; rebooting replays the same pending message. Corrected readers use the configured envelope ceiling, normally 1 MiB. |
Repeated terminal command cannot accept script data |
Upload replay can race the finished command's pending event replay. Corrected clients ignore stale script frames for confirmed local terminal commands and let events drain, without modifying script bytes or relaunching. Upgrade the client and retain spool/history. A terminal-script error for a still-live command remains a protocol failure. |
Repeated signal command ...: command is not running |
An older client disconnects on a late signal instead of persisting its rejection. Upgrade the client while retaining config/spool/history; corrected builds acknowledge the signal without disconnecting. Do not wipe state or reissue the original command. |
| QUEUED persists | Check connection, supported shell, queue/running capacity, acceptance events, and queue expiry. Queuing does not promise execution, and queue TTL is not a runtime timeout. |
| CLI deadline / broken stream | Work may still run. Query the captured issue and resume history via the API. For uncertain admission, retry only with the same request ID and payload. |
| CLI exit 0 but task failed | Inspect remote terminal lifecycle, rejection/detail, and output. A successful follow stream is not a successful process. |
| INVALID_ARGUMENT / unsupported shell | Verify argument order, UUIDv7 format, profile composition, explicit shell and advertised support, target CWD, and upload size/hash. Current CLI discards an unknown --shell parse error and can fall back to the server default; use only verified names. |
| CONFLICT / AlreadyExists | Same ID with changed content/target, or a client-instance conflict. Stop retrying unchanged assumptions; inspect the original command or pending claimant. Do not assign a new ID to duplicate an uncertain side effect. |
| CODE_ALREADY_EXECUTED / reclaimed history | Inspect remaining execution/tombstone evidence. Missing full output does not establish that the operation never ran; require a deliberate new operation before resubmission. |
| PROTOCOL_ERROR / incompatible peers | Compare versions and negotiated capabilities, then inspect bounded server/client logs. Preserve IDs and durable records rather than rewriting event sequences or instance state. |
| REJECTED / context or elevation unavailable | Code did not launch. Inspect attempted/effective context and rejection. Fix prerequisites; a new intentional execution needs a new command request ID. |
| Stdin accepted but no response | Check writeSeq, stdin acknowledgement, consumer buffering, newline mode, queue state, EOF, and whether the command is already terminal. There is no PTY. |
| Signal admitted but process remains | Inspect signal-result events and whole-Job lifecycle. TERM can be ignored and escalated; API acknowledgement is not signal delivery or completed termination. |
| CAPACITY_EXHAUSTED / ResourceExhausted | Identify command/client/server quota versus actual filesystem free space. Lower payload/queue pressure, wait for eligible retention, or adjust authorized capacity; repeated retries without a change will not help. |
| Dirty readiness / output incomplete | List incidents, isolate the affected scope, preserve evidence, and use the recovery procedure below. A process can remain live while one scope is unhealthy. |
Treat retryable as permission to consider a bounded retry, not proof that
immediate repeated calls will succeed. Do not parse error text into invented
stable exit-code classes: the current CLI normally exits 1 for its own errors
and exposes remote lifecycle separately.
On Windows, SCM startup and reconnect diagnostics are written to
%ProgramData%\RVBox\service-startup.log, independently of optional
observability logging. Inspect recent entries around the failure before
restarting. The tray's log action may select the configured runtime log instead;
an empty/missing runtime log does not mean the service recorded no diagnostics.
Corrected servers log bounded WebSocket read/reconciliation errors without
command payloads, and count agent_message_too_big failures. Earlier clients
reported the placeholder daemon version v1; use the installed executable's
--version or PE product version when the server cannot identify the release.
If the server log says it received a StatusMessageTooBig close from the
client, inspect outbound traffic: the old client reader rejected it. Corrected
servers send scripts in 16 KiB chunks, allowing pending uploads to reach older
clients without lowering inbound validation limits. Large inline commands,
stdin, and reconciliation responses still need the updated client reader.
Windows identity, GUI, and shells
--elevated is the only execution-context request toggle. Selection and
fallback happen in the Windows client before launch, never by retrying user
code after it starts:
| Usable active session | Normal request | Elevated request |
|---|---|---|
| Exactly one | active-user | active-user-elevated → active-system → local-system |
| None, or ambiguous multiple candidates | local-service | local-system |
Read windows_requested_elevated, attempted/effective contexts, effective SID,
session ID/user SID, and selection detail from command status. active-system
is SYSTEM in the active session, not the user's elevated profile. SYSTEM in
Session 0 cannot substitute for an interactive desktop. Fallback completes
before authorization; a later command failure does not trigger another role.
GUI, audio, display, and per-user shell operations need the intended interactive session and desktop. NTFS ACLs, HKCU/profile, mapped drives, and network identity follow the effective token. A single-owner laptop may have a split admin token, but do not assume it. If a UI does not appear, verify both token/session and window station/desktop; a process listing alone is insufficient.
Builds containing the interactive-desktop fix explicitly target
winsta0\default for active-user, active-user-elevated, and active-system.
They verify the launcher's actual desktop/access, use a headless command console,
and request normal GUI startup visibility. Session 0 routes remain
noninteractive. See Windows GUI workflows for a bundled
visible-window probe, /wait examples, and limits involving locks, focus,
privileges, shell policy, and applications that hand off to an existing process.
Opening folders and Explorer exit code 1
For shell folder opening, use start "" "%USERPROFILE%" through --shell cmd.
The first quoted argument is the empty window title. This was verified to open
the active user's home and return 0. Direct explorer.exe PATH was observed
returning 1 while another Explorer process displayed the requested folder.
That raw exit code remains a failed command; do not globally reinterpret 1 as
success or assume it always proves a successful handoff. For GUI tasks, inspect
the actual window/effect as well. start reports launch, not later application
success. A window delegated to another process does not share the launcher's
lifetime; killing the completed launch command is not a way to close that
window. See Microsoft's start documentation.
PowerShell policy and encoding
The reviewed PowerShell wrapper uses -NoLogo -NoProfile -NonInteractive -File;
it does not bypass script policy. A Windows baseline with Restricted policy can
therefore reject --shell powershell before user code executes. pwsh selects
the same configured shell path; it is not a workaround. Check the configured
executable and effective execution policies. For a compatible task, use cmd or
an explicit PowerShell -Command invocation; do not silently relax machine or
Group Policy settings. A process-scoped policy exception is a deliberate
deployment choice, not a change this skill makes automatically.
Use uploaded scripts to avoid layers of quoting. Current Windows wrappers use UTF-8 for cmd and UTF-8 with BOM for PowerShell; cmd's execution/output still depends on its code page. Upload alone does not fix Unicode interpretation. Do not assume Linux quoting or console encoding transfers unchanged. Inspect raw API bytes when text rendering looks corrupt.
Storage, output loss, and recovery
The per-command budget includes compressed command-owned data, not only stdout: scripts, stdin, metadata, events, and retained output also consume it. Separate limits apply per command, per client, and to the server, plus physical free-space floors. Active and otherwise essential state cannot simply be evicted. Eligible terminal commands and their owned data rotate by quota/age; default terminal retention is 30 days. Audit/resolved incident history has a separate budget, normally 100 MiB. Read actual config before treating any default as current.
For a dirty scope:
- Inspect
rv storage incidentsand full API incident data, including scope, data-loss flag, and automatically-repairable flag. Page through all records. - Preserve logs and a consistent copy before an authorized repair or loss acceptance. Healthy scopes can remain usable during scoped recovery.
- Request
storage repaironly for a safe repairable case. Current online resolution is not a general-purpose reconstruction tool. When the loss is understood and authorized,storage acknowledge INCIDENT_ID 'NOTE'records acceptance; it does not recover lost bytes or erase incident history. - Recheck incident state, readiness, and the affected command/output. For an unrepairable or inaccessible store, use the deployment's backup/restore path; do not invent unavailable offline repair flags.
Output truncation/incompleteness and command lifecycle are separate facts. Retention and late client reconciliation can change what evidence remains. Never clear live databases, WALs, client instance state, or segments to suppress an error, make an ID reusable, or pretend a command did not execute.