Files
rvbox/skills/rvbox/references/troubleshooting.md
T

12 KiB

Troubleshooting

Contents

  • Triage by symptom
  • Windows identity, GUI, and shells
  • Storage, output loss, and recovery

Triage by symptom

Record the deployment route, versions, client ID, issue/request IDs, exact arguments, timestamps, status/event details, and relevant daemon logs. Keep the initial evidence; do not restart services or resubmit work merely to inspect it.

Symptom Check / next action
Local sandbox denies Docker/SSH Use the established authorized execution route. This does not establish a remote outage. Preserve SSH host verification.
Socket missing / connection refused Confirm namespace, server process, explicit socket config, container mounts, and startup logs. A running proxy alone does not prove the server is listening.
Socket permission denied Run as the existing socket owner through the deployment route; inspect directory traversal and mount permissions. Do not chmod the control socket world-accessible.
Client missing or NOT_FOUND List all client pages; verify the exact registered ID and server instance. A hostname assumption or wrong Compose project can select different state. Retention can also remove command history.
Disconnected / OFFLINE / UNAVAILABLE Separate server recovery from client connectivity. Inspect SCM, WSS hostname/path, proxy WebSocket upgrade, certificate/time/CA, DNS, reconnect logs, and pending instance conflicts. Missing optional HTTP health is not proof of a stopped daemon.
StatusMessageTooBig / read limited at 32769 bytes after output or during reconnect Early v1 WebSocket readers kept the library's 32 KiB default despite allowing larger protocol envelopes. Upgrade the server first to accept pending output, then the client for large incoming dispatch/history. Preserve spool/history; rebooting replays the same pending message. Corrected readers use the configured envelope ceiling, normally 1 MiB.
Repeated terminal command cannot accept script data Upload replay can race the finished command's pending event replay. Corrected clients ignore stale script frames for confirmed local terminal commands and let events drain, without modifying script bytes or relaunching. Upgrade the client and retain spool/history. A terminal-script error for a still-live command remains a protocol failure.
Repeated signal command ...: command is not running An older client disconnects on a late signal instead of persisting its rejection. Upgrade the client while retaining config/spool/history; corrected builds acknowledge the signal without disconnecting. Do not wipe state or reissue the original command.
QUEUED persists Check connection, supported shell, queue/running capacity, acceptance events, and queue expiry. Queuing does not promise execution, and queue TTL is not a runtime timeout.
CLI deadline / broken stream Work may still run. Query the captured issue and resume history via the API. For uncertain admission, retry only with the same request ID and payload.
CLI exit 0 but task failed Inspect remote terminal lifecycle, rejection/detail, and output. A successful follow stream is not a successful process.
INVALID_ARGUMENT / unsupported shell Verify argument order, UUIDv7 format, profile composition, explicit shell and advertised support, target CWD, and upload size/hash. Current CLI discards an unknown --shell parse error and can fall back to the server default; use only verified names.
CONFLICT / AlreadyExists Same ID with changed content/target, or a client-instance conflict. Stop retrying unchanged assumptions; inspect the original command or pending claimant. Do not assign a new ID to duplicate an uncertain side effect.
CODE_ALREADY_EXECUTED / reclaimed history Inspect remaining execution/tombstone evidence. Missing full output does not establish that the operation never ran; require a deliberate new operation before resubmission.
PROTOCOL_ERROR / incompatible peers Compare versions and negotiated capabilities, then inspect bounded server/client logs. Preserve IDs and durable records rather than rewriting event sequences or instance state.
REJECTED / context or elevation unavailable Code did not launch. Inspect attempted/effective context and rejection. Fix prerequisites; a new intentional execution needs a new command request ID.
Stdin accepted but no response Check writeSeq, stdin acknowledgement, consumer buffering, newline mode, queue state, EOF, and whether the command is already terminal. There is no PTY.
Signal admitted but process remains Inspect signal-result events and whole-Job lifecycle. TERM can be ignored and escalated; API acknowledgement is not signal delivery or completed termination.
CAPACITY_EXHAUSTED / ResourceExhausted Identify command/client/server quota versus actual filesystem free space. Lower payload/queue pressure, wait for eligible retention, or adjust authorized capacity; repeated retries without a change will not help.
Dirty readiness / output incomplete List incidents, isolate the affected scope, preserve evidence, and use the recovery procedure below. A process can remain live while one scope is unhealthy.

Treat retryable as permission to consider a bounded retry, not proof that immediate repeated calls will succeed. Do not parse error text into invented stable exit-code classes: the current CLI normally exits 1 for its own errors and exposes remote lifecycle separately.

On Windows, SCM startup and reconnect diagnostics are written to %ProgramData%\RVBox\service-startup.log, independently of optional observability logging. Inspect recent entries around the failure before restarting. The tray's log action may select the configured runtime log instead; an empty/missing runtime log does not mean the service recorded no diagnostics. Corrected servers log bounded WebSocket read/reconciliation errors without command payloads, and count agent_message_too_big failures. Earlier clients reported the placeholder daemon version v1; use the installed executable's --version or PE product version when the server cannot identify the release.

If the server log says it received a StatusMessageTooBig close from the client, inspect outbound traffic: the old client reader rejected it. Corrected servers send scripts in 16 KiB chunks, allowing pending uploads to reach older clients without lowering inbound validation limits. Large inline commands, stdin, and reconciliation responses still need the updated client reader.

Windows identity, GUI, and shells

--elevated is the only execution-context request toggle. Selection and fallback happen in the Windows client before launch, never by retrying user code after it starts:

Usable active session Normal request Elevated request
Exactly one active-user active-user-elevated → active-system → local-system
None, or ambiguous multiple candidates local-service local-system

Read windows_requested_elevated, attempted/effective contexts, effective SID, session ID/user SID, and selection detail from command status. active-system is SYSTEM in the active session, not the user's elevated profile. SYSTEM in Session 0 cannot substitute for an interactive desktop. Fallback completes before authorization; a later command failure does not trigger another role.

GUI, audio, display, and per-user shell operations need the intended interactive session and desktop. NTFS ACLs, HKCU/profile, mapped drives, and network identity follow the effective token. A single-owner laptop may have a split admin token, but do not assume it. If a UI does not appear, verify both token/session and window station/desktop; a process listing alone is insufficient.

Builds containing the interactive-desktop fix explicitly target winsta0\default for active-user, active-user-elevated, and active-system. They verify the launcher's actual desktop/access, use a headless command console, and request normal GUI startup visibility. Session 0 routes remain noninteractive. See Windows GUI workflows for a bundled visible-window probe, /wait examples, and limits involving locks, focus, privileges, shell policy, and applications that hand off to an existing process.

Opening folders and Explorer exit code 1

For shell folder opening, use start "" "%USERPROFILE%" through --shell cmd. The first quoted argument is the empty window title. This was verified to open the active user's home and return 0. Direct explorer.exe PATH was observed returning 1 while another Explorer process displayed the requested folder. That raw exit code remains a failed command; do not globally reinterpret 1 as success or assume it always proves a successful handoff. For GUI tasks, inspect the actual window/effect as well. start reports launch, not later application success. A window delegated to another process does not share the launcher's lifetime; killing the completed launch command is not a way to close that window. See Microsoft's start documentation.

PowerShell policy and encoding

The reviewed PowerShell wrapper uses -NoLogo -NoProfile -NonInteractive -File; it does not bypass script policy. A Windows baseline with Restricted policy can therefore reject --shell powershell before user code executes. pwsh selects the same configured shell path; it is not a workaround. Check the configured executable and effective execution policies. For a compatible task, use cmd or an explicit PowerShell -Command invocation; do not silently relax machine or Group Policy settings. A process-scoped policy exception is a deliberate deployment choice, not a change this skill makes automatically.

Use uploaded scripts to avoid layers of quoting. Current Windows wrappers use UTF-8 for cmd and UTF-8 with BOM for PowerShell; cmd's execution/output still depends on its code page. Upload alone does not fix Unicode interpretation. Do not assume Linux quoting or console encoding transfers unchanged. Inspect raw API bytes when text rendering looks corrupt.

Storage, output loss, and recovery

The per-command budget includes compressed command-owned data, not only stdout: scripts, stdin, metadata, events, and retained output also consume it. Separate limits apply per command, per client, and to the server, plus physical free-space floors. Active and otherwise essential state cannot simply be evicted. Eligible terminal commands and their owned data rotate by quota/age; default terminal retention is 30 days. Audit/resolved incident history has a separate budget, normally 100 MiB. Read actual config before treating any default as current.

For a dirty scope:

  1. Inspect rv storage incidents and full API incident data, including scope, data-loss flag, and automatically-repairable flag. Page through all records.
  2. Preserve logs and a consistent copy before an authorized repair or loss acceptance. Healthy scopes can remain usable during scoped recovery.
  3. Request storage repair only for a safe repairable case. Current online resolution is not a general-purpose reconstruction tool. When the loss is understood and authorized, storage acknowledge INCIDENT_ID 'NOTE' records acceptance; it does not recover lost bytes or erase incident history.
  4. Recheck incident state, readiness, and the affected command/output. For an unrepairable or inaccessible store, use the deployment's backup/restore path; do not invent unavailable offline repair flags.

Output truncation/incompleteness and command lifecycle are separate facts. Retention and late client reconciliation can change what evidence remains. Never clear live databases, WALs, client instance state, or segments to suppress an error, make an ID reusable, or pretend a command did not execute.