docs: expand practical test coverage matrix
This commit is contained in:
@@ -189,7 +189,7 @@ developers call rather than duplicating orchestration in workflow YAML:
|
|||||||
scripts/test-unit [--package PATTERN] [--run REGEXP] [--race]
|
scripts/test-unit [--package PATTERN] [--run REGEXP] [--race]
|
||||||
scripts/test-integration [--suite NAME|all] [--run-id ID] [--resume]
|
scripts/test-integration [--suite NAME|all] [--run-id ID] [--resume]
|
||||||
scripts/test-e2e [--scenario NAME|all] [--run-id ID] [--resume]
|
scripts/test-e2e [--scenario NAME|all] [--run-id ID] [--resume]
|
||||||
scripts/test-env doctor|status|logs|collect|recover|reuse|stop|reset|purge|gc ...
|
scripts/test-env doctor|coverage|status|logs|collect|recover|reuse|stop|reset|purge|gc ...
|
||||||
scripts/windows/test-host.ps1 Prepare|Status|Run|Collect|Stop|Reset
|
scripts/windows/test-host.ps1 Prepare|Status|Run|Collect|Stop|Reset
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -286,6 +286,8 @@ and preserves only the manifest plus state required for `status`, `logs`, or
|
|||||||
|
|
||||||
- `doctor [--e2e]` is read-only and validates local/container prerequisites and,
|
- `doctor [--e2e]` is read-only and validates local/container prerequisites and,
|
||||||
when requested, the native Windows runner before allocating a run;
|
when requested, the native Windows runner before allocating a run;
|
||||||
|
- `coverage` is read-only and validates the coverage inventory against listed
|
||||||
|
unit/integration/E2E cases and their required host/layer metadata;
|
||||||
- `status RUN_ID` is read-only and reports steps, health, disk usage, owned
|
- `status RUN_ID` is read-only and reports steps, health, disk usage, owned
|
||||||
Compose/Windows resources, and whether resume is valid;
|
Compose/Windows resources, and whether resume is valid;
|
||||||
- `logs RUN_ID [COMPONENT]` reads bounded/tail output; `collect RUN_ID` writes a
|
- `logs RUN_ID [COMPONENT]` reads bounded/tail output; `collect RUN_ID` writes a
|
||||||
@@ -344,6 +346,425 @@ stable nonzero exit codes, and a final summary giving the run ID, seed, report
|
|||||||
path, resume command when applicable, and exact cleanup command. Document the
|
path, resume command when applicable, and exact cleanup command. Document the
|
||||||
same workflow in `docs/testing.md` during Phase 0.
|
same workflow in `docs/testing.md` during Phase 0.
|
||||||
|
|
||||||
|
### 2.5 Traceable practical-scenario coverage catalogue
|
||||||
|
|
||||||
|
Create `test/coverage.toml` as the executable coverage inventory. Every concrete
|
||||||
|
case has a stable ID, class (`happy`, `hostile-recovery`, or `race`), owning
|
||||||
|
requirement/plan section, layer, suite, supported platforms, privilege/session
|
||||||
|
fixture, speed class, implementation phase, and case name. The ranges below are
|
||||||
|
shorthand; the inventory expands them into one row per case. `scripts/test-env
|
||||||
|
coverage` rejects duplicate IDs, missing case implementations, cases that cannot
|
||||||
|
be listed by their suite, and normative requirements with no test. Test code
|
||||||
|
references the ID in its name or metadata, and failures print it.
|
||||||
|
|
||||||
|
Use line coverage as a warning signal, not as a substitute for scenarios.
|
||||||
|
Nevertheless, pure policy/domain/protocol/config packages must maintain at least
|
||||||
|
90% statement coverage and core store/session/runtime packages at least 80%.
|
||||||
|
Platform syscall wrappers report coverage separately and are judged primarily by
|
||||||
|
native integration cases. Any excluded unreachable/generated/error branch needs
|
||||||
|
a reviewed explanation in the inventory. CI reports coverage changes by package;
|
||||||
|
a new critical branch may not lower coverage merely because the repository-wide
|
||||||
|
percentage remains above threshold.
|
||||||
|
|
||||||
|
For every numeric/time/size limit, generate `minimum`, `minimum+1`, normal,
|
||||||
|
`maximum-1`, `maximum`, and `maximum+1` cases where meaningful, plus zero,
|
||||||
|
negative decoded values where the source type permits them, integer overflow,
|
||||||
|
and unit-conversion boundaries. For every state machine, exhaustively generate
|
||||||
|
all state/event pairs and then add the concurrent interleavings listed below.
|
||||||
|
Pairwise generation covers independent dimensions such as shell, source type,
|
||||||
|
foreground/background, privilege intent, online/offline state, and output kind;
|
||||||
|
use a full Cartesian product only where interactions affect an invariant.
|
||||||
|
|
||||||
|
#### Happy-path cases
|
||||||
|
|
||||||
|
- `HP-CFG-01..12` — Parse every annotated server/client/Windows TOML; apply
|
||||||
|
compiled defaults and explicit flag precedence; normalize Unicode/space-
|
||||||
|
containing absolute paths; validate Unix and Windows shell paths; advertise
|
||||||
|
only current-platform shells; render redacted effective configuration; run
|
||||||
|
`--check-config` without opening listeners/stores; create the first Windows
|
||||||
|
template; start service live-but-not-ready for placeholder routing; restart
|
||||||
|
into ready state after valid configuration; preserve config across upgrade.
|
||||||
|
- `HP-PROTO-01..10` — Negotiate exact and overlapping protocol ranges; ignore
|
||||||
|
unknown optional fields; round-trip every envelope and control message;
|
||||||
|
transfer maximum legal binary output/script chunks; Zstandard round-trip
|
||||||
|
empty/compressible/incompressible data; accept Unicode command/output bytes;
|
||||||
|
preserve client-observed and server-receipt timestamps; encode/decode every
|
||||||
|
structured error and optional field without losing presence; store/forward the
|
||||||
|
single `elevated` request bit unchanged while accepting Windows attempted/
|
||||||
|
effective contexts only as client-originated result metadata.
|
||||||
|
- `HP-CTL-01..18` — List/get clients and commands; run foreground/background
|
||||||
|
command text and scripts; print UUID before following; resume follow; paginate
|
||||||
|
commands and byte slices; select stdout/stderr; append text/raw/file stdin;
|
||||||
|
close stdin; send TERM/KILL successfully to Windows and parse every portable
|
||||||
|
signal name for later Unix delivery; inspect Windows identity,
|
||||||
|
expiry, truncation, incomplete output, incidents, and takeover; repair and
|
||||||
|
acknowledge incidents; exercise matching gRPC and JSON-RPC unary behavior;
|
||||||
|
accept optional global `--request-id` and ignore it for reads.
|
||||||
|
- `HP-IDEM-01..10` — Generate UUIDv7 at the correct owner; accept externally
|
||||||
|
supplied UUIDv7; return the same result for identical run/stdin/close/signal/
|
||||||
|
incident/takeover retries; map run request ID directly to `issue_uuid`; retain
|
||||||
|
one mutation result through transport reconnect; suppress a matching command
|
||||||
|
tombstone replay; preserve UUIDv7 FIFO order for equal-millisecond issuance.
|
||||||
|
- `HP-SES-01..14` — Register a new client; reconnect the same durable instance;
|
||||||
|
fence its old connection; accept a different instance when none is live;
|
||||||
|
authorize and consume an exact live-instance takeover; expire an unused grant;
|
||||||
|
advertise/reconcile capacity; dispatch into running and queued slots; requeue
|
||||||
|
a transient rejection; resume event and script delivery; reconcile active and
|
||||||
|
terminal-unacknowledged records; discard a server-confirmed terminal client
|
||||||
|
record; keep multiple clients fair.
|
||||||
|
- `HP-CLIENT-01..14` — Create and retain one durable client-instance ID; start
|
||||||
|
with an empty spool; durably admit and acknowledge commands; advertise queue/
|
||||||
|
running capacity; dequeue fairly; assign and replay event sequences; compact
|
||||||
|
acknowledged events; retain terminal-unacknowledged records; discard them only
|
||||||
|
after reconciliation/acknowledgement; enforce local command/aggregate quotas;
|
||||||
|
restart with clean recovery; reconnect with saved backoff/session inputs; stop
|
||||||
|
all supervised Jobs on orderly service shutdown.
|
||||||
|
- `HP-CMD-01..18` — Execute `cmd` and PowerShell command text and uploaded
|
||||||
|
scripts; use foreground/background modes; apply requested/omitted CWD;
|
||||||
|
materialize identity-scoped default CWD; overlay an empty/small/large legal
|
||||||
|
environment; handle empty/multiline text, shell metacharacters, PowerShell
|
||||||
|
Unicode, and space/non-ASCII executable/CWD/wrapper paths; return exit codes 0
|
||||||
|
and nonzero; record every public lifecycle; run 1 and configured maximum
|
||||||
|
concurrent commands; queue then start work; execute short and descendant-
|
||||||
|
producing commands; collect resource snapshots; apply every supported
|
||||||
|
individual/composed resource profile before release.
|
||||||
|
- `HP-WINCTX-01..14` — With one active standard user run normal as
|
||||||
|
`ACTIVE_USER`; with a split-token administrator run normal filtered and
|
||||||
|
elevated through the linked full token; accept an already-full admin token;
|
||||||
|
build/verify a restricted medium token when UAC is off; run elevated as
|
||||||
|
`ACTIVE_SYSTEM` after linked-token unavailability; use `LOCAL_SYSTEM` as final
|
||||||
|
elevated fallback; with no active session run normal as `LOCAL_SERVICE` and
|
||||||
|
elevated as `LOCAL_SYSTEM`; persist attempts/effective SIDs/session; omit
|
||||||
|
session fields for Session 0; load/unload active profile; build the effective
|
||||||
|
environment; access ACL-permitted drive, localhost, and network resources.
|
||||||
|
- `HP-LAUNCH-01..16` — Authenticate the per-command launcher pipe; create the
|
||||||
|
launcher and shell suspended; assign the Job before release; persist/flush
|
||||||
|
prepared then authorized state; resume exactly once; capture both streams;
|
||||||
|
write and close stdin; keep descendants in the Job; observe root and complete-
|
||||||
|
tree exit; drain pipes to EOF; deliver CTRL_BREAK through the verified helper;
|
||||||
|
escalate TERM after grace; terminate immediately on KILL; enforce Job limits;
|
||||||
|
remove wrappers/scripts after terminal acknowledgement; preserve effective
|
||||||
|
identity on later lifecycle events.
|
||||||
|
- `HP-OUT-01..14` — Preserve binary/invalid-UTF-8 bytes, empty writes, partial
|
||||||
|
lines, no-newline output, long lines, interleaved stdout/stderr, exact chunk
|
||||||
|
boundaries, cumulative event acknowledgements, reconnect replay, byte-offset
|
||||||
|
pagination within an event, live follow after historical output, normal drain,
|
||||||
|
explicit incomplete marker, and normal client/server compression reuse.
|
||||||
|
- `HP-STORE-01..20` — Initialize/migrate SQLite; append inline and segmented
|
||||||
|
payloads; group commit; reopen cleanly; query stable cursors; charge command/
|
||||||
|
client/server bytes; rotate output window; evict whole oldest terminal UUID;
|
||||||
|
protect active commands; reserve closeout; reclaim terminal age; create and
|
||||||
|
cap tombstones; rotate audit by bytes/age; maintain incident history; run safe
|
||||||
|
repair; acknowledge loss; derive/clear dirty state; checkpoint WAL; backup and
|
||||||
|
restore a quiesced store.
|
||||||
|
- `HP-FLOW-01..12` — Send heartbeat while idle; reset activity on any valid
|
||||||
|
frame/Pong; reset reconnect backoff after stability; keep Ping/Pong/Close and
|
||||||
|
acknowledgements ahead of saturated data; apply fair writer/persistence
|
||||||
|
scheduling; cross high then low watermarks; retain assigned events; summarize
|
||||||
|
unsequenced client loss; use closeout reserve; enforce per-command/session send
|
||||||
|
windows; reconnect with deterministic full jitter.
|
||||||
|
- `HP-SVC-01..16` — Idempotently install/start/stop/restart/uninstall the SCM
|
||||||
|
service; boot before login; preserve data on uninstall; change Automatic/
|
||||||
|
Manual in SCM; start one tray per session from the Run key; reconnect tray
|
||||||
|
after Explorer restart; exit tray without stopping service; read status/log;
|
||||||
|
perform an elevated configuration action; rotate service logs; operate without
|
||||||
|
tray; run Session 0 on Server Core; attach console for human CLI modes; display
|
||||||
|
native error/dialog when no parent console exists.
|
||||||
|
- `HP-OPS-01..12` — Expose liveness during recovery and readiness afterward;
|
||||||
|
keep healthy scopes usable; emit bounded-cardinality metrics; rotate/redact
|
||||||
|
logs and audits; report storage and cache usage; perform orderly shutdown;
|
||||||
|
resume after service/server restart; validate deployment examples; upgrade
|
||||||
|
with config check/snapshot/migration; restore prior version/config on an
|
||||||
|
intentionally failed pre-migration check; generate checksums/package metadata.
|
||||||
|
- `HP-HARNESS-01..14` — Doctor a cold and cached environment; create/list a run;
|
||||||
|
execute one case/scenario; collect JSON/JUnit/report; stop and resume; recover
|
||||||
|
an interrupted controller; reuse immutable definitions into fresh state;
|
||||||
|
reset; purge one run; dry-run and execute age GC; purge all confirmed RVBox
|
||||||
|
runs; retain caches by default; explicitly clear caches/images; preserve file
|
||||||
|
ownership and unrelated labelled fixtures.
|
||||||
|
|
||||||
|
#### Bad, hostile, and abnormal-recovery cases
|
||||||
|
|
||||||
|
- `BH-CFG-01..24` — Reject missing/unknown/duplicate keys, wrong TOML types,
|
||||||
|
invalid UTF-8, bare/overflow/negative durations, illegal zeros, limit and
|
||||||
|
watermark inversions, incompatible profiles, relative paths, missing/wrong-
|
||||||
|
type shell files, changed shell identity before launch, disallowed CWD,
|
||||||
|
insecure state/config/work ownership or ACL, symlink/reparse-point escape,
|
||||||
|
reserved device/ADS/NT-object path misuse, disallowed or inaccessible UNC
|
||||||
|
paths, bad URL/TLS name, listener collision,
|
||||||
|
malformed Windows quoting, and non-loopback JSON-RPC without explicit enable;
|
||||||
|
prove no listener, store mutation, or child starts after static failure.
|
||||||
|
- `BH-PROTO-01..28` — Reject text WebSocket messages, invalid fragmentation,
|
||||||
|
empty/unknown payload, incompatible major/minor, absent/wrong fencing fields,
|
||||||
|
oversize envelope/spec/control/HTTP body/chunk/script, protobuf recursion/
|
||||||
|
length abuse, integer overflow, invalid enum/oneof combinations, malformed
|
||||||
|
UUID/client ID, invalid declared compressed sizes, truncated/corrupt Zstandard,
|
||||||
|
compression bombs, impossible event sequence/revision, changed duplicate
|
||||||
|
payload, out-of-order/overlap/hole script chunks, post-commit chunks, SHA/size
|
||||||
|
mismatch, invalid stdin sequence, and unsupported signal; close/fence only the
|
||||||
|
offending peer and keep daemons alive.
|
||||||
|
- `BH-TLSNET-01..18` — Reject wrong CA/name/expired/not-yet-valid certificate,
|
||||||
|
plaintext endpoint, wrong nginx path/upgrade, and unreachable proxy; recover
|
||||||
|
from DNS refusal, TCP reset, TLS failure, proxy restart, half-open connection,
|
||||||
|
packet loss/duplication/reordering, extreme latency, slow reads/writes,
|
||||||
|
permanently blocked writer, write deadline, missing Pong, clock jump, and
|
||||||
|
reconnect storm without goroutine/file-descriptor growth or data loss.
|
||||||
|
- `BH-CTL-01..24` — Reject malformed/empty/batch JSON-RPC where unsupported,
|
||||||
|
unknown method, invalid params/base64/timestamp, oversized body, tampered or
|
||||||
|
filter-mismatched cursor, page-size overflow, unknown client/command/incident,
|
||||||
|
illegal lifecycle action, stdin after close/terminal, signal before eligible
|
||||||
|
state, takeover for wrong/stale instance, repair of irreparable evidence,
|
||||||
|
acknowledgement without/beyond note bound, reused request ID with changed
|
||||||
|
method/target/content, read-only request ID leakage into domain, and client
|
||||||
|
disconnect during follow; never cancel work merely because the caller exits.
|
||||||
|
- `BH-SES-01..24` — Reject stale/unknown generation, stale output after fencing,
|
||||||
|
live different-instance collision without exact grant, mismatched/expired/
|
||||||
|
consumed grant, capacity lies/overflow, dispatch to wrong generation,
|
||||||
|
permanent client rejection, reconnect snapshot missing accepted/running state,
|
||||||
|
matching tombstone against stale server state, hash/revision/terminal
|
||||||
|
contradiction, incomplete reconciliation snapshot, client dirty-store claim,
|
||||||
|
lost/repeated result, offline target, queue capacity exhaustion, and one noisy/
|
||||||
|
corrupt client; isolate the scope and keep unrelated clients/control responsive.
|
||||||
|
- `BH-CLIENT-01..28` — Handle missing/corrupt/permission-denied client-instance
|
||||||
|
identity, duplicate daemon lock, spool schema/checksum/segment corruption,
|
||||||
|
uncommitted spool tail, missing acknowledged bytes, stale accepted record,
|
||||||
|
incomplete prepared/authorized process metadata, unknown live process, local
|
||||||
|
queue full, per-command/aggregate spool full, client disk floor, script temp
|
||||||
|
orphan, tombstone overflow, event-sequence counter drift, impossible server Ack,
|
||||||
|
dirty local storage, server rejecting registration/version, repeated connect
|
||||||
|
refusal, shutdown deadline, service kill, and config/shell change across
|
||||||
|
restart. Do not claim a complete reconciliation snapshot or accept work until
|
||||||
|
essential local state is trustworthy or explicitly resolved.
|
||||||
|
- `BH-IDEM-01..16` — Reject noncanonical/non-v7 IDs, all-zero/wrong-length binary
|
||||||
|
forms, same UUID with changed immutable execution spec/script bytes, same
|
||||||
|
request ID across mutation kinds/targets, duplicate sequence with changed
|
||||||
|
bytes, and replay older than full history but still in tombstones; define and
|
||||||
|
test best-effort behavior after tombstone horizon expiry; never create a second
|
||||||
|
process or mutation result on any retained duplicate.
|
||||||
|
- `BH-SCRIPT-01..18` — Treat filename as display only; reject path separators,
|
||||||
|
traversal, reserved names, NUL/control abuse, excessive metadata, changed
|
||||||
|
replay bytes, sparse/overlapping writes, premature commit, digest mismatch,
|
||||||
|
quota exhaustion, disk-full, permission/share violation, wrapper replacement,
|
||||||
|
symlink/reparse race, shell executable replacement, and cancellation during
|
||||||
|
upload; clean only owned temporary files and preserve durable rejection detail.
|
||||||
|
- `BH-WINCTX-01..26` — Handle no linked token, standard user elevation request,
|
||||||
|
Administrator Protection/approval-only policy, disabled/removed account,
|
||||||
|
locked/disconnected/logging-off session, `WTSQueryUserToken` failure, multiple
|
||||||
|
ambiguous active sessions, console-session change, session-ID reuse with a new
|
||||||
|
logon SID, token SID/session/integrity/type mismatch, restricted-token creation
|
||||||
|
failure, profile load/unload failure, environment block failure, LocalService
|
||||||
|
logon failure, `TokenSessionId` failure, active-SYSTEM failure, final LocalSystem
|
||||||
|
failure, explicit CWD denial, identity-work-root tampering, absent mapped drive/
|
||||||
|
HKCU/network credentials, and Session 0 interactive API failure. Follow only
|
||||||
|
the allowed pre-launch chain and emit a bounded decision record with no
|
||||||
|
effective context if exhausted.
|
||||||
|
- `BH-LAUNCH-01..34` — Reject spoofed/remote/early launcher or signal-helper pipe
|
||||||
|
clients, wrong PID/creation time/SID/session/generation/checksum/length,
|
||||||
|
inherited unrelated handles, launcher outside Job, breakaway attempt, shell/
|
||||||
|
wrapper/CWD/profile validation failure, invalid or case-colliding Windows
|
||||||
|
environment keys, oversized environment block, unencodable wrapper content,
|
||||||
|
process creation/assignment/resume failure, helper attach/delivery refusal,
|
||||||
|
PID reuse, root disappearance, child
|
||||||
|
holding pipes forever, descendant escape attempt, output read/write error,
|
||||||
|
stdin broken pipe, Job accounting failure, unsupported Job limit, service death
|
||||||
|
before/after preparation/authorization, launcher death at every handshake, and
|
||||||
|
incomplete EOF. Before authorization reject/cancel safely; after durable
|
||||||
|
authorization terminate/interrupt and never redispatch.
|
||||||
|
- `BH-OUTFLOW-01..24` — Survive output flood from one/many commands, tiny writes,
|
||||||
|
incompressible data, invalid bytes, compressor error, client raw backlog,
|
||||||
|
offline spool full, pinned send window full, server ingress full, command/
|
||||||
|
client/global quota full, physical disk floor, loss-marker reservation
|
||||||
|
pressure, missing EOF, corrupt spool record, repeated Ack, Ack beyond sent
|
||||||
|
sequence, server retention removing resume point, and slow follower. Preserve
|
||||||
|
lifecycle/control progress and expose every loss/incomplete range honestly.
|
||||||
|
- `BH-STORE-01..36` — Recover uncommitted tails and marked eviction; detect short
|
||||||
|
committed files, checksum/frame corruption, missing segment, wrong file type,
|
||||||
|
path replacement, SQLite corruption/lock/busy/readonly, WAL/shm loss, schema or
|
||||||
|
migration checksum mismatch, partial migration, fsync/rename/directory-sync
|
||||||
|
failure, ENOSPC at every append/closeout/audit phase, counter/reservation drift,
|
||||||
|
orphan temp/tombstone files, interrupted audit/age rotation, tombstone cap,
|
||||||
|
filesystem free-space breach, permission change, backup inconsistency, and
|
||||||
|
restore mismatch. Gate only affected scopes, expose incidents promptly, and
|
||||||
|
require acknowledgement before clearing known loss.
|
||||||
|
- `BH-RET-01..20` — Reject new command when its own cap cannot fit; reject one
|
||||||
|
client/global admission when active/non-evictable data alone fills the tier;
|
||||||
|
evict terminal commands with no output; charge scripts/stdin/metadata/history/
|
||||||
|
idempotency data; never partly evict a command; preserve active commands,
|
||||||
|
tombstones, audit, unresolved incidents, assigned client events, and closeout
|
||||||
|
reserve; handle no eligible victim; retain truncation metadata; apply 30-day
|
||||||
|
and independent audit age/byte rotation exactly once.
|
||||||
|
- `BH-SVC-01..28` — Reject non-admin install/config/uninstall, untrusted binary/
|
||||||
|
config path, direct internal service/launcher/helper mode, wrong SCM launch,
|
||||||
|
duplicate/corrupt service registration, insecure ProgramData ACL, malicious
|
||||||
|
tray pipe client, spoofed admin claim, cross-session tray action, oversized/
|
||||||
|
stalled IPC, second tray mutex, config/log path substitution, Run-key quoting
|
||||||
|
attack, startup config invalidity, network unavailable at boot, tray absent/
|
||||||
|
crashed, Explorer absent, service start timeout, forced stop with active Jobs,
|
||||||
|
uninstall interruption, log disk-full/rotation sharing violation, and Server
|
||||||
|
Core without GUI. Keep service live/not-ready where designed and never let tray
|
||||||
|
failure own daemon state.
|
||||||
|
- `BH-SEC-01..22` — Exercise command/script path traversal, SQL metacharacters,
|
||||||
|
log/terminal escape injection, Unicode confusables in IDs, environment-value
|
||||||
|
redaction, secret-looking payloads in metrics/logs/audits, malicious PATH/
|
||||||
|
COMSPEC/PATHEXT/file association, unsafe executable replacement, named-pipe ACL
|
||||||
|
bypass, remote pipe access, inherited-handle leakage, decompression allocation
|
||||||
|
abuse, cursor forgery, JSON-RPC bind warning, unauthenticated JSON-RPC authority
|
||||||
|
as the explicit v1 behavior, and least-access state/work/log ACLs. Tests assert
|
||||||
|
the documented insecure boundary rather than pretending authentication exists.
|
||||||
|
- `BH-OPS-01..20` — Handle asynchronous recovery timeout, one dirty scope, metrics
|
||||||
|
scrape during churn, log sink failure, full audit budget, invalid health path,
|
||||||
|
shutdown deadline, kill during shutdown, restart loop, incompatible/failed
|
||||||
|
upgrade, downgrade attempt, backup destination full, restore with wrong config,
|
||||||
|
clock rollback/forward/DST, hostname/client-ID change, and missing OS diagnostic
|
||||||
|
data. Liveness stays meaningful, readiness stays conservative, and absence is
|
||||||
|
never fabricated as zero/healthy data.
|
||||||
|
- `BH-HARNESS-01..26` — Reject invalid/traversal run ID, corrupt/truncated/foreign
|
||||||
|
manifest, commit/image/config/VM mismatch, live lock stealing, stale lock with
|
||||||
|
live owner, unlabelled or wrong-repository Docker resource, symlinked run root,
|
||||||
|
path outside `.test-runs`, Windows resource outside test root, secret in report,
|
||||||
|
oversized artifact, no disk, unavailable image/runner, port collision, lost
|
||||||
|
runner, interrupted setup/collect/cleanup, failed Docker/SCM removal, unsafe
|
||||||
|
resume, reuse attempting mutable-state copy, GC of resumable run, purge without
|
||||||
|
confirmation, and `--all` with unrelated resources. Fail closed and print an
|
||||||
|
exact recovery/cleanup instruction without invoking global prune.
|
||||||
|
|
||||||
|
#### Obscure race, crash-window, and interleaving cases
|
||||||
|
|
||||||
|
All race cases run under deterministic schedule hooks first, then selected cases
|
||||||
|
run repeatedly with the Go race detector or native process concurrency. A case
|
||||||
|
must assert durable state, process count/identity, quota counters, emitted event
|
||||||
|
order, and leaked goroutine/handle/file counts—not only the CLI exit status.
|
||||||
|
|
||||||
|
- `RC-DOM-01..18` — Concurrent identical/conflicting command creation; mutation
|
||||||
|
retry while original commits; cancel versus queue dispatch, acceptance,
|
||||||
|
preparation, authorization, running, and terminal commit; signal versus cancel/
|
||||||
|
terminal; stdin write versus close/terminal; late resource snapshot versus
|
||||||
|
terminal; duplicate terminal events; command revision increments from multiple
|
||||||
|
controllers; UUIDv7 generation in one millisecond and across clock rollback.
|
||||||
|
- `RC-SES-01..20` — Same-instance reconnects cross; old read/write loops race
|
||||||
|
fencing; different-instance reconnect races grant creation/expiry/consumption;
|
||||||
|
session closes while dispatch reserves capacity; capacity update crosses
|
||||||
|
acceptance/rejection; reconnect crosses queue expiry; stale output arrives
|
||||||
|
before/after new generation commit; reconciliation result is lost while new
|
||||||
|
dispatch wakes; two clients contend for global fairness; heartbeat timeout
|
||||||
|
crosses Pong/read activity/write deadline/server shutdown.
|
||||||
|
- `RC-CLIENT-01..22` — Duplicate dispatch crosses durable local admission;
|
||||||
|
acceptance acknowledgement crosses service crash; queue dequeue crosses cancel/
|
||||||
|
shutdown; output/event sequence assignment crosses Ack/reconnect/compaction;
|
||||||
|
terminal acknowledgement crosses local cleanup/tombstone insertion; server
|
||||||
|
discard result crosses retransmission; local quota reservation crosses script/
|
||||||
|
stdin/output writes; reconnect crosses config restart; identity/spool recovery
|
||||||
|
crosses connection startup; two commands become runnable as capacity changes;
|
||||||
|
stop-all crosses a new prepared Job. Preserve at-most-once execution, monotonic
|
||||||
|
event order, and accurate advertised capacity.
|
||||||
|
- `RC-STORE-01..28` — Event append races Ack/query/retention; terminal commit
|
||||||
|
races output drain and whole-command eviction; eviction races pagination,
|
||||||
|
follow, idempotency retry, and age rotation; tombstone insertion races full-
|
||||||
|
record deletion/replay; audit rotation races incident resolution; repair races
|
||||||
|
startup recovery/retention/backup; command/client/global reservations cross
|
||||||
|
exact caps concurrently; closeout consumes its reserve while disk floor trips;
|
||||||
|
group commit crosses process kill; WAL checkpoint crosses readers/shutdown;
|
||||||
|
two failures create incidents for one scope without reopening resolved history.
|
||||||
|
- `RC-OUT-01..20` — stdout/stderr readers race root exit/descendant exit; child
|
||||||
|
inherits pipe while grace expires; raw backlog crosses high/low repeatedly;
|
||||||
|
compressor completion reorders across commands but not within one command;
|
||||||
|
Ack arrives during reconnect/spool compaction; output assignment races client
|
||||||
|
loss conversion; server retention marker races live follow; terminal event
|
||||||
|
races final output/incomplete marker; slow follower disconnects while page read
|
||||||
|
crosses segment eviction.
|
||||||
|
- `RC-SCRIPT-01..12` — Duplicate chunks arrive concurrently; commit crosses last
|
||||||
|
fsync, cancellation, quota eviction pressure, client restart, digest failure,
|
||||||
|
and launch eligibility; cleanup crosses terminal acknowledgement/service death;
|
||||||
|
wrapper identity is swapped between validation and creation/revalidation;
|
||||||
|
identical upload resumes from competing old/new sessions.
|
||||||
|
- `RC-WINCTX-01..18` — User logs on/off, locks/unlocks, disconnects/reconnects,
|
||||||
|
or changes console session during enumeration/token construction/revalidation;
|
||||||
|
session ID is reused with a different logon SID; multiple RDP sessions become
|
||||||
|
active/ambiguous; linked-token policy changes; profile unload races Job exit;
|
||||||
|
active-user CWD disappears or ACL changes before release. Re-enumerate at most
|
||||||
|
once as specified, never retarget silently, and never fall back after
|
||||||
|
`launch_prepared`.
|
||||||
|
- `RC-LAUNCH-01..30` — Cancellation/service stop/client crash at every instruction
|
||||||
|
boundary among durable acceptance, token selection, Job/pipe creation,
|
||||||
|
suspended launcher creation, Job assignment, pipe authentication, suspended
|
||||||
|
shell report, prepared fsync, authorized fsync, release send, resume, running
|
||||||
|
acknowledgement, root exit, drain, terminal fsync, EventAck, cleanup, and
|
||||||
|
tombstone insertion. Duplicate dispatch after each restart must yield zero or
|
||||||
|
one process, never two.
|
||||||
|
- `RC-SIGNAL-01..16` — TERM/KILL/cancel collide with launcher connection, shell
|
||||||
|
resume, root exit, descendant-only Job, helper pipe authentication,
|
||||||
|
`AttachConsole`, CTRL_BREAK delivery, grace timeout, Job termination, service
|
||||||
|
shutdown, PID reuse, and repeated signal revision. Record the actual attempted/
|
||||||
|
escalated result and never signal an unrelated PID/console.
|
||||||
|
- `RC-SVC-01..20` — SCM start/stop/restart/uninstall cross recovery, command
|
||||||
|
admission, prepared launch, and active Jobs; two installers/configurators race;
|
||||||
|
tray startup crosses logon/logoff/Explorer restart/service restart; multiple
|
||||||
|
trays contend for the per-session mutex; pipe disconnect crosses a mutating
|
||||||
|
helper result; log rotation crosses tray open/read; config edit crosses explicit
|
||||||
|
restart; Automatic/Manual change crosses SCM query.
|
||||||
|
- `RC-OPS-01..14` — Health/metrics/read RPCs cross recovery state transitions,
|
||||||
|
dirty resolution, retention, and shutdown; log/audit rotation crosses process
|
||||||
|
crash; backup crosses group commit and dispatch pause; upgrade crosses queued/
|
||||||
|
active commands and rollback; wall-clock changes cross queue TTL/retention while
|
||||||
|
monotonic heartbeat remains correct.
|
||||||
|
- `RC-HARNESS-01..18` — Controller dies before/after manifest fsync, resource
|
||||||
|
creation/label recording, Windows lease, fault checkpoint, result write,
|
||||||
|
artifact collection, stop/reset/purge, and lock release; two cleanup commands
|
||||||
|
race; status/collect runs during cleanup; resume crosses a late old controller;
|
||||||
|
GC crosses a newly resumed run. Recovery must converge idempotently and never
|
||||||
|
delete an unrelated resource.
|
||||||
|
|
||||||
|
For each crash-window family, implement a loop that enumerates named hooks rather
|
||||||
|
than hand-selecting a few attractive points. Persist the hook name before
|
||||||
|
triggering death, restart from the real store/spool, replay the same external
|
||||||
|
request, and check the invariant matrix. A newly added durable write or external
|
||||||
|
side effect must add a hook and coverage row before merging.
|
||||||
|
|
||||||
|
### 2.6 Native Windows test-host requirements
|
||||||
|
|
||||||
|
Native Windows is mandatory for the Phase 4/5 gates. Development can begin with
|
||||||
|
unit tests and cross-compilation before a host is connected, but the Windows
|
||||||
|
supervisor/service/tray implementation cannot be called complete without it.
|
||||||
|
Prefer disposable or snapshot-resettable VMs over a personal workstation.
|
||||||
|
|
||||||
|
Provision at least one primary interactive host with:
|
||||||
|
|
||||||
|
- current Windows 11 Pro/Enterprise, Desktop Experience, Explorer, `cmd.exe`,
|
||||||
|
Windows PowerShell 5.1, UAC enabled, and all current updates captured in a
|
||||||
|
named clean snapshot;
|
||||||
|
- at least 4 vCPU, 8 GiB RAM, and 60 GiB free disk recommended (2 vCPU, 4 GiB,
|
||||||
|
and 30 GiB free is the minimum smoke lane), with reboot and snapshot-revert
|
||||||
|
authority;
|
||||||
|
- one local standard account and one traditional split-token local administrator,
|
||||||
|
plus a separate automation principal able to install/control the test service;
|
||||||
|
- a preconfigured CI runner, OpenSSH, or WinRM management channel reachable only
|
||||||
|
from the test controller; credentials live outside the repository and run
|
||||||
|
manifest;
|
||||||
|
- bidirectional reachability to the Linux nginx test endpoint, stable DNS or a
|
||||||
|
supplied address, time synchronization, and permission to transfer the
|
||||||
|
CI-built binary/config/CA into a dedicated test root;
|
||||||
|
- no valuable user data, credentials, mapped drives, or production services,
|
||||||
|
because tests deliberately change SCM/Run-key state, ACLs, UAC fixtures, logon
|
||||||
|
sessions, kill processes, fill bounded scratch storage, reboot, and revert.
|
||||||
|
|
||||||
|
Maintain snapshot/policy variants for logged-out, standard-user active,
|
||||||
|
split-token-admin active, UAC disabled/already-full admin, and Administrator
|
||||||
|
Protection when supported. The harness must restore the baseline after any
|
||||||
|
variant that changes machine policy. A Windows Server Desktop Experience VM with
|
||||||
|
multiple simultaneous `WTSActive` sessions is required for the final native
|
||||||
|
ambiguous-session case; until available, keep its exhaustive selector unit test
|
||||||
|
mandatory and mark only that native case blocked. Add a Server Core VM for the
|
||||||
|
headless Session 0 release smoke. Before publishing, also exercise the oldest
|
||||||
|
supported Windows 10 or Server 2016 baseline; it need not be the everyday runner.
|
||||||
|
|
||||||
|
A physical Windows machine is optional. It is useful for an additional real
|
||||||
|
display/audio/DDC command smoke, but RVBox only guarantees correct token/session/
|
||||||
|
process execution—not success of arbitrary vendor hardware APIs—so physical
|
||||||
|
hardware is not a release blocker. `test-host.ps1 Prepare` must inventory the
|
||||||
|
host against this checklist and refuse destructive suites unless the machine is
|
||||||
|
explicitly marked disposable/resettable and the clean snapshot identity is
|
||||||
|
recorded.
|
||||||
|
|
||||||
## 3. Repository and build bootstrap (Phase 0)
|
## 3. Repository and build bootstrap (Phase 0)
|
||||||
|
|
||||||
### 3.1 Establish the repository layout
|
### 3.1 Establish the repository layout
|
||||||
@@ -384,6 +805,7 @@ scripts/
|
|||||||
test-env # doctor/recover/reuse/cleanup by exact run ID
|
test-env # doctor/recover/reuse/cleanup by exact run ID
|
||||||
windows/test-host.ps1 # native Windows host lifecycle adapter
|
windows/test-host.ps1 # native Windows host lifecycle adapter
|
||||||
test/
|
test/
|
||||||
|
coverage.toml # requirement-to-case inventory with stable IDs
|
||||||
harness/ # shared manifest, journal, orchestration, and reporting
|
harness/ # shared manifest, journal, orchestration, and reporting
|
||||||
defaults.toml # local resource/time/artifact budgets
|
defaults.toml # local resource/time/artifact budgets
|
||||||
integration/ # cross-package/real-resource suite definitions
|
integration/ # cross-package/real-resource suite definitions
|
||||||
@@ -425,8 +847,8 @@ Add:
|
|||||||
resource limits, run-ID labels, ephemeral PKI, fault proxy, and artifact/state
|
resource limits, run-ID labels, ephemeral PKI, fault proxy, and artifact/state
|
||||||
mounts. It accepts only values produced by the validated harness manifest and
|
mounts. It accepts only values produced by the validated harness manifest and
|
||||||
never uses an implicit default project name.
|
never uses an implicit default project name.
|
||||||
- `Makefile`: `generate`, `fmt`, `lint`, `test`, `test-race`, `test-integration`,
|
- `Makefile`: `generate`, `fmt`, `lint`, `test`, `test-race`, `test-coverage`,
|
||||||
`test-e2e`, `test-status`, `build`, and `verify`. `test` aliases the unit
|
`test-integration`, `test-e2e`, `test-status`, `build`, and `verify`. `test` aliases the unit
|
||||||
layer; integration/E2E targets delegate to the checked-in scripts and print
|
layer; integration/E2E targets delegate to the checked-in scripts and print
|
||||||
their run ID. Each target invokes the container workflow and must not silently
|
their run ID. Each target invokes the container workflow and must not silently
|
||||||
fall back to host tools.
|
fall back to host tools.
|
||||||
@@ -451,11 +873,13 @@ Add CI (or a repository script ready for CI) that runs, in order:
|
|||||||
v1 compatibility baseline; enable `buf breaking` against that baseline
|
v1 compatibility baseline; enable `buf breaking` against that baseline
|
||||||
immediately after it merges, not against the obsolete draft.
|
immediately after it merges, not against the obsolete draft.
|
||||||
2. Generation freshness check.
|
2. Generation freshness check.
|
||||||
3. `go fmt`, `go vet`, static analysis, and unit tests.
|
3. Coverage-inventory validation: every required stable ID maps to a listed
|
||||||
4. Race tests for server/client concurrency packages.
|
case and every normative critical invariant has at least one case.
|
||||||
5. Linux server/storage/session integration tests in Compose; no Linux client
|
4. `go fmt`, `go vet`, static analysis, and unit tests with per-package coverage.
|
||||||
|
5. Race tests for server/client concurrency packages.
|
||||||
|
6. Linux server/storage/session integration tests in Compose; no Linux client
|
||||||
supervisor is required at this stage.
|
supervisor is required at this stage.
|
||||||
6. Cross-compilation checks for Windows packages in the container plus native
|
7. Cross-compilation checks for Windows packages in the container plus native
|
||||||
Windows build/runtime smoke tests on a Windows runner. A Windows runner is a
|
Windows build/runtime smoke tests on a Windows runner. A Windows runner is a
|
||||||
Phase 0 prerequisite, not a later optional enhancement.
|
Phase 0 prerequisite, not a later optional enhancement.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user