test: add reproducible Windows VM bundle harness
This commit is contained in:
@@ -190,7 +190,9 @@ scripts/test-unit [--package PATTERN] [--run REGEXP] [--race]
|
||||
scripts/test-integration [--suite NAME|all] [--run-id ID] [--resume]
|
||||
scripts/test-e2e [--scenario NAME|all] [--run-id ID] [--resume]
|
||||
scripts/test-env doctor|coverage|status|logs|collect|recover|reuse|stop|reset|purge|gc ...
|
||||
scripts/windows/test-host.ps1 Prepare|Status|Run|Collect|Stop|Reset
|
||||
scripts/build build|verify (pinned toolchain; host-safe)
|
||||
scripts/windows/build-test-bundle --run-id ID --config FILE [--ca FILE]
|
||||
scripts/windows/test-host status|prepare|stage|run|collect|stop|reset|recover
|
||||
```
|
||||
|
||||
Scripts are thin, reviewed orchestration wrappers. `scripts/test-unit` invokes
|
||||
@@ -199,6 +201,13 @@ commands call a shared Go harness under `test/harness` so manifest parsing,
|
||||
timeouts, process control, reporting, and cleanup logic are not reimplemented in
|
||||
shell and PowerShell. The harness itself is built in the toolchain container;
|
||||
no Go/Buf/protoc/SQLite SDK or package manager is installed on the host.
|
||||
`scripts/build` is the corresponding host-safe wrapper around the established
|
||||
containerized `make build`/`make verify` targets. A native run uses
|
||||
`scripts/windows/build-test-bundle` to make one non-secret bundle below its
|
||||
exact `.test-runs/<run-id>` directory, with source-commit and SHA-256 manifest;
|
||||
it refuses replacement rather than overwriting an earlier bundle. This is not a
|
||||
release publisher: signing, version resources, and public checksum publication
|
||||
remain Phase 8 gates.
|
||||
|
||||
The first integration/E2E invocation creates a filesystem-safe random run ID,
|
||||
or validates an explicitly supplied one, and records
|
||||
@@ -257,12 +266,14 @@ copies prior mutable test state.
|
||||
The Linux controller exposes bounded fault controls for frame drop/delay,
|
||||
listener interruption, process termination at named durability checkpoints,
|
||||
filesystem quota/error simulation, and monotonic/wall-clock advancement in the
|
||||
deterministic peer. Native Windows `test-host.ps1` validates administrator state,
|
||||
OS build, UAC/policy fixture, active sessions, service identity, and test-root
|
||||
ownership before running. It uses a test-specific ProgramData root and an
|
||||
exclusive host lease; production-name SCM/Run-key installation tests run only in
|
||||
a resettable VM snapshot. Passwords and VM access credentials come from the CI
|
||||
secret store or interactive prompt, never arguments persisted in the manifest.
|
||||
deterministic peer. Native Windows `test-host` validates administrator state, OS
|
||||
build, UAC/policy fixture, active sessions, service identity, VM/snapshot UUIDs,
|
||||
and test-root ownership before running. It is a POSIX controller which uses SSH
|
||||
to run `VBoxManage` on the Linux fixture host; it uses a test-specific
|
||||
ProgramData root and an exclusive remote host lease. Production-name SCM/Run-key
|
||||
installation tests run only in a resettable VM snapshot. Passwords and VM access
|
||||
credentials come from the CI secret store or interactive prompt, never arguments
|
||||
persisted in the manifest.
|
||||
|
||||
The E2E controller owns the canonical run manifest on Linux and passes the same
|
||||
run ID plus a one-run configuration bundle to the preconfigured Windows runner.
|
||||
@@ -761,7 +772,7 @@ supported Windows 10 or Server 2016 baseline; it need not be the everyday runner
|
||||
A physical Windows machine is optional. It is useful for an additional real
|
||||
display/audio/DDC command smoke, but RVBox only guarantees correct token/session/
|
||||
process execution—not success of arbitrary vendor hardware APIs—so physical
|
||||
hardware is not a release blocker. `test-host.ps1 Prepare` must inventory the
|
||||
hardware is not a release blocker. `test-host prepare` must inventory the
|
||||
host against this checklist and refuse destructive suites unless the machine is
|
||||
explicitly marked disposable/resettable and the clean snapshot identity is
|
||||
recorded.
|
||||
@@ -787,7 +798,7 @@ mirror; update both documents when the fixture is reprovisioned.
|
||||
| VM name / UUID | `rvbox-win10-test` / `6cdc114f-71e5-4167-a394-e922e14e6f5c` |
|
||||
| VM group / config | `/RVBox/Tests`; `/home/cabbage/VirtualBox VMs/RVBox/Tests/rvbox-win10-test/rvbox-win10-test.vbox` |
|
||||
| Guest OS | Windows 10 Pro 22H2, build `19045.2006`, en-US, BIOS boot |
|
||||
| Guest Additions | `7.2.16r174877`; Guest Control readiness requires `GuestAdditionsRunLevel=3` |
|
||||
| Guest Additions | `7.2.16r174877`; readiness requires published Guest Additions version and Windows OS-release properties (this build does not publish a RunLevel property) |
|
||||
| Resources | 2 vCPU, 4096 MiB RAM, 64 MiB VRAM, `VBoxSVGA`, 3D acceleration disabled, 40 GiB dynamically allocated VDI |
|
||||
| Disk / source media | `/home/cabbage/VMs/rvbox-win10-test.vdi`; source ISO `/media/Data2/Downloaded/Win10_22H2_English_x64.iso` (Windows image index 6) |
|
||||
| Devices | Audio (`none`), playback/capture, USB (OHCI/EHCI/xHCI), clipboard/file transfer, drag-and-drop, and shared folders disabled; Intel 82540EM NIC, cable connected |
|
||||
@@ -802,10 +813,23 @@ The guest password, SSH key, and any host account secret are test secrets. Keep
|
||||
them in the operator/CI secret store or a mode-600 password file outside the
|
||||
repository; never put them in this plan, a command-line argument, a run
|
||||
manifest, or collected logs. `VBoxManage guestcontrol` supports
|
||||
`--passwordfile`; prefer that option over an inline password. Every operator or
|
||||
agent must set `RVBOX_TEST_GUEST_PASSWORD_FILE` to the absolute path of that
|
||||
host-side file before a native run. The account name and VM metadata above are
|
||||
not credentials.
|
||||
`--passwordfile`; prefer that option over an inline password. The documented
|
||||
fixture's host-local password-file path is a controller default and may be
|
||||
overridden with `RVBOX_TEST_GUEST_PASSWORD_FILE`; the password value is never a
|
||||
default or repository value. The account name and VM metadata above are not
|
||||
credentials.
|
||||
|
||||
Guest Control uses this split-token administrator's medium-integrity token; its
|
||||
Administrators SID is deny-only. Do not bypass UAC to create the service during
|
||||
automation. A one-time interactive elevated fixture bootstrap must install the
|
||||
test `RVBoxClient` service in the reset baseline and grant `rvboxtest` only
|
||||
query/change-config/start/stop service access plus write access to its dedicated
|
||||
test subtree. Each native run then changes that bootstrapped service's explicit
|
||||
image/configuration and starts it through SCM. This is fixture administration,
|
||||
not a second product worker/elevation broker or a Task Scheduler dependency.
|
||||
Keep the exact service SDDL and test-subtree ACL with the fixture record before
|
||||
taking its replacement baseline snapshot. The production installer/UAC flow is
|
||||
tested separately through an interactive elevated lane.
|
||||
|
||||
For this provisioned lane, the approved host-only credential-file location is
|
||||
`/home/cabbage/.local/share/rvbox-secrets/rvbox-win10-test.password`. It must
|
||||
@@ -853,15 +877,18 @@ ssh "$RVBOX_TEST_VBOX_HOST" \
|
||||
ssh "$RVBOX_TEST_VBOX_HOST" \
|
||||
'VBoxManage startvm "rvbox-win10-test" --type headless'
|
||||
|
||||
# Poll these properties before a test; GuestAdditionsRunLevel=3 is required
|
||||
# for Guest Control, and LoggedInUsers/NoLoggedInUsers selects the session case.
|
||||
# Poll these properties before a test. This fixture requires Guest Additions
|
||||
# version plus Windows OS-release properties; LoggedInUsers/NoLoggedInUsers
|
||||
# selects the session case.
|
||||
ssh "$RVBOX_TEST_VBOX_HOST" \
|
||||
'VBoxManage guestproperty enumerate "rvbox-win10-test"'
|
||||
```
|
||||
|
||||
The harness must wait for the VM to report `running`, then poll Guest
|
||||
Properties until Guest Additions is ready and the requested login fixture is
|
||||
observed. NAT address `10.0.2.15` was observed during provisioning but is DHCP
|
||||
Properties until the Guest Additions version and Windows OS-release properties
|
||||
are present and the requested login fixture is observed. Do not require a
|
||||
`GuestAdditionsRunLevel` value: VirtualBox Guest Additions `7.2.16r174877` on
|
||||
this fixture does not publish it. NAT address `10.0.2.15` was observed during provisioning but is DHCP
|
||||
state, not an identity or a stable endpoint; use Guest Control for management
|
||||
and discover any test networking separately. A failed readiness poll is a
|
||||
stopped-resumable run, not permission to start a second VM with the same name.
|
||||
@@ -964,7 +991,7 @@ scripts/
|
||||
test-integration # resumable component-suite entry point
|
||||
test-e2e # resumable production-shaped scenario entry point
|
||||
test-env # doctor/recover/reuse/cleanup by exact run ID
|
||||
windows/test-host.ps1 # native Windows host lifecycle adapter
|
||||
windows/test-host # POSIX controller for the native Windows VM lifecycle
|
||||
test/
|
||||
coverage.toml # requirement-to-case inventory with stable IDs
|
||||
harness/ # shared manifest, journal, orchestration, and reporting
|
||||
@@ -2196,7 +2223,7 @@ The native `windows-supervisor` and `windows-service-tray` integration suites us
|
||||
real Windows subprocesses and APIs for both shells, CWD/environment overlays,
|
||||
concurrent output without newlines, stdin ordering/close, Job tree termination,
|
||||
script materialization/cleanup, service shutdown interruption, and every token/
|
||||
session context. They run through `scripts/windows/test-host.ps1` under the
|
||||
session context. They run through `scripts/windows/test-host` under the
|
||||
exclusive host lease and journal every machine-wide mutation for reset.
|
||||
|
||||
Add a table-driven crash suite for every acceptance, script-upload, launch,
|
||||
|
||||
+55
-19
@@ -24,7 +24,7 @@ observation.
|
||||
| Snapshot folder | `/home/cabbage/VirtualBox VMs/RVBox/Tests/rvbox-win10-test/Snapshots` |
|
||||
| Guest OS | Windows 10 Pro 22H2, build `19045.2006`, en-US, BIOS boot |
|
||||
| Guest account | Local `rvboxtest`; split-token local administrator; console session 1 was observed during provisioning |
|
||||
| Guest Additions | `7.2.16r174877`; Guest Control readiness requires `GuestAdditionsRunLevel=3` |
|
||||
| Guest Additions | `7.2.16r174877`; readiness requires published Guest Additions version and Windows OS-release properties (this build does not publish a RunLevel property) |
|
||||
| Last observed state | `poweroff`; current snapshot `baseline-disk-first` |
|
||||
|
||||
The Guest Control credential is test-only. The account name is safe to record,
|
||||
@@ -108,35 +108,71 @@ the stale unregistered `win10_dev` configuration (its disk is missing).
|
||||
|
||||
## Harness contract
|
||||
|
||||
The checked-in PowerShell adapter is [`scripts/windows/test-host.ps1`](../scripts/windows/test-host.ps1).
|
||||
Its actions are `Prepare`, `Status`, `Run`, `Collect`, `Stop`, and `Reset`.
|
||||
The adapter takes identity and credentials only from the host environment:
|
||||
The canonical adapter is the POSIX controller script
|
||||
[`scripts/windows/test-host`](../scripts/windows/test-host). It runs from the
|
||||
Linux controller and invokes `VBoxManage` only through SSH on Helium; the
|
||||
fixture host is Arch Linux and does not provide PowerShell. Its actions are
|
||||
`status`, `prepare`, `stage`, `run`, `collect`, `stop`, `reset`, and `recover`.
|
||||
The legacy [`test-host.ps1`](../scripts/windows/test-host.ps1) is retained only
|
||||
as a reference for a future Windows-hosted fixture and is not the Helium lane.
|
||||
|
||||
```powershell
|
||||
$env:RVBOX_WINDOWS_VM = 'rvbox-win10-test'
|
||||
$env:RVBOX_WINDOWS_BASELINE_SNAPSHOT = 'baseline-disk-first'
|
||||
$env:RVBOX_WINDOWS_GUEST_USER = 'rvboxtest'
|
||||
$env:RVBOX_WINDOWS_GUEST_PASSWORD_FILE = 'C:\secure\rvbox-win10-test.password'
|
||||
```
|
||||
|
||||
The implementation plan also uses the equivalent `RVBOX_TEST_*` names for
|
||||
controllers that run the lifecycle over SSH:
|
||||
The adapter takes identity and credentials only from its environment:
|
||||
|
||||
```sh
|
||||
export RVBOX_TEST_VBOX_HOST=helium-remote
|
||||
export RVBOX_TEST_VBOX_VM=rvbox-win10-test
|
||||
export RVBOX_TEST_VBOX_VM_UUID=6cdc114f-71e5-4167-a394-e922e14e6f5c
|
||||
export RVBOX_TEST_VBOX_SNAPSHOT=baseline-disk-first
|
||||
export RVBOX_TEST_VBOX_SNAPSHOT_UUID=9430a9a4-754a-4b22-beaa-8dfd90043f5b
|
||||
export RVBOX_TEST_GUEST_USER=rvboxtest
|
||||
export RVBOX_TEST_GUEST_PASSWORD_FILE=/home/cabbage/.local/share/rvbox-secrets/rvbox-win10-test.password
|
||||
```
|
||||
|
||||
Those values are the controller defaults for this one documented fixture, so a
|
||||
normal Helium run does not need to export them. They remain overrideable for a
|
||||
separately recorded fixture. The default contains only the host-local password
|
||||
*file path*, never the password value.
|
||||
|
||||
The provisioned lane's backend is SSH plus `VBoxManage` plus Guest Control; it
|
||||
does not require WinRM, OpenSSH inside Windows, or a stable guest IP. Guest
|
||||
commands must pass explicit paths and argument vectors. With this VirtualBox
|
||||
build, `--wait-stdout` and `--wait-stderr` are supported but `--wait-exit` is
|
||||
not; use `closeprocess` when a process wait cannot complete. For `cmd.exe`
|
||||
payloads, include `--unquoted-args` so Windows backslashes and the single `/c`
|
||||
payload are preserved.
|
||||
does not require WinRM, OpenSSH inside Windows, or a stable guest IP. The
|
||||
adapter first validates the VM and snapshot UUIDs, acquires its exclusive lease
|
||||
on Helium, and writes a non-secret step report to the run directory. Guest
|
||||
commands use exact console-safe executable paths and argument vectors. With
|
||||
this VirtualBox build, `--wait-stdout` and `--wait-stderr` are supported but
|
||||
`--wait-exit` is not; use `closeprocess` when a process wait cannot complete.
|
||||
For `cmd.exe` payloads, include `--unquoted-args` so Windows backslashes and
|
||||
the single `/c` payload are preserved.
|
||||
|
||||
`stage` accepts one versioned non-secret test bundle and copies it first to an
|
||||
exact host staging directory, then to
|
||||
`C:\\ProgramData\\RVBox\\test-runs\\<run-id>`. `run` never directly executes the
|
||||
GUI-subsystem `rvbox.exe` through Guest Control. It uses `sc.exe` and other
|
||||
console-safe management tools to start/query/stop the installed RVBox service,
|
||||
then checks the service's real health endpoint, named-pipe response, durable
|
||||
state, and agent-server results. Before a WSS scenario it performs a bounded
|
||||
guest-to-nginx connectivity and CA-trust probe. The resulting service and
|
||||
artifact paths are recorded in the run report and reclaimed by the snapshot
|
||||
reset rather than broad guest deletion.
|
||||
|
||||
### Required one-time service bootstrap
|
||||
|
||||
Guest Control launches `rvboxtest` with its filtered, medium-integrity UAC
|
||||
token: the Administrators SID is deny-only. The harness must not bypass UAC to
|
||||
create services. Before native service tests can run, an operator must use a
|
||||
trusted interactive elevated session to install one test-only `RVBoxClient`
|
||||
service in the baseline and grant only `rvboxtest` the service rights required
|
||||
by the harness: query status/configuration, change its image/configuration,
|
||||
start, and stop. The test account also needs write access only to the dedicated
|
||||
`C:\\ProgramData\\RVBox\\test-runs` subtree; SYSTEM retains ownership of normal
|
||||
RVBox state and logs. Record the resulting service SDDL and subtree ACL in this
|
||||
document before taking a new reset snapshot.
|
||||
|
||||
Each run then stages an exact bundle, changes the bootstrapped service image to
|
||||
that run's explicit `--service --config` command line, and starts it through
|
||||
SCM. The one-time bootstrap is fixture administration, not a second RVBox
|
||||
process, runtime elevation broker, or Task Scheduler mechanism. Native tests
|
||||
for the production installer/UAC flow remain a separately interactive test;
|
||||
they cannot be automated through this filtered Guest Control token.
|
||||
|
||||
## Scope and known limitations
|
||||
|
||||
|
||||
+70
-17
@@ -15,6 +15,30 @@ Run focused unit tests with:
|
||||
scripts/test-unit --package ./internal/domain --run UUIDv7 --race
|
||||
```
|
||||
|
||||
Build all supported binaries without installing `make` or Go on the host:
|
||||
|
||||
```sh
|
||||
scripts/build build
|
||||
```
|
||||
|
||||
For a native Windows run, create the immutable per-run test bundle with the
|
||||
pinned toolchain. Supply a test-specific config whose endpoint and CA path are
|
||||
valid for that run; the command refuses to replace an existing bundle.
|
||||
|
||||
```sh
|
||||
scripts/windows/build-test-bundle \
|
||||
--run-id windows-smoke \
|
||||
--config .test-runs/windows-smoke/client.toml \
|
||||
--ca .test-runs/windows-smoke/ca.pem
|
||||
scripts/windows/test-host prepare --run-id windows-smoke
|
||||
scripts/windows/test-host stage --run-id windows-smoke \
|
||||
--bundle .test-runs/windows-smoke/windows-bundle
|
||||
```
|
||||
|
||||
The bundle manifest records the source commit and SHA-256 of every bundled
|
||||
file. It is a test artifact, not a signed release package; release signing,
|
||||
version resources, and publication are Phase 8 gates.
|
||||
|
||||
The integration harness provides the Phase 0 `sample` suite, the incremental
|
||||
Phase 2 `store` suite, and the incremental Phase 3 `server-session` suite.
|
||||
The resumable E2E harness adds `smoke`, `script`, `recovery`, and `all`
|
||||
@@ -97,20 +121,49 @@ than treating it as a stable endpoint. VRDE is enabled at
|
||||
`192.168.50.162:3389` for diagnostics, while native Windows RDP is disabled in
|
||||
the baseline.
|
||||
|
||||
The exact headless VirtualBox/Guest Control adapter is
|
||||
`scripts/windows/test-host.ps1`; it takes the VM name, baseline snapshot, and
|
||||
guest identity/password-file only from host environment variables, acquires an
|
||||
exclusive lease, and never writes secrets to the repository. Set
|
||||
`RVBOX_WINDOWS_GUEST_PASSWORD_FILE` to a mode-600 file outside the repository;
|
||||
the adapter passes it with VirtualBox `--passwordfile` and never accepts an
|
||||
inline password. Use `Prepare`, `Run`,
|
||||
`Collect`, `Stop`, and `Reset` in that order for a native run. The VM is the
|
||||
minimum smoke lane, so deferred native multi-session/ambiguous-session, Server
|
||||
Core, and older-build entries remain explicitly blocked until their own
|
||||
fixtures exist. A previous Windows 10 fixture probe found that VirtualBox Guest
|
||||
Control 7.2.16 rejects the GUI-subsystem `rvbox.exe` as a directly runnable
|
||||
guest executable; wrapping it through `cmd.exe` exited the RVBox process but
|
||||
left the Guest Control wrapper waiting. This remains an adapter completion-path
|
||||
limitation: native runtime coverage needs a service-driven or
|
||||
console-compatible guest runner. Wine or a protocol stub is not treated as
|
||||
equivalent coverage.
|
||||
The canonical headless VirtualBox/Guest Control adapter is
|
||||
`scripts/windows/test-host`. It is a POSIX controller script because the
|
||||
fixture's VirtualBox host is Arch Linux and has no PowerShell runtime. The
|
||||
controller connects to Helium over SSH; `VBoxManage` and the host-only password
|
||||
file never need to exist on the Linux development controller. It takes the VM
|
||||
identity, baseline snapshot, guest identity, and password-file only from host
|
||||
environment variables, acquires an exclusive remote lease, and never writes
|
||||
secrets to the repository, run manifest, or command line. Set
|
||||
`RVBOX_TEST_GUEST_PASSWORD_FILE` to the mode-600 host-side file; the adapter
|
||||
passes it only as VirtualBox `--passwordfile`.
|
||||
The provisioned fixture's non-secret identity values, including the host-local
|
||||
password-file path, are safe defaults in that script and may be overridden for
|
||||
another documented fixture; the password itself is never embedded.
|
||||
|
||||
The native lifecycle is `status`, `prepare`, `stage`, `run`, `collect`,
|
||||
`stop`, and `reset`. `prepare` verifies the VM and snapshot UUIDs, restores the
|
||||
baseline, starts headless, waits for Guest Additions, and records the selected
|
||||
login fixture. `stage` copies a versioned non-secret test bundle through a
|
||||
run-specific host directory to a run-specific guest directory. `run` starts
|
||||
the real RVBox SCM service and observes it through SCM, the local health/tray
|
||||
pipe, durable artifacts, and the test agent endpoint; it must not invoke the
|
||||
GUI-subsystem `rvbox.exe` directly through Guest Control. `collect` obtains
|
||||
only bounded/redacted artifacts, and `reset` restores the exact baseline and
|
||||
leaves the VM powered off.
|
||||
|
||||
This service-driven protocol is required because VirtualBox Guest Control
|
||||
7.2.16 does not reliably complete a direct GUI-subsystem `rvbox.exe` run;
|
||||
wrapping it in `cmd.exe` can leave the Guest Control wrapper waiting. Guest
|
||||
Control is therefore limited to console-safe setup tools (`sc.exe`, `whoami`,
|
||||
`query`, bounded file operations) and artifact collection. A native E2E run
|
||||
also performs a bounded guest-to-nginx HTTPS/TCP readiness probe before it
|
||||
starts the service. Wine and protocol stubs are not equivalent Windows
|
||||
coverage.
|
||||
|
||||
The fixture needs a one-time operator-approved service bootstrap before the
|
||||
SCM smoke lane can run: Guest Control supplies the split-token administrator's
|
||||
medium UAC token, so it cannot safely install services. The bootstrap installs
|
||||
the test service in the reset baseline and grants the test account only the SCM
|
||||
query/change-config/start/stop rights plus access to its dedicated test subtree.
|
||||
Each run then reconfigures and starts that one service. This does not add a
|
||||
product broker or use Task Scheduler; it is fixture setup. See
|
||||
[testing-vm.md](testing-vm.md) for the exact boundary.
|
||||
|
||||
The VM is the minimum smoke lane, so native multi-session/ambiguous-session,
|
||||
Server Core, and older-build entries remain explicitly blocked until their own
|
||||
fixtures exist. Their pure selector tests remain mandatory.
|
||||
|
||||
Reference in New Issue
Block a user