diff --git a/docs/implementation-plan.v1.md b/docs/implementation-plan.v1.md index cbbf4a4..0821097 100644 --- a/docs/implementation-plan.v1.md +++ b/docs/implementation-plan.v1.md @@ -888,7 +888,8 @@ invisible Guest Control session must never answer it. When that bounded manual step is necessary, use the tracked Docker-only [`test/rdp-access`](../test/rdp-access/README.md) helper. It starts a self-signed HTTPS Guacamole gateway only after `test-host prepare` holds the -fixture lease; VRDE remains loopback-only on Helium and its SSH tunnel is bound +fixture lease; VRDE remains loopback-only on Helium and its reconnecting SSH +watchdog/tunnel is bound only to the helper's private Docker gateway. Stop the helper before the normal `test-host reset`. It is a recovery interface, not a product component or a replacement for Guest Control/native test automation. diff --git a/docs/testing-vm.md b/docs/testing-vm.md index e08cb26..6c54079 100644 --- a/docs/testing-vm.md +++ b/docs/testing-vm.md @@ -88,7 +88,8 @@ and collection. Do not expose the VM's RDP endpoints beyond the test LAN. For the rare interactive UAC/manual-recovery step, use the Docker-only helper in [`test/rdp-access`](../test/rdp-access/README.md). It creates a temporary self-signed HTTPS Guacamole gateway while retaining VRDE on Helium loopback and -the SSH tunnel on a private Docker gateway. Follow its full lease/prepare/up/ +the reconnecting SSH watchdog/tunnel on a private Docker gateway. Follow its +full lease/prepare/up/ down/reset lifecycle; it is not an alternative to the native test controller. ## Snapshots and reset contract diff --git a/test/rdp-access/README.md b/test/rdp-access/README.md index 2ec0b8e..f028687 100644 --- a/test/rdp-access/README.md +++ b/test/rdp-access/README.md @@ -12,7 +12,7 @@ browser -- HTTPS/self-signed --> nginx + Guacamole containers | private Docker gateway | -controller SSH tunnel --> Helium 127.0.0.1:3389 --> VirtualBox VRDE --> VM console +controller SSH watchdog/tunnel --> Helium 127.0.0.1:3389 --> VirtualBox VRDE --> VM console ``` Only the HTTPS listener can be made public, and that requires an explicit @@ -53,7 +53,12 @@ For localhost-only use, omit `--bind` and `--public-host`. The default listener is `127.0.0.1:5002`; use a local SSH forward or a browser on the controller. Choose alternate ports with `--http-port` and `--tunnel-port` if either is in use. The VM must already be running. `up` checks the documented VM/snapshot -identity but intentionally does not restore, start, stop, or reset the VM. +identity but intentionally does not restore, start, stop, or reset the VM. It +starts a small host-side watchdog for the SSH master. The watchdog reconnects +after a transient Helium/SSH failure while preserving the same Docker-gateway +listener, so an already-open Guacamole session can recover without restarting +the containers. Its PID, stop marker, and diagnostic log are kept under the +ignored `.runtime/` directory. All fixture-specific values have embedded, working defaults: the `helium-remote` SSH alias, Helium's loopback VRDE endpoint (`127.0.0.1:3389`), @@ -66,6 +71,18 @@ port (`54001`). They can be overridden without editing tracked files through Helium loopback (`127.0.0.1` or `localhost`) so an override cannot accidentally turn the diagnostic server into a remote target. +If `up` is run again while the Compose services are still running, it is +idempotent: an existing healthy tunnel is reused, and a missing tunnel is +recreated in place. `status` reports a helper-owned tunnel as +`private_tunnel=active` and a listener supplied by an external/interactive +SSH supervisor as `private_tunnel=active_external`. A missing listener is +reported as `private_tunnel=inactive`; inspect `.runtime/tunnel.log` and run +`up` again to trigger a bounded reconnect attempt. + +The XML mapping intentionally uses Guacamole's `${GUAC_PASSWORD}` connection +parameter token. Guacamole resolves this to the password entered at web login; +it is not a host environment variable and must remain in the template. + After the interactive action, close the browser connection and remove the temporary access path before releasing the fixture lease: @@ -74,9 +91,10 @@ test/rdp-access/rdp-access down scripts/windows/test-host reset --run-id interactive-rdp ``` -`down` stops containers and the SSH master/tunnel but retains the one-day -certificate and password verifier for a quick restart. To remove all generated -state, including the certificate and verifier: +`down` stops containers and the SSH watchdog/master/tunnel but retains the +one-day certificate and password verifier for a quick restart. To remove all +generated state, including the certificate, verifier, watchdog PID, and tunnel +log: ```sh test/rdp-access/rdp-access clean @@ -101,14 +119,16 @@ test/rdp-access/rdp-access logs --tail=100 test/rdp-access/rdp-access url ``` -If the browser reaches Guacamole but stays on “Waiting for response”, verify -that the VM is running and the private tunnel is active with `status`. This -helper already uses `security=rdp` and disables Guacamole's GFX extension, -which are required by the fixture's legacy VRDE server. Do not switch the -helper to native Windows RDP: `TermService` is intentionally disabled in the -baseline. If VRDE remains unusable, stop this helper and use Guest Control for -the deterministic portion of the work; record the blocked interactive step in -the native test report. +If the browser reaches Guacamole but stays on “Waiting for response”, run +`status` first. Confirm `private_tunnel=active` (or +`active_external`), then check the VM state and the last lines of +`.runtime/tunnel.log`. A tunnel can be recreated without losing the Compose +stack by running `up` again. This helper already uses `security=rdp` and +disables Guacamole's GFX extension, which are required by the fixture's legacy +VRDE server. Do not switch the helper to native Windows RDP: +`TermService` is intentionally disabled in the baseline. If VRDE remains +unusable, stop this helper and use Guest Control for the deterministic portion +of the work; record the blocked interactive step in the native test report. The helper requires Docker/Docker Compose, SSH access through the existing `helium-remote` alias, and the fixture password file documented in diff --git a/test/rdp-access/rdp-access b/test/rdp-access/rdp-access index ac801af..a4ed2b6 100755 --- a/test/rdp-access/rdp-access +++ b/test/rdp-access/rdp-access @@ -29,7 +29,7 @@ Actions: status show gateway, tunnel, and fixture status without changing anything url print the current browser URL logs follow or print Compose logs (pass Docker Compose log options) - down stop containers and the private SSH tunnel; retain generated state + down stop containers and the private SSH watchdog/tunnel; retain generated state clean run down and delete generated state; pass --images to also remove the exact unused Guacamole/nginx images @@ -79,12 +79,42 @@ compose() { socket_path=$runtime_dir/ssh-control.socket session_file=$runtime_dir/session.env cert_name_file=$runtime_dir/cert-name +tunnel_stop_file=$runtime_dir/tunnel.stop +tunnel_pid_file=$runtime_dir/tunnel-watchdog.pid +tunnel_log_file=$runtime_dir/tunnel.log stop_tunnel() { + mkdir -p "$runtime_dir" + : >"$tunnel_stop_file" if [ -S "$socket_path" ]; then - ssh -S "$socket_path" -O exit "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1 || true + ssh -S "$socket_path" -O exit -o ConnectTimeout=5 "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1 || true + fi + if [ -f "$tunnel_pid_file" ]; then + tunnel_pid=$(cat "$tunnel_pid_file" 2>/dev/null || true) + case $tunnel_pid in + ''|*[!0-9]*) ;; + *) + # Never trust a stale numeric PID file. Confirm that it is + # still this helper's watchdog before signalling it. + tunnel_cmd=$(ps -p "$tunnel_pid" -o args= 2>/dev/null || true) + case $tunnel_cmd in + *rdp-tunnel-watchdog*"$tunnel_stop_file"*) + kill "$tunnel_pid" >/dev/null 2>&1 || true + # The watchdog normally exits within its two-second + # retry interval. Do not leave a stale reconnect loop + # behind, but only force the verified process. + sleep 1 + tunnel_cmd=$(ps -p "$tunnel_pid" -o args= 2>/dev/null || true) + case $tunnel_cmd in + *rdp-tunnel-watchdog*"$tunnel_stop_file"*) kill -KILL "$tunnel_pid" >/dev/null 2>&1 || true ;; + esac + ;; + esac + ;; + esac fi rm -f "$socket_path" + rm -f "$tunnel_pid_file" "$tunnel_stop_file" } gateway_for_network() { @@ -152,19 +182,111 @@ render_mapping() { printf 'url=https://%s:%s/guacamole/\n' "$RDP_ACCESS_PUBLIC_HOST" "$RDP_ACCESS_HTTP_PORT" >"$session_file" printf 'docker_gateway=%s\n' "$docker_gateway" >>"$session_file" printf 'tunnel_port=%s\n' "$RDP_ACCESS_TUNNEL_PORT" >>"$session_file" + printf 'bind=%s\n' "$RDP_ACCESS_BIND" >>"$session_file" + printf 'http_port=%s\n' "$RDP_ACCESS_HTTP_PORT" >>"$session_file" + printf 'public_host=%s\n' "$RDP_ACCESS_PUBLIC_HOST" >>"$session_file" + printf 'web_user=%s\n' "$RDP_ACCESS_WEB_USER" >>"$session_file" + printf 'rdp_user=%s\n' "$RDP_ACCESS_RDP_USER" >>"$session_file" chmod 600 "$session_file" } +session_has() { + grep -Fqx "$1=$2" "$session_file" +} + +session_matches_current() { + [ -f "$session_file" ] || return 1 + session_has url "https://$RDP_ACCESS_PUBLIC_HOST:$RDP_ACCESS_HTTP_PORT/guacamole/" || return 1 + session_has docker_gateway "$docker_gateway" || return 1 + session_has tunnel_port "$RDP_ACCESS_TUNNEL_PORT" || return 1 + session_has bind "$RDP_ACCESS_BIND" || return 1 + session_has http_port "$RDP_ACCESS_HTTP_PORT" || return 1 + session_has public_host "$RDP_ACCESS_PUBLIC_HOST" || return 1 + session_has web_user "$RDP_ACCESS_WEB_USER" || return 1 + session_has rdp_user "$RDP_ACCESS_RDP_USER" +} + start_tunnel() { stop_tunnel - ssh -M -S "$socket_path" -fN \ - -o BatchMode=yes \ - -o ExitOnForwardFailure=yes \ - -o ServerAliveInterval=30 \ - -o ServerAliveCountMax=3 \ - -L "$docker_gateway:$RDP_ACCESS_TUNNEL_PORT:$RDP_ACCESS_VRDE_HOST:$RDP_ACCESS_VRDE_PORT" \ - "$RVBOX_TEST_VBOX_HOST" - ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1 || fail "private VRDE tunnel did not start" + umask 077 + : >"$tunnel_log_file" + chmod 600 "$tunnel_log_file" + + # Keep the SSH master outside the short-lived controller process. Helium + # may drop an idle or metered route; the watchdog reconnects it and keeps + # the same local listener for Guacamole. The stop marker is deliberately + # inside .runtime so down/clean can terminate the loop deterministically. + watchdog_script=' + set -eu + stop_file=$1 + socket=$2 + host=$3 + bind=$4 + tunnel_port=$5 + vrde_host=$6 + vrde_port=$7 + while [ ! -e "$stop_file" ]; do + if [ -S "$socket" ] && ssh -S "$socket" -O check "$host" >/dev/null 2>&1; then + sleep 2 + continue + fi + rm -f "$socket" + ssh -M -S "$socket" -fN \ + -o BatchMode=yes \ + -o ExitOnForwardFailure=yes \ + -o ServerAliveInterval=30 \ + -o ServerAliveCountMax=3 \ + -L "$bind:$tunnel_port:$vrde_host:$vrde_port" \ + "$host" || true + sleep 2 + done + ' + if command -v setsid >/dev/null 2>&1; then + setsid nohup sh -c "$watchdog_script" rdp-tunnel-watchdog \ + "$tunnel_stop_file" "$socket_path" "$RVBOX_TEST_VBOX_HOST" \ + "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" "$RDP_ACCESS_VRDE_HOST" \ + "$RDP_ACCESS_VRDE_PORT" >"$tunnel_log_file" 2>&1 & + else + nohup sh -c "$watchdog_script" rdp-tunnel-watchdog \ + "$tunnel_stop_file" "$socket_path" "$RVBOX_TEST_VBOX_HOST" \ + "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" "$RDP_ACCESS_VRDE_HOST" \ + "$RDP_ACCESS_VRDE_PORT" >"$tunnel_log_file" 2>&1 & + fi + tunnel_watchdog_pid=$! + printf '%s\n' "$tunnel_watchdog_pid" >"$tunnel_pid_file" + chmod 600 "$tunnel_pid_file" + + # A listener is useful only after the SSH master has authenticated and + # bound the Docker-network gateway. Give the reconnect loop a bounded + # window, then fail with a clear recovery path. + attempts=0 + while [ "$attempts" -lt 20 ]; do + if [ -S "$socket_path" ] && ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1; then + return 0 + fi + if command -v nc >/dev/null 2>&1 && nc -z -w 2 "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" >/dev/null 2>&1; then + return 0 + fi + attempts=$((attempts + 1)) + sleep 1 + done + stop_tunnel + return 1 +} + +tunnel_listener_active() { + [ -n "${docker_gateway-}" ] || return 1 + command -v nc >/dev/null 2>&1 || return 1 + nc -z -w 2 "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" >/dev/null 2>&1 +} + +tunnel_control_active() { + [ -S "$socket_path" ] || return 1 + ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1 +} + +tunnel_active() { + tunnel_control_active || tunnel_listener_active } action=${1-} @@ -220,7 +342,22 @@ case $action in docker compose version >/dev/null assert_fixture_running if compose ps -q | grep -q .; then - fail "gateway already exists; use status or down first" + # A short-lived controller (or a dropped SSH route) can leave the + # Compose stack running after its tunnel has disappeared. Reuse + # that stack and repair only the private forwarding path instead + # of forcing the operator to tear down a usable Guacamole session. + mkdir -p "$runtime_dir" + docker_gateway=$(gateway_for_network) || fail "could not determine private Docker gateway" + session_matches_current || fail "gateway already exists with a different configuration; run down first, then run up with the desired options" + if tunnel_active; then + printf 'Guacamole stack and private VRDE tunnel are already active.\n' + elif start_tunnel; then + printf 'Existing Guacamole stack reused; private VRDE tunnel restored.\n' + else + fail "existing Guacamole stack was found but the private VRDE tunnel could not be restored; inspect $tunnel_log_file" + fi + if [ -f "$session_file" ]; then sed -n '1p' "$session_file"; fi + exit 0 fi mkdir -p "$runtime_dir" compose up -d guacd @@ -248,8 +385,16 @@ case $action in "$repo_root/scripts/windows/test-host" status || true if [ -f "$session_file" ]; then sed -n '1p' "$session_file"; fi compose ps - if [ -S "$socket_path" ] && ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1; then + docker_gateway=$(gateway_for_network 2>/dev/null || true) + if tunnel_control_active; then printf 'private_tunnel=active\n' + printf 'private_tunnel_supervisor=helper\n' + elif tunnel_listener_active; then + # A tunnel started by an interactive shell or another supervisor + # has no control socket owned by this helper, but it is still a + # valid path when the Docker listener is reachable. + printf 'private_tunnel=active_external\n' + printf 'private_tunnel_supervisor=external\n' else printf 'private_tunnel=inactive\n' fi