fix: keep Guacamole VRDE tunnel alive

docs/implementation-plan.v1.md

@@ -888,7 +888,8 @@ invisible Guest Control session must never answer it.
 When that bounded manual step is necessary, use the tracked Docker-only
 [`test/rdp-access`](../test/rdp-access/README.md) helper. It starts a
 self-signed HTTPS Guacamole gateway only after `test-host prepare` holds the
-fixture lease; VRDE remains loopback-only on Helium and its SSH tunnel is bound
+fixture lease; VRDE remains loopback-only on Helium and its reconnecting SSH
+watchdog/tunnel is bound
 only to the helper's private Docker gateway. Stop the helper before the normal
 `test-host reset`. It is a recovery interface, not a product component or a
 replacement for Guest Control/native test automation.

docs/testing-vm.md

@@ -88,7 +88,8 @@ and collection. Do not expose the VM's RDP endpoints beyond the test LAN.
 For the rare interactive UAC/manual-recovery step, use the Docker-only helper
 in [`test/rdp-access`](../test/rdp-access/README.md). It creates a temporary
 self-signed HTTPS Guacamole gateway while retaining VRDE on Helium loopback and
-the SSH tunnel on a private Docker gateway. Follow its full lease/prepare/up/
+the reconnecting SSH watchdog/tunnel on a private Docker gateway. Follow its
+full lease/prepare/up/
 down/reset lifecycle; it is not an alternative to the native test controller.
 
 ## Snapshots and reset contract

test/rdp-access/README.md

@@ -12,7 +12,7 @@ browser -- HTTPS/self-signed --> nginx + Guacamole containers
                                       |
                                  private Docker gateway
                                       |
-controller SSH tunnel --> Helium 127.0.0.1:3389 --> VirtualBox VRDE --> VM console
+controller SSH watchdog/tunnel --> Helium 127.0.0.1:3389 --> VirtualBox VRDE --> VM console
 ```
 
 Only the HTTPS listener can be made public, and that requires an explicit
@@ -53,7 +53,12 @@ For localhost-only use, omit `--bind` and `--public-host`. The default listener
 is `127.0.0.1:5002`; use a local SSH forward or a browser on the controller.
 Choose alternate ports with `--http-port` and `--tunnel-port` if either is in
 use. The VM must already be running. `up` checks the documented VM/snapshot
-identity but intentionally does not restore, start, stop, or reset the VM.
+identity but intentionally does not restore, start, stop, or reset the VM. It
+starts a small host-side watchdog for the SSH master. The watchdog reconnects
+after a transient Helium/SSH failure while preserving the same Docker-gateway
+listener, so an already-open Guacamole session can recover without restarting
+the containers. Its PID, stop marker, and diagnostic log are kept under the
+ignored `.runtime/` directory.
 
 All fixture-specific values have embedded, working defaults: the
 `helium-remote` SSH alias, Helium's loopback VRDE endpoint (`127.0.0.1:3389`),
@@ -66,6 +71,18 @@ port (`54001`). They can be overridden without editing tracked files through
 Helium loopback (`127.0.0.1` or `localhost`) so an override cannot accidentally
 turn the diagnostic server into a remote target.
 
+If `up` is run again while the Compose services are still running, it is
+idempotent: an existing healthy tunnel is reused, and a missing tunnel is
+recreated in place. `status` reports a helper-owned tunnel as
+`private_tunnel=active` and a listener supplied by an external/interactive
+SSH supervisor as `private_tunnel=active_external`. A missing listener is
+reported as `private_tunnel=inactive`; inspect `.runtime/tunnel.log` and run
+`up` again to trigger a bounded reconnect attempt.
+
+The XML mapping intentionally uses Guacamole's `${GUAC_PASSWORD}` connection
+parameter token. Guacamole resolves this to the password entered at web login;
+it is not a host environment variable and must remain in the template.
+
 After the interactive action, close the browser connection and remove the
 temporary access path before releasing the fixture lease:
 
@@ -74,9 +91,10 @@ test/rdp-access/rdp-access down
 scripts/windows/test-host reset --run-id interactive-rdp
 ```
 
-`down` stops containers and the SSH master/tunnel but retains the one-day
+`down` stops containers and the SSH watchdog/master/tunnel but retains the
-certificate and password verifier for a quick restart. To remove all generated
+one-day certificate and password verifier for a quick restart. To remove all
-state, including the certificate and verifier:
+generated state, including the certificate, verifier, watchdog PID, and tunnel
+log:
 
 ```sh
 test/rdp-access/rdp-access clean
@@ -101,14 +119,16 @@ test/rdp-access/rdp-access logs --tail=100
 test/rdp-access/rdp-access url
 ```
 
-If the browser reaches Guacamole but stays on “Waiting for response”, verify
+If the browser reaches Guacamole but stays on “Waiting for response”, run
-that the VM is running and the private tunnel is active with `status`. This
+`status` first. Confirm `private_tunnel=active` (or
-helper already uses `security=rdp` and disables Guacamole's GFX extension,
+`active_external`), then check the VM state and the last lines of
-which are required by the fixture's legacy VRDE server. Do not switch the
+`.runtime/tunnel.log`. A tunnel can be recreated without losing the Compose
-helper to native Windows RDP: `TermService` is intentionally disabled in the
+stack by running `up` again. This helper already uses `security=rdp` and
-baseline. If VRDE remains unusable, stop this helper and use Guest Control for
+disables Guacamole's GFX extension, which are required by the fixture's legacy
-the deterministic portion of the work; record the blocked interactive step in
+VRDE server. Do not switch the helper to native Windows RDP:
-the native test report.
+`TermService` is intentionally disabled in the baseline. If VRDE remains
+unusable, stop this helper and use Guest Control for the deterministic portion
+of the work; record the blocked interactive step in the native test report.
 
 The helper requires Docker/Docker Compose, SSH access through the existing
 `helium-remote` alias, and the fixture password file documented in

test/rdp-access/rdp-access

@@ -29,7 +29,7 @@ Actions:
   status   show gateway, tunnel, and fixture status without changing anything
   url      print the current browser URL
   logs     follow or print Compose logs (pass Docker Compose log options)
-  down     stop containers and the private SSH tunnel; retain generated state
+  down     stop containers and the private SSH watchdog/tunnel; retain generated state
   clean    run down and delete generated state; pass --images to also remove
            the exact unused Guacamole/nginx images
 
@@ -79,12 +79,42 @@ compose() {
 socket_path=$runtime_dir/ssh-control.socket
 session_file=$runtime_dir/session.env
 cert_name_file=$runtime_dir/cert-name
+tunnel_stop_file=$runtime_dir/tunnel.stop
+tunnel_pid_file=$runtime_dir/tunnel-watchdog.pid
+tunnel_log_file=$runtime_dir/tunnel.log
 
 stop_tunnel() {
+    mkdir -p "$runtime_dir"
+    : >"$tunnel_stop_file"
     if [ -S "$socket_path" ]; then
-        ssh -S "$socket_path" -O exit "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1 || true
+        ssh -S "$socket_path" -O exit -o ConnectTimeout=5 "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1 || true
+    fi
+    if [ -f "$tunnel_pid_file" ]; then
+        tunnel_pid=$(cat "$tunnel_pid_file" 2>/dev/null || true)
+        case $tunnel_pid in
+            ''|*[!0-9]*) ;;
+            *)
+                # Never trust a stale numeric PID file. Confirm that it is
+                # still this helper's watchdog before signalling it.
+                tunnel_cmd=$(ps -p "$tunnel_pid" -o args= 2>/dev/null || true)
+                case $tunnel_cmd in
+                    *rdp-tunnel-watchdog*"$tunnel_stop_file"*)
+                        kill "$tunnel_pid" >/dev/null 2>&1 || true
+                        # The watchdog normally exits within its two-second
+                        # retry interval. Do not leave a stale reconnect loop
+                        # behind, but only force the verified process.
+                        sleep 1
+                        tunnel_cmd=$(ps -p "$tunnel_pid" -o args= 2>/dev/null || true)
+                        case $tunnel_cmd in
+                            *rdp-tunnel-watchdog*"$tunnel_stop_file"*) kill -KILL "$tunnel_pid" >/dev/null 2>&1 || true ;;
+                        esac
+                        ;;
+                esac
+                ;;
+        esac
     fi
     rm -f "$socket_path"
+    rm -f "$tunnel_pid_file" "$tunnel_stop_file"
 }
 
 gateway_for_network() {
@@ -152,19 +182,111 @@ render_mapping() {
     printf 'url=https://%s:%s/guacamole/\n' "$RDP_ACCESS_PUBLIC_HOST" "$RDP_ACCESS_HTTP_PORT" >"$session_file"
     printf 'docker_gateway=%s\n' "$docker_gateway" >>"$session_file"
     printf 'tunnel_port=%s\n' "$RDP_ACCESS_TUNNEL_PORT" >>"$session_file"
+    printf 'bind=%s\n' "$RDP_ACCESS_BIND" >>"$session_file"
+    printf 'http_port=%s\n' "$RDP_ACCESS_HTTP_PORT" >>"$session_file"
+    printf 'public_host=%s\n' "$RDP_ACCESS_PUBLIC_HOST" >>"$session_file"
+    printf 'web_user=%s\n' "$RDP_ACCESS_WEB_USER" >>"$session_file"
+    printf 'rdp_user=%s\n' "$RDP_ACCESS_RDP_USER" >>"$session_file"
     chmod 600 "$session_file"
 }
 
+session_has() {
+    grep -Fqx "$1=$2" "$session_file"
+}
+
+session_matches_current() {
+    [ -f "$session_file" ] || return 1
+    session_has url "https://$RDP_ACCESS_PUBLIC_HOST:$RDP_ACCESS_HTTP_PORT/guacamole/" || return 1
+    session_has docker_gateway "$docker_gateway" || return 1
+    session_has tunnel_port "$RDP_ACCESS_TUNNEL_PORT" || return 1
+    session_has bind "$RDP_ACCESS_BIND" || return 1
+    session_has http_port "$RDP_ACCESS_HTTP_PORT" || return 1
+    session_has public_host "$RDP_ACCESS_PUBLIC_HOST" || return 1
+    session_has web_user "$RDP_ACCESS_WEB_USER" || return 1
+    session_has rdp_user "$RDP_ACCESS_RDP_USER"
+}
+
 start_tunnel() {
     stop_tunnel
-    ssh -M -S "$socket_path" -fN \
+    umask 077
-        -o BatchMode=yes \
+    : >"$tunnel_log_file"
-        -o ExitOnForwardFailure=yes \
+    chmod 600 "$tunnel_log_file"
-        -o ServerAliveInterval=30 \
+
-        -o ServerAliveCountMax=3 \
+    # Keep the SSH master outside the short-lived controller process. Helium
-        -L "$docker_gateway:$RDP_ACCESS_TUNNEL_PORT:$RDP_ACCESS_VRDE_HOST:$RDP_ACCESS_VRDE_PORT" \
+    # may drop an idle or metered route; the watchdog reconnects it and keeps
-        "$RVBOX_TEST_VBOX_HOST"
+    # the same local listener for Guacamole. The stop marker is deliberately
-    ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1 || fail "private VRDE tunnel did not start"
+    # inside .runtime so down/clean can terminate the loop deterministically.
+    watchdog_script='
+        set -eu
+        stop_file=$1
+        socket=$2
+        host=$3
+        bind=$4
+        tunnel_port=$5
+        vrde_host=$6
+        vrde_port=$7
+        while [ ! -e "$stop_file" ]; do
+            if [ -S "$socket" ] && ssh -S "$socket" -O check "$host" >/dev/null 2>&1; then
+                sleep 2
+                continue
+            fi
+            rm -f "$socket"
+            ssh -M -S "$socket" -fN \
+                -o BatchMode=yes \
+                -o ExitOnForwardFailure=yes \
+                -o ServerAliveInterval=30 \
+                -o ServerAliveCountMax=3 \
+                -L "$bind:$tunnel_port:$vrde_host:$vrde_port" \
+                "$host" || true
+            sleep 2
+        done
+    '
+    if command -v setsid >/dev/null 2>&1; then
+        setsid nohup sh -c "$watchdog_script" rdp-tunnel-watchdog \
+            "$tunnel_stop_file" "$socket_path" "$RVBOX_TEST_VBOX_HOST" \
+            "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" "$RDP_ACCESS_VRDE_HOST" \
+            "$RDP_ACCESS_VRDE_PORT" </dev/null >>"$tunnel_log_file" 2>&1 &
+    else
+        nohup sh -c "$watchdog_script" rdp-tunnel-watchdog \
+            "$tunnel_stop_file" "$socket_path" "$RVBOX_TEST_VBOX_HOST" \
+            "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" "$RDP_ACCESS_VRDE_HOST" \
+            "$RDP_ACCESS_VRDE_PORT" </dev/null >>"$tunnel_log_file" 2>&1 &
+    fi
+    tunnel_watchdog_pid=$!
+    printf '%s\n' "$tunnel_watchdog_pid" >"$tunnel_pid_file"
+    chmod 600 "$tunnel_pid_file"
+
+    # A listener is useful only after the SSH master has authenticated and
+    # bound the Docker-network gateway. Give the reconnect loop a bounded
+    # window, then fail with a clear recovery path.
+    attempts=0
+    while [ "$attempts" -lt 20 ]; do
+        if [ -S "$socket_path" ] && ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1; then
+            return 0
+        fi
+        if command -v nc >/dev/null 2>&1 && nc -z -w 2 "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" >/dev/null 2>&1; then
+            return 0
+        fi
+        attempts=$((attempts + 1))
+        sleep 1
+    done
+    stop_tunnel
+    return 1
+}
+
+tunnel_listener_active() {
+    [ -n "${docker_gateway-}" ] || return 1
+    command -v nc >/dev/null 2>&1 || return 1
+    nc -z -w 2 "$docker_gateway" "$RDP_ACCESS_TUNNEL_PORT" >/dev/null 2>&1
+}
+
+tunnel_control_active() {
+    [ -S "$socket_path" ] || return 1
+    ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1
+}
+
+tunnel_active() {
+    tunnel_control_active || tunnel_listener_active
 }
 
 action=${1-}
@@ -220,7 +342,22 @@ case $action in
         docker compose version >/dev/null
         assert_fixture_running
         if compose ps -q | grep -q .; then
-            fail "gateway already exists; use status or down first"
+            # A short-lived controller (or a dropped SSH route) can leave the
+            # Compose stack running after its tunnel has disappeared. Reuse
+            # that stack and repair only the private forwarding path instead
+            # of forcing the operator to tear down a usable Guacamole session.
+            mkdir -p "$runtime_dir"
+            docker_gateway=$(gateway_for_network) || fail "could not determine private Docker gateway"
+            session_matches_current || fail "gateway already exists with a different configuration; run down first, then run up with the desired options"
+            if tunnel_active; then
+                printf 'Guacamole stack and private VRDE tunnel are already active.\n'
+            elif start_tunnel; then
+                printf 'Existing Guacamole stack reused; private VRDE tunnel restored.\n'
+            else
+                fail "existing Guacamole stack was found but the private VRDE tunnel could not be restored; inspect $tunnel_log_file"
+            fi
+            if [ -f "$session_file" ]; then sed -n '1p' "$session_file"; fi
+            exit 0
         fi
         mkdir -p "$runtime_dir"
         compose up -d guacd
@@ -248,8 +385,16 @@ case $action in
         "$repo_root/scripts/windows/test-host" status || true
         if [ -f "$session_file" ]; then sed -n '1p' "$session_file"; fi
         compose ps
-        if [ -S "$socket_path" ] && ssh -S "$socket_path" -O check "$RVBOX_TEST_VBOX_HOST" >/dev/null 2>&1; then
+        docker_gateway=$(gateway_for_network 2>/dev/null || true)
+        if tunnel_control_active; then
             printf 'private_tunnel=active\n'
+            printf 'private_tunnel_supervisor=helper\n'
+        elif tunnel_listener_active; then
+            # A tunnel started by an interactive shell or another supervisor
+            # has no control socket owned by this helper, but it is still a
+            # valid path when the Docker listener is reachable.
+            printf 'private_tunnel=active_external\n'
+            printf 'private_tunnel_supervisor=external\n'
         else
             printf 'private_tunnel=inactive\n'
         fi