fix: reconnect after silent control heartbeat loss

deploy/production/README.md

@@ -13,10 +13,37 @@ initial x1/x2/lithium topology.
   `archive_control_token`, `qb_password`, and `syncthing_api_key`, each a
   regular non-empty file with mode `0600`.
 
-The Syncthing mounts intentionally reproduce each instance's `/var/syncthing`
-layout, including nested data binds. This lets route discovery and route
-provisioning use one safe API-to-local path mapping without altering an
-existing Syncthing configuration.
+## Hardlink-safe bind-mount topology
+
+For an archive source, the qB content path and every Syncthing route used for
+staging must resolve through the **same container mount**. Matching host
+filesystem device IDs alone is insufficient: two separate Docker bind mounts
+have different mount IDs and `link(2)` may return `EXDEV` across them. The
+client deliberately treats that case as copy-only and performs a full payload
+free-space check.
+
+When a Syncthing route is physically nested below the qB root, mount the qB
+root once and map the exact Syncthing API folder through it:
+
+```yaml
+volumes:
+  - /srv/downloads:/data/qb
+  - /srv/syncthing-config:/data/sync
+```
+
+```toml
+[syncthing]
+api_root = "/var/syncthing"
+local_root = "/data/sync"
+local_path_overrides = { "/var/syncthing/Downloads/Sync" = "/data/qb/Sync" }
+```
+
+Do **not** additionally mount `/srv/downloads/Sync` at a path beneath
+`/data/sync`. The override is the authoritative mapping for that folder and
+keeps qB source files and staging destinations in one mount namespace. Use
+the folder's normalized API-visible **path** as the override key (for example,
+Syncthing `~/Downloads/Sync` becomes `/var/syncthing/Downloads/Sync`); do not
+use the folder ID.
 
 Before starting a stack, validate it with:
 

deploy/production/lithium/compose.yaml

@@ -2,7 +2,7 @@ name: archive-control-archive
 
 services:
   archive-client:
-    image: sodium/archive-clients:v0.1.11
+    image: sodium/archive-clients:v0.1.13
     user: "1000:1000"
     restart: unless-stopped
     command: ["--config", "/etc/archive-control/client.toml"]

deploy/production/x1/compose.yaml

@@ -2,7 +2,7 @@ name: archive-control-cache
 
 services:
   archive-client:
-    image: sodium/archive-clients:v0.1.11
+    image: sodium/archive-clients:v0.1.13
     user: "1001:1001"
     restart: unless-stopped
     network_mode: host

deploy/production/x2/client.toml

@@ -39,4 +39,8 @@ endpoint = "http://127.0.0.1:8384"
 api_key_file = "/run/secrets/syncthing_api_key"
 api_root = "/var/syncthing"
 local_root = "/data/sync"
+# `DownloadsSync-X2` is ~/Downloads/Sync on the host, nested below the qB
+# root. Resolve it through /data/qb rather than a second nested bind mount so
+# source staging can hardlink it.
+local_path_overrides = { "/var/syncthing/Downloads/Sync" = "/data/qb/Sync" }
 advertised_addresses = ["dynamic"]

deploy/production/x2/compose.yaml

@@ -2,7 +2,7 @@ name: archive-control-cache
 
 services:
   archive-client:
-    image: sodium/archive-clients:v0.1.11
+    image: sodium/archive-clients:v0.1.13
     user: "1001:1001"
     restart: unless-stopped
     network_mode: host
@@ -16,4 +16,3 @@ services:
       - ./backups:/var/backups/archive-control
       - /home/ubuntu/Downloads:/data/qb
       - /home/ubuntu/compose/syncthing/st_home:/data/sync
-      - /home/ubuntu/Downloads/Sync:/data/sync/Downloads/Sync

docs/deployment-and-usage.md

@@ -128,7 +128,12 @@ advertised_addresses = ["dynamic"]
 When an existing Syncthing folder is physically nested in the qB data root,
 use a `local_path_overrides` entry to map that exact Syncthing API path through
 the same client bind mount. This enables hardlinks without creating two Docker
-mount boundaries for the same host files.
+mount boundaries for the same host files. Do not add a second bind mount for
+the nested folder: Linux treats it as a distinct mount even when it has the
+same `st_dev`, and the client correctly falls back to copy-only capacity
+accounting. The override key is the normalized Syncthing folder path beneath
+`api_root`, not its folder ID. See the production deployment README for the
+required compose and override pattern.
 
 The remaining node examples omit optional `[connection]`, `[jobs]`, and
 `[backup]` tables and therefore use these same defaults; deployments may
@@ -218,7 +223,7 @@ cache/archive routes according to policy.
 ```yaml
 services:
   archive-client:
-    image: sodium/archive-clients:v0.1.11
+    image: sodium/archive-clients:v0.1.13
     user: "1001:1001"
     restart: unless-stopped
     command: ["archive-client", "--config", "/etc/archive-control/client.toml"]

docs/service-apis.md

@@ -89,9 +89,11 @@ left unchanged.
 
 Syncthing may serialize a folder path relative to its home as `~/...`. The
 client normalizes that notation beneath the configured API-visible sync root
-before applying the API-to-local root mapping. Deployments must mirror
-Syncthing's nested bind mounts into the client so the normalized API path and
-the client filesystem path refer to the same bytes.
+before applying the API-to-local root mapping. When that folder is nested
+below qB's content root, an exact `local_path_overrides` entry must map it
+through the qB bind mount. Do not mirror it as a second nested client bind
+mount: it denotes the same host bytes but a distinct mount namespace boundary,
+which prevents hardlink staging.
 
 Provisioning uses idempotent device and folder configuration updates. Each
 client receives its peer device ID and optional advertised addresses

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "archive-clients"
-version = "0.1.11"
+version = "0.1.13"
 requires-python = ">=3.11"
 dependencies = ["protobuf==7.35.1", "websockets==16.0"]
 

src/archive_clients/daemon.py

@@ -11,6 +11,7 @@ from pathlib import Path, PurePosixPath
 from typing import Any
 
 from websockets.asyncio.client import connect
+from websockets.exceptions import ConnectionClosedOK
 
 from archive_clients.backup import SQLiteBackupManager
 from archive_clients.config import ClientConfig
@@ -205,7 +206,23 @@ class ArchiveClientDaemon:
             command_tasks: set[asyncio.Task[None]] = set()
             try:
                 await self._resume_commands(outbound, command_tasks)
-                async for frame in websocket:
+                # A proxy can leave the TCP/WebSocket socket apparently open
+                # after the control server has discarded its session.  The
+                # server then cannot deliver durable commands and its pending
+                # outbox remains stranded unless the client independently
+                # detects the missing application heartbeats and reconnects.
+                while True:
+                    try:
+                        frame = await asyncio.wait_for(
+                            websocket.recv(),
+                            self.config.connection.offline_timeout,
+                        )
+                    except ConnectionClosedOK:
+                        return
+                    except asyncio.TimeoutError as exc:
+                        raise RuntimeError(
+                            "control heartbeat timed out"
+                        ) from exc
                     await self._handle(decode(frame), outbound, command_tasks)
             finally:
                 writer.cancel()

src/archive_clients/syncthing.py

@@ -192,11 +192,19 @@ class SyncthingTransferObserver:
                 for relative_path, expected_bytes in declared_files
             )
             observed_fraction = observed_bytes / total if total else 1.0
-            # Folder completion remains useful when Syncthing has already
-            # atomically published a file but still reports it in its need
-            # queue; the allocated-byte estimate is needed for sparse temp
-            # files that are pre-sized before their blocks arrive.
-            fraction = min(observed_fraction, float(raw_fraction) / 100)
+            # ``/db/completion`` is folder-wide.  It can be near zero when a
+            # route contains a freshly-created item even though the tracked
+            # job's temporary payload has already received many blocks.  For
+            # an incomplete temporary file, its allocated blocks are the only
+            # job-specific signal, so do not cap them with that unrelated
+            # folder aggregate.  Once every declared file is atomically
+            # present, retain the completion value as a conservative guard
+            # until the need queue has caught up.
+            fraction = (
+                observed_fraction
+                if observed_fraction < 1.0
+                else min(1.0, float(raw_fraction) / 100)
+            )
             completed_bytes = int(total * fraction)
         if complete:
             fraction = 1.0

tests/test_daemon.py

@@ -22,6 +22,53 @@ from archive_control.v1 import (
 
 
 class DaemonTransportTests(unittest.IsolatedAsyncioTestCase):
+    async def test_silent_control_connection_ends_for_reconnect(self):
+        """A lost server heartbeat must not leave durable commands stranded."""
+
+        async def control(websocket):
+            registration = decode(await websocket.recv())
+            self.assertEqual(registration.WhichOneof("payload"), "register_request")
+            response = new_envelope()
+            response.correlation_id = registration.message_id
+            response.register_response.status = (
+                client_pb2.REGISTRATION_STATUS_ACCEPTED
+            )
+            response.register_response.negotiated_version.major = 1
+            await websocket.send(encode(response))
+            # Deliberately keep TCP/WebSocket open but send no application
+            # heartbeats. This models a stale proxy/server-side session.
+            await websocket.wait_closed()
+
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            token = root / "token"
+            token.write_text("shared-secret", encoding="utf-8")
+            os.chmod(token, 0o600)
+            async with serve(control, "127.0.0.1", 0, ping_interval=None) as server:
+                port = server.sockets[0].getsockname()[1]
+                service = ServiceConfig(
+                    "http://local", PurePosixPath("/api"), root,
+                )
+                config = ClientConfig(
+                    "cache-1", "Cache 1", "cache",
+                    f"ws://127.0.0.1:{port}", token,
+                    root / "state.db", root / "backups", service, service,
+                    ConnectionConfig(
+                        registration_timeout=1,
+                        offline_timeout=0.05,
+                        reconnect_initial=0.01,
+                        reconnect_max=0.01,
+                        reconnect_jitter=False,
+                    ),
+                )
+                probe = FilesystemProbe(root, True, True, True, True, True)
+                daemon = ArchiveClientDaemon(config, [probe, probe], [])
+                await asyncio.to_thread(daemon.store.initialize)
+                with self.assertRaisesRegex(
+                    RuntimeError, "control heartbeat timed out"
+                ):
+                    await asyncio.wait_for(daemon._connection(), 1)
+
     async def test_eviction_assignment_and_steps_are_admitted(self):
         with tempfile.TemporaryDirectory() as directory:
             root = Path(directory)