archive-clients/README.md

Archive Clients

One Python daemon runs as either an archive or cache node for Archive Control. Both roles use the same code and sodium/archive-clients image.

The current foundation provides strict TOML configuration, file-backed secrets, API/local root mapping, fail-fast permission and sparse-file probes, a durable SQLite command inbox, bounded one-writer WebSocket output, registration-first authentication, heartbeat handling, duplicate-command acknowledgements, and indefinite capped exponential reconnect with optional jitter. The state DB has an exclusive process lease plus checksummed, integrity-verified online backups with recent/daily/weekly retention. Startup probes report qBittorrent/Web API/libtorrent versions, Syncthing version/device identity, service health, and per-root hardlink/reflink/sparse support without exposing local paths to control.

The first read-only discovery layer strictly decodes v1/v2/hybrid metainfo, derives canonical resource identities and selection ranges, builds lazy content trees, and reports only safe Syncthing folders below the configured root. Renamed/noncanonical torrent paths remain visible but are marked noncanonical so later job preflight can reject them without hiding the resource. The qBittorrent adapter uses cookie authentication, bounded responses, one reauthentication attempt on session expiry, and hash-scoped file/metainfo fetches. Its mutation boundary adds targets stopped, applies only selected vs skipped state, supports qBittorrent 4 pause and qBittorrent 5 stop semantics, performs guarded full rechecks, and removes entries without requesting data deletion. It never logs credentials, cookies, response bodies, or endpoints. Durable inventory commands now stream bounded atomic summary, lookup, and content-tree chunks. Page tokens are guarded by timestamp-independent snapshot revisions, stale trees fail explicitly, and slow scans run outside the socket reader so heartbeat acknowledgements remain responsive. Durable EnsureRoute commands use scoped Syncthing device/folder REST updates, safe API/local path mapping, exact pairwise configuration read-back, and bidirectional nonce/acknowledgement files. Route attempt nonces, ownership, and ordered updates survive restart; accepted attempts resume on reconnect without concurrent duplicate execution. Conflicting existing folders and foreign data are reported without being overwritten. The transfer storage foundation now publishes isolated per-job namespaces with canonical protobuf-JSON manifests and digest-bound ready markers. Selected regular files and metainfo artifacts are materialized with hardlink, reflink, then sparse-aware copy fallback; every intent/result is journaled in SQLite. Receiver materialization never overwrites a path, can reuse a same-size regular file for later qBittorrent verification, and replays completed operations idempotently after a database reopen. The Syncthing transfer observer requests scoped rescans and combines verified local manifest payloads, folder completion, and outstanding job-prefix needs; an advisory percentage alone is never treated as transfer completion.

archive-client --config /etc/archive-control/client.toml --check-config
archive-client --config /etc/archive-control/client.toml --mode archive
archive-client-backup --database /var/lib/archive-control/client.db \
  --backup-dir /var/backups/archive-control list

--mode accepts only archive or cache and overrides the configured role. Secrets must be regular files without group/world permissions. The daemon never stores them in SQLite or sends the shared token after registration.

The daemon executes heartbeat, inventory, state-snapshot, route-provisioning, archive/unarchive assignment and step commands, compensation, staging cleanup, and safe cache eviction. Durable journals and job cursors make interrupted work replayable; unsupported or invalid assignments are rejected explicitly and never falsely acknowledged as accepted.

Run tests and build using containers:

docker build --target test -t archive-clients-test .
docker run --rm archive-clients-test
docker build -t sodium/archive-clients:dev .

The local cross-project route test is under e2e/. It brings up isolated 2×2 cache/archive node stacks plus the standalone control core and verifies all four eager-mesh routes, transfer/eviction/unarchive workflows, and the adversarial matrix without live infrastructure or Telegram. See e2e/README.md.

The complete cross-project design, protocol, workflow, safety, deployment, Telegram UX, and testing documentation is published under docs/.

Generated bindings are pinned to archive-control-proto commit 4ec852014dad74606d4078b3ae1aa208c814b033.