docs: publish archive control design set

2026-07-23 13:39:09 +00:00
parent 6ca94a546e
commit 7e295cff3a
12 changed files with 1960 additions and 7 deletions
@@ -0,0 +1,169 @@
+# System design
+
+## Scope and invariants
+
+Archive Control moves already-complete BitTorrent file selections between hot
+cache nodes and durable archive nodes. It does not download missing content,
+change global qBittorrent or Syncthing settings, or decide retention on archive
+nodes.
+
+The non-negotiable invariants are:
+
+1. A committed archive copy is never deleted by a transfer rollback or cleanup.
+2. A cache placement is evicted only by an explicit eviction job after fresh
+   proof that every selected file is covered by online archive placements.
+3. A target is committed only after qBittorrent verifies the selected union in
+   stopped state without attempting a download.
+4. One control daemon is the only cross-node orchestrator. Clients execute one
+   commanded local step at a time.
+5. Every destructive transition is durable, attributable, retryable, and
+   guarded by an expected revision or external-state fingerprint.
+6. File operations stay beneath configured roots, never follow symlinks, and
+   never recursively delete a content tree.
+
+## Components
+
+### Archive control core
+
+The existing mogic-bot process hosts a new `archive_control/` package parallel
+to `remote_control/`. Its core has no Telegram imports and exposes an
+`ArchiveControlService` facade used by:
+
+- the production Telegram handler;
+- a test-only HTTP adapter used by local E2E tests;
+- the WebSocket connection manager for clients.
+
+Internally the core contains configuration, SQLite persistence, connection
+registry, durable command outbox, protocol codec, scheduler, route coordinator,
+inventory coordinator, job state machines, backup manager, and event/view
+projection. The existing bot lifecycle starts and stops these tasks; no second
+bot or independent control process is introduced.
+
+### Archive client
+
+One Python image and codebase runs with role `archive` or `cache`. Each instance
+contains:
+
+- a reconnecting control connection and durable command inbox;
+- a local SQLite journal and backup manager;
+- qBittorrent and Syncthing adapters;
+- safe path and file-operation modules;
+- route discovery/provisioning;
+- inventory and content-tree responders;
+- transfer and eviction step executors.
+
+The role controls policy, not the available transport mechanics. Cache clients
+may source archive jobs, receive unarchive jobs, and perform eviction. Archive
+clients may receive archive jobs and source unarchive jobs.
+
+### External local services
+
+Clients talk to their locally configured qBittorrent and Syncthing APIs. API
+paths may differ from paths inside the client container, so each root is
+configured as an API-visible path plus its daemon-local equivalent. The
+mapping must be one-to-one and is validated at startup.
+
+## Logical data flow
+
+```text
+Telegram handler ─┐
+                  ├─ ArchiveControlService ─ SQLite/control scheduler
+test HTTP adapter ┘             │
+                                │ versioned protobuf JSON over WebSocket
+               ┌────────────────┴─────────────────┐
+        cache client                         archive client
+        ├─ qB adapter                        ├─ qB adapter
+        ├─ Syncthing adapter                 ├─ Syncthing adapter
+        ├─ SQLite journal                    ├─ SQLite journal
+        └─ local file ops                    └─ local file ops
+               └──────── pairwise Syncthing route ────────┘
+```
+
+The control service persists intent and the job's current event-sequence lease
+before emitting a command. The client
+persists command acceptance before acknowledging it. Completion is a later
+event. This separates delivery from execution and makes reconnect replay safe.
+
+## Persistent model
+
+### Control database
+
+The control database contains at least:
+
+- `clients` and last known capabilities/health;
+- connection generations and heartbeat timestamps;
+- discovered/provisioned routes and verification state;
+- immutable job definitions and mutable job projections;
+- per-job ordered events;
+- durable commands, delivery attempts, and acknowledgements;
+- resource reservations;
+- cache/archive placements and generations;
+- bounded inventory snapshots and revisions;
+- persisted Telegram sessions and confirmation nonces;
+- hidden-record tombstones and backup metadata.
+
+### Client database
+
+Each client stores:
+
+- accepted command IDs and their stable acknowledgements;
+- job definitions, local step journals, revisions, and emitted events;
+- file-operation provenance and compensation state;
+- staging manifests and ready-marker observations;
+- route-creation ownership and verification state;
+- last control connection/session cursors;
+- backup metadata.
+
+Credentials, shared tokens, qBittorrent passwords, Syncthing API keys, and
+Telegram secrets are never stored in either database.
+
+### Placements are not job history
+
+A placement is current, verified residency of a resource selection on one
+node. It is keyed by canonical resource identity and client ID and includes a
+generation plus verified file-index set. Jobs mutate placements, but deleting
+or hiding a job record never changes placement data. A resource may have many
+archive and cache placements.
+
+## Scheduling and ownership
+
+Jobs enter a durable FIFO queue while holding a global reservation on every
+known v1/v2 alias of their resource identity. Only one transfer or eviction for
+a resource may be active or queued at a time. This prevents incompatible
+baselines even when jobs would use different nodes or observe different sides
+of a hybrid identity.
+
+Defaults allow one active data-moving job per client and one per route. A job
+must acquire its source client, target client, route, and resource reservation
+atomically. Disjoint node pairs may run concurrently. Route setup is a
+preflight activity and does not permit a data step to bypass these leases.
+
+Offline nodes do not prevent unrelated jobs from being listed or run. A job
+requiring an offline node remains waiting indefinitely; it does not consume an
+active data slot until work can proceed.
+
+## State and recovery
+
+Control state is authoritative for orchestration; neither client independently
+advances to the next step. On reconnect:
+
+1. The client registers with active-job cursors.
+2. Control compares revisions and per-job event sequences.
+3. Missing events are replayed when available; gaps or disagreement trigger a
+   full client/job snapshot.
+4. Control observes relevant qBittorrent, Syncthing, staging, and manifest
+   state before selecting retry, resume, compensation, cleanup, or manual
+   intervention.
+5. A command is reissued with its original ID when the acceptance result is
+   uncertain.
+
+A client never guesses that a job committed. If a database is lost, evidence
+from a verified target, immutable manifest, ready marker, and file-operation
+journal may reconstruct state only when it proves one unique outcome. Otherwise
+the resource remains reserved and the job fails closed for manual
+reconciliation.
+
+Routine no-progress and connectivity conditions become `WAITING` or `STALLED`,
+not failures. Permanent protocol, validation, integrity, unsafe-path,
+unsupported-partfile, or permission conditions fail with a stable code and a
+clear operator-facing reason.