archive-clients/docs/system-design.md

# System design

## Scope and invariants

Archive Control moves already-complete BitTorrent file selections between hot
cache nodes and durable archive nodes. It does not download missing content,
change global qBittorrent or Syncthing settings, or decide retention on archive
nodes.

The non-negotiable invariants are:

1. A committed archive copy is never deleted by a transfer rollback or cleanup.
2. A cache placement is evicted only by an explicit eviction job after fresh
   proof that every selected file is covered by online archive placements.
3. A target is committed only after qBittorrent verifies the selected union in
   stopped state without attempting a download.
4. One control daemon is the only cross-node orchestrator. Clients execute one
   commanded local step at a time.
5. Every destructive transition is durable, attributable, retryable, and
   guarded by an expected revision or external-state fingerprint.
6. File operations stay beneath configured roots, never follow symlinks, and
   never recursively delete a content tree.

## Components

### Archive control core

The existing mogic-bot process hosts a new `archive_control/` package parallel
to `remote_control/`. Its core has no Telegram imports and exposes an
`ArchiveControlService` facade used by:

- the production Telegram handler;
- a test-only HTTP adapter used by local E2E tests;
- the WebSocket connection manager for clients.

Internally the core contains configuration, SQLite persistence, connection
registry, durable command outbox, protocol codec, scheduler, route coordinator,
inventory coordinator, job state machines, backup manager, and event/view
projection. The existing bot lifecycle starts and stops these tasks; no second
bot or independent control process is introduced.

### Archive client

One Python image and codebase runs with role `archive` or `cache`. Each instance
contains:

- a reconnecting control connection and durable command inbox;
- a local SQLite journal and backup manager;
- qBittorrent and Syncthing adapters;
- safe path and file-operation modules;
- route discovery/provisioning;
- inventory and content-tree responders;
- transfer and eviction step executors.

The role controls policy, not the available transport mechanics. Cache clients
may source archive jobs, receive unarchive jobs, and perform eviction. Archive
clients may receive archive jobs and source unarchive jobs.

### External local services

Clients talk to their locally configured qBittorrent and Syncthing APIs. API
paths may differ from paths inside the client container, so each root is
configured as an API-visible path plus its daemon-local equivalent. The
mapping must be one-to-one and is validated at startup.

## Logical data flow

```text
Telegram handler ─┐
                  ├─ ArchiveControlService ─ SQLite/control scheduler
test HTTP adapter ┘             │
                                │ versioned protobuf JSON over WebSocket
               ┌────────────────┴─────────────────┐
        cache client                         archive client
        ├─ qB adapter                        ├─ qB adapter
        ├─ Syncthing adapter                 ├─ Syncthing adapter
        ├─ SQLite journal                    ├─ SQLite journal
        └─ local file ops                    └─ local file ops
               └──────── pairwise Syncthing route ────────┘
```

The control service persists intent and the job's current event-sequence lease
before emitting a command. The client
persists command acceptance before acknowledging it. Completion is a later
event. This separates delivery from execution and makes reconnect replay safe.

## Persistent model

### Control database

The control database contains at least:

- `clients` and last known capabilities/health;
- connection generations and heartbeat timestamps;
- discovered/provisioned routes and verification state;
- immutable job definitions and mutable job projections;
- per-job ordered events;
- durable commands, delivery attempts, and acknowledgements;
- resource reservations;
- cache/archive placements and generations;
- bounded inventory snapshots and revisions;
- persisted Telegram sessions and confirmation nonces;
- hidden-record tombstones and backup metadata.

### Client database

Each client stores:

- accepted command IDs and their stable acknowledgements;
- job definitions, local step journals, revisions, and emitted events;
- file-operation provenance and compensation state;
- staging manifests and ready-marker observations;
- route-creation ownership and verification state;
- last control connection/session cursors;
- backup metadata.

Credentials, shared tokens, qBittorrent passwords, Syncthing API keys, and
Telegram secrets are never stored in either database.

### Placements are not job history

A placement is current, verified residency of a resource selection on one
node. It is keyed by canonical resource identity and client ID and includes a
generation plus verified file-index set. Jobs mutate placements, but deleting
or hiding a job record never changes placement data. A resource may have many
archive and cache placements.

## Scheduling and ownership

Jobs enter a durable FIFO queue while holding a global reservation on every
known v1/v2 alias of their resource identity. Only one transfer or eviction for
a resource may be active or queued at a time. This prevents incompatible
baselines even when jobs would use different nodes or observe different sides
of a hybrid identity.

By default every eligible queued job is claimed in the same scheduler pass and
independent jobs run concurrently on each client; physical service and storage
capacity are the limit. Set `enforce_concurrency_limits=true` on control to
enable the optional one-per-client/one-per-route gates. A job always retains
its resource reservation, and commands for the same job remain serialized.
Route setup is a preflight activity and does not permit a data step to bypass
the resource reservation.

Offline nodes do not prevent unrelated jobs from being listed or run. A job
requiring an offline node remains waiting indefinitely; it does not consume an
active data slot until work can proceed.

## State and recovery

Control state is authoritative for orchestration; neither client independently
advances to the next step. On reconnect:

1. The client registers with active-job cursors.
2. Control compares revisions and per-job event sequences.
3. Missing events are replayed when available; gaps or disagreement trigger a
   full client/job snapshot.
4. Control observes relevant qBittorrent, Syncthing, staging, and manifest
   state before selecting retry, resume, compensation, cleanup, or manual
   intervention.

If a connection disappears while a synchronous job operation is in progress,
the client journals events locally, reconnects indefinitely, and serializes a
replayed command behind the in-flight operation for that job. The replay reads
the journal rather than repeating the data operation. This applies to every
transfer step, including post-commit staging cleanup.
5. A command is reissued with its original ID when the acceptance result is
   uncertain.

A client never guesses that a job committed. If a database is lost, evidence
from a verified target, immutable manifest, ready marker, and file-operation
journal may reconstruct state only when it proves one unique outcome. Otherwise
the resource remains reserved and the job fails closed for manual
reconciliation.

Routine no-progress and connectivity conditions become `WAITING` or `STALLED`,
not failures. Permanent protocol, validation, integrity, unsafe-path,
unsupported-partfile, or permission conditions fail with a stable code and a
clear operator-facing reason.