diff --git a/README.md b/README.md index d142149..0a71ca3 100644 --- a/README.md +++ b/README.md @@ -57,12 +57,11 @@ archive-client-backup --database /var/lib/archive-control/client.db \ Secrets must be regular files without group/world permissions. The daemon never stores them in SQLite or sends the shared token after registration. -The daemon currently executes heartbeat, inventory, state-snapshot, and -route-provisioning commands. Transfer storage primitives are implemented and -tested but are not yet wired to assignment/step commands; those and eviction -commands remain durably rejected as unsupported until their orchestration and -qBittorrent executors are added. They are never falsely acknowledged as -accepted. +The daemon executes heartbeat, inventory, state-snapshot, route-provisioning, +archive/unarchive assignment and step commands, compensation, staging cleanup, +and safe cache eviction. Durable journals and job cursors make interrupted +work replayable; unsupported or invalid assignments are rejected explicitly +and never falsely acknowledged as accepted. Run tests and build using containers: @@ -74,7 +73,12 @@ docker build -t sodium/archive-clients:dev . The local cross-project route test is under `e2e/`. It brings up isolated 2×2 cache/archive node stacks plus the standalone control core and verifies all -four eager-mesh routes without Telegram. See `e2e/README.md`. +four eager-mesh routes, transfer/eviction/unarchive workflows, and the +adversarial matrix without live infrastructure or Telegram. See +`e2e/README.md`. + +The complete cross-project design, protocol, workflow, safety, deployment, +Telegram UX, and testing documentation is published under [`docs/`](docs/). Generated bindings are pinned to archive-control-proto commit `4ec852014dad74606d4078b3ae1aa208c814b033`. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..c703cfc --- /dev/null +++ b/docs/README.md @@ -0,0 +1,41 @@ +# Archive Control design documentation + +This directory is the implementation contract for Archive Control. These +documents incorporate the decisions made during the design review and +supersede conflicting details in the original private problem statement. That +statement is intentionally not published because it contains deployment +credentials and host-specific details. + +## Reading order + +1. [Decisions](decisions.md) +2. [Implementation plan](implementation-plan.md) +3. [System design](system-design.md) +4. [Resource and workflow model](workflows.md) +5. [Daemon protocol](protocol.md) +6. [Storage safety and recovery](storage-safety.md) +7. [qBittorrent and Syncthing integration](service-apis.md) +8. [Configuration, deployment, and usage](deployment-and-usage.md) +9. [Telegram UX](telegram-ux.md) +10. [Testing strategy](testing.md) + +The independent protobuf source of truth lives in the +`cabbage/archive-control-proto` repository. Generated bindings in this +repository record the exact source commit. + +## Implementation status + +As of 2026-07-23, implementation phases 0–7 are complete across the proto, +control, and client repositories. This includes registration, recovery, +on-demand inventory, automatic route provisioning, transfer preview and delta +planning, selective archive/unarchive merging, durable execution and +compensation, safe cache eviction, database backup/restore, and the integrated +Telegram flows. + +The local 2×2 topology passes archive, eviction, and unarchive happy paths plus +selective/partial merges, multiple archive coverage, restart/replay, +pure-v2/hybrid identities, stalls, disk exhaustion, hostile preconditions, and +protocol duplication/reordering. The integrated bot also has an isolated dummy +backend for interactive Telegram conversation review and trace export. +Remaining work is the broader release/chaos matrix and the separately approved +staged deployment described in phases 8–9. diff --git a/docs/decisions.md b/docs/decisions.md new file mode 100644 index 0000000..e79ff11 --- /dev/null +++ b/docs/decisions.md @@ -0,0 +1,137 @@ +# Agreed design decisions + +This is the concise decision ledger from the design interview. Later documents +expand these rules but must not contradict them. + +## Roles and authority + +- Client roles are `archive` and `cache`. `archive` and `unarchive` are job + operations, not daemon roles. +- Titan's control daemon is the sole cross-node orchestrator. Clients never + command one another and do not advance to the next workflow step without a + persisted Titan command. +- Archive nodes retain committed resources. Cache eviction is a distinct, + explicit operation. +- Transfer and eviction jobs are durable, idempotent, recoverable, and + serialized globally per torrent resource. + +## Connection and protocol + +- Every WebSocket connection starts with a registration containing client ID, + role, protocol/capabilities, and the single pre-shared token. The token occurs + in no later message. Invalid registration closes the connection. +- A new valid registration for a duplicate client ID disconnects the old + connection. +- Clients reconnect forever with configurable capped exponential backoff. +- Commands are at-least-once, durably deduplicated, and separate acceptance + acknowledgements from completion events. +- Protobuf JSON uses a versioned `archive_control.v1` envelope. Per-job event + sequences support gap detection and snapshot reconciliation. + +## State, recovery, and scheduling + +- Titan and every client have SQLite state. They reconcile active jobs on every + reconnect and fail closed when lost state cannot be proven from journals, + manifests, and observed external state. +- SQLite backups use the online backup API. Defaults are every six hours, 12 + recent, 14 daily, and eight weekly copies, plus pre/post-migration backups. +- Default concurrency is one active data-moving job per client and per route. + Disjoint node pairs may run concurrently. Queueing is durable FIFO. +- A queued job owns a per-resource reservation. Cancelling it removes only the + queued record/reservation and never sends cleanup commands. +- Connectivity or transfer stalls wait indefinitely. A configurable 30-minute + no-progress threshold marks the job `STALLED` without losing its percentage. + +## Resources and selections + +- Resource identity supports BitTorrent v1, v2, and hybrid info hashes. A + qBittorrent-local hash is a separate adapter identifier. Hybrid hash aliases + share reservations and are joined only when no supplied hash conflicts. +- Only selected versus skipped matters; qBittorrent priority levels are not + preserved. +- Placements are keyed by resource and node and contain a verified set of + selected torrent file indices. Multiple cache and archive placements are + allowed. +- Transfer jobs have one source and one target. A requested selection is + expanded to immutable file indices. Targets merge the missing delta into + their existing selection transactionally. +- Initial Telegram selection supports either all source-selected complete + entries or one file/directory. The protocol supports multiple entries from + v1. +- Full source and target file inventories are fetched immediately before + staging, even for the normal `Confirm` path. The previewed baseline and delta + are persisted immutably and must still match at execution. + +## Transfer safety + +- Each transfer uses `/.archive-control/jobs//` with a + payload, exported torrent, protobuf-JSON manifest, auxiliary artifacts, and a + ready marker. +- File materialization tries hard link, then reflink, then safe copy. It never + follows symlinks or special files and never escapes configured roots. +- Pre-existing target paths are never overwritten. Same-size files may be + reused only after qBittorrent's stopped recheck proves them valid. +- New target torrents are explicitly placed under the configured qB root, + added stopped with manual torrent management, assigned selected/skipped + states, fully rechecked, and started only after successful verification. +- Existing target merges preserve a durable baseline. Failure restores the + baseline selection/state and removes only attributable delta data. +- qBittorrent/libtorrent partfiles are transferred or merged only through a + tested version-aware adapter. An unsafe format fails job preflight with a + clear reason. +- Sparse support is probed once at client startup for configured roots. Jobs + involving sparse files require semantic sparse support on every leg; exact + physical extent layouts are not required. +- Permission handling is fail-fast. The daemon never chmods/chowns source data. + +## Commit, cleanup, and deletion + +- A transfer commits after the target's selected union passes a stopped full + recheck without downloading. After commit, the new placement is retained. +- Staging cleanup is post-commit. Its failure is `CLEANUP_REQUIRED`, not a + failed transfer, and cannot cause committed archive data to be deleted. +- Archive jobs no longer delete their source cache placement. +- Eviction is a durable job. It requires fresh, complete archive coverage for + every selected cache file; coverage may be the union of multiple online + archive nodes. There is no force override initially. +- Eviction removes the qBittorrent entry without data deletion, checks exact + path overlap with remaining torrents, unlinks eligible known files, and uses + non-recursive bottom-up directory removal. It never blindly removes a tree. + +## Syncthing routes + +- Client config supplies Syncthing endpoint/API key and separate API-visible + and daemon-local sync roots. Compatible `sendreceive` folders beneath the + root are discovered automatically. +- Syncthing's native folder ID is the route ID. Routes are dedicated to exactly + two devices: one archive and one cache client. +- Missing routes can be created `on_demand` (default) or by `eager_mesh`. + Clients automatically pair missing Syncthing devices and create/verify + folders. Optional advertised addresses default to `dynamic`. +- Route setup timeout defaults to 30 minutes. Archive Control never removes a + verified route merely because a node is absent. + +## User interface and retention + +- Main actions are Archive, Unarchive, Evict Cache, and Job Status. +- Telegram state is shared across authorized admins. Jobs/events store no + Telegram user owner. +- Existing private-chat `@username` authorization remains compatible; optional + numeric user-ID pins harden configured entries when available. +- Archive Control's Telegram view/filter/confirmation sessions persist in + SQLite. Destructive confirmations use single-use nonces. +- Telegram progress edits are coalesced to at most once every five seconds by + default, with immediate important transitions. +- Clear/Remove hides completed records without touching placements. Minimal + idempotency tombstones remain for 30 days by default. + +## Delivery and testing + +- `archive-control-proto` is drafted and reviewed before application code. +- Host tooling stays light. Buf, protoc, tests, and builds run in Docker and + preserve host UID/GID `1001:1001`. +- The client image is `sodium/archive-clients`, published for linux/amd64 and + linux/arm64 with immutable release tags. +- Cross-project E2E lives in this coordination repository and simulates two + cache nodes, two archive nodes, and a thin control node. It never mutates the + live x1/x2/lithium services. diff --git a/docs/deployment-and-usage.md b/docs/deployment-and-usage.md new file mode 100644 index 0000000..38db01f --- /dev/null +++ b/docs/deployment-and-usage.md @@ -0,0 +1,267 @@ +# Configuration, deployment, and usage + +## Configuration rules + +Both daemons parse strict TOML: unknown keys, duplicate keys, invalid types, +unsafe relative paths, and missing required secrets fail startup. Effective +precedence is command line, environment, TOML, then documented default. Secret +fields support a direct environment value or a `*_file` path, never both. +Environment interpolation is explicit `${NAME}` syntax; an unset value fails. + +`--print-effective-config` emits a validated, fully resolved configuration with +all secrets redacted. Durations accept a documented unit string. Paths shown to +the APIs and paths visible inside the daemon are configured as pairs and tested +for consistent mapping before registration. + +Important defaults: + +- registration deadline `10s`; +- heartbeat interval `15s`, offline timeout `45s`; +- unacknowledged command retry timeout `15s`, three total attempts; +- reconnect `1s` initial, `60s` cap, jitter enabled; +- route policy `on_demand`, setup timeout `30m`; +- stall warning `30m`, with no automatic failure timeout; +- one active data job per client and route; +- Telegram coalesce window `5s`, UI session TTL `24h`; +- tombstone retention `30d`; +- database backup interval `6h`, retention `12/14/8`. + +## Control configuration example + +```toml +[archive_control] +enabled = true +listen = "127.0.0.1:8765" +websocket_path = "/archive_control" +shared_token_file = "/run/secrets/archive_control_token" +database = "/var/lib/mogic/archive-control.db" +backup_dir = "/var/backups/mogic/archive-control" +registration_timeout = "10s" +heartbeat_interval = "15s" +offline_timeout = "45s" +command_ack_timeout = "15s" # Delay before resending the same command ID. +command_max_attempts = 3 # Total sends, including the initial attempt. +stall_after = "30m" # Warning state only; jobs continue waiting. +route_policy = "on_demand" # Alternative: eager_mesh. +route_setup_timeout = "30m" +max_active_per_client = 1 +max_active_per_route = 1 +max_envelope_bytes = 1048576 +inventory_chunk_bytes = 524288 + +[archive_control.backup] +interval = "6h" +recent = 12 +daily = 14 +weekly = 8 + +[archive_control.telegram] +progress_coalesce = "5s" +session_ttl = "24h" +tombstone_retention = "30d" +# Existing username allowlist remains authoritative when no ID is pinned. +admin_user_ids = { "@alice" = 123456789 } +``` + +The test-only HTTP adapter has a separate `test_http.enabled` flag that is +false by default and omitted from production Compose configuration. + +```toml +[archive_control.test_http] +enabled = false # Enable only in the isolated local E2E control stack. +listen = "127.0.0.1:0" # Non-loopback binds are rejected. +max_body_bytes = 1048576 +``` + +## Client configuration examples + +All nodes use the same image. The examples show two active cache and two active +archive nodes. Values are illustrative; credentials are mounted secrets. + +```toml +# cache-1.toml +client_id = "cache-1" +display_name = "Cache 1" +role = "cache" +control_endpoint = "ws://control:8765/archive_control" +shared_token_file = "/run/secrets/archive_control_token" +state_db = "/var/lib/archive-control/client.db" +backup_dir = "/var/backups/archive-control" + +[connection] +registration_timeout = "10s" # First response must arrive within this window. +heartbeat_interval = "15s" # Server may negotiate a different effective value. +offline_timeout = "45s" +reconnect_initial = "1s" +reconnect_max = "60s" # Retry forever, never wait longer than this. +reconnect_reset_after = "60s" +reconnect_jitter = true + +[jobs] +stall_after = "30m" # Warning only; no automatic job failure. +verification_timeout = "30m" # Stopped qB full-recheck deadline. +poll_interval = "1s" # Active qB/Syncthing observation interval. +# Worst-case copy fallback must leave this many bytes free. +free_space_reserve_bytes = 1073741824 + +[backup] +interval = "6h" +recent = 12 +daily = 14 +weekly = 8 + +[qbittorrent] +endpoint = "http://qbittorrent:8080" +username = "${QB_USER}" +password_file = "/run/secrets/qb_password" +api_root = "/downloads" +local_root = "/data/qb" + +[syncthing] +endpoint = "http://syncthing:8384" +api_key_file = "/run/secrets/syncthing_api_key" +api_root = "/sync" +local_root = "/data/sync" +advertised_addresses = ["dynamic"] +``` + +The remaining node examples omit optional `[connection]`, `[jobs]`, and +`[backup]` tables and therefore use these same defaults; deployments may +override them per node. + +```toml +# cache-2.toml: same service layout, distinct stable identity/state volumes. +client_id = "cache-2" +display_name = "Cache 2" +role = "cache" +control_endpoint = "ws://control:8765/archive_control" +shared_token_file = "/run/secrets/archive_control_token" +state_db = "/var/lib/archive-control/client.db" +backup_dir = "/var/backups/archive-control" + +[qbittorrent] +endpoint = "http://qbittorrent:8080" +username = "${QB_USER}" +password_file = "/run/secrets/qb_password" +api_root = "/downloads" +local_root = "/data/qb" + +[syncthing] +endpoint = "http://syncthing:8384" +api_key_file = "/run/secrets/syncthing_api_key" +api_root = "/sync" +local_root = "/data/sync" +advertised_addresses = ["dynamic"] +``` + +```toml +# archive-1.toml +client_id = "archive-1" +display_name = "Archive 1" +role = "archive" +control_endpoint = "ws://control:8765/archive_control" +shared_token_file = "/run/secrets/archive_control_token" +state_db = "/var/lib/archive-control/client.db" +backup_dir = "/var/backups/archive-control" + +[qbittorrent] +endpoint = "http://qbittorrent:8080" +username = "${QB_USER}" +password_file = "/run/secrets/qb_password" +api_root = "/archive" +local_root = "/data/archive" + +[syncthing] +endpoint = "http://syncthing:8384" +api_key_file = "/run/secrets/syncthing_api_key" +api_root = "/sync" +local_root = "/data/sync" +advertised_addresses = ["dynamic"] +``` + +```toml +# archive-2.toml +client_id = "archive-2" +display_name = "Archive 2" +role = "archive" +control_endpoint = "ws://control:8765/archive_control" +shared_token_file = "/run/secrets/archive_control_token" +state_db = "/var/lib/archive-control/client.db" +backup_dir = "/var/backups/archive-control" + +[qbittorrent] +endpoint = "http://qbittorrent:8080" +username = "${QB_USER}" +password_file = "/run/secrets/qb_password" +api_root = "/archive" +local_root = "/data/archive" + +[syncthing] +endpoint = "http://syncthing:8384" +api_key_file = "/run/secrets/syncthing_api_key" +api_root = "/sync" +local_root = "/data/sync" +advertised_addresses = ["dynamic"] +``` + +No explicit route matrix is configured. Clients discover existing compatible +folders, report their Syncthing device ID, and control creates missing +cache/archive routes according to policy. + +## Client Compose example + +```yaml +services: + archive-client: + image: sodium/archive-clients:v0.1.0 + user: "1001:1001" + restart: unless-stopped + command: ["archive-client", "--config", "/etc/archive-control/client.toml"] + environment: + QB_USER: admin + volumes: + - ./client.toml:/etc/archive-control/client.toml:ro + - ./secrets:/run/secrets:ro + - ./state:/var/lib/archive-control + - ./backups:/var/backups/archive-control + - ./qb-data:/data/qb + - ./sync-data:/data/sync +``` + +Archive nodes mount `/data/archive` instead of `/data/qb`. Mounts must line up +with the configured local roots and the service containers' API-visible roots. +The daemon runs unprivileged; directory ownership is prepared by the operator, +and mismatch causes fail-fast startup. + +## Image and release policy + +The production client image contains application source and dependencies; it +does not bind-mount source code. A multi-stage Dockerfile uses a slim Python +base and removes build-only packages/caches. Buildx publishes +`sodium/archive-clients:` for `linux/amd64` and `linux/arm64`, records +the multi-platform digest, and may update `latest`. Deployments pin the +immutable semantic tag or digest. + +Generated protobuf Python bindings are committed in each consumer with the +exact `archive-control-proto` tag/commit recorded. Release order is proto, +control consumer, client consumer, E2E, then image publication. + +## Operator usage + +1. Prepare local qB, sync, state, backup, config, and secret mounts with the + container UID/GID (default example `1001:1001`). +2. Run a config validation/effective-config command before starting. +3. Start clients. Confirm service health, sparse/root probes, and registration. +4. Allow route discovery/provisioning to complete when first needed. +5. Use Telegram Archive/Unarchive; review the exact source, target, and + selection before confirming. +6. Use Evict Cache separately only when its fresh coverage check is enabled. +7. Treat orange Cleanup Required as a committed transfer needing staging + cleanup; do not manually remove the target placement. +8. Verify database backups periodically with the offline tooling. + +The relevant runtime must be stopped before restore. The control maintenance +entry point and the client's `archive-client-backup` command support `list`, +`verify `, and `restore `; restore verifies the copy, preserves +the replaced database and SQLite sidecars with a timestamped `suspect` name, +and installs the selected database atomically. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 0000000..d22db8b --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,276 @@ +# Actionable implementation plan + +Implementation status (2026-07-23): phases 0–7 are implemented in the proto, +control, and client worktrees. Phase 8 has a passing local 2×2 happy path, +deterministic selective/recovery matrix, and adversarial v2/fault matrix: +complementary partial placements merge correctly, eviction uses archive +coverage unions, queued and active cancellation preserve their safety +boundaries, restarts replay durable work, and hostile preconditions fail +without committing. Pure-v2/hybrid round trips, recoverable stalls, +capacity-guard failure, and duplicate/reordered WebSocket events also pass from +a clean topology. The remaining fixture and fault categories listed below are +still release gates; phase 9 remains intentionally deferred until the complete +release matrix passes. + +No application implementation begins until Phase 0 is reviewed. Within every +later phase, tests and documentation are part of the deliverable rather than a +follow-up task. + +## Phase 0 — Freeze the contract + +### Actions + +1. Convert the decision ledger into the system, workflow, safety, service API, + deployment, Telegram, and test documents in this directory. +2. Create the independent `archive-control-proto` Git repository. +3. Define all v1 resource, selection, route, inventory, job, progress, + command/event, reconciliation, and envelope messages. +4. Add canonical protobuf-JSON examples for registration, inventory queries, + route creation, transfer start/progress/commit, cancellation, eviction, and + recovery snapshots. +5. Run Dockerized Buf format/lint and protoc descriptor compilation. +6. Configure the Gitea remote `cabbage/archive-control-proto` with its dedicated + SSH key. Tag the reviewed baseline `v0.1.0` only after review. + +### Acceptance gate + +- No unresolved `TODO` changes wire meaning or destructive behavior. +- Every workflow transition maps to a protobuf command/event. +- JSON examples parse through generated descriptors. +- Documentation and protobuf enums/state machines agree. + +## Phase 1 — Build the control core inside mogic-bot + +### Actions + +1. Add `archive_control/` parallel to `remote_control/`, with no imports from + Telegram in its core service, scheduler, protocol, persistence, or + orchestration layers. +2. Add strict configuration parsing to the existing bot configuration, + including shared token, WebSocket bind/path, database/backup paths, + heartbeat, scheduling, route policy, retention, and Telegram refresh knobs. +3. Add SQLite schema migrations for clients, connections, routes, jobs, + commands, job events, reservations, placements, UI sessions, and + tombstones. Enable foreign keys, WAL, busy timeout, and transactional schema + migration. +4. Implement online backups, integrity verification, retention, and offline + verify/restore commands. +5. Implement WebSocket registration-first authentication, duplicate-client + replacement, heartbeats, bounded send queues, envelope validation, and + protocol version negotiation. +6. Implement durable command outbox/inbox semantics, acknowledgements, + per-job event sequencing, snapshots, and reconnect reconciliation. +7. Implement the global resource reservation and FIFO per-client/per-route + scheduler. +8. Add the thin test-only HTTP adapter over `ArchiveControlService`. + +### Acceptance gate + +- Bot startup/shutdown cleanly owns Archive Control tasks in its existing + asyncio lifecycle. +- Existing bot tests and workflows are unchanged. +- Restart, duplicate registration, lost acknowledgement, event duplication, + sequence gap, and DB backup/restore tests pass. +- The test HTTP adapter can create/query/cancel synthetic jobs without Telegram. + +## Phase 2 — Build the archive-client foundation + +### Actions + +1. Scaffold `archive-clients` with a `src/` layout, Python 3.11, strict TOML + parsing, environment/file secrets, CLI overrides, structured redacted logs, + and console entry points. +2. Implement `--mode archive|cache`, stable client ID, state/backup DBs, and + Docker-friendly shutdown handling. +3. Implement capped exponential reconnect with jitter, registration, + heartbeat acknowledgements, one-writer WebSocket output, durable command + deduplication, and active-job snapshots. +4. Implement API-visible-to-local root mapping and path-confinement helpers. +5. Probe qBittorrent, Syncthing, filesystem/link/reflink, sparse, and permission + capabilities at startup; advertise a precise health/capability report. +6. Add multi-stage Dockerfile and Compose example. Run unprivileged and mount + only config, secrets, DB/backups, qB data, and sync data. +7. Build amd64 first; keep the Dockerfile architecture-neutral for the release + buildx phase. + +### Acceptance gate + +- Invalid/unknown configuration fails before connecting. +- Secrets never appear in logs or SQLite. +- Restart and duplicate-command tests prove idempotent client behavior. +- Both daemon roles start from the same image and differ only by config/mode. + +## Phase 3 — Service adapters and on-demand discovery + +### Actions + +1. Implement qBittorrent cookie authentication and capability adapters for + supported 4.5–4.6 and 5.x APIs. +2. Implement on-demand torrent summaries, scoped hash lookup, lazy complete + content trees, v1/v2 identity validation, torrent export, stopped add, + selection application, recheck monitoring, and entry-only deletion. +3. Implement Syncthing API-key authentication, version/config/status/events, + folder/device discovery, per-file progress, need/file completion checks, and + rescan support. +4. Discover only healthy pairwise `sendreceive` routes under the configured + roots. +5. Implement idempotent device pairing and folder creation using advertised + addresses, plus bidirectional nonce verification and 30-minute default + timeout. +6. Implement scoped, chunked, revisioned inventory and tree queries; do not add + background cluster inventory polling. + +### Acceptance gate + +- Contract fixtures cover API version differences and permanent/transient + errors. +- A pair of isolated client/Syncthing stacks can auto-create and verify a route. +- Large synthetic inventories stream in bounded chunks and interrupted results + are not committed by Titan. + +## Phase 4 — Safe file operation and manifest layer + +### Actions + +1. Define the on-disk job staging layout and atomically written protobuf-JSON + transfer manifest/ready marker. +2. Implement safe traversal using normalized torrent paths, no-follow checks, + root confinement, regular-file-only validation, and duplicate/case-collision + detection. +3. Implement hardlink → reflink → safe-copy fallback with per-file journaling, + progress, cancellation points, and restart reconciliation. +4. Implement exact logical/allocated size accounting, free-space/reserve checks, + and startup sparse capability enforcement. +5. Implement target never-overwrite behavior and provenance for created versus + pre-existing reused files. +6. Implement tested qBittorrent/libtorrent partfile adapters, compatibility + gates, baseline backups, and safe merge/rollback behavior. + +### Acceptance gate + +- Tests cover cross-filesystem fallback, insufficient space, permission errors, + symlink races, special files, path escape, pre-existing collisions, sparse + artifacts, and supported/unsupported partfiles. +- Cancelling/restarting every file-operation phase leaves a deterministic + journal and no uncontrolled deletion. + +## Phase 5 — Transfer orchestration + +### Actions + +1. Implement fresh source/target tree preflight, requested-selection expansion, + immutable file-index snapshots, exact delta calculation, and no-op rejection. +2. Implement source staging, per-job Syncthing progress, destination completion + proof, and target materialization. +3. Implement new-target stopped/manual add and existing-target baseline/delta + transaction. +4. Apply selected/skipped state only, recheck the union while stopped, monitor + for any download attempt, and restore/start the correct run state. +5. Commit placement generation only after verification; treat staging removal + as post-commit cleanup with retryable `CLEANUP_REQUIRED` state. +6. Implement cooperative cancellation/compensation for every pre-commit phase + and non-destructive post-commit cleanup behavior. + +### Acceptance gate + +- Full, selective, subset, and incremental merge scenarios pass in both + directions. +- A target never downloads data during verification. +- Failure injection at every transition preserves source and target baselines. +- Archive jobs retain their source cache placements. + +## Phase 6 — Eviction and placement reconciliation + +### Actions + +1. Implement on-demand cache placement listing and fresh archive coverage sets. +2. Require complete per-file coverage across one or more online archive nodes; + expose disabled reasons and provide no force override. +3. Implement eviction jobs: coverage recheck, qB entry-only removal, remaining + torrent exact-path overlap snapshot, safe file unlink, partfile ownership + check, and non-recursive bottom-up directory removal. +4. Reconcile observed placements with successful job generations and external + qBittorrent changes without background polling. + +### Acceptance gate + +- Eviction cannot proceed with any uncovered selected file. +- Shared exact paths, shared directories, unknown files, and unrelated + partfiles survive. +- Retrying after interruption is idempotent and an archive placement is never + deleted. + +## Phase 7 — Telegram integration + +### Actions + +1. Add `ArchiveControlHandler` without changing existing handler behavior. +2. Implement Archive, Unarchive, Evict Cache, and Job Status flows with one + edited message, pagers, chained literal filters, cancel/back behavior, and + single-use confirmation nonces. +3. Implement normal-all and one-entry tree selection; keep protobuf support for + multiple selections. +4. Persist UI sessions and restore live status subscriptions after bot restart. +5. Inherit private-chat username authorization and optionally pin configured + numeric IDs. Keep jobs shared and ownerless. +6. Render five-step transfer and three-step eviction progress, including stale, + waiting, stalled percentage, cleanup-required, failure reason, paging, + Clear Finished, and Remove One. +7. Coalesce routine edits to the configurable five-second default and handle + Telegram rate limits. + +### Acceptance gate + +- Telegram and test HTTP adapters produce identical service-layer outcomes. +- Stale/double callbacks are harmless. +- Bot restart preserves active UI/status behavior. +- Existing mogic-bot regression tests pass. + +## Phase 8 — Cross-project E2E and chaos testing + +### Actions + +1. Create five Compose projects under this repository's `e2e/`: control, + cache-1, cache-2, archive-1, and archive-2. +2. Give every data node isolated qBittorrent, Syncthing, archive-client, config, + state, backup, and data volumes on one dedicated test network. +3. Configure static advertised Syncthing service addresses while testing + automatic device/folder pairing. +4. Generate small v1/v2/hybrid torrents and fixtures for complementary and + overlapping selections, partfiles, sparse files, path collisions, shared + directories, and permission errors. +5. Drive scenarios with curl against the test HTTP adapter. +6. Inject control/client/service restarts, network loss, duplicate commands, + stale snapshots, stalls, disk exhaustion, cancellation, and DB restore. +7. Tear down only test resources identified by unique project names/labels. + +### Acceptance gate + +- The complete acceptance matrix in `testing.md` passes repeatedly without + live infrastructure. +- No test uses credentials or paths from x1, x2, lithium, or Titan production. +- Test teardown is scope-validated and leaves unrelated Docker resources alone. + +## Phase 9 — Release and staged deployment + +### Actions + +1. Run Buf breaking checks against the last proto release and tag compatible + repositories. +2. Build `sodium/archive-clients` for linux/amd64 and linux/arm64 with OCI + source/protocol labels; publish an immutable semantic-version tag and the + stable `latest` alias. +3. Record image digest and provide pinned Compose examples. +4. Back up real qBittorrent/Syncthing and Archive Control state before a + separately approved deployment. +5. Deploy one cache/archive pair in observation mode, validate on-demand + inventory and route discovery, then run a small non-critical canary transfer. +6. Exercise recovery and eviction only after canary verification and explicit + approval, then expand node-by-node. + +### Acceptance gate + +- Multi-platform manifest resolves to tested images and the runtime image is + reasonably small. +- Rollback instructions and prior images/config/DB backups are verified. +- No production qBittorrent/Syncthing global setting is modified. diff --git a/docs/protocol.md b/docs/protocol.md new file mode 100644 index 0000000..69163d1 --- /dev/null +++ b/docs/protocol.md @@ -0,0 +1,162 @@ +# Daemon protocol + +The source of truth is `archive_control.v1` in the sibling +`archive-control-proto` repository. This document defines behavior that cannot +be expressed by field declarations alone. There is no separately maintained +JSON schema. + +## Framing and encoding + +- Each WebSocket text frame contains exactly one `Envelope` encoded with the + canonical protobuf JSON mapping. +- Binary frames, batches, unknown top-level payloads, malformed JSON, duplicate + JSON keys, non-finite numbers, invalid timestamps, and over-limit envelopes + are protocol errors. +- UUID fields use lowercase canonical UUID text. Hash fields use validated + lowercase hexadecimal text. +- Protobuf `uint64` values use JSON strings according to the protobuf JSON + mapping. Enum names are emitted symbolically. +- Receivers may ignore unknown JSON fields only after a compatible minor + version has been negotiated. An unknown required command or job operation is + rejected explicitly. +- Implementations apply negotiated envelope and inventory chunk limits before + allocation and use bounded outbound queues. + +## Registration and connection ownership + +`RegisterRequest` must be the first application message on every new +connection. It carries protocol version, stable client ID/name, role, +connection-instance UUID, capabilities, current active-job cursors, and the +single shared token. + +The server validates the entire request and constant-time token comparison +before accepting other messages. A rejected token or malformed first message +causes a rejection when safe to send and then immediate connection closure. The +token is forbidden in all later messages and is redacted before any structured +logging, error rendering, metrics label, trace, or database write. + +A valid new registration for an already connected client atomically becomes +the current connection generation. The old connection is closed, and messages +arriving from its stale generation cannot mutate state. + +Clients retry registration indefinitely until accepted or the process exits. +The default delay begins at one second, doubles with jitter, and caps at 60 +seconds. A connection stable for 60 seconds resets the backoff. + +## Version and feature negotiation + +The envelope and registration both state the sender version. Major `1` accepts +only major `1`; a different major is rejected. The negotiated minor is the +highest mutually supported minor no greater than either endpoint's advertised +minor. + +Capabilities, not version guesses, gate optional behavior such as route +provisioning, sparse files, partfile migration, or eviction. A required missing +feature fails the job preflight with `UNSUPPORTED`. Compatible changes add +fields or enum values; released field numbers and enum values are never reused, +and removals reserve them. Buf breaking checks enforce file-level compatibility. + +## Heartbeats and liveness + +Default settings are: + +| Setting | Default | +| --- | ---: | +| Registration deadline | 10 s | +| Heartbeat interval | 15 s | +| Offline threshold | 45 s | +| Unacknowledged command retry | 15 s | +| Maximum command send attempts | 3 total | +| Reconnect initial/cap | 1 s / 60 s | +| Stable reset window | 60 s | + +The server communicates the effective heartbeat values in `RegisterResponse`. +Heartbeat sequence acknowledgements prove application-level responsiveness. +Missing the offline threshold marks the client offline, releases connection +ownership, and moves affected jobs to waiting; it does not fail them. + +## Reliable commands + +Control generates a UUID `command_id`, persists the command and intended +client, then sends it at least once. A client transactionally inserts an unseen +ID and its immutable payload before replying `ACCEPTED`. A repeat with the same +payload returns the stored acknowledgement as `DUPLICATE`; the same ID with a +different payload is a protocol conflict. + +`CommandAck` confirms durable acceptance or rejection only. It never means the +step completed. Completion is represented by ordered `JobEvent`, +`InventoryChunk`, `RouteUpdate`, or `JobSnapshot` messages. Control retries an +unacknowledged command with its original ID after reconnect or timeout, up to +the configured total attempt limit. A late acknowledgement remains valid after +automatic delivery stops. + +Route updates use a stable update UUID and a sequence monotonic within their +route-creation command. Conflicting duplicates or gaps are reconciled with the +client state snapshot before a route can become ready. + +Every mutating command includes an expected job revision or immutable job +definition plus the expected last global event sequence. A stale revision or +sequence is rejected without side effects. The control daemon sends only one +active step command for a job and waits for its persisted completion event +before commanding the next participant. This single-writer lease lets whichever +client owns the current step allocate the next global per-job event sequence; +the following command starts from the sequence control has durably accepted. + +## Events, progress, and reconciliation + +Each client-originated job event has a unique ID, monotonically increasing +global per-job `sequence`, and resulting job revision. Duplicate IDs/sequences +are idempotent only when their complete content matches. Control never grants +concurrent event-writer leases for one job. A gap or conflicting duplicate +pauses destructive orchestration and requests snapshots. + +`fraction_complete` is current-step progress and +`overall_fraction_complete` is the weighted five- or three-step job progress; +both are clamped to `[0,1]`. Byte counts are authoritative when known. +`STALLED` retains both fractions, byte counters, and current step. Progress +timestamp movement alone does not prove content movement. + +On registration, active-job cursors provide the client's revision, last event +sequence, state, and commit flag. Reconciliation applies these rules: + +1. Equal cursors resume normal delivery. +2. A client behind receives safe replay/snapshot commands. +3. Control behind requests and validates the client's full job snapshot. +4. Conflicting commit evidence reserves the resource and requires manual + reconciliation; neither side performs cleanup. +5. In-flight command IDs are compared before any resend. + +## Inventory protocol + +Inventory is requested only for a user flow, job preflight, active-job +monitoring, route discovery, or reconciliation. Registration does not upload a +full torrent inventory. + +Queries declare a scope, resource IDs/filter, page bounds, and optional expected +revision. Responses form an atomic snapshot stream identified by query ID, +snapshot ID, revision, monotonically increasing chunk index, and final marker. +Control commits the result only after every chunk through `last_chunk` is +present. A disconnect or error discards the partial snapshot. Full content +trees are fetched for one chosen resource and from both relevant endpoints +before staging. File entries carry index, logical/allocated/completed bytes, +selection, sparse/padding flags, and the resource content revision required to +validate the preview. + +## Stable errors + +Machine behavior uses `ErrorCode`, `retryable`, and structured details; the +human message explains the concrete condition without secrets. Transient +unavailability/timeouts may wait or retry. Invalid input, stale fingerprints, +path conflicts, permissions, unsupported formats, integrity failures, and +unsafe state are permanent for the current attempt. Unknown internal errors +must not be mislabeled retryable. + +## Manifest encoding + +`TransferManifest` and `ReadyMarker` use the same canonical protobuf JSON +mapping, written atomically. The manifest describes immutable source/target +identity, route, requested set, target baseline, transfer delta, file paths and +sizes, sparse flags, materialization provenance, fingerprints, and auxiliary +artifacts. The ready marker is written last and includes a SHA-256 digest of the +exact manifest bytes. A receiver ignores a job directory until both exist and +the digest matches. diff --git a/docs/service-apis.md b/docs/service-apis.md new file mode 100644 index 0000000..ab06086 --- /dev/null +++ b/docs/service-apis.md @@ -0,0 +1,142 @@ +# qBittorrent and Syncthing integration + +Adapters are version-aware boundaries. Core orchestration consumes normalized +resource, route, and progress models and never branches on raw service response +shapes. + +## qBittorrent Web API + +The implementation targets supported qBittorrent 4.5–4.6 and 5.x releases and +detects the application and Web API versions at startup. The authoritative +references are the official [5.0 WebUI API](https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-%28qBittorrent-5.0%29) +and [4.1-compatible WebUI API](https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-%28qBittorrent-4.1%29). + +### Adapter capabilities + +| Need | Web API family | Archive Control rule | +| --- | --- | --- | +| Authenticate | `auth/login`, cookie session | Log no credentials/cookies; reauthenticate once on expiry | +| Detect compatibility | `app/version`, `app/webapiVersion`, build info | Advertise exact versions and select adapter | +| List/lookup | `torrents/info`, properties | On-demand and hash-scoped where possible | +| Read file state | `torrents/files` | Normalize indices, paths, selected/skipped, size, progress | +| Export metainfo | `torrents/export` | Required before staging; fail if exact metainfo unavailable | +| Add target | `torrents/add` | Explicit save path, stopped/paused, manual management | +| Set selection | `torrents/filePrio` / selection endpoint | Encode only selected vs skipped, never preserve priority levels | +| Recheck | `torrents/recheck` plus state polling | Keep stopped; any download activity fails the job | +| Remove entry | `torrents/delete` | Eviction uses `deleteFiles=false` only | + +The adapter accounts for terminology/behavior changes such as paused versus +stopped states. It does not mutate qBittorrent preferences, categories, tags, +limits, queueing defaults, or global save-path behavior. + +### Inventory normalization + +For each torrent the client derives canonical v1/v2 identity from reliable API +and exported-metainfo data, keeps the qB-local hash separately, and calculates: + +- total and selected-complete logical bytes; +- normalized selected and selected-complete file-index sets; +- torrent runtime state; +- canonical path flag and content revision; +- a save-path fingerprint based on validated path mapping, never a leaked + absolute host path. + +qBittorrent 5 may expose a pure-v2 or hybrid torrent under the first 20 bytes +of its v2 hash while separately advertising full `infohash_v1` and +`infohash_v2` values. Lookup matches all advertised identities, rejects +ambiguous matches, validates the canonical hashes from exported metainfo, and +uses only qBittorrent's returned local hash for later mutation calls. + +Complete content trees are built only for a chosen resource or job preflight. +All entries retain qBittorrent's stable torrent file index. Renamed/noncanonical +content paths are rejected in v1 because they cannot be transported and merged +without ambiguity. + +### Verification guard + +The client captures transfer counters and state before recheck, issues recheck, +and watches until success, timeout/stall, or failure. Success requires every +selected file complete and no missing/error pieces. Any downloading state or +increase in received payload bytes causes immediate stop and a failed job with +the affected path/piece detail available from the adapter when possible. + +## Syncthing REST and events APIs + +Clients authenticate using the configured API key header and discover exact +service version/capabilities. Primary references are Syncthing's official +[REST conventions](https://docs.syncthing.net/dev/rest.html), +[configuration model](https://docs.syncthing.net/users/config.html), and +[DownloadProgress event](https://docs.syncthing.net/events/downloadprogress.html). + +### Discovery and route management + +The adapter reads system status/version and configuration, then selects folders +that: + +- are `sendreceive`; +- map beneath the configured API-visible and local sync roots; +- contain exactly the local device and one peer device; +- pair one registered cache with one registered archive client; +- have a safe relative filesystem path. + +The folder's native ID becomes `route_id`. Multi-device, send-only, +receive-only, out-of-root, or ambiguous folders are reported as unsupported and +left unchanged. + +Provisioning uses idempotent device and folder configuration updates. Each +client receives its peer device ID and optional advertised addresses +(`dynamic` by default). It changes only the specific device/folder objects +owned by the provisioning command. Read-back verification must agree before +nonce exchange begins. + +### Transfer progress and completion + +`DownloadProgress` events are filtered to the job's exact staging prefix and +used for live block-level byte estimates. Their percentage is advisory. The +adapter falls back to coarser polling if events are unavailable. + +Final completion combines: + +- job-prefix files present and matching manifest logical sizes; +- Syncthing folder/local-global completion from the official + [`db/completion`](https://docs.syncthing.net/rest/db-completion-get.html) + response; +- per-file state from [`db/file`](https://docs.syncthing.net/v2.0.0/rest/db-file-get.html) + where supported; +- no relevant outstanding entries from + [`db/need`](https://docs.syncthing.net/v1.19.2/rest/db-need-get.html); +- valid manifest digest and ready marker. + +The client can request a scoped rescan after publishing/cleanup. It does not +declare transfer success from a `100%` progress event alone. + +## Error normalization + +Each adapter maps transport errors, authentication failure, incompatible API, +missing torrent/folder, stale external state, permission failures, and service +health into stable protocol error codes. Responses and exception text are +redacted before logging because service products may echo URLs, headers, or +filesystem paths. Authentication failures are permanent until configuration +changes; transient service absence moves an active job to waiting. + +## Bot-free control adapter + +The loopback-only test adapter calls the same `ArchiveControlService` used by +Telegram: + +- `GET /test/v1/resources?operation=archive|unarchive|evict_cache` performs a + complete, paged, on-demand summary scan and reconciles placements. +- `GET /test/v1/resources/{info_hash}/tree?client_id={client}` fetches one + complete revisioned content tree. +- `POST /test/v1/jobs/preview` fetches fresh source and target summaries and + full trees, expands optional `selected_paths` or + `selected_file_indices`, and returns an immutable definition plus its + `preview_revision`. +- `POST /test/v1/jobs` consumes `{preview_revision, definition}` and rejects a + deterministic protobuf SHA-256 mismatch. +- Job list/detail/cancel/hide/clear endpoints expose the shared ownerless job + state, while `GET /test/v1/placements` exposes reconciled generations. + +Lower-level inventory-query, route, scheduler, and backup endpoints remain for +contract and failure-injection tests. This adapter must never bind outside +loopback. diff --git a/docs/storage-safety.md b/docs/storage-safety.md new file mode 100644 index 0000000..8851a71 --- /dev/null +++ b/docs/storage-safety.md @@ -0,0 +1,153 @@ +# Storage safety, rollback, and recovery + +## Path trust boundary + +qBittorrent metadata, Syncthing configuration, transfer manifests, and +filesystem directory entries are all untrusted inputs. Before any access, the +client: + +1. validates the canonical torrent-relative path; +2. rejects absolute paths, empty/`.`/`..` components, NULs, platform separators, + duplicate normalized names, and configured case-fold collisions; +3. resolves the API-visible path through one validated root mapping; +4. traverses without following symlinks and checks the resulting descriptor is + a regular file or an explicitly created directory; +5. verifies every operation remains beneath the local configured root. + +Sockets, devices, FIFOs, symlinks, and other special entries fail preflight. +Permission or ownership mismatch is fail-fast. Archive Control never changes +source ownership or mode to make a job pass. + +## Isolated staging + +Every route stores transient jobs under: + +```text +/.archive-control/jobs// +├── payload/ +├── metainfo/source.torrent +├── artifacts/... +├── manifest.json +└── ready.json +``` + +Temporary writes use an attempt-specific name, fsync where durability matters, +and atomic rename. The manifest records every expected file and method. The +ready marker is the final publish action. Job IDs and known subpaths are used +directly; cleanup never expands a glob or recursively targets the sync root. + +## Materialization fallback + +For each file, source staging and target materialization try: + +1. hard link when the same filesystem and permissions allow it; +2. reflink/clone when supported; +3. bounded buffered copy into a temporary file followed by atomic placement. + +The journal records intent before mutation and completion afterward, including +source identity, destination, method, logical/allocated sizes, and whether the +target pre-existed. Restarts reconcile the journal against file descriptors and +the manifest instead of blindly repeating operations. + +Free-space checks account for methods that allocate data plus a configurable +reserve. A mid-copy space failure leaves only an attributable temporary file. +No operation overwrites an existing target. A same-type, same-size regular file +may be provisionally reused, but qBittorrent's full recheck is the integrity +authority; any other collision fails before mutation. + +## Sparse files + +At startup, each client probes semantic sparse-file behavior once for every +configured qB and Syncthing local root. Capability is advertised with the +connection. A job with any sparse source file requires sparse support for every +staging and destination root involved. If a target leg lacks it, the job fails +clearly before transfer. + +Sparse files are retained as sparse, without attempting to reproduce an exact +physical hole map. Hardlink/reflink naturally preserve semantics; the copy +implementation uses a sparse-aware strategy and verifies logical length and +allocation behavior. Exact allocated-block counts need not match between +filesystems. + +## qBittorrent transaction boundary + +For a new target, the client exports source metainfo, places candidate data, +adds the torrent stopped with manual management and explicit save path, applies +the exact union selection, and starts a full recheck. It monitors both state and +transfer counters. Any download transition or received payload byte is a hard +failure and the torrent is stopped immediately. + +For an existing target, the durable baseline includes its selected indices, +run state, local identifier, save-path fingerprint, content revision, and +partfile state. The job adds only the delta. Before commit, rollback restores +that baseline and removes only current-job-created, unshared files. Source +torrent state is never changed by transfer logic. + +The target placement commits only after the selected union verifies completely +while stopped. The placement generation and commit event are persisted in the +same control transaction. Once committed, target content is outside rollback +scope. + +## Partfiles + +Libtorrent partfiles may contain data for skipped or non-materialized torrent +files and their layout varies by version. Archive Control therefore uses a +version-aware adapter keyed by observed qBittorrent/libtorrent version and a +tested format signature. The adapter must support export/import and, for an +existing target, merge with a recoverable baseline. + +Unknown, ambiguous, or incompatible partfile state fails the job at live +preflight with `UNSUPPORTED` or `PRECONDITION_FAILED` and a precise reason. It +is never copied over an existing partfile blindly. Rollback restores the +baseline. Eviction removes a partfile only when its ownership by the removed +torrent is unambiguous. + +## Safe eviction deletion + +Before unlinking, the client persists: + +- the confirmed cache placement generation and file set; +- fresh archive coverage proofs; +- the exact torrent manifest and local paths; +- qBittorrent entry-removal result; +- exact paths referenced by all remaining torrents; +- per-file unlink and per-directory `rmdir` results. + +The qBittorrent entry is removed with `deleteFiles=false`. For each owned file, +the client refuses removal if another remaining torrent references the exact +path. Eligible known files are unlinked individually. It then attempts +non-recursive `rmdir` deepest-first for known owned directories. `ENOTEMPTY` +means retain and continue; unknown files or directories are never traversed or +deleted. There is no recursive delete fallback. + +## Compensation by phase + +| Point | Safe outcome | +| --- | --- | +| Queued, no command | Delete queue row/reservation only | +| Source staging | Stop; unlink only journaled staging artifacts | +| Syncthing transfer | Stop publishing; clean only isolated job namespace | +| Target materialization | Restore target baseline; remove attributable delta | +| qB recheck before commit | Stop/remove newly added entry without data deletion, then compensate delta | +| After commit | Keep placement; retry staging cleanup only | +| Eviction after qB removal | Resume safe per-file unlink; never re-add or delete archives automatically | + +If attribution is uncertain, retain data and require manual intervention. + +## SQLite durability and backup + +Both daemons enable foreign keys, WAL, a bounded busy timeout, and explicit +transactions. Schema migrations create an online backup before and after the +migration and fail atomically. + +The default recurring policy runs SQLite's online backup API every six hours, +verifies checksum and SQLite integrity, writes via temporary file plus atomic +rename, and retains 12 recent, 14 daily, and eight weekly copies. Backups live +outside qBittorrent and Syncthing roots. Configuration warns if the backup and +database share a filesystem, but may permit it for small installations. + +Offline tooling provides list, verify, and restore. Restore requires stopped +daemon access, verifies the chosen backup, preserves the suspect database under +a timestamped name, installs the replacement atomically, and runs integrity and +schema checks before normal startup. Backups contain no configured secrets. + diff --git a/docs/system-design.md b/docs/system-design.md new file mode 100644 index 0000000..79a52bb --- /dev/null +++ b/docs/system-design.md @@ -0,0 +1,169 @@ +# System design + +## Scope and invariants + +Archive Control moves already-complete BitTorrent file selections between hot +cache nodes and durable archive nodes. It does not download missing content, +change global qBittorrent or Syncthing settings, or decide retention on archive +nodes. + +The non-negotiable invariants are: + +1. A committed archive copy is never deleted by a transfer rollback or cleanup. +2. A cache placement is evicted only by an explicit eviction job after fresh + proof that every selected file is covered by online archive placements. +3. A target is committed only after qBittorrent verifies the selected union in + stopped state without attempting a download. +4. One control daemon is the only cross-node orchestrator. Clients execute one + commanded local step at a time. +5. Every destructive transition is durable, attributable, retryable, and + guarded by an expected revision or external-state fingerprint. +6. File operations stay beneath configured roots, never follow symlinks, and + never recursively delete a content tree. + +## Components + +### Archive control core + +The existing mogic-bot process hosts a new `archive_control/` package parallel +to `remote_control/`. Its core has no Telegram imports and exposes an +`ArchiveControlService` facade used by: + +- the production Telegram handler; +- a test-only HTTP adapter used by local E2E tests; +- the WebSocket connection manager for clients. + +Internally the core contains configuration, SQLite persistence, connection +registry, durable command outbox, protocol codec, scheduler, route coordinator, +inventory coordinator, job state machines, backup manager, and event/view +projection. The existing bot lifecycle starts and stops these tasks; no second +bot or independent control process is introduced. + +### Archive client + +One Python image and codebase runs with role `archive` or `cache`. Each instance +contains: + +- a reconnecting control connection and durable command inbox; +- a local SQLite journal and backup manager; +- qBittorrent and Syncthing adapters; +- safe path and file-operation modules; +- route discovery/provisioning; +- inventory and content-tree responders; +- transfer and eviction step executors. + +The role controls policy, not the available transport mechanics. Cache clients +may source archive jobs, receive unarchive jobs, and perform eviction. Archive +clients may receive archive jobs and source unarchive jobs. + +### External local services + +Clients talk to their locally configured qBittorrent and Syncthing APIs. API +paths may differ from paths inside the client container, so each root is +configured as an API-visible path plus its daemon-local equivalent. The +mapping must be one-to-one and is validated at startup. + +## Logical data flow + +```text +Telegram handler ─┐ + ├─ ArchiveControlService ─ SQLite/control scheduler +test HTTP adapter ┘ │ + │ versioned protobuf JSON over WebSocket + ┌────────────────┴─────────────────┐ + cache client archive client + ├─ qB adapter ├─ qB adapter + ├─ Syncthing adapter ├─ Syncthing adapter + ├─ SQLite journal ├─ SQLite journal + └─ local file ops └─ local file ops + └──────── pairwise Syncthing route ────────┘ +``` + +The control service persists intent and the job's current event-sequence lease +before emitting a command. The client +persists command acceptance before acknowledging it. Completion is a later +event. This separates delivery from execution and makes reconnect replay safe. + +## Persistent model + +### Control database + +The control database contains at least: + +- `clients` and last known capabilities/health; +- connection generations and heartbeat timestamps; +- discovered/provisioned routes and verification state; +- immutable job definitions and mutable job projections; +- per-job ordered events; +- durable commands, delivery attempts, and acknowledgements; +- resource reservations; +- cache/archive placements and generations; +- bounded inventory snapshots and revisions; +- persisted Telegram sessions and confirmation nonces; +- hidden-record tombstones and backup metadata. + +### Client database + +Each client stores: + +- accepted command IDs and their stable acknowledgements; +- job definitions, local step journals, revisions, and emitted events; +- file-operation provenance and compensation state; +- staging manifests and ready-marker observations; +- route-creation ownership and verification state; +- last control connection/session cursors; +- backup metadata. + +Credentials, shared tokens, qBittorrent passwords, Syncthing API keys, and +Telegram secrets are never stored in either database. + +### Placements are not job history + +A placement is current, verified residency of a resource selection on one +node. It is keyed by canonical resource identity and client ID and includes a +generation plus verified file-index set. Jobs mutate placements, but deleting +or hiding a job record never changes placement data. A resource may have many +archive and cache placements. + +## Scheduling and ownership + +Jobs enter a durable FIFO queue while holding a global reservation on every +known v1/v2 alias of their resource identity. Only one transfer or eviction for +a resource may be active or queued at a time. This prevents incompatible +baselines even when jobs would use different nodes or observe different sides +of a hybrid identity. + +Defaults allow one active data-moving job per client and one per route. A job +must acquire its source client, target client, route, and resource reservation +atomically. Disjoint node pairs may run concurrently. Route setup is a +preflight activity and does not permit a data step to bypass these leases. + +Offline nodes do not prevent unrelated jobs from being listed or run. A job +requiring an offline node remains waiting indefinitely; it does not consume an +active data slot until work can proceed. + +## State and recovery + +Control state is authoritative for orchestration; neither client independently +advances to the next step. On reconnect: + +1. The client registers with active-job cursors. +2. Control compares revisions and per-job event sequences. +3. Missing events are replayed when available; gaps or disagreement trigger a + full client/job snapshot. +4. Control observes relevant qBittorrent, Syncthing, staging, and manifest + state before selecting retry, resume, compensation, cleanup, or manual + intervention. +5. A command is reissued with its original ID when the acceptance result is + uncertain. + +A client never guesses that a job committed. If a database is lost, evidence +from a verified target, immutable manifest, ready marker, and file-operation +journal may reconstruct state only when it proves one unique outcome. Otherwise +the resource remains reserved and the job fails closed for manual +reconciliation. + +Routine no-progress and connectivity conditions become `WAITING` or `STALLED`, +not failures. Permanent protocol, validation, integrity, unsafe-path, +unsupported-partfile, or permission conditions fail with a stable code and a +clear operator-facing reason. diff --git a/docs/telegram-ux.md b/docs/telegram-ux.md new file mode 100644 index 0000000..32afe6b --- /dev/null +++ b/docs/telegram-ux.md @@ -0,0 +1,177 @@ +# Telegram UX + +Archive Control adds a separate handler/module to the existing bot. Existing +commands and conversations retain their current behavior. + +## Authorization and shared state + +The handler accepts private-chat users through the existing `@username` +allowlist for compatibility. A configured numeric Telegram user ID may pin an +entry; when pinned, both the ID and current allowlist rule must match. This +mitigates username reassignment while permitting a staged migration. + +Jobs, placements, events, queues, and routes are shared administrative state and +store no initiating-user identity. UI session state records only what is needed +to resume the conversation/view and expires after 24 hours by default. + +## Entry point and navigation + +The existing “Archive/Unarchive Resources” entry opens one message: + +```text +What do you want to do? + +[ Archive ] [ Unarchive ] +[ Evict Cache ] [ Job Status ] +``` + +All subsequent pages edit this message. `Cancel` closes the active selection +flow. `Back` returns one level while retaining validated filters/selections. +Callback payloads contain opaque session/action IDs, not resource names or +paths, and are validated against persisted session revision and expiry. + +## Archive and unarchive flow + +### 1. Select source resource + +Control queries relevant online nodes on demand. It groups identical resource +and selection replicas into one display choice while showing distinct +selections separately. The list is stable-sorted and paged: + +```text +Which cached resource do you want to archive? (page 1/3) +filters: `documentary`, `2025` + +1. `10.0 GB` Resource A — cache-1 +2. `840 MB` Resource B — cache-2 + +[ 1 ] [ 2 ] +[ Prev ] [ Next ] +[ Remove filters ] [ Cancel ] +``` + +Size is selected-complete logical content, not total torrent size. New messages +add literal, case-insensitive filters and narrow the current results; Remove +filters resets the chain. If no item remains, the message explains whether no +source is online, no selected-complete data exists, or filters eliminated all +results. + +### 2. Select source replica and target + +If multiple identical-selection replicas exist, control may choose a healthy +source automatically using deterministic availability/route/free-space rules; +otherwise it lists the source replicas. One eligible target is selected +automatically; multiple targets are paged. Offline targets are not offered. + +A resource already on a target is still eligible when the requested selection +has an unverified delta. The UI says that it will extend an existing placement. + +### 3. Confirm or choose one entry + +Before showing confirmation, control obtains fresh full trees from source and +target and calculates the transfer delta. + +```text +Archive Resource A from cache-1 to archive-2? +Requested: 7 files, 10.0 GB +Already verified there: 2 files, 3.1 GB +Will transfer: 5 files, 6.9 GB + +[ Confirm ] +[ Choose one entry… ] [ Cancel ] +``` + +Choose one entry opens a paged/filterable tree. Only source-selected-complete +files/directories are enabled. Selecting a directory expands it to descendant +file indices, returns to a freshly recomputed confirmation, and clearly states +the chosen subtree. The protocol supports multiple entries from v1, but the +initial UI exposes one. + +Confirmation uses a persisted, single-use nonce bound to session revision, +operation, resource, source, target, selection, fingerprints, and placement +generation. Double taps or stale callbacks cannot create a second job. + +## Evict Cache flow + +The first page lists current cache placements. Selecting one triggers fresh +queries to the cache and relevant online archive nodes. Each row/confirmation +shows coverage: + +- enabled when the union of verified archives covers every selected cache file; +- disabled with missing file count/size and unavailable archive explanation + otherwise. + +There is no override button. Confirmation creates a durable three-step +eviction job; it never clears an archive placement or a transfer history row. + +## Job Status + +Status pages show live and retained jobs with operation, resource, endpoints, +state, current-step and overall percentages, bytes/s where meaningful, and +failure reason. +Routine edits are coalesced to one per five seconds by default. Step changes, +stall transitions, failures, commit, cleanup-required, and terminal states edit +immediately subject to Telegram rate-limit backoff. + +State symbols: + +| Symbol | Meaning | +| --- | --- | +| 🟡 | queued/not started | +| 🔵 | active, waiting, or stalled | +| 🟠 | committed but cleanup required | +| 🟢 | succeeded | +| 🔴 | failed or cancelled | + +Transfer jobs display five numbered steps: Source Stage/HardLink, Syncthing, +Target Merge/Copy, QB Verification, and Staging Cleanup. Route setup and +preflight are named, unnumbered pre-steps. Eviction displays Verify Archive +Coverage, QB Remove Entry, and Safe File Unlink. + +A stalled job keeps its last overall percentage: + +```text +🔵 Archive: Resource A — cache-1 → archive-2 +[2/5 Syncthing: 63.42%, overall 32.68%, stalled, last progress 38m ago] +Waiting indefinitely; retry occurs when progress/service connectivity returns. +``` + +Failure messages include stable phase and clear reason, while hiding secrets +and host-only paths. + +## Clear and remove behavior + +`Clear Finished` hides only green successful jobs and leaves a 30-day default +tombstone. Orange Cleanup Required is not clearable as finished. + +`Remove One` first replaces buttons with current-page job numbers and Cancel. +A successful green row hides immediately. A queued job confirmation deletes +only its queue record/reservation. An active, failed-with-residue, or cancelling +job requires a second confirmation and begins/continues compensation; its row +remains until safe cleanup is known. A committed job can only retry staging +cleanup. No history action deletes a cache placement; that is exclusively Evict +Cache. + +## Restart and rate-limit behavior + +Sessions, pages, filters, chosen entries, confirmation bindings, and live status +views are persisted. On bot restart, active status messages resume updates when +the stored chat/message remains editable; otherwise users can reopen Job Status +without losing job state. “message is not modified” is treated as success. +Telegram retry-after values pause/coalesce edits without affecting orchestration. + +## Interactive test entry + +An opt-in `Archive Control (Test)` entry reuses this same handler with an +independent callback namespace and SQLite database. Its scripted backend +provides comprehensive, empty, and slow-route scenarios without contacting +control/client daemons, qBittorrent, or Syncthing. The production entry, +callback namespace, database, jobs, placements, and other bot features are +unchanged. + +The test entry exposes administrator-only `/actest` controls to reset a +scenario, move a synthetic job through displayed states, fail the next selected +backend call, complete a slow route, inspect recent trace events, or export the +complete JSONL trace. The trace contains callback and filter inputs, session +state transitions, exact message/keyboard renders, backend results/errors, and +job transitions, so a UI report can be reproduced without live storage nodes. diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..8916cc4 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,245 @@ +# Testing strategy + +Tests run locally in containers and never inspect or mutate live x1, x2, +lithium, or Titan services. Host tooling follows `playground/ENV.md`: Docker and +Compose provide runtimes/build tools and write generated files as UID/GID +`1001:1001`. + +## Current happy-path coverage + +The initial happy-path pass is tracked by executable capability rather than by +declaring an unfinished phase complete: + +As of 2026-07-23, the full control/bot suite passes 102 tests, the full client +suite passes 60 tests, and a clean five-project Compose run passes automatic +2×2 route creation plus archive → covered eviction → unarchive with SHA-256 +verification at both materialized targets. + +| Capability | Unit/contract | Component/E2E | +| --- | --- | --- | +| Registration, heartbeat, replay, snapshots | Covered | Control/client integration covered | +| On-demand inventory and content trees | Covered | WebSocket integration covered | +| Automatic pairwise route provisioning | Covered | Local 2×2 topology covered | +| Manifest/ready-marker publication | Covered | Receiver filesystem round trip covered | +| Hardlink, reflink, sparse-copy fallback | Covered | Local filesystem round trip covered | +| Existing same-size target reuse | Covered | Local filesystem round trip covered | +| qB stopped add, selection, guarded recheck, start/resume, entry-only delete | Covered for 4.x/5.x semantics | Current pinned service covered | +| Syncthing scoped rescan and completion proof | Covered | Real route transfer covered | +| Transfer command state machine and qB verification | Covered | Archive and unarchive covered | +| Archive/unarchive placement commit | Covered | Both directions covered | +| Cache eviction | Coverage union, inode/path safety, idempotency covered | Real retained-source eviction covered | +| Telegram archive-control flows | Persistence, nonce, stale callback, stalled rendering, dummy controls/trace covered | Real handler with isolated dummy backend covered; live Telegram remains release smoke | + +Cancellation, rollback provenance, shared/unknown eviction paths, +case-collisions, and stale preview guards now have focused tests. The broader +chaos matrix remains a release gate rather than part of the initial happy-path +run. + +The executable complex E2E matrix additionally covers complementary partial +archive placements, selective cache-placement merging, repeated placement +generation after eviction, two-archive coverage union, +uncovered/no-op/stale-preview rejection, exact-path sharing and unknown file +retention, queued record-only cancellation, precommit rollback after target +materialization, control restart after source staging, durable command replay +after a client disconnect, and a no-downloaded-bytes assertion at the target. +It also mutates the source selection and creates an absent target after job +confirmation; both hostile races must fail before commit with the precise +precondition reason while retaining the source. A wrong-size pre-existing +target and an unwritable target root likewise fail without commit; the latter +is wrapped in fixture cleanup that restores the original directory mode. + +The second adversarial matrix also passes from clean state. It covers pure-v2 +and hybrid archive/evict/unarchive round trips, including qBittorrent's +truncated-v2 primary key and hybrid v1/v2 aliases; a disconnected target that +becomes `STALLED` at 20% and resumes successfully; bounded disk exhaustion with +an exact precommit failure and intact source; and duplicate/reordered protocol +delivery from a separate WebSocket probe process. + +The remaining Compose expansion includes supported +partfiles, sparse-capability mismatch, case/type collisions, broader +service-level network partitions, and database backup/restore during active +recovery. Focused unit/component tests already cover several of these +invariants, but they remain unchecked in the full five-project topology. + +## Test layers + +### Proto contract + +- Buf format, lint, descriptor build, and breaking checks. +- Decode every canonical protobuf-JSON fixture through the descriptor. +- Golden round trips for envelopes and on-disk manifests. +- Reject duplicate JSON keys, unknown major versions, invalid hashes/ranges, + malformed UUIDs/timestamps, over-limit frames, token outside registration, + and unknown required commands. +- Consumer tests assert the generated binding header records the exact proto + release/commit. + +### Unit and property tests + +Control tests cover state transitions, hybrid-identity alias reservations, +FIFO scheduling, +placement unions, coverage-set calculation, command retries, event gap +reconciliation, UI nonce idempotency, retention, and backup selection. + +Client tests cover path normalization/confinement, file-index range algebra, +tree expansion, root mapping, capability probes, version adapters, manifest +digesting, journal replay, progress aggregation, and safe directory removal. +Property/fuzz inputs emphasize path traversal, Unicode/case collisions, +overlapping selections, reordered/duplicated events, and partial journals. + +### Adapter contract tests + +Pinned qBittorrent and Syncthing container versions cover every supported API +family. Assertions include stopped add, exact selection, export, full recheck, +download-attempt detection, entry-only delete, folder/device idempotency, +events fallback, need/completion proof, and redacted errors. Partfile adapters +are tested only against explicitly supported qBittorrent/libtorrent fixtures. + +### Component integration tests + +- Control plus fake clients: reconnect, duplicate ID replacement, lost ack, + replay, sequence gaps, old connection fencing, queue leases, and restart. +- One client plus qB/Syncthing: inventory, staging, sparse/copy fallbacks, + service restart, permission failure, and database recovery. +- Two clients plus two Syncthing services: automatic device/folder creation, + bidirectional nonce verification, timeout, and existing-route discovery. + +## Cross-project E2E topology + +The coordination repository owns uniquely named Compose projects: + +```text +e2e/ +├── control/ # mogic archive-control core + test HTTP adapter +├── cache-1/ # qBittorrent + Syncthing + cache client +├── cache-2/ # qBittorrent + Syncthing + cache client +├── archive-1/ # qBittorrent + Syncthing + archive client +├── archive-2/ # qBittorrent + Syncthing + archive client +├── fixtures/ +├── scenarios/ +└── scripts/ +``` + +Each data-node project has isolated config, state, backup, qB data, sync data, +and service volumes. All join one dedicated, uniquely labeled test network. +Syncthing advertises static Compose service addresses in E2E; production +defaults remain `dynamic`. The topology deliberately starts without explicit +route folders so tests exercise automatic 2×2 pairing. + +Teardown resolves exact project names/labels and stops only those resources. +Volume deletion is a separate explicit test action, never an unscoped prune. + +## Thin test HTTP adapter + +The adapter is enabled only in the control E2E configuration and delegates to +`ArchiveControlService`; it contains no orchestration logic. Initial endpoints: + +| Method/path | Purpose | +| --- | --- | +| `GET /test/v1/clients` | connection/health/capability view | +| `GET /test/v1/routes` | route discovery/provisioning state | +| `GET /test/v1/resources?operation=...` | on-demand eligible source listing | +| `GET /test/v1/resources/{id}/tree?...` | scoped complete tree | +| `POST /test/v1/jobs/preview` | validate and calculate exact delta/coverage | +| `POST /test/v1/jobs` | create using preview revision and idempotency key | +| `GET /test/v1/jobs/{id}` | projected state and progress | +| `POST /test/v1/jobs/{id}/cancel` | queued removal or active compensation | +| `POST /test/v1/jobs/{id}/hide` | UI/history-only removal | +| `POST /test/v1/scheduler/pause` | stop automatic advancement at a durable boundary | +| `POST /test/v1/scheduler/advance` | dispatch exactly the next eligible command | +| `POST /test/v1/scheduler/resume` | restore automatic advancement | +| `POST /test/v1/protocol/order-probe` | create and lease one synthetic ordering-test job | +| `POST /test/v1/maintenance/backup` | request and observe a test backup | + +JSON objects are adapter DTOs, not a second daemon protocol. Test scripts use +curl, poll bounded state conditions, and dump control/client/service logs on +failure. Scheduler gating is available only through the loopback, opt-in test +adapter and is absent from normal daemon and Telegram workflows. + +## Interactive Telegram UX harness + +The integrated bot has an opt-in `Archive Control (Test)` entry backed by a +scripted in-process service. It runs the production conversation handler and +planner, but never starts a control listener and never contacts a client +daemon, qBittorrent, or Syncthing. Its UI sessions, scenario state, synthetic +jobs, and append-only trace share one dedicated SQLite test database that +configuration validation forbids from being the production control database. + +The `comprehensive` scenario provides enough resources, targets, and retained +jobs to exercise resource/job pagination, filtering, Unicode and long names, +full and partial placements, selective entry previews, archive, unarchive, +covered eviction, job removal, every significant displayed job state, and +single-use confirmations. `empty` covers empty lists. `route_slow` keeps a +newly requested route pending until an administrator explicitly marks it ready. + +The existing Telegram manager allowlist and optional numeric-ID pin apply. +Private-chat controls are: + +```text +/actest status +/actest reset [comprehensive|empty|route_slow] +/actest advance +/actest fail-next [message] +/actest route-ready +/actest trace [limit] +/actest export +``` + +`advance` accepts queued, preparing, running, waiting, stalled, cancelling, +cleanup_required, succeeded, failed, and cancelled. `fail-next` injects one +deterministic planner-style backend failure. Reset clears only the test +sessions, jobs, scenario state, and trace. State otherwise survives bot +restarts. + +The SQLite trace records callback receipt and validated callback state before +and after handling, text filters, complete rendered text and inline keyboards, +backend calls/results/errors, test-control commands, and job transitions. +`trace` shows a bounded recent view in chat; `export` sends the complete trace +as JSON Lines for offline diagnosis. + +## Fixture matrix + +Fixtures include small deterministic v1, v2, and hybrid torrents with: + +- full, complementary, overlapping, and one-directory selections; +- same-size corrupt pre-existing targets and type/size collisions; +- nested shared directories and exact file paths shared by multiple torrents; +- unknown files/directories inside a resource-owned directory; +- supported, unsupported, and ambiguous partfiles; +- sparse files plus a simulated non-sparse target capability; +- canonical and renamed/noncanonical qBittorrent paths; +- hardlink, reflink, and copy-only filesystem conditions; +- permissions and space-reserve failures. + +## Acceptance scenarios + +| Area | Required scenarios | +| --- | --- | +| Archive | new target, selective target, source retained, second archive replica | +| Merge | complementary sources build union, overlap transfers delta only, no-op rejected, selected-incomplete target rejected | +| Unarchive | new cache, merge existing cache, archive always retained, repeat after prior eviction | +| Eviction | one-archive coverage, union coverage, missing coverage disabled, shared/unknown content retained | +| Routes | on-demand 2×2 creation, eager mesh, existing match, slow success, timeout, restart during provisioning | +| Recovery | restart every step on source/target/control, lost DB with proof, ambiguous loss fails closed | +| Messaging | duplicate command/event, lost ack, sequence gap, stale revision, duplicate client ID | +| Safety | attempted download, corrupt same-size file, symlink/path escape, special file, partfile mismatch | +| Operations | stall retains percent, offline wait, cancellation each phase, postcommit cleanup retry | +| Database | online backup under load, pre/post migration, retention, corrupt backup rejection, offline restore | +| UI | pagination/filter chain, stale/double callback, persisted session, clear vs evict separation | + +Every scenario asserts final qB selections/states, placement generations, +manifests/journals, filesystem ownership/provenance, job event order, absence of +unexpected received bytes, and preservation of unrelated files. + +## Fault injection and pass criteria + +Scenario hooks pause or restart each container, disconnect network edges, +duplicate/reorder protocol frames, fill a bounded test filesystem, change an +external qB selection, and kill processes between intent and completion journal +writes. Tests use eventual assertions with explicit deadlines; long production +waits are shortened by test-only configuration without altering state logic. + +A release candidate passes only when the whole matrix succeeds repeatedly from +clean volumes and from retained/restarted volumes, existing mogic-bot regression +tests remain green, Buf reports no unintended break, and both amd64 and arm64 +client images start and pass smoke/config checks. diff --git a/docs/workflows.md b/docs/workflows.md new file mode 100644 index 0000000..375c3c6 --- /dev/null +++ b/docs/workflows.md @@ -0,0 +1,180 @@ +# Resource and workflow model + +## Resource identity and file selection + +A resource is identified by its BitTorrent metainfo identity: + +- v1: lowercase 40-character SHA-1 info hash; +- v2: lowercase 64-character SHA-256 info hash; +- hybrid: both values. + +The qBittorrent-local torrent hash is retained only as an adapter identifier. +Identity matching never relies on display name or filesystem path. Control +maintains aliases for both hashes of a hybrid torrent: one matching hash joins +records only when any other supplied hash does not conflict, and the normalized +identity is then enriched with both values. Reservations cover every known +alias so v1-only and hybrid observations cannot schedule concurrently. + +A selection is a normalized set of torrent file indices. Ranges are sorted, +coalesced, and immutable after job creation. Only selected/skipped state is +portable; qBittorrent priority levels are deliberately ignored. A directory +choice expands to all currently available selected-complete descendant file +indices before confirmation. + +For every transfer, control fetches complete source and target content trees +immediately before staging. It validates the source fingerprint, expands the +requested selection, reads the target's verified baseline, and computes: + +```text +transfer delta = requested source files - verified target files +target union = verified target files + transfer delta +``` + +An empty delta is a no-op and is rejected rather than recorded as a successful +transfer. Separate one-source/one-target jobs can merge complementary +selections into the same placement over time. + +## Job creation + +All creation paths follow the same service-layer transaction: + +1. Resolve a fresh source summary and suitable online targets. +2. Optionally resolve one file or directory; otherwise use all + selected-complete source files. +3. Fetch full source and target trees and compute the exact delta. +4. Validate role, route feasibility, canonical paths, generation, completion, + disk capacity, and advertised capabilities. Partfile compatibility is not an + eligibility filter at this stage. +5. Show a final confirmation containing operation, resource, source, target, + requested size/count, and whether this extends an existing placement. +6. Consume a single-use confirmation nonce and atomically insert the immutable + job, resource reservation, and initial event. + +The previewed requested selection, verified target baseline, exact transfer +delta, and their logical byte counts become immutable fields in the job +definition. Execution recomputes and compares them rather than silently +adapting to external changes. + +Partfile handling is deliberately checked again immediately before the first +data step. A resource is not hidden as ineligible merely because its partfile +format may be unsafe; the accepted job fails with an explicit reason when the +live preflight proves it cannot be handled. + +## Route readiness + +Each data route is one `sendreceive` Syncthing folder shared by exactly two +devices: one cache client and one archive client. The Syncthing folder ID is the +route ID. + +At registration, clients report compatible folders below their configured sync +root. Control pairs matching route IDs. It ignores and never modifies +multi-device or wrong-type folders. + +If a needed pair has no route, the default `on_demand` policy performs: + +1. Generate a route/folder ID and deterministic relative folder path. +2. Send each client the peer device ID and advertised addresses. +3. Have each client idempotently add the peer device and its side of the same + folder. +4. Wait for both Syncthing instances to report a healthy shared folder. +5. Exchange unique nonce files in both directions and require both observations. +6. Mark the route ready and refresh registrations/routes. + +The default timeout is 30 minutes and is configurable. A timeout leaves a +diagnosable provisioning state. Rollback may remove only configuration created +by that attempt and only while ownership is unambiguous. A previously verified +route is never automatically deleted. `eager_mesh` runs the same process for +all missing online cache/archive pairs. + +## Archive and unarchive transfers + +Archive and unarchive share one transfer state machine; only source/target roles +and placement policy differ. + +### Preflight + +- Revalidate both resource fingerprints and placement generations. +- Require source-selected files to be complete and regular, canonical entries. +- Reject noncanonical qBittorrent file/folder renames in v1. +- Confirm target path types/sizes do not conflict. +- For an existing target, require every currently selected baseline file to be + complete and verified. Unexpected selected-incomplete files fail as stale + external state rather than being downloaded or silently deselected. +- Confirm free space/reserve, permissions, sparse capability, supported + partfile format, route health, and negotiated protocol features. +- Export the source `.torrent` metainfo. +- Persist the exact source, target baseline, requested selection, and delta. + +### Five visible steps + +1. **Source Stage/HardLink.** Create the isolated job directory and materialize + only delta files plus required metainfo/auxiliary data. Try hard link, + reflink, then copy, journaling every result. Write the manifest atomically + and the ready marker last. +2. **Syncthing.** Wait for the destination to receive the complete job prefix. + Report job-scoped percentage and approximate speed, but do not infer success + from a progress percentage alone. +3. **Target Merge/Copy.** Materialize the delta beneath the configured qB root. + Reuse same-size pre-existing regular files without overwriting them; record + provenance so verification or rollback can distinguish them. +4. **QB Verification.** Add a missing torrent stopped with manual management + and explicit save path, or apply a transactional delta to an existing + torrent. Set selected/skipped state, run a full recheck of the target union, + and fail immediately if qBittorrent attempts to download. Successful recheck + atomically advances the placement generation and is the commit point. +5. **Staging Cleanup.** Remove only job-owned staging artifacts from both + endpoints. A post-commit cleanup failure produces `CLEANUP_REQUIRED`; it + never rolls back or deletes the committed placement. + +Archive keeps the cache source. Unarchive always keeps the archive source. A +new target is started only after successful verification so it can seed; an +existing target returns to its captured stopped/running state. Archive Control +does not copy tags, categories, rate limits, queue position, or other +source-local metadata. + +### Existing target merge + +Before changing an existing target, the client durably records its selection, +run state, qB identifier, content revision, file provenance, and partfile +baseline. Failure before commit restores that baseline and removes only files +created by the current job that remain unreferenced. Pre-existing paths are +never removed. Success stores the verified union, allowing later jobs from +other sources to add complementary file indices. + +## Evict cache + +Eviction is independent of transfer history and operates on one whole cache +placement in v1. + +1. Query the cache placement and all relevant online archive placements fresh. +2. Build per-file archive coverage; the union may span multiple archive nodes. +3. Disable confirmation and explain missing files if coverage is incomplete. +4. On confirmation, reserve the resource and persist the exact cache + fingerprint/generation plus archive proofs. +5. Revalidate those proofs immediately before deletion. +6. Ask qBittorrent to remove the cache entry with data deletion disabled. +7. Snapshot exact paths referenced by all remaining local torrents. +8. Unlink only manifest-owned files not exactly referenced by another torrent. +9. Remove a partfile only when ownership by the evicted torrent is + unambiguous. +10. Attempt non-recursive `rmdir` from deepest known owned directory upward; + retain directories containing unknown files, subdirectories, or other + application data. + +There is no force override in v1. Retrying any step observes prior effects and +continues safely. Eviction never changes archive placements. + +## Cancellation, removal, and display retention + +- Cancelling a queued job deletes the queue record and releases its reservation + only; no client command or file operation occurs. +- Cancelling an active pre-commit job requests cooperative stop and runs + attributable compensation. The source placement remains intact. +- After commit, cancellation can only drive staging cleanup. It cannot undo the + committed target. +- A failed or cancelling job stays visible until its cleanup/compensation state + is known. +- Clear Finished and removal of a successful record affect only UI visibility. + They never touch qBittorrent, files, routes, or placements. +- Hidden jobs retain a minimal idempotency tombstone for 30 days by default; + detailed events may be purged sooner under configured retention.