archive-clients/docs/implementation-plan.md

Actionable implementation plan

Implementation status (2026-07-23): phases 0–7 are implemented in the proto, control, and client worktrees. Phase 8 has a passing local 2×2 happy path, deterministic selective/recovery matrix, and adversarial v2/fault matrix: complementary partial placements merge correctly, eviction uses archive coverage unions, queued and active cancellation preserve their safety boundaries, restarts replay durable work, and hostile preconditions fail without committing. Pure-v2/hybrid round trips, recoverable stalls, capacity-guard failure, and duplicate/reordered WebSocket events also pass from a clean topology. The remaining fixture and fault categories listed below are still release gates; phase 9 remains intentionally deferred until the complete release matrix passes.

No application implementation begins until Phase 0 is reviewed. Within every later phase, tests and documentation are part of the deliverable rather than a follow-up task.

Phase 0 — Freeze the contract

Actions

1. Convert the decision ledger into the system, workflow, safety, service API, deployment, Telegram, and test documents in this directory.
2. Create the independent archive-control-proto Git repository.
3. Define all v1 resource, selection, route, inventory, job, progress, command/event, reconciliation, and envelope messages.
4. Add canonical protobuf-JSON examples for registration, inventory queries, route creation, transfer start/progress/commit, cancellation, eviction, and recovery snapshots.
5. Run Dockerized Buf format/lint and protoc descriptor compilation.
6. Configure the Gitea remote cabbage/archive-control-proto with its dedicated SSH key. Tag the reviewed baseline v0.1.0 only after review.

Acceptance gate

• No unresolved TODO changes wire meaning or destructive behavior.
• Every workflow transition maps to a protobuf command/event.
• JSON examples parse through generated descriptors.
• Documentation and protobuf enums/state machines agree.

Phase 1 — Build the control core inside mogic-bot

Actions

1. Add archive_control/ parallel to remote_control/, with no imports from Telegram in its core service, scheduler, protocol, persistence, or orchestration layers.
2. Add strict configuration parsing to the existing bot configuration, including shared token, WebSocket bind/path, database/backup paths, heartbeat, scheduling, route policy, retention, and Telegram refresh knobs.
3. Add SQLite schema migrations for clients, connections, routes, jobs, commands, job events, reservations, placements, UI sessions, and tombstones. Enable foreign keys, WAL, busy timeout, and transactional schema migration.
4. Implement online backups, integrity verification, retention, and offline verify/restore commands.
5. Implement WebSocket registration-first authentication, duplicate-client replacement, heartbeats, bounded send queues, envelope validation, and protocol version negotiation.
6. Implement durable command outbox/inbox semantics, acknowledgements, per-job event sequencing, snapshots, and reconnect reconciliation.
7. Implement the global resource reservation and FIFO per-client/per-route scheduler.
8. Add the thin test-only HTTP adapter over ArchiveControlService.

Acceptance gate

• Bot startup/shutdown cleanly owns Archive Control tasks in its existing asyncio lifecycle.
• Existing bot tests and workflows are unchanged.
• Restart, duplicate registration, lost acknowledgement, event duplication, sequence gap, and DB backup/restore tests pass.
• The test HTTP adapter can create/query/cancel synthetic jobs without Telegram.

Phase 2 — Build the archive-client foundation

Actions

1. Scaffold archive-clients with a src/ layout, Python 3.11, strict TOML parsing, environment/file secrets, CLI overrides, structured redacted logs, and console entry points.
2. Implement --mode archive|cache, stable client ID, state/backup DBs, and Docker-friendly shutdown handling.
3. Implement capped exponential reconnect with jitter, registration, heartbeat acknowledgements, one-writer WebSocket output, durable command deduplication, and active-job snapshots.
4. Implement API-visible-to-local root mapping and path-confinement helpers.
5. Probe qBittorrent, Syncthing, filesystem/link/reflink, sparse, and permission capabilities at startup; advertise a precise health/capability report.
6. Add multi-stage Dockerfile and Compose example. Run unprivileged and mount only config, secrets, DB/backups, qB data, and sync data.
7. Build amd64 first; keep the Dockerfile architecture-neutral for the release buildx phase.

Acceptance gate

• Invalid/unknown configuration fails before connecting.
• Secrets never appear in logs or SQLite.
• Restart and duplicate-command tests prove idempotent client behavior.
• Both daemon roles start from the same image and differ only by config/mode.

Phase 3 — Service adapters and on-demand discovery

Actions

1. Implement qBittorrent cookie authentication and capability adapters for supported 4.5–4.6 and 5.x APIs.
2. Implement on-demand torrent summaries, scoped hash lookup, lazy complete content trees, v1/v2 identity validation, torrent export, stopped add, selection application, recheck monitoring, and entry-only deletion.
3. Implement Syncthing API-key authentication, version/config/status/events, folder/device discovery, per-file progress, need/file completion checks, and rescan support.
4. Discover only healthy pairwise sendreceive routes under the configured roots.
5. Implement idempotent device pairing and folder creation using advertised addresses, plus bidirectional nonce verification and 30-minute default timeout.
6. Implement scoped, chunked, revisioned inventory and tree queries; do not add background cluster inventory polling.

Acceptance gate

• Contract fixtures cover API version differences and permanent/transient errors.
• A pair of isolated client/Syncthing stacks can auto-create and verify a route.
• Large synthetic inventories stream in bounded chunks and interrupted results are not committed by Titan.

Phase 4 — Safe file operation and manifest layer

Actions

1. Define the on-disk job staging layout and atomically written protobuf-JSON transfer manifest/ready marker.
2. Implement safe traversal using normalized torrent paths, no-follow checks, root confinement, regular-file-only validation, and duplicate/case-collision detection.
3. Implement hardlink → reflink → safe-copy fallback with per-file journaling, progress, cancellation points, and restart reconciliation.
4. Implement exact logical/allocated size accounting, free-space/reserve checks, and startup sparse capability enforcement.
5. Implement target never-overwrite behavior and provenance for created versus pre-existing reused files.
6. Implement tested qBittorrent/libtorrent partfile adapters, compatibility gates, baseline backups, and safe merge/rollback behavior.

Acceptance gate

• Tests cover cross-filesystem fallback, insufficient space, permission errors, symlink races, special files, path escape, pre-existing collisions, sparse artifacts, and supported/unsupported partfiles.
• Cancelling/restarting every file-operation phase leaves a deterministic journal and no uncontrolled deletion.

Phase 5 — Transfer orchestration

Actions

1. Implement fresh source/target tree preflight, requested-selection expansion, immutable file-index snapshots, exact delta calculation, and no-op rejection.
2. Implement source staging, per-job Syncthing progress, destination completion proof, and target materialization.
3. Implement new-target stopped/manual add and existing-target baseline/delta transaction.
4. Apply selected/skipped state only, recheck the union while stopped, monitor for any download attempt, and restore/start the correct run state.
5. Commit placement generation only after verification; treat staging removal as post-commit cleanup with retryable CLEANUP_REQUIRED state.
6. Implement cooperative cancellation/compensation for every pre-commit phase and non-destructive post-commit cleanup behavior.

Acceptance gate

• Full, selective, subset, and incremental merge scenarios pass in both directions.
• A target never downloads data during verification.
• Failure injection at every transition preserves source and target baselines.
• Archive jobs retain their source cache placements.

Phase 6 — Eviction and placement reconciliation

Actions

1. Implement on-demand cache placement listing and fresh archive coverage sets.
2. Require complete per-file coverage across one or more online archive nodes; expose disabled reasons and provide no force override.
3. Implement eviction jobs: coverage recheck, qB entry-only removal, remaining torrent exact-path overlap snapshot, safe file unlink, partfile ownership check, and non-recursive bottom-up directory removal.
4. Reconcile observed placements with successful job generations and external qBittorrent changes without background polling.

Acceptance gate

• Eviction cannot proceed with any uncovered selected file.
• Shared exact paths, shared directories, unknown files, and unrelated partfiles survive.
• Retrying after interruption is idempotent and an archive placement is never deleted.

Phase 7 — Telegram integration

Actions

1. Add ArchiveControlHandler without changing existing handler behavior.
2. Implement Archive, Unarchive, Evict Cache, and Job Status flows with one edited message, pagers, chained literal filters, cancel/back behavior, and single-use confirmation nonces.
3. Implement normal-all and one-entry tree selection; keep protobuf support for multiple selections.
4. Persist UI sessions and restore live status subscriptions after bot restart.
5. Inherit private-chat username authorization and optionally pin configured numeric IDs. Keep jobs shared and ownerless.
6. Render five-step transfer and three-step eviction progress, including stale, waiting, stalled percentage, cleanup-required, failure reason, paging, Clear Finished, and Remove One.
7. Coalesce routine edits to the configurable five-second default and handle Telegram rate limits.

Acceptance gate

• Telegram and test HTTP adapters produce identical service-layer outcomes.
• Stale/double callbacks are harmless.
• Bot restart preserves active UI/status behavior.
• Existing mogic-bot regression tests pass.

Phase 8 — Cross-project E2E and chaos testing

Actions

1. Create five Compose projects under this repository's e2e/: control, cache-1, cache-2, archive-1, and archive-2.
2. Give every data node isolated qBittorrent, Syncthing, archive-client, config, state, backup, and data volumes on one dedicated test network.
3. Configure static advertised Syncthing service addresses while testing automatic device/folder pairing.
4. Generate small v1/v2/hybrid torrents and fixtures for complementary and overlapping selections, partfiles, sparse files, path collisions, shared directories, and permission errors.
5. Drive scenarios with curl against the test HTTP adapter.
6. Inject control/client/service restarts, network loss, duplicate commands, stale snapshots, stalls, disk exhaustion, cancellation, and DB restore.
7. Tear down only test resources identified by unique project names/labels.

Acceptance gate

• The complete acceptance matrix in testing.md passes repeatedly without live infrastructure.
• No test uses credentials or paths from x1, x2, lithium, or Titan production.
• Test teardown is scope-validated and leaves unrelated Docker resources alone.

Phase 9 — Release and staged deployment

Actions

1. Run Buf breaking checks against the last proto release and tag compatible repositories.
2. Build sodium/archive-clients for linux/amd64 and linux/arm64 with OCI source/protocol labels; publish an immutable semantic-version tag and the stable latest alias.
3. Record image digest and provide pinned Compose examples.
4. Back up real qBittorrent/Syncthing and Archive Control state before a separately approved deployment.
5. Deploy one cache/archive pair in observation mode, validate on-demand inventory and route discovery, then run a small non-critical canary transfer.
6. Exercise recovery and eviction only after canary verification and explicit approval, then expand node-by-node.

Acceptance gate

• Multi-platform manifest resolves to tested images and the runtime image is reasonably small.
• Rollback instructions and prior images/config/DB backups are verified.
• No production qBittorrent/Syncthing global setting is modified.