277 lines
12 KiB
Markdown
277 lines
12 KiB
Markdown
# Actionable implementation plan
|
||
|
||
Implementation status (2026-07-23): phases 0–7 are implemented in the proto,
|
||
control, and client worktrees. Phase 8 has a passing local 2×2 happy path,
|
||
deterministic selective/recovery matrix, and adversarial v2/fault matrix:
|
||
complementary partial placements merge correctly, eviction uses archive
|
||
coverage unions, queued and active cancellation preserve their safety
|
||
boundaries, restarts replay durable work, and hostile preconditions fail
|
||
without committing. Pure-v2/hybrid round trips, recoverable stalls,
|
||
capacity-guard failure, and duplicate/reordered WebSocket events also pass from
|
||
a clean topology. The remaining fixture and fault categories listed below are
|
||
still release gates; phase 9 remains intentionally deferred until the complete
|
||
release matrix passes.
|
||
|
||
No application implementation begins until Phase 0 is reviewed. Within every
|
||
later phase, tests and documentation are part of the deliverable rather than a
|
||
follow-up task.
|
||
|
||
## Phase 0 — Freeze the contract
|
||
|
||
### Actions
|
||
|
||
1. Convert the decision ledger into the system, workflow, safety, service API,
|
||
deployment, Telegram, and test documents in this directory.
|
||
2. Create the independent `archive-control-proto` Git repository.
|
||
3. Define all v1 resource, selection, route, inventory, job, progress,
|
||
command/event, reconciliation, and envelope messages.
|
||
4. Add canonical protobuf-JSON examples for registration, inventory queries,
|
||
route creation, transfer start/progress/commit, cancellation, eviction, and
|
||
recovery snapshots.
|
||
5. Run Dockerized Buf format/lint and protoc descriptor compilation.
|
||
6. Configure the Gitea remote `cabbage/archive-control-proto` with its dedicated
|
||
SSH key. Tag the reviewed baseline `v0.1.0` only after review.
|
||
|
||
### Acceptance gate
|
||
|
||
- No unresolved `TODO` changes wire meaning or destructive behavior.
|
||
- Every workflow transition maps to a protobuf command/event.
|
||
- JSON examples parse through generated descriptors.
|
||
- Documentation and protobuf enums/state machines agree.
|
||
|
||
## Phase 1 — Build the control core inside mogic-bot
|
||
|
||
### Actions
|
||
|
||
1. Add `archive_control/` parallel to `remote_control/`, with no imports from
|
||
Telegram in its core service, scheduler, protocol, persistence, or
|
||
orchestration layers.
|
||
2. Add strict configuration parsing to the existing bot configuration,
|
||
including shared token, WebSocket bind/path, database/backup paths,
|
||
heartbeat, scheduling, route policy, retention, and Telegram refresh knobs.
|
||
3. Add SQLite schema migrations for clients, connections, routes, jobs,
|
||
commands, job events, reservations, placements, UI sessions, and
|
||
tombstones. Enable foreign keys, WAL, busy timeout, and transactional schema
|
||
migration.
|
||
4. Implement online backups, integrity verification, retention, and offline
|
||
verify/restore commands.
|
||
5. Implement WebSocket registration-first authentication, duplicate-client
|
||
replacement, heartbeats, bounded send queues, envelope validation, and
|
||
protocol version negotiation.
|
||
6. Implement durable command outbox/inbox semantics, acknowledgements,
|
||
per-job event sequencing, snapshots, and reconnect reconciliation.
|
||
7. Implement the global resource reservation and FIFO per-client/per-route
|
||
scheduler.
|
||
8. Add the thin test-only HTTP adapter over `ArchiveControlService`.
|
||
|
||
### Acceptance gate
|
||
|
||
- Bot startup/shutdown cleanly owns Archive Control tasks in its existing
|
||
asyncio lifecycle.
|
||
- Existing bot tests and workflows are unchanged.
|
||
- Restart, duplicate registration, lost acknowledgement, event duplication,
|
||
sequence gap, and DB backup/restore tests pass.
|
||
- The test HTTP adapter can create/query/cancel synthetic jobs without Telegram.
|
||
|
||
## Phase 2 — Build the archive-client foundation
|
||
|
||
### Actions
|
||
|
||
1. Scaffold `archive-clients` with a `src/` layout, Python 3.11, strict TOML
|
||
parsing, environment/file secrets, CLI overrides, structured redacted logs,
|
||
and console entry points.
|
||
2. Implement `--mode archive|cache`, stable client ID, state/backup DBs, and
|
||
Docker-friendly shutdown handling.
|
||
3. Implement capped exponential reconnect with jitter, registration,
|
||
heartbeat acknowledgements, one-writer WebSocket output, durable command
|
||
deduplication, and active-job snapshots.
|
||
4. Implement API-visible-to-local root mapping and path-confinement helpers.
|
||
5. Probe qBittorrent, Syncthing, filesystem/link/reflink, sparse, and permission
|
||
capabilities at startup; advertise a precise health/capability report.
|
||
6. Add multi-stage Dockerfile and Compose example. Run unprivileged and mount
|
||
only config, secrets, DB/backups, qB data, and sync data.
|
||
7. Build amd64 first; keep the Dockerfile architecture-neutral for the release
|
||
buildx phase.
|
||
|
||
### Acceptance gate
|
||
|
||
- Invalid/unknown configuration fails before connecting.
|
||
- Secrets never appear in logs or SQLite.
|
||
- Restart and duplicate-command tests prove idempotent client behavior.
|
||
- Both daemon roles start from the same image and differ only by config/mode.
|
||
|
||
## Phase 3 — Service adapters and on-demand discovery
|
||
|
||
### Actions
|
||
|
||
1. Implement qBittorrent cookie authentication and capability adapters for
|
||
supported 4.5–4.6 and 5.x APIs.
|
||
2. Implement on-demand torrent summaries, scoped hash lookup, lazy complete
|
||
content trees, v1/v2 identity validation, torrent export, stopped add,
|
||
selection application, recheck monitoring, and entry-only deletion.
|
||
3. Implement Syncthing API-key authentication, version/config/status/events,
|
||
folder/device discovery, per-file progress, need/file completion checks, and
|
||
rescan support.
|
||
4. Discover only healthy pairwise `sendreceive` routes under the configured
|
||
roots.
|
||
5. Implement idempotent device pairing and folder creation using advertised
|
||
addresses, plus bidirectional nonce verification and 30-minute default
|
||
timeout.
|
||
6. Implement scoped, chunked, revisioned inventory and tree queries; do not add
|
||
background cluster inventory polling.
|
||
|
||
### Acceptance gate
|
||
|
||
- Contract fixtures cover API version differences and permanent/transient
|
||
errors.
|
||
- A pair of isolated client/Syncthing stacks can auto-create and verify a route.
|
||
- Large synthetic inventories stream in bounded chunks and interrupted results
|
||
are not committed by Titan.
|
||
|
||
## Phase 4 — Safe file operation and manifest layer
|
||
|
||
### Actions
|
||
|
||
1. Define the on-disk job staging layout and atomically written protobuf-JSON
|
||
transfer manifest/ready marker.
|
||
2. Implement safe traversal using normalized torrent paths, no-follow checks,
|
||
root confinement, regular-file-only validation, and duplicate/case-collision
|
||
detection.
|
||
3. Implement hardlink → reflink → safe-copy fallback with per-file journaling,
|
||
progress, cancellation points, and restart reconciliation.
|
||
4. Implement exact logical/allocated size accounting, free-space/reserve checks,
|
||
and startup sparse capability enforcement.
|
||
5. Implement target never-overwrite behavior and provenance for created versus
|
||
pre-existing reused files.
|
||
6. Implement tested qBittorrent/libtorrent partfile adapters, compatibility
|
||
gates, baseline backups, and safe merge/rollback behavior.
|
||
|
||
### Acceptance gate
|
||
|
||
- Tests cover cross-filesystem fallback, insufficient space, permission errors,
|
||
symlink races, special files, path escape, pre-existing collisions, sparse
|
||
artifacts, and supported/unsupported partfiles.
|
||
- Cancelling/restarting every file-operation phase leaves a deterministic
|
||
journal and no uncontrolled deletion.
|
||
|
||
## Phase 5 — Transfer orchestration
|
||
|
||
### Actions
|
||
|
||
1. Implement fresh source/target tree preflight, requested-selection expansion,
|
||
immutable file-index snapshots, exact delta calculation, and no-op rejection.
|
||
2. Implement source staging, per-job Syncthing progress, destination completion
|
||
proof, and target materialization.
|
||
3. Implement new-target stopped/manual add and existing-target baseline/delta
|
||
transaction.
|
||
4. Apply selected/skipped state only, recheck the union while stopped, monitor
|
||
for any download attempt, and restore/start the correct run state.
|
||
5. Commit placement generation only after verification; treat staging removal
|
||
as post-commit cleanup with retryable `CLEANUP_REQUIRED` state.
|
||
6. Implement cooperative cancellation/compensation for every pre-commit phase
|
||
and non-destructive post-commit cleanup behavior.
|
||
|
||
### Acceptance gate
|
||
|
||
- Full, selective, subset, and incremental merge scenarios pass in both
|
||
directions.
|
||
- A target never downloads data during verification.
|
||
- Failure injection at every transition preserves source and target baselines.
|
||
- Archive jobs retain their source cache placements.
|
||
|
||
## Phase 6 — Eviction and placement reconciliation
|
||
|
||
### Actions
|
||
|
||
1. Implement on-demand cache placement listing and fresh archive coverage sets.
|
||
2. Require complete per-file coverage across one or more online archive nodes;
|
||
expose disabled reasons and provide no force override.
|
||
3. Implement eviction jobs: coverage recheck, qB entry-only removal, remaining
|
||
torrent exact-path overlap snapshot, safe file unlink, partfile ownership
|
||
check, and non-recursive bottom-up directory removal.
|
||
4. Reconcile observed placements with successful job generations and external
|
||
qBittorrent changes without background polling.
|
||
|
||
### Acceptance gate
|
||
|
||
- Eviction cannot proceed with any uncovered selected file.
|
||
- Shared exact paths, shared directories, unknown files, and unrelated
|
||
partfiles survive.
|
||
- Retrying after interruption is idempotent and an archive placement is never
|
||
deleted.
|
||
|
||
## Phase 7 — Telegram integration
|
||
|
||
### Actions
|
||
|
||
1. Add `ArchiveControlHandler` without changing existing handler behavior.
|
||
2. Implement Archive, Unarchive, Evict Cache, and Job Status flows with one
|
||
edited message, pagers, chained literal filters, cancel/back behavior, and
|
||
single-use confirmation nonces.
|
||
3. Implement normal-all and one-entry tree selection; keep protobuf support for
|
||
multiple selections.
|
||
4. Persist UI sessions and restore live status subscriptions after bot restart.
|
||
5. Inherit private-chat username authorization and optionally pin configured
|
||
numeric IDs. Keep jobs shared and ownerless.
|
||
6. Render five-step transfer and three-step eviction progress, including stale,
|
||
waiting, stalled percentage, cleanup-required, failure reason, paging,
|
||
Clear Finished, and Remove One.
|
||
7. Coalesce routine edits to the configurable five-second default and handle
|
||
Telegram rate limits.
|
||
|
||
### Acceptance gate
|
||
|
||
- Telegram and test HTTP adapters produce identical service-layer outcomes.
|
||
- Stale/double callbacks are harmless.
|
||
- Bot restart preserves active UI/status behavior.
|
||
- Existing mogic-bot regression tests pass.
|
||
|
||
## Phase 8 — Cross-project E2E and chaos testing
|
||
|
||
### Actions
|
||
|
||
1. Create five Compose projects under this repository's `e2e/`: control,
|
||
cache-1, cache-2, archive-1, and archive-2.
|
||
2. Give every data node isolated qBittorrent, Syncthing, archive-client, config,
|
||
state, backup, and data volumes on one dedicated test network.
|
||
3. Configure static advertised Syncthing service addresses while testing
|
||
automatic device/folder pairing.
|
||
4. Generate small v1/v2/hybrid torrents and fixtures for complementary and
|
||
overlapping selections, partfiles, sparse files, path collisions, shared
|
||
directories, and permission errors.
|
||
5. Drive scenarios with curl against the test HTTP adapter.
|
||
6. Inject control/client/service restarts, network loss, duplicate commands,
|
||
stale snapshots, stalls, disk exhaustion, cancellation, and DB restore.
|
||
7. Tear down only test resources identified by unique project names/labels.
|
||
|
||
### Acceptance gate
|
||
|
||
- The complete acceptance matrix in `testing.md` passes repeatedly without
|
||
live infrastructure.
|
||
- No test uses credentials or paths from x1, x2, lithium, or Titan production.
|
||
- Test teardown is scope-validated and leaves unrelated Docker resources alone.
|
||
|
||
## Phase 9 — Release and staged deployment
|
||
|
||
### Actions
|
||
|
||
1. Run Buf breaking checks against the last proto release and tag compatible
|
||
repositories.
|
||
2. Build `sodium/archive-clients` for linux/amd64 and linux/arm64 with OCI
|
||
source/protocol labels; publish an immutable semantic-version tag and the
|
||
stable `latest` alias.
|
||
3. Record image digest and provide pinned Compose examples.
|
||
4. Back up real qBittorrent/Syncthing and Archive Control state before a
|
||
separately approved deployment.
|
||
5. Deploy one cache/archive pair in observation mode, validate on-demand
|
||
inventory and route discovery, then run a small non-critical canary transfer.
|
||
6. Exercise recovery and eviction only after canary verification and explicit
|
||
approval, then expand node-by-node.
|
||
|
||
### Acceptance gate
|
||
|
||
- Multi-platform manifest resolves to tested images and the runtime image is
|
||
reasonably small.
|
||
- Rollback instructions and prior images/config/DB backups are verified.
|
||
- No production qBittorrent/Syncthing global setting is modified.
|