docs: publish archive control design set
This commit is contained in:
@@ -57,12 +57,11 @@ archive-client-backup --database /var/lib/archive-control/client.db \
|
||||
Secrets must be regular files without group/world permissions. The daemon never
|
||||
stores them in SQLite or sends the shared token after registration.
|
||||
|
||||
The daemon currently executes heartbeat, inventory, state-snapshot, and
|
||||
route-provisioning commands. Transfer storage primitives are implemented and
|
||||
tested but are not yet wired to assignment/step commands; those and eviction
|
||||
commands remain durably rejected as unsupported until their orchestration and
|
||||
qBittorrent executors are added. They are never falsely acknowledged as
|
||||
accepted.
|
||||
The daemon executes heartbeat, inventory, state-snapshot, route-provisioning,
|
||||
archive/unarchive assignment and step commands, compensation, staging cleanup,
|
||||
and safe cache eviction. Durable journals and job cursors make interrupted
|
||||
work replayable; unsupported or invalid assignments are rejected explicitly
|
||||
and never falsely acknowledged as accepted.
|
||||
|
||||
Run tests and build using containers:
|
||||
|
||||
@@ -74,7 +73,12 @@ docker build -t sodium/archive-clients:dev .
|
||||
|
||||
The local cross-project route test is under `e2e/`. It brings up isolated
|
||||
2×2 cache/archive node stacks plus the standalone control core and verifies all
|
||||
four eager-mesh routes without Telegram. See `e2e/README.md`.
|
||||
four eager-mesh routes, transfer/eviction/unarchive workflows, and the
|
||||
adversarial matrix without live infrastructure or Telegram. See
|
||||
`e2e/README.md`.
|
||||
|
||||
The complete cross-project design, protocol, workflow, safety, deployment,
|
||||
Telegram UX, and testing documentation is published under [`docs/`](docs/).
|
||||
|
||||
Generated bindings are pinned to archive-control-proto commit
|
||||
`4ec852014dad74606d4078b3ae1aa208c814b033`.
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
# Archive Control design documentation
|
||||
|
||||
This directory is the implementation contract for Archive Control. These
|
||||
documents incorporate the decisions made during the design review and
|
||||
supersede conflicting details in the original private problem statement. That
|
||||
statement is intentionally not published because it contains deployment
|
||||
credentials and host-specific details.
|
||||
|
||||
## Reading order
|
||||
|
||||
1. [Decisions](decisions.md)
|
||||
2. [Implementation plan](implementation-plan.md)
|
||||
3. [System design](system-design.md)
|
||||
4. [Resource and workflow model](workflows.md)
|
||||
5. [Daemon protocol](protocol.md)
|
||||
6. [Storage safety and recovery](storage-safety.md)
|
||||
7. [qBittorrent and Syncthing integration](service-apis.md)
|
||||
8. [Configuration, deployment, and usage](deployment-and-usage.md)
|
||||
9. [Telegram UX](telegram-ux.md)
|
||||
10. [Testing strategy](testing.md)
|
||||
|
||||
The independent protobuf source of truth lives in the
|
||||
`cabbage/archive-control-proto` repository. Generated bindings in this
|
||||
repository record the exact source commit.
|
||||
|
||||
## Implementation status
|
||||
|
||||
As of 2026-07-23, implementation phases 0–7 are complete across the proto,
|
||||
control, and client repositories. This includes registration, recovery,
|
||||
on-demand inventory, automatic route provisioning, transfer preview and delta
|
||||
planning, selective archive/unarchive merging, durable execution and
|
||||
compensation, safe cache eviction, database backup/restore, and the integrated
|
||||
Telegram flows.
|
||||
|
||||
The local 2×2 topology passes archive, eviction, and unarchive happy paths plus
|
||||
selective/partial merges, multiple archive coverage, restart/replay,
|
||||
pure-v2/hybrid identities, stalls, disk exhaustion, hostile preconditions, and
|
||||
protocol duplication/reordering. The integrated bot also has an isolated dummy
|
||||
backend for interactive Telegram conversation review and trace export.
|
||||
Remaining work is the broader release/chaos matrix and the separately approved
|
||||
staged deployment described in phases 8–9.
|
||||
@@ -0,0 +1,137 @@
|
||||
# Agreed design decisions
|
||||
|
||||
This is the concise decision ledger from the design interview. Later documents
|
||||
expand these rules but must not contradict them.
|
||||
|
||||
## Roles and authority
|
||||
|
||||
- Client roles are `archive` and `cache`. `archive` and `unarchive` are job
|
||||
operations, not daemon roles.
|
||||
- Titan's control daemon is the sole cross-node orchestrator. Clients never
|
||||
command one another and do not advance to the next workflow step without a
|
||||
persisted Titan command.
|
||||
- Archive nodes retain committed resources. Cache eviction is a distinct,
|
||||
explicit operation.
|
||||
- Transfer and eviction jobs are durable, idempotent, recoverable, and
|
||||
serialized globally per torrent resource.
|
||||
|
||||
## Connection and protocol
|
||||
|
||||
- Every WebSocket connection starts with a registration containing client ID,
|
||||
role, protocol/capabilities, and the single pre-shared token. The token occurs
|
||||
in no later message. Invalid registration closes the connection.
|
||||
- A new valid registration for a duplicate client ID disconnects the old
|
||||
connection.
|
||||
- Clients reconnect forever with configurable capped exponential backoff.
|
||||
- Commands are at-least-once, durably deduplicated, and separate acceptance
|
||||
acknowledgements from completion events.
|
||||
- Protobuf JSON uses a versioned `archive_control.v1` envelope. Per-job event
|
||||
sequences support gap detection and snapshot reconciliation.
|
||||
|
||||
## State, recovery, and scheduling
|
||||
|
||||
- Titan and every client have SQLite state. They reconcile active jobs on every
|
||||
reconnect and fail closed when lost state cannot be proven from journals,
|
||||
manifests, and observed external state.
|
||||
- SQLite backups use the online backup API. Defaults are every six hours, 12
|
||||
recent, 14 daily, and eight weekly copies, plus pre/post-migration backups.
|
||||
- Default concurrency is one active data-moving job per client and per route.
|
||||
Disjoint node pairs may run concurrently. Queueing is durable FIFO.
|
||||
- A queued job owns a per-resource reservation. Cancelling it removes only the
|
||||
queued record/reservation and never sends cleanup commands.
|
||||
- Connectivity or transfer stalls wait indefinitely. A configurable 30-minute
|
||||
no-progress threshold marks the job `STALLED` without losing its percentage.
|
||||
|
||||
## Resources and selections
|
||||
|
||||
- Resource identity supports BitTorrent v1, v2, and hybrid info hashes. A
|
||||
qBittorrent-local hash is a separate adapter identifier. Hybrid hash aliases
|
||||
share reservations and are joined only when no supplied hash conflicts.
|
||||
- Only selected versus skipped matters; qBittorrent priority levels are not
|
||||
preserved.
|
||||
- Placements are keyed by resource and node and contain a verified set of
|
||||
selected torrent file indices. Multiple cache and archive placements are
|
||||
allowed.
|
||||
- Transfer jobs have one source and one target. A requested selection is
|
||||
expanded to immutable file indices. Targets merge the missing delta into
|
||||
their existing selection transactionally.
|
||||
- Initial Telegram selection supports either all source-selected complete
|
||||
entries or one file/directory. The protocol supports multiple entries from
|
||||
v1.
|
||||
- Full source and target file inventories are fetched immediately before
|
||||
staging, even for the normal `Confirm` path. The previewed baseline and delta
|
||||
are persisted immutably and must still match at execution.
|
||||
|
||||
## Transfer safety
|
||||
|
||||
- Each transfer uses `<sync-folder>/.archive-control/jobs/<job-id>/` with a
|
||||
payload, exported torrent, protobuf-JSON manifest, auxiliary artifacts, and a
|
||||
ready marker.
|
||||
- File materialization tries hard link, then reflink, then safe copy. It never
|
||||
follows symlinks or special files and never escapes configured roots.
|
||||
- Pre-existing target paths are never overwritten. Same-size files may be
|
||||
reused only after qBittorrent's stopped recheck proves them valid.
|
||||
- New target torrents are explicitly placed under the configured qB root,
|
||||
added stopped with manual torrent management, assigned selected/skipped
|
||||
states, fully rechecked, and started only after successful verification.
|
||||
- Existing target merges preserve a durable baseline. Failure restores the
|
||||
baseline selection/state and removes only attributable delta data.
|
||||
- qBittorrent/libtorrent partfiles are transferred or merged only through a
|
||||
tested version-aware adapter. An unsafe format fails job preflight with a
|
||||
clear reason.
|
||||
- Sparse support is probed once at client startup for configured roots. Jobs
|
||||
involving sparse files require semantic sparse support on every leg; exact
|
||||
physical extent layouts are not required.
|
||||
- Permission handling is fail-fast. The daemon never chmods/chowns source data.
|
||||
|
||||
## Commit, cleanup, and deletion
|
||||
|
||||
- A transfer commits after the target's selected union passes a stopped full
|
||||
recheck without downloading. After commit, the new placement is retained.
|
||||
- Staging cleanup is post-commit. Its failure is `CLEANUP_REQUIRED`, not a
|
||||
failed transfer, and cannot cause committed archive data to be deleted.
|
||||
- Archive jobs no longer delete their source cache placement.
|
||||
- Eviction is a durable job. It requires fresh, complete archive coverage for
|
||||
every selected cache file; coverage may be the union of multiple online
|
||||
archive nodes. There is no force override initially.
|
||||
- Eviction removes the qBittorrent entry without data deletion, checks exact
|
||||
path overlap with remaining torrents, unlinks eligible known files, and uses
|
||||
non-recursive bottom-up directory removal. It never blindly removes a tree.
|
||||
|
||||
## Syncthing routes
|
||||
|
||||
- Client config supplies Syncthing endpoint/API key and separate API-visible
|
||||
and daemon-local sync roots. Compatible `sendreceive` folders beneath the
|
||||
root are discovered automatically.
|
||||
- Syncthing's native folder ID is the route ID. Routes are dedicated to exactly
|
||||
two devices: one archive and one cache client.
|
||||
- Missing routes can be created `on_demand` (default) or by `eager_mesh`.
|
||||
Clients automatically pair missing Syncthing devices and create/verify
|
||||
folders. Optional advertised addresses default to `dynamic`.
|
||||
- Route setup timeout defaults to 30 minutes. Archive Control never removes a
|
||||
verified route merely because a node is absent.
|
||||
|
||||
## User interface and retention
|
||||
|
||||
- Main actions are Archive, Unarchive, Evict Cache, and Job Status.
|
||||
- Telegram state is shared across authorized admins. Jobs/events store no
|
||||
Telegram user owner.
|
||||
- Existing private-chat `@username` authorization remains compatible; optional
|
||||
numeric user-ID pins harden configured entries when available.
|
||||
- Archive Control's Telegram view/filter/confirmation sessions persist in
|
||||
SQLite. Destructive confirmations use single-use nonces.
|
||||
- Telegram progress edits are coalesced to at most once every five seconds by
|
||||
default, with immediate important transitions.
|
||||
- Clear/Remove hides completed records without touching placements. Minimal
|
||||
idempotency tombstones remain for 30 days by default.
|
||||
|
||||
## Delivery and testing
|
||||
|
||||
- `archive-control-proto` is drafted and reviewed before application code.
|
||||
- Host tooling stays light. Buf, protoc, tests, and builds run in Docker and
|
||||
preserve host UID/GID `1001:1001`.
|
||||
- The client image is `sodium/archive-clients`, published for linux/amd64 and
|
||||
linux/arm64 with immutable release tags.
|
||||
- Cross-project E2E lives in this coordination repository and simulates two
|
||||
cache nodes, two archive nodes, and a thin control node. It never mutates the
|
||||
live x1/x2/lithium services.
|
||||
@@ -0,0 +1,267 @@
|
||||
# Configuration, deployment, and usage
|
||||
|
||||
## Configuration rules
|
||||
|
||||
Both daemons parse strict TOML: unknown keys, duplicate keys, invalid types,
|
||||
unsafe relative paths, and missing required secrets fail startup. Effective
|
||||
precedence is command line, environment, TOML, then documented default. Secret
|
||||
fields support a direct environment value or a `*_file` path, never both.
|
||||
Environment interpolation is explicit `${NAME}` syntax; an unset value fails.
|
||||
|
||||
`--print-effective-config` emits a validated, fully resolved configuration with
|
||||
all secrets redacted. Durations accept a documented unit string. Paths shown to
|
||||
the APIs and paths visible inside the daemon are configured as pairs and tested
|
||||
for consistent mapping before registration.
|
||||
|
||||
Important defaults:
|
||||
|
||||
- registration deadline `10s`;
|
||||
- heartbeat interval `15s`, offline timeout `45s`;
|
||||
- unacknowledged command retry timeout `15s`, three total attempts;
|
||||
- reconnect `1s` initial, `60s` cap, jitter enabled;
|
||||
- route policy `on_demand`, setup timeout `30m`;
|
||||
- stall warning `30m`, with no automatic failure timeout;
|
||||
- one active data job per client and route;
|
||||
- Telegram coalesce window `5s`, UI session TTL `24h`;
|
||||
- tombstone retention `30d`;
|
||||
- database backup interval `6h`, retention `12/14/8`.
|
||||
|
||||
## Control configuration example
|
||||
|
||||
```toml
|
||||
[archive_control]
|
||||
enabled = true
|
||||
listen = "127.0.0.1:8765"
|
||||
websocket_path = "/archive_control"
|
||||
shared_token_file = "/run/secrets/archive_control_token"
|
||||
database = "/var/lib/mogic/archive-control.db"
|
||||
backup_dir = "/var/backups/mogic/archive-control"
|
||||
registration_timeout = "10s"
|
||||
heartbeat_interval = "15s"
|
||||
offline_timeout = "45s"
|
||||
command_ack_timeout = "15s" # Delay before resending the same command ID.
|
||||
command_max_attempts = 3 # Total sends, including the initial attempt.
|
||||
stall_after = "30m" # Warning state only; jobs continue waiting.
|
||||
route_policy = "on_demand" # Alternative: eager_mesh.
|
||||
route_setup_timeout = "30m"
|
||||
max_active_per_client = 1
|
||||
max_active_per_route = 1
|
||||
max_envelope_bytes = 1048576
|
||||
inventory_chunk_bytes = 524288
|
||||
|
||||
[archive_control.backup]
|
||||
interval = "6h"
|
||||
recent = 12
|
||||
daily = 14
|
||||
weekly = 8
|
||||
|
||||
[archive_control.telegram]
|
||||
progress_coalesce = "5s"
|
||||
session_ttl = "24h"
|
||||
tombstone_retention = "30d"
|
||||
# Existing username allowlist remains authoritative when no ID is pinned.
|
||||
admin_user_ids = { "@alice" = 123456789 }
|
||||
```
|
||||
|
||||
The test-only HTTP adapter has a separate `test_http.enabled` flag that is
|
||||
false by default and omitted from production Compose configuration.
|
||||
|
||||
```toml
|
||||
[archive_control.test_http]
|
||||
enabled = false # Enable only in the isolated local E2E control stack.
|
||||
listen = "127.0.0.1:0" # Non-loopback binds are rejected.
|
||||
max_body_bytes = 1048576
|
||||
```
|
||||
|
||||
## Client configuration examples
|
||||
|
||||
All nodes use the same image. The examples show two active cache and two active
|
||||
archive nodes. Values are illustrative; credentials are mounted secrets.
|
||||
|
||||
```toml
|
||||
# cache-1.toml
|
||||
client_id = "cache-1"
|
||||
display_name = "Cache 1"
|
||||
role = "cache"
|
||||
control_endpoint = "ws://control:8765/archive_control"
|
||||
shared_token_file = "/run/secrets/archive_control_token"
|
||||
state_db = "/var/lib/archive-control/client.db"
|
||||
backup_dir = "/var/backups/archive-control"
|
||||
|
||||
[connection]
|
||||
registration_timeout = "10s" # First response must arrive within this window.
|
||||
heartbeat_interval = "15s" # Server may negotiate a different effective value.
|
||||
offline_timeout = "45s"
|
||||
reconnect_initial = "1s"
|
||||
reconnect_max = "60s" # Retry forever, never wait longer than this.
|
||||
reconnect_reset_after = "60s"
|
||||
reconnect_jitter = true
|
||||
|
||||
[jobs]
|
||||
stall_after = "30m" # Warning only; no automatic job failure.
|
||||
verification_timeout = "30m" # Stopped qB full-recheck deadline.
|
||||
poll_interval = "1s" # Active qB/Syncthing observation interval.
|
||||
# Worst-case copy fallback must leave this many bytes free.
|
||||
free_space_reserve_bytes = 1073741824
|
||||
|
||||
[backup]
|
||||
interval = "6h"
|
||||
recent = 12
|
||||
daily = 14
|
||||
weekly = 8
|
||||
|
||||
[qbittorrent]
|
||||
endpoint = "http://qbittorrent:8080"
|
||||
username = "${QB_USER}"
|
||||
password_file = "/run/secrets/qb_password"
|
||||
api_root = "/downloads"
|
||||
local_root = "/data/qb"
|
||||
|
||||
[syncthing]
|
||||
endpoint = "http://syncthing:8384"
|
||||
api_key_file = "/run/secrets/syncthing_api_key"
|
||||
api_root = "/sync"
|
||||
local_root = "/data/sync"
|
||||
advertised_addresses = ["dynamic"]
|
||||
```
|
||||
|
||||
The remaining node examples omit optional `[connection]`, `[jobs]`, and
|
||||
`[backup]` tables and therefore use these same defaults; deployments may
|
||||
override them per node.
|
||||
|
||||
```toml
|
||||
# cache-2.toml: same service layout, distinct stable identity/state volumes.
|
||||
client_id = "cache-2"
|
||||
display_name = "Cache 2"
|
||||
role = "cache"
|
||||
control_endpoint = "ws://control:8765/archive_control"
|
||||
shared_token_file = "/run/secrets/archive_control_token"
|
||||
state_db = "/var/lib/archive-control/client.db"
|
||||
backup_dir = "/var/backups/archive-control"
|
||||
|
||||
[qbittorrent]
|
||||
endpoint = "http://qbittorrent:8080"
|
||||
username = "${QB_USER}"
|
||||
password_file = "/run/secrets/qb_password"
|
||||
api_root = "/downloads"
|
||||
local_root = "/data/qb"
|
||||
|
||||
[syncthing]
|
||||
endpoint = "http://syncthing:8384"
|
||||
api_key_file = "/run/secrets/syncthing_api_key"
|
||||
api_root = "/sync"
|
||||
local_root = "/data/sync"
|
||||
advertised_addresses = ["dynamic"]
|
||||
```
|
||||
|
||||
```toml
|
||||
# archive-1.toml
|
||||
client_id = "archive-1"
|
||||
display_name = "Archive 1"
|
||||
role = "archive"
|
||||
control_endpoint = "ws://control:8765/archive_control"
|
||||
shared_token_file = "/run/secrets/archive_control_token"
|
||||
state_db = "/var/lib/archive-control/client.db"
|
||||
backup_dir = "/var/backups/archive-control"
|
||||
|
||||
[qbittorrent]
|
||||
endpoint = "http://qbittorrent:8080"
|
||||
username = "${QB_USER}"
|
||||
password_file = "/run/secrets/qb_password"
|
||||
api_root = "/archive"
|
||||
local_root = "/data/archive"
|
||||
|
||||
[syncthing]
|
||||
endpoint = "http://syncthing:8384"
|
||||
api_key_file = "/run/secrets/syncthing_api_key"
|
||||
api_root = "/sync"
|
||||
local_root = "/data/sync"
|
||||
advertised_addresses = ["dynamic"]
|
||||
```
|
||||
|
||||
```toml
|
||||
# archive-2.toml
|
||||
client_id = "archive-2"
|
||||
display_name = "Archive 2"
|
||||
role = "archive"
|
||||
control_endpoint = "ws://control:8765/archive_control"
|
||||
shared_token_file = "/run/secrets/archive_control_token"
|
||||
state_db = "/var/lib/archive-control/client.db"
|
||||
backup_dir = "/var/backups/archive-control"
|
||||
|
||||
[qbittorrent]
|
||||
endpoint = "http://qbittorrent:8080"
|
||||
username = "${QB_USER}"
|
||||
password_file = "/run/secrets/qb_password"
|
||||
api_root = "/archive"
|
||||
local_root = "/data/archive"
|
||||
|
||||
[syncthing]
|
||||
endpoint = "http://syncthing:8384"
|
||||
api_key_file = "/run/secrets/syncthing_api_key"
|
||||
api_root = "/sync"
|
||||
local_root = "/data/sync"
|
||||
advertised_addresses = ["dynamic"]
|
||||
```
|
||||
|
||||
No explicit route matrix is configured. Clients discover existing compatible
|
||||
folders, report their Syncthing device ID, and control creates missing
|
||||
cache/archive routes according to policy.
|
||||
|
||||
## Client Compose example
|
||||
|
||||
```yaml
|
||||
services:
|
||||
archive-client:
|
||||
image: sodium/archive-clients:v0.1.0
|
||||
user: "1001:1001"
|
||||
restart: unless-stopped
|
||||
command: ["archive-client", "--config", "/etc/archive-control/client.toml"]
|
||||
environment:
|
||||
QB_USER: admin
|
||||
volumes:
|
||||
- ./client.toml:/etc/archive-control/client.toml:ro
|
||||
- ./secrets:/run/secrets:ro
|
||||
- ./state:/var/lib/archive-control
|
||||
- ./backups:/var/backups/archive-control
|
||||
- ./qb-data:/data/qb
|
||||
- ./sync-data:/data/sync
|
||||
```
|
||||
|
||||
Archive nodes mount `/data/archive` instead of `/data/qb`. Mounts must line up
|
||||
with the configured local roots and the service containers' API-visible roots.
|
||||
The daemon runs unprivileged; directory ownership is prepared by the operator,
|
||||
and mismatch causes fail-fast startup.
|
||||
|
||||
## Image and release policy
|
||||
|
||||
The production client image contains application source and dependencies; it
|
||||
does not bind-mount source code. A multi-stage Dockerfile uses a slim Python
|
||||
base and removes build-only packages/caches. Buildx publishes
|
||||
`sodium/archive-clients:<semver>` for `linux/amd64` and `linux/arm64`, records
|
||||
the multi-platform digest, and may update `latest`. Deployments pin the
|
||||
immutable semantic tag or digest.
|
||||
|
||||
Generated protobuf Python bindings are committed in each consumer with the
|
||||
exact `archive-control-proto` tag/commit recorded. Release order is proto,
|
||||
control consumer, client consumer, E2E, then image publication.
|
||||
|
||||
## Operator usage
|
||||
|
||||
1. Prepare local qB, sync, state, backup, config, and secret mounts with the
|
||||
container UID/GID (default example `1001:1001`).
|
||||
2. Run a config validation/effective-config command before starting.
|
||||
3. Start clients. Confirm service health, sparse/root probes, and registration.
|
||||
4. Allow route discovery/provisioning to complete when first needed.
|
||||
5. Use Telegram Archive/Unarchive; review the exact source, target, and
|
||||
selection before confirming.
|
||||
6. Use Evict Cache separately only when its fresh coverage check is enabled.
|
||||
7. Treat orange Cleanup Required as a committed transfer needing staging
|
||||
cleanup; do not manually remove the target placement.
|
||||
8. Verify database backups periodically with the offline tooling.
|
||||
|
||||
The relevant runtime must be stopped before restore. The control maintenance
|
||||
entry point and the client's `archive-client-backup` command support `list`,
|
||||
`verify <backup>`, and `restore <backup>`; restore verifies the copy, preserves
|
||||
the replaced database and SQLite sidecars with a timestamped `suspect` name,
|
||||
and installs the selected database atomically.
|
||||
@@ -0,0 +1,276 @@
|
||||
# Actionable implementation plan
|
||||
|
||||
Implementation status (2026-07-23): phases 0–7 are implemented in the proto,
|
||||
control, and client worktrees. Phase 8 has a passing local 2×2 happy path,
|
||||
deterministic selective/recovery matrix, and adversarial v2/fault matrix:
|
||||
complementary partial placements merge correctly, eviction uses archive
|
||||
coverage unions, queued and active cancellation preserve their safety
|
||||
boundaries, restarts replay durable work, and hostile preconditions fail
|
||||
without committing. Pure-v2/hybrid round trips, recoverable stalls,
|
||||
capacity-guard failure, and duplicate/reordered WebSocket events also pass from
|
||||
a clean topology. The remaining fixture and fault categories listed below are
|
||||
still release gates; phase 9 remains intentionally deferred until the complete
|
||||
release matrix passes.
|
||||
|
||||
No application implementation begins until Phase 0 is reviewed. Within every
|
||||
later phase, tests and documentation are part of the deliverable rather than a
|
||||
follow-up task.
|
||||
|
||||
## Phase 0 — Freeze the contract
|
||||
|
||||
### Actions
|
||||
|
||||
1. Convert the decision ledger into the system, workflow, safety, service API,
|
||||
deployment, Telegram, and test documents in this directory.
|
||||
2. Create the independent `archive-control-proto` Git repository.
|
||||
3. Define all v1 resource, selection, route, inventory, job, progress,
|
||||
command/event, reconciliation, and envelope messages.
|
||||
4. Add canonical protobuf-JSON examples for registration, inventory queries,
|
||||
route creation, transfer start/progress/commit, cancellation, eviction, and
|
||||
recovery snapshots.
|
||||
5. Run Dockerized Buf format/lint and protoc descriptor compilation.
|
||||
6. Configure the Gitea remote `cabbage/archive-control-proto` with its dedicated
|
||||
SSH key. Tag the reviewed baseline `v0.1.0` only after review.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- No unresolved `TODO` changes wire meaning or destructive behavior.
|
||||
- Every workflow transition maps to a protobuf command/event.
|
||||
- JSON examples parse through generated descriptors.
|
||||
- Documentation and protobuf enums/state machines agree.
|
||||
|
||||
## Phase 1 — Build the control core inside mogic-bot
|
||||
|
||||
### Actions
|
||||
|
||||
1. Add `archive_control/` parallel to `remote_control/`, with no imports from
|
||||
Telegram in its core service, scheduler, protocol, persistence, or
|
||||
orchestration layers.
|
||||
2. Add strict configuration parsing to the existing bot configuration,
|
||||
including shared token, WebSocket bind/path, database/backup paths,
|
||||
heartbeat, scheduling, route policy, retention, and Telegram refresh knobs.
|
||||
3. Add SQLite schema migrations for clients, connections, routes, jobs,
|
||||
commands, job events, reservations, placements, UI sessions, and
|
||||
tombstones. Enable foreign keys, WAL, busy timeout, and transactional schema
|
||||
migration.
|
||||
4. Implement online backups, integrity verification, retention, and offline
|
||||
verify/restore commands.
|
||||
5. Implement WebSocket registration-first authentication, duplicate-client
|
||||
replacement, heartbeats, bounded send queues, envelope validation, and
|
||||
protocol version negotiation.
|
||||
6. Implement durable command outbox/inbox semantics, acknowledgements,
|
||||
per-job event sequencing, snapshots, and reconnect reconciliation.
|
||||
7. Implement the global resource reservation and FIFO per-client/per-route
|
||||
scheduler.
|
||||
8. Add the thin test-only HTTP adapter over `ArchiveControlService`.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Bot startup/shutdown cleanly owns Archive Control tasks in its existing
|
||||
asyncio lifecycle.
|
||||
- Existing bot tests and workflows are unchanged.
|
||||
- Restart, duplicate registration, lost acknowledgement, event duplication,
|
||||
sequence gap, and DB backup/restore tests pass.
|
||||
- The test HTTP adapter can create/query/cancel synthetic jobs without Telegram.
|
||||
|
||||
## Phase 2 — Build the archive-client foundation
|
||||
|
||||
### Actions
|
||||
|
||||
1. Scaffold `archive-clients` with a `src/` layout, Python 3.11, strict TOML
|
||||
parsing, environment/file secrets, CLI overrides, structured redacted logs,
|
||||
and console entry points.
|
||||
2. Implement `--mode archive|cache`, stable client ID, state/backup DBs, and
|
||||
Docker-friendly shutdown handling.
|
||||
3. Implement capped exponential reconnect with jitter, registration,
|
||||
heartbeat acknowledgements, one-writer WebSocket output, durable command
|
||||
deduplication, and active-job snapshots.
|
||||
4. Implement API-visible-to-local root mapping and path-confinement helpers.
|
||||
5. Probe qBittorrent, Syncthing, filesystem/link/reflink, sparse, and permission
|
||||
capabilities at startup; advertise a precise health/capability report.
|
||||
6. Add multi-stage Dockerfile and Compose example. Run unprivileged and mount
|
||||
only config, secrets, DB/backups, qB data, and sync data.
|
||||
7. Build amd64 first; keep the Dockerfile architecture-neutral for the release
|
||||
buildx phase.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Invalid/unknown configuration fails before connecting.
|
||||
- Secrets never appear in logs or SQLite.
|
||||
- Restart and duplicate-command tests prove idempotent client behavior.
|
||||
- Both daemon roles start from the same image and differ only by config/mode.
|
||||
|
||||
## Phase 3 — Service adapters and on-demand discovery
|
||||
|
||||
### Actions
|
||||
|
||||
1. Implement qBittorrent cookie authentication and capability adapters for
|
||||
supported 4.5–4.6 and 5.x APIs.
|
||||
2. Implement on-demand torrent summaries, scoped hash lookup, lazy complete
|
||||
content trees, v1/v2 identity validation, torrent export, stopped add,
|
||||
selection application, recheck monitoring, and entry-only deletion.
|
||||
3. Implement Syncthing API-key authentication, version/config/status/events,
|
||||
folder/device discovery, per-file progress, need/file completion checks, and
|
||||
rescan support.
|
||||
4. Discover only healthy pairwise `sendreceive` routes under the configured
|
||||
roots.
|
||||
5. Implement idempotent device pairing and folder creation using advertised
|
||||
addresses, plus bidirectional nonce verification and 30-minute default
|
||||
timeout.
|
||||
6. Implement scoped, chunked, revisioned inventory and tree queries; do not add
|
||||
background cluster inventory polling.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Contract fixtures cover API version differences and permanent/transient
|
||||
errors.
|
||||
- A pair of isolated client/Syncthing stacks can auto-create and verify a route.
|
||||
- Large synthetic inventories stream in bounded chunks and interrupted results
|
||||
are not committed by Titan.
|
||||
|
||||
## Phase 4 — Safe file operation and manifest layer
|
||||
|
||||
### Actions
|
||||
|
||||
1. Define the on-disk job staging layout and atomically written protobuf-JSON
|
||||
transfer manifest/ready marker.
|
||||
2. Implement safe traversal using normalized torrent paths, no-follow checks,
|
||||
root confinement, regular-file-only validation, and duplicate/case-collision
|
||||
detection.
|
||||
3. Implement hardlink → reflink → safe-copy fallback with per-file journaling,
|
||||
progress, cancellation points, and restart reconciliation.
|
||||
4. Implement exact logical/allocated size accounting, free-space/reserve checks,
|
||||
and startup sparse capability enforcement.
|
||||
5. Implement target never-overwrite behavior and provenance for created versus
|
||||
pre-existing reused files.
|
||||
6. Implement tested qBittorrent/libtorrent partfile adapters, compatibility
|
||||
gates, baseline backups, and safe merge/rollback behavior.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Tests cover cross-filesystem fallback, insufficient space, permission errors,
|
||||
symlink races, special files, path escape, pre-existing collisions, sparse
|
||||
artifacts, and supported/unsupported partfiles.
|
||||
- Cancelling/restarting every file-operation phase leaves a deterministic
|
||||
journal and no uncontrolled deletion.
|
||||
|
||||
## Phase 5 — Transfer orchestration
|
||||
|
||||
### Actions
|
||||
|
||||
1. Implement fresh source/target tree preflight, requested-selection expansion,
|
||||
immutable file-index snapshots, exact delta calculation, and no-op rejection.
|
||||
2. Implement source staging, per-job Syncthing progress, destination completion
|
||||
proof, and target materialization.
|
||||
3. Implement new-target stopped/manual add and existing-target baseline/delta
|
||||
transaction.
|
||||
4. Apply selected/skipped state only, recheck the union while stopped, monitor
|
||||
for any download attempt, and restore/start the correct run state.
|
||||
5. Commit placement generation only after verification; treat staging removal
|
||||
as post-commit cleanup with retryable `CLEANUP_REQUIRED` state.
|
||||
6. Implement cooperative cancellation/compensation for every pre-commit phase
|
||||
and non-destructive post-commit cleanup behavior.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Full, selective, subset, and incremental merge scenarios pass in both
|
||||
directions.
|
||||
- A target never downloads data during verification.
|
||||
- Failure injection at every transition preserves source and target baselines.
|
||||
- Archive jobs retain their source cache placements.
|
||||
|
||||
## Phase 6 — Eviction and placement reconciliation
|
||||
|
||||
### Actions
|
||||
|
||||
1. Implement on-demand cache placement listing and fresh archive coverage sets.
|
||||
2. Require complete per-file coverage across one or more online archive nodes;
|
||||
expose disabled reasons and provide no force override.
|
||||
3. Implement eviction jobs: coverage recheck, qB entry-only removal, remaining
|
||||
torrent exact-path overlap snapshot, safe file unlink, partfile ownership
|
||||
check, and non-recursive bottom-up directory removal.
|
||||
4. Reconcile observed placements with successful job generations and external
|
||||
qBittorrent changes without background polling.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Eviction cannot proceed with any uncovered selected file.
|
||||
- Shared exact paths, shared directories, unknown files, and unrelated
|
||||
partfiles survive.
|
||||
- Retrying after interruption is idempotent and an archive placement is never
|
||||
deleted.
|
||||
|
||||
## Phase 7 — Telegram integration
|
||||
|
||||
### Actions
|
||||
|
||||
1. Add `ArchiveControlHandler` without changing existing handler behavior.
|
||||
2. Implement Archive, Unarchive, Evict Cache, and Job Status flows with one
|
||||
edited message, pagers, chained literal filters, cancel/back behavior, and
|
||||
single-use confirmation nonces.
|
||||
3. Implement normal-all and one-entry tree selection; keep protobuf support for
|
||||
multiple selections.
|
||||
4. Persist UI sessions and restore live status subscriptions after bot restart.
|
||||
5. Inherit private-chat username authorization and optionally pin configured
|
||||
numeric IDs. Keep jobs shared and ownerless.
|
||||
6. Render five-step transfer and three-step eviction progress, including stale,
|
||||
waiting, stalled percentage, cleanup-required, failure reason, paging,
|
||||
Clear Finished, and Remove One.
|
||||
7. Coalesce routine edits to the configurable five-second default and handle
|
||||
Telegram rate limits.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Telegram and test HTTP adapters produce identical service-layer outcomes.
|
||||
- Stale/double callbacks are harmless.
|
||||
- Bot restart preserves active UI/status behavior.
|
||||
- Existing mogic-bot regression tests pass.
|
||||
|
||||
## Phase 8 — Cross-project E2E and chaos testing
|
||||
|
||||
### Actions
|
||||
|
||||
1. Create five Compose projects under this repository's `e2e/`: control,
|
||||
cache-1, cache-2, archive-1, and archive-2.
|
||||
2. Give every data node isolated qBittorrent, Syncthing, archive-client, config,
|
||||
state, backup, and data volumes on one dedicated test network.
|
||||
3. Configure static advertised Syncthing service addresses while testing
|
||||
automatic device/folder pairing.
|
||||
4. Generate small v1/v2/hybrid torrents and fixtures for complementary and
|
||||
overlapping selections, partfiles, sparse files, path collisions, shared
|
||||
directories, and permission errors.
|
||||
5. Drive scenarios with curl against the test HTTP adapter.
|
||||
6. Inject control/client/service restarts, network loss, duplicate commands,
|
||||
stale snapshots, stalls, disk exhaustion, cancellation, and DB restore.
|
||||
7. Tear down only test resources identified by unique project names/labels.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- The complete acceptance matrix in `testing.md` passes repeatedly without
|
||||
live infrastructure.
|
||||
- No test uses credentials or paths from x1, x2, lithium, or Titan production.
|
||||
- Test teardown is scope-validated and leaves unrelated Docker resources alone.
|
||||
|
||||
## Phase 9 — Release and staged deployment
|
||||
|
||||
### Actions
|
||||
|
||||
1. Run Buf breaking checks against the last proto release and tag compatible
|
||||
repositories.
|
||||
2. Build `sodium/archive-clients` for linux/amd64 and linux/arm64 with OCI
|
||||
source/protocol labels; publish an immutable semantic-version tag and the
|
||||
stable `latest` alias.
|
||||
3. Record image digest and provide pinned Compose examples.
|
||||
4. Back up real qBittorrent/Syncthing and Archive Control state before a
|
||||
separately approved deployment.
|
||||
5. Deploy one cache/archive pair in observation mode, validate on-demand
|
||||
inventory and route discovery, then run a small non-critical canary transfer.
|
||||
6. Exercise recovery and eviction only after canary verification and explicit
|
||||
approval, then expand node-by-node.
|
||||
|
||||
### Acceptance gate
|
||||
|
||||
- Multi-platform manifest resolves to tested images and the runtime image is
|
||||
reasonably small.
|
||||
- Rollback instructions and prior images/config/DB backups are verified.
|
||||
- No production qBittorrent/Syncthing global setting is modified.
|
||||
@@ -0,0 +1,162 @@
|
||||
# Daemon protocol
|
||||
|
||||
The source of truth is `archive_control.v1` in the sibling
|
||||
`archive-control-proto` repository. This document defines behavior that cannot
|
||||
be expressed by field declarations alone. There is no separately maintained
|
||||
JSON schema.
|
||||
|
||||
## Framing and encoding
|
||||
|
||||
- Each WebSocket text frame contains exactly one `Envelope` encoded with the
|
||||
canonical protobuf JSON mapping.
|
||||
- Binary frames, batches, unknown top-level payloads, malformed JSON, duplicate
|
||||
JSON keys, non-finite numbers, invalid timestamps, and over-limit envelopes
|
||||
are protocol errors.
|
||||
- UUID fields use lowercase canonical UUID text. Hash fields use validated
|
||||
lowercase hexadecimal text.
|
||||
- Protobuf `uint64` values use JSON strings according to the protobuf JSON
|
||||
mapping. Enum names are emitted symbolically.
|
||||
- Receivers may ignore unknown JSON fields only after a compatible minor
|
||||
version has been negotiated. An unknown required command or job operation is
|
||||
rejected explicitly.
|
||||
- Implementations apply negotiated envelope and inventory chunk limits before
|
||||
allocation and use bounded outbound queues.
|
||||
|
||||
## Registration and connection ownership
|
||||
|
||||
`RegisterRequest` must be the first application message on every new
|
||||
connection. It carries protocol version, stable client ID/name, role,
|
||||
connection-instance UUID, capabilities, current active-job cursors, and the
|
||||
single shared token.
|
||||
|
||||
The server validates the entire request and constant-time token comparison
|
||||
before accepting other messages. A rejected token or malformed first message
|
||||
causes a rejection when safe to send and then immediate connection closure. The
|
||||
token is forbidden in all later messages and is redacted before any structured
|
||||
logging, error rendering, metrics label, trace, or database write.
|
||||
|
||||
A valid new registration for an already connected client atomically becomes
|
||||
the current connection generation. The old connection is closed, and messages
|
||||
arriving from its stale generation cannot mutate state.
|
||||
|
||||
Clients retry registration indefinitely until accepted or the process exits.
|
||||
The default delay begins at one second, doubles with jitter, and caps at 60
|
||||
seconds. A connection stable for 60 seconds resets the backoff.
|
||||
|
||||
## Version and feature negotiation
|
||||
|
||||
The envelope and registration both state the sender version. Major `1` accepts
|
||||
only major `1`; a different major is rejected. The negotiated minor is the
|
||||
highest mutually supported minor no greater than either endpoint's advertised
|
||||
minor.
|
||||
|
||||
Capabilities, not version guesses, gate optional behavior such as route
|
||||
provisioning, sparse files, partfile migration, or eviction. A required missing
|
||||
feature fails the job preflight with `UNSUPPORTED`. Compatible changes add
|
||||
fields or enum values; released field numbers and enum values are never reused,
|
||||
and removals reserve them. Buf breaking checks enforce file-level compatibility.
|
||||
|
||||
## Heartbeats and liveness
|
||||
|
||||
Default settings are:
|
||||
|
||||
| Setting | Default |
|
||||
| --- | ---: |
|
||||
| Registration deadline | 10 s |
|
||||
| Heartbeat interval | 15 s |
|
||||
| Offline threshold | 45 s |
|
||||
| Unacknowledged command retry | 15 s |
|
||||
| Maximum command send attempts | 3 total |
|
||||
| Reconnect initial/cap | 1 s / 60 s |
|
||||
| Stable reset window | 60 s |
|
||||
|
||||
The server communicates the effective heartbeat values in `RegisterResponse`.
|
||||
Heartbeat sequence acknowledgements prove application-level responsiveness.
|
||||
Missing the offline threshold marks the client offline, releases connection
|
||||
ownership, and moves affected jobs to waiting; it does not fail them.
|
||||
|
||||
## Reliable commands
|
||||
|
||||
Control generates a UUID `command_id`, persists the command and intended
|
||||
client, then sends it at least once. A client transactionally inserts an unseen
|
||||
ID and its immutable payload before replying `ACCEPTED`. A repeat with the same
|
||||
payload returns the stored acknowledgement as `DUPLICATE`; the same ID with a
|
||||
different payload is a protocol conflict.
|
||||
|
||||
`CommandAck` confirms durable acceptance or rejection only. It never means the
|
||||
step completed. Completion is represented by ordered `JobEvent`,
|
||||
`InventoryChunk`, `RouteUpdate`, or `JobSnapshot` messages. Control retries an
|
||||
unacknowledged command with its original ID after reconnect or timeout, up to
|
||||
the configured total attempt limit. A late acknowledgement remains valid after
|
||||
automatic delivery stops.
|
||||
|
||||
Route updates use a stable update UUID and a sequence monotonic within their
|
||||
route-creation command. Conflicting duplicates or gaps are reconciled with the
|
||||
client state snapshot before a route can become ready.
|
||||
|
||||
Every mutating command includes an expected job revision or immutable job
|
||||
definition plus the expected last global event sequence. A stale revision or
|
||||
sequence is rejected without side effects. The control daemon sends only one
|
||||
active step command for a job and waits for its persisted completion event
|
||||
before commanding the next participant. This single-writer lease lets whichever
|
||||
client owns the current step allocate the next global per-job event sequence;
|
||||
the following command starts from the sequence control has durably accepted.
|
||||
|
||||
## Events, progress, and reconciliation
|
||||
|
||||
Each client-originated job event has a unique ID, monotonically increasing
|
||||
global per-job `sequence`, and resulting job revision. Duplicate IDs/sequences
|
||||
are idempotent only when their complete content matches. Control never grants
|
||||
concurrent event-writer leases for one job. A gap or conflicting duplicate
|
||||
pauses destructive orchestration and requests snapshots.
|
||||
|
||||
`fraction_complete` is current-step progress and
|
||||
`overall_fraction_complete` is the weighted five- or three-step job progress;
|
||||
both are clamped to `[0,1]`. Byte counts are authoritative when known.
|
||||
`STALLED` retains both fractions, byte counters, and current step. Progress
|
||||
timestamp movement alone does not prove content movement.
|
||||
|
||||
On registration, active-job cursors provide the client's revision, last event
|
||||
sequence, state, and commit flag. Reconciliation applies these rules:
|
||||
|
||||
1. Equal cursors resume normal delivery.
|
||||
2. A client behind receives safe replay/snapshot commands.
|
||||
3. Control behind requests and validates the client's full job snapshot.
|
||||
4. Conflicting commit evidence reserves the resource and requires manual
|
||||
reconciliation; neither side performs cleanup.
|
||||
5. In-flight command IDs are compared before any resend.
|
||||
|
||||
## Inventory protocol
|
||||
|
||||
Inventory is requested only for a user flow, job preflight, active-job
|
||||
monitoring, route discovery, or reconciliation. Registration does not upload a
|
||||
full torrent inventory.
|
||||
|
||||
Queries declare a scope, resource IDs/filter, page bounds, and optional expected
|
||||
revision. Responses form an atomic snapshot stream identified by query ID,
|
||||
snapshot ID, revision, monotonically increasing chunk index, and final marker.
|
||||
Control commits the result only after every chunk through `last_chunk` is
|
||||
present. A disconnect or error discards the partial snapshot. Full content
|
||||
trees are fetched for one chosen resource and from both relevant endpoints
|
||||
before staging. File entries carry index, logical/allocated/completed bytes,
|
||||
selection, sparse/padding flags, and the resource content revision required to
|
||||
validate the preview.
|
||||
|
||||
## Stable errors
|
||||
|
||||
Machine behavior uses `ErrorCode`, `retryable`, and structured details; the
|
||||
human message explains the concrete condition without secrets. Transient
|
||||
unavailability/timeouts may wait or retry. Invalid input, stale fingerprints,
|
||||
path conflicts, permissions, unsupported formats, integrity failures, and
|
||||
unsafe state are permanent for the current attempt. Unknown internal errors
|
||||
must not be mislabeled retryable.
|
||||
|
||||
## Manifest encoding
|
||||
|
||||
`TransferManifest` and `ReadyMarker` use the same canonical protobuf JSON
|
||||
mapping, written atomically. The manifest describes immutable source/target
|
||||
identity, route, requested set, target baseline, transfer delta, file paths and
|
||||
sizes, sparse flags, materialization provenance, fingerprints, and auxiliary
|
||||
artifacts. The ready marker is written last and includes a SHA-256 digest of the
|
||||
exact manifest bytes. A receiver ignores a job directory until both exist and
|
||||
the digest matches.
|
||||
@@ -0,0 +1,142 @@
|
||||
# qBittorrent and Syncthing integration
|
||||
|
||||
Adapters are version-aware boundaries. Core orchestration consumes normalized
|
||||
resource, route, and progress models and never branches on raw service response
|
||||
shapes.
|
||||
|
||||
## qBittorrent Web API
|
||||
|
||||
The implementation targets supported qBittorrent 4.5–4.6 and 5.x releases and
|
||||
detects the application and Web API versions at startup. The authoritative
|
||||
references are the official [5.0 WebUI API](https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-%28qBittorrent-5.0%29)
|
||||
and [4.1-compatible WebUI API](https://github.com/qbittorrent/qBittorrent/wiki/WebUI-API-%28qBittorrent-4.1%29).
|
||||
|
||||
### Adapter capabilities
|
||||
|
||||
| Need | Web API family | Archive Control rule |
|
||||
| --- | --- | --- |
|
||||
| Authenticate | `auth/login`, cookie session | Log no credentials/cookies; reauthenticate once on expiry |
|
||||
| Detect compatibility | `app/version`, `app/webapiVersion`, build info | Advertise exact versions and select adapter |
|
||||
| List/lookup | `torrents/info`, properties | On-demand and hash-scoped where possible |
|
||||
| Read file state | `torrents/files` | Normalize indices, paths, selected/skipped, size, progress |
|
||||
| Export metainfo | `torrents/export` | Required before staging; fail if exact metainfo unavailable |
|
||||
| Add target | `torrents/add` | Explicit save path, stopped/paused, manual management |
|
||||
| Set selection | `torrents/filePrio` / selection endpoint | Encode only selected vs skipped, never preserve priority levels |
|
||||
| Recheck | `torrents/recheck` plus state polling | Keep stopped; any download activity fails the job |
|
||||
| Remove entry | `torrents/delete` | Eviction uses `deleteFiles=false` only |
|
||||
|
||||
The adapter accounts for terminology/behavior changes such as paused versus
|
||||
stopped states. It does not mutate qBittorrent preferences, categories, tags,
|
||||
limits, queueing defaults, or global save-path behavior.
|
||||
|
||||
### Inventory normalization
|
||||
|
||||
For each torrent the client derives canonical v1/v2 identity from reliable API
|
||||
and exported-metainfo data, keeps the qB-local hash separately, and calculates:
|
||||
|
||||
- total and selected-complete logical bytes;
|
||||
- normalized selected and selected-complete file-index sets;
|
||||
- torrent runtime state;
|
||||
- canonical path flag and content revision;
|
||||
- a save-path fingerprint based on validated path mapping, never a leaked
|
||||
absolute host path.
|
||||
|
||||
qBittorrent 5 may expose a pure-v2 or hybrid torrent under the first 20 bytes
|
||||
of its v2 hash while separately advertising full `infohash_v1` and
|
||||
`infohash_v2` values. Lookup matches all advertised identities, rejects
|
||||
ambiguous matches, validates the canonical hashes from exported metainfo, and
|
||||
uses only qBittorrent's returned local hash for later mutation calls.
|
||||
|
||||
Complete content trees are built only for a chosen resource or job preflight.
|
||||
All entries retain qBittorrent's stable torrent file index. Renamed/noncanonical
|
||||
content paths are rejected in v1 because they cannot be transported and merged
|
||||
without ambiguity.
|
||||
|
||||
### Verification guard
|
||||
|
||||
The client captures transfer counters and state before recheck, issues recheck,
|
||||
and watches until success, timeout/stall, or failure. Success requires every
|
||||
selected file complete and no missing/error pieces. Any downloading state or
|
||||
increase in received payload bytes causes immediate stop and a failed job with
|
||||
the affected path/piece detail available from the adapter when possible.
|
||||
|
||||
## Syncthing REST and events APIs
|
||||
|
||||
Clients authenticate using the configured API key header and discover exact
|
||||
service version/capabilities. Primary references are Syncthing's official
|
||||
[REST conventions](https://docs.syncthing.net/dev/rest.html),
|
||||
[configuration model](https://docs.syncthing.net/users/config.html), and
|
||||
[DownloadProgress event](https://docs.syncthing.net/events/downloadprogress.html).
|
||||
|
||||
### Discovery and route management
|
||||
|
||||
The adapter reads system status/version and configuration, then selects folders
|
||||
that:
|
||||
|
||||
- are `sendreceive`;
|
||||
- map beneath the configured API-visible and local sync roots;
|
||||
- contain exactly the local device and one peer device;
|
||||
- pair one registered cache with one registered archive client;
|
||||
- have a safe relative filesystem path.
|
||||
|
||||
The folder's native ID becomes `route_id`. Multi-device, send-only,
|
||||
receive-only, out-of-root, or ambiguous folders are reported as unsupported and
|
||||
left unchanged.
|
||||
|
||||
Provisioning uses idempotent device and folder configuration updates. Each
|
||||
client receives its peer device ID and optional advertised addresses
|
||||
(`dynamic` by default). It changes only the specific device/folder objects
|
||||
owned by the provisioning command. Read-back verification must agree before
|
||||
nonce exchange begins.
|
||||
|
||||
### Transfer progress and completion
|
||||
|
||||
`DownloadProgress` events are filtered to the job's exact staging prefix and
|
||||
used for live block-level byte estimates. Their percentage is advisory. The
|
||||
adapter falls back to coarser polling if events are unavailable.
|
||||
|
||||
Final completion combines:
|
||||
|
||||
- job-prefix files present and matching manifest logical sizes;
|
||||
- Syncthing folder/local-global completion from the official
|
||||
[`db/completion`](https://docs.syncthing.net/rest/db-completion-get.html)
|
||||
response;
|
||||
- per-file state from [`db/file`](https://docs.syncthing.net/v2.0.0/rest/db-file-get.html)
|
||||
where supported;
|
||||
- no relevant outstanding entries from
|
||||
[`db/need`](https://docs.syncthing.net/v1.19.2/rest/db-need-get.html);
|
||||
- valid manifest digest and ready marker.
|
||||
|
||||
The client can request a scoped rescan after publishing/cleanup. It does not
|
||||
declare transfer success from a `100%` progress event alone.
|
||||
|
||||
## Error normalization
|
||||
|
||||
Each adapter maps transport errors, authentication failure, incompatible API,
|
||||
missing torrent/folder, stale external state, permission failures, and service
|
||||
health into stable protocol error codes. Responses and exception text are
|
||||
redacted before logging because service products may echo URLs, headers, or
|
||||
filesystem paths. Authentication failures are permanent until configuration
|
||||
changes; transient service absence moves an active job to waiting.
|
||||
|
||||
## Bot-free control adapter
|
||||
|
||||
The loopback-only test adapter calls the same `ArchiveControlService` used by
|
||||
Telegram:
|
||||
|
||||
- `GET /test/v1/resources?operation=archive|unarchive|evict_cache` performs a
|
||||
complete, paged, on-demand summary scan and reconciles placements.
|
||||
- `GET /test/v1/resources/{info_hash}/tree?client_id={client}` fetches one
|
||||
complete revisioned content tree.
|
||||
- `POST /test/v1/jobs/preview` fetches fresh source and target summaries and
|
||||
full trees, expands optional `selected_paths` or
|
||||
`selected_file_indices`, and returns an immutable definition plus its
|
||||
`preview_revision`.
|
||||
- `POST /test/v1/jobs` consumes `{preview_revision, definition}` and rejects a
|
||||
deterministic protobuf SHA-256 mismatch.
|
||||
- Job list/detail/cancel/hide/clear endpoints expose the shared ownerless job
|
||||
state, while `GET /test/v1/placements` exposes reconciled generations.
|
||||
|
||||
Lower-level inventory-query, route, scheduler, and backup endpoints remain for
|
||||
contract and failure-injection tests. This adapter must never bind outside
|
||||
loopback.
|
||||
@@ -0,0 +1,153 @@
|
||||
# Storage safety, rollback, and recovery
|
||||
|
||||
## Path trust boundary
|
||||
|
||||
qBittorrent metadata, Syncthing configuration, transfer manifests, and
|
||||
filesystem directory entries are all untrusted inputs. Before any access, the
|
||||
client:
|
||||
|
||||
1. validates the canonical torrent-relative path;
|
||||
2. rejects absolute paths, empty/`.`/`..` components, NULs, platform separators,
|
||||
duplicate normalized names, and configured case-fold collisions;
|
||||
3. resolves the API-visible path through one validated root mapping;
|
||||
4. traverses without following symlinks and checks the resulting descriptor is
|
||||
a regular file or an explicitly created directory;
|
||||
5. verifies every operation remains beneath the local configured root.
|
||||
|
||||
Sockets, devices, FIFOs, symlinks, and other special entries fail preflight.
|
||||
Permission or ownership mismatch is fail-fast. Archive Control never changes
|
||||
source ownership or mode to make a job pass.
|
||||
|
||||
## Isolated staging
|
||||
|
||||
Every route stores transient jobs under:
|
||||
|
||||
```text
|
||||
<sync-root>/.archive-control/jobs/<job-id>/
|
||||
├── payload/<original torrent hierarchy>
|
||||
├── metainfo/source.torrent
|
||||
├── artifacts/...
|
||||
├── manifest.json
|
||||
└── ready.json
|
||||
```
|
||||
|
||||
Temporary writes use an attempt-specific name, fsync where durability matters,
|
||||
and atomic rename. The manifest records every expected file and method. The
|
||||
ready marker is the final publish action. Job IDs and known subpaths are used
|
||||
directly; cleanup never expands a glob or recursively targets the sync root.
|
||||
|
||||
## Materialization fallback
|
||||
|
||||
For each file, source staging and target materialization try:
|
||||
|
||||
1. hard link when the same filesystem and permissions allow it;
|
||||
2. reflink/clone when supported;
|
||||
3. bounded buffered copy into a temporary file followed by atomic placement.
|
||||
|
||||
The journal records intent before mutation and completion afterward, including
|
||||
source identity, destination, method, logical/allocated sizes, and whether the
|
||||
target pre-existed. Restarts reconcile the journal against file descriptors and
|
||||
the manifest instead of blindly repeating operations.
|
||||
|
||||
Free-space checks account for methods that allocate data plus a configurable
|
||||
reserve. A mid-copy space failure leaves only an attributable temporary file.
|
||||
No operation overwrites an existing target. A same-type, same-size regular file
|
||||
may be provisionally reused, but qBittorrent's full recheck is the integrity
|
||||
authority; any other collision fails before mutation.
|
||||
|
||||
## Sparse files
|
||||
|
||||
At startup, each client probes semantic sparse-file behavior once for every
|
||||
configured qB and Syncthing local root. Capability is advertised with the
|
||||
connection. A job with any sparse source file requires sparse support for every
|
||||
staging and destination root involved. If a target leg lacks it, the job fails
|
||||
clearly before transfer.
|
||||
|
||||
Sparse files are retained as sparse, without attempting to reproduce an exact
|
||||
physical hole map. Hardlink/reflink naturally preserve semantics; the copy
|
||||
implementation uses a sparse-aware strategy and verifies logical length and
|
||||
allocation behavior. Exact allocated-block counts need not match between
|
||||
filesystems.
|
||||
|
||||
## qBittorrent transaction boundary
|
||||
|
||||
For a new target, the client exports source metainfo, places candidate data,
|
||||
adds the torrent stopped with manual management and explicit save path, applies
|
||||
the exact union selection, and starts a full recheck. It monitors both state and
|
||||
transfer counters. Any download transition or received payload byte is a hard
|
||||
failure and the torrent is stopped immediately.
|
||||
|
||||
For an existing target, the durable baseline includes its selected indices,
|
||||
run state, local identifier, save-path fingerprint, content revision, and
|
||||
partfile state. The job adds only the delta. Before commit, rollback restores
|
||||
that baseline and removes only current-job-created, unshared files. Source
|
||||
torrent state is never changed by transfer logic.
|
||||
|
||||
The target placement commits only after the selected union verifies completely
|
||||
while stopped. The placement generation and commit event are persisted in the
|
||||
same control transaction. Once committed, target content is outside rollback
|
||||
scope.
|
||||
|
||||
## Partfiles
|
||||
|
||||
Libtorrent partfiles may contain data for skipped or non-materialized torrent
|
||||
files and their layout varies by version. Archive Control therefore uses a
|
||||
version-aware adapter keyed by observed qBittorrent/libtorrent version and a
|
||||
tested format signature. The adapter must support export/import and, for an
|
||||
existing target, merge with a recoverable baseline.
|
||||
|
||||
Unknown, ambiguous, or incompatible partfile state fails the job at live
|
||||
preflight with `UNSUPPORTED` or `PRECONDITION_FAILED` and a precise reason. It
|
||||
is never copied over an existing partfile blindly. Rollback restores the
|
||||
baseline. Eviction removes a partfile only when its ownership by the removed
|
||||
torrent is unambiguous.
|
||||
|
||||
## Safe eviction deletion
|
||||
|
||||
Before unlinking, the client persists:
|
||||
|
||||
- the confirmed cache placement generation and file set;
|
||||
- fresh archive coverage proofs;
|
||||
- the exact torrent manifest and local paths;
|
||||
- qBittorrent entry-removal result;
|
||||
- exact paths referenced by all remaining torrents;
|
||||
- per-file unlink and per-directory `rmdir` results.
|
||||
|
||||
The qBittorrent entry is removed with `deleteFiles=false`. For each owned file,
|
||||
the client refuses removal if another remaining torrent references the exact
|
||||
path. Eligible known files are unlinked individually. It then attempts
|
||||
non-recursive `rmdir` deepest-first for known owned directories. `ENOTEMPTY`
|
||||
means retain and continue; unknown files or directories are never traversed or
|
||||
deleted. There is no recursive delete fallback.
|
||||
|
||||
## Compensation by phase
|
||||
|
||||
| Point | Safe outcome |
|
||||
| --- | --- |
|
||||
| Queued, no command | Delete queue row/reservation only |
|
||||
| Source staging | Stop; unlink only journaled staging artifacts |
|
||||
| Syncthing transfer | Stop publishing; clean only isolated job namespace |
|
||||
| Target materialization | Restore target baseline; remove attributable delta |
|
||||
| qB recheck before commit | Stop/remove newly added entry without data deletion, then compensate delta |
|
||||
| After commit | Keep placement; retry staging cleanup only |
|
||||
| Eviction after qB removal | Resume safe per-file unlink; never re-add or delete archives automatically |
|
||||
|
||||
If attribution is uncertain, retain data and require manual intervention.
|
||||
|
||||
## SQLite durability and backup
|
||||
|
||||
Both daemons enable foreign keys, WAL, a bounded busy timeout, and explicit
|
||||
transactions. Schema migrations create an online backup before and after the
|
||||
migration and fail atomically.
|
||||
|
||||
The default recurring policy runs SQLite's online backup API every six hours,
|
||||
verifies checksum and SQLite integrity, writes via temporary file plus atomic
|
||||
rename, and retains 12 recent, 14 daily, and eight weekly copies. Backups live
|
||||
outside qBittorrent and Syncthing roots. Configuration warns if the backup and
|
||||
database share a filesystem, but may permit it for small installations.
|
||||
|
||||
Offline tooling provides list, verify, and restore. Restore requires stopped
|
||||
daemon access, verifies the chosen backup, preserves the suspect database under
|
||||
a timestamped name, installs the replacement atomically, and runs integrity and
|
||||
schema checks before normal startup. Backups contain no configured secrets.
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
# System design
|
||||
|
||||
## Scope and invariants
|
||||
|
||||
Archive Control moves already-complete BitTorrent file selections between hot
|
||||
cache nodes and durable archive nodes. It does not download missing content,
|
||||
change global qBittorrent or Syncthing settings, or decide retention on archive
|
||||
nodes.
|
||||
|
||||
The non-negotiable invariants are:
|
||||
|
||||
1. A committed archive copy is never deleted by a transfer rollback or cleanup.
|
||||
2. A cache placement is evicted only by an explicit eviction job after fresh
|
||||
proof that every selected file is covered by online archive placements.
|
||||
3. A target is committed only after qBittorrent verifies the selected union in
|
||||
stopped state without attempting a download.
|
||||
4. One control daemon is the only cross-node orchestrator. Clients execute one
|
||||
commanded local step at a time.
|
||||
5. Every destructive transition is durable, attributable, retryable, and
|
||||
guarded by an expected revision or external-state fingerprint.
|
||||
6. File operations stay beneath configured roots, never follow symlinks, and
|
||||
never recursively delete a content tree.
|
||||
|
||||
## Components
|
||||
|
||||
### Archive control core
|
||||
|
||||
The existing mogic-bot process hosts a new `archive_control/` package parallel
|
||||
to `remote_control/`. Its core has no Telegram imports and exposes an
|
||||
`ArchiveControlService` facade used by:
|
||||
|
||||
- the production Telegram handler;
|
||||
- a test-only HTTP adapter used by local E2E tests;
|
||||
- the WebSocket connection manager for clients.
|
||||
|
||||
Internally the core contains configuration, SQLite persistence, connection
|
||||
registry, durable command outbox, protocol codec, scheduler, route coordinator,
|
||||
inventory coordinator, job state machines, backup manager, and event/view
|
||||
projection. The existing bot lifecycle starts and stops these tasks; no second
|
||||
bot or independent control process is introduced.
|
||||
|
||||
### Archive client
|
||||
|
||||
One Python image and codebase runs with role `archive` or `cache`. Each instance
|
||||
contains:
|
||||
|
||||
- a reconnecting control connection and durable command inbox;
|
||||
- a local SQLite journal and backup manager;
|
||||
- qBittorrent and Syncthing adapters;
|
||||
- safe path and file-operation modules;
|
||||
- route discovery/provisioning;
|
||||
- inventory and content-tree responders;
|
||||
- transfer and eviction step executors.
|
||||
|
||||
The role controls policy, not the available transport mechanics. Cache clients
|
||||
may source archive jobs, receive unarchive jobs, and perform eviction. Archive
|
||||
clients may receive archive jobs and source unarchive jobs.
|
||||
|
||||
### External local services
|
||||
|
||||
Clients talk to their locally configured qBittorrent and Syncthing APIs. API
|
||||
paths may differ from paths inside the client container, so each root is
|
||||
configured as an API-visible path plus its daemon-local equivalent. The
|
||||
mapping must be one-to-one and is validated at startup.
|
||||
|
||||
## Logical data flow
|
||||
|
||||
```text
|
||||
Telegram handler ─┐
|
||||
├─ ArchiveControlService ─ SQLite/control scheduler
|
||||
test HTTP adapter ┘ │
|
||||
│ versioned protobuf JSON over WebSocket
|
||||
┌────────────────┴─────────────────┐
|
||||
cache client archive client
|
||||
├─ qB adapter ├─ qB adapter
|
||||
├─ Syncthing adapter ├─ Syncthing adapter
|
||||
├─ SQLite journal ├─ SQLite journal
|
||||
└─ local file ops └─ local file ops
|
||||
└──────── pairwise Syncthing route ────────┘
|
||||
```
|
||||
|
||||
The control service persists intent and the job's current event-sequence lease
|
||||
before emitting a command. The client
|
||||
persists command acceptance before acknowledging it. Completion is a later
|
||||
event. This separates delivery from execution and makes reconnect replay safe.
|
||||
|
||||
## Persistent model
|
||||
|
||||
### Control database
|
||||
|
||||
The control database contains at least:
|
||||
|
||||
- `clients` and last known capabilities/health;
|
||||
- connection generations and heartbeat timestamps;
|
||||
- discovered/provisioned routes and verification state;
|
||||
- immutable job definitions and mutable job projections;
|
||||
- per-job ordered events;
|
||||
- durable commands, delivery attempts, and acknowledgements;
|
||||
- resource reservations;
|
||||
- cache/archive placements and generations;
|
||||
- bounded inventory snapshots and revisions;
|
||||
- persisted Telegram sessions and confirmation nonces;
|
||||
- hidden-record tombstones and backup metadata.
|
||||
|
||||
### Client database
|
||||
|
||||
Each client stores:
|
||||
|
||||
- accepted command IDs and their stable acknowledgements;
|
||||
- job definitions, local step journals, revisions, and emitted events;
|
||||
- file-operation provenance and compensation state;
|
||||
- staging manifests and ready-marker observations;
|
||||
- route-creation ownership and verification state;
|
||||
- last control connection/session cursors;
|
||||
- backup metadata.
|
||||
|
||||
Credentials, shared tokens, qBittorrent passwords, Syncthing API keys, and
|
||||
Telegram secrets are never stored in either database.
|
||||
|
||||
### Placements are not job history
|
||||
|
||||
A placement is current, verified residency of a resource selection on one
|
||||
node. It is keyed by canonical resource identity and client ID and includes a
|
||||
generation plus verified file-index set. Jobs mutate placements, but deleting
|
||||
or hiding a job record never changes placement data. A resource may have many
|
||||
archive and cache placements.
|
||||
|
||||
## Scheduling and ownership
|
||||
|
||||
Jobs enter a durable FIFO queue while holding a global reservation on every
|
||||
known v1/v2 alias of their resource identity. Only one transfer or eviction for
|
||||
a resource may be active or queued at a time. This prevents incompatible
|
||||
baselines even when jobs would use different nodes or observe different sides
|
||||
of a hybrid identity.
|
||||
|
||||
Defaults allow one active data-moving job per client and one per route. A job
|
||||
must acquire its source client, target client, route, and resource reservation
|
||||
atomically. Disjoint node pairs may run concurrently. Route setup is a
|
||||
preflight activity and does not permit a data step to bypass these leases.
|
||||
|
||||
Offline nodes do not prevent unrelated jobs from being listed or run. A job
|
||||
requiring an offline node remains waiting indefinitely; it does not consume an
|
||||
active data slot until work can proceed.
|
||||
|
||||
## State and recovery
|
||||
|
||||
Control state is authoritative for orchestration; neither client independently
|
||||
advances to the next step. On reconnect:
|
||||
|
||||
1. The client registers with active-job cursors.
|
||||
2. Control compares revisions and per-job event sequences.
|
||||
3. Missing events are replayed when available; gaps or disagreement trigger a
|
||||
full client/job snapshot.
|
||||
4. Control observes relevant qBittorrent, Syncthing, staging, and manifest
|
||||
state before selecting retry, resume, compensation, cleanup, or manual
|
||||
intervention.
|
||||
5. A command is reissued with its original ID when the acceptance result is
|
||||
uncertain.
|
||||
|
||||
A client never guesses that a job committed. If a database is lost, evidence
|
||||
from a verified target, immutable manifest, ready marker, and file-operation
|
||||
journal may reconstruct state only when it proves one unique outcome. Otherwise
|
||||
the resource remains reserved and the job fails closed for manual
|
||||
reconciliation.
|
||||
|
||||
Routine no-progress and connectivity conditions become `WAITING` or `STALLED`,
|
||||
not failures. Permanent protocol, validation, integrity, unsafe-path,
|
||||
unsupported-partfile, or permission conditions fail with a stable code and a
|
||||
clear operator-facing reason.
|
||||
@@ -0,0 +1,177 @@
|
||||
# Telegram UX
|
||||
|
||||
Archive Control adds a separate handler/module to the existing bot. Existing
|
||||
commands and conversations retain their current behavior.
|
||||
|
||||
## Authorization and shared state
|
||||
|
||||
The handler accepts private-chat users through the existing `@username`
|
||||
allowlist for compatibility. A configured numeric Telegram user ID may pin an
|
||||
entry; when pinned, both the ID and current allowlist rule must match. This
|
||||
mitigates username reassignment while permitting a staged migration.
|
||||
|
||||
Jobs, placements, events, queues, and routes are shared administrative state and
|
||||
store no initiating-user identity. UI session state records only what is needed
|
||||
to resume the conversation/view and expires after 24 hours by default.
|
||||
|
||||
## Entry point and navigation
|
||||
|
||||
The existing “Archive/Unarchive Resources” entry opens one message:
|
||||
|
||||
```text
|
||||
What do you want to do?
|
||||
|
||||
[ Archive ] [ Unarchive ]
|
||||
[ Evict Cache ] [ Job Status ]
|
||||
```
|
||||
|
||||
All subsequent pages edit this message. `Cancel` closes the active selection
|
||||
flow. `Back` returns one level while retaining validated filters/selections.
|
||||
Callback payloads contain opaque session/action IDs, not resource names or
|
||||
paths, and are validated against persisted session revision and expiry.
|
||||
|
||||
## Archive and unarchive flow
|
||||
|
||||
### 1. Select source resource
|
||||
|
||||
Control queries relevant online nodes on demand. It groups identical resource
|
||||
and selection replicas into one display choice while showing distinct
|
||||
selections separately. The list is stable-sorted and paged:
|
||||
|
||||
```text
|
||||
Which cached resource do you want to archive? (page 1/3)
|
||||
filters: `documentary`, `2025`
|
||||
|
||||
1. `10.0 GB` Resource A — cache-1
|
||||
2. `840 MB` Resource B — cache-2
|
||||
|
||||
[ 1 ] [ 2 ]
|
||||
[ Prev ] [ Next ]
|
||||
[ Remove filters ] [ Cancel ]
|
||||
```
|
||||
|
||||
Size is selected-complete logical content, not total torrent size. New messages
|
||||
add literal, case-insensitive filters and narrow the current results; Remove
|
||||
filters resets the chain. If no item remains, the message explains whether no
|
||||
source is online, no selected-complete data exists, or filters eliminated all
|
||||
results.
|
||||
|
||||
### 2. Select source replica and target
|
||||
|
||||
If multiple identical-selection replicas exist, control may choose a healthy
|
||||
source automatically using deterministic availability/route/free-space rules;
|
||||
otherwise it lists the source replicas. One eligible target is selected
|
||||
automatically; multiple targets are paged. Offline targets are not offered.
|
||||
|
||||
A resource already on a target is still eligible when the requested selection
|
||||
has an unverified delta. The UI says that it will extend an existing placement.
|
||||
|
||||
### 3. Confirm or choose one entry
|
||||
|
||||
Before showing confirmation, control obtains fresh full trees from source and
|
||||
target and calculates the transfer delta.
|
||||
|
||||
```text
|
||||
Archive Resource A from cache-1 to archive-2?
|
||||
Requested: 7 files, 10.0 GB
|
||||
Already verified there: 2 files, 3.1 GB
|
||||
Will transfer: 5 files, 6.9 GB
|
||||
|
||||
[ Confirm ]
|
||||
[ Choose one entry… ] [ Cancel ]
|
||||
```
|
||||
|
||||
Choose one entry opens a paged/filterable tree. Only source-selected-complete
|
||||
files/directories are enabled. Selecting a directory expands it to descendant
|
||||
file indices, returns to a freshly recomputed confirmation, and clearly states
|
||||
the chosen subtree. The protocol supports multiple entries from v1, but the
|
||||
initial UI exposes one.
|
||||
|
||||
Confirmation uses a persisted, single-use nonce bound to session revision,
|
||||
operation, resource, source, target, selection, fingerprints, and placement
|
||||
generation. Double taps or stale callbacks cannot create a second job.
|
||||
|
||||
## Evict Cache flow
|
||||
|
||||
The first page lists current cache placements. Selecting one triggers fresh
|
||||
queries to the cache and relevant online archive nodes. Each row/confirmation
|
||||
shows coverage:
|
||||
|
||||
- enabled when the union of verified archives covers every selected cache file;
|
||||
- disabled with missing file count/size and unavailable archive explanation
|
||||
otherwise.
|
||||
|
||||
There is no override button. Confirmation creates a durable three-step
|
||||
eviction job; it never clears an archive placement or a transfer history row.
|
||||
|
||||
## Job Status
|
||||
|
||||
Status pages show live and retained jobs with operation, resource, endpoints,
|
||||
state, current-step and overall percentages, bytes/s where meaningful, and
|
||||
failure reason.
|
||||
Routine edits are coalesced to one per five seconds by default. Step changes,
|
||||
stall transitions, failures, commit, cleanup-required, and terminal states edit
|
||||
immediately subject to Telegram rate-limit backoff.
|
||||
|
||||
State symbols:
|
||||
|
||||
| Symbol | Meaning |
|
||||
| --- | --- |
|
||||
| 🟡 | queued/not started |
|
||||
| 🔵 | active, waiting, or stalled |
|
||||
| 🟠 | committed but cleanup required |
|
||||
| 🟢 | succeeded |
|
||||
| 🔴 | failed or cancelled |
|
||||
|
||||
Transfer jobs display five numbered steps: Source Stage/HardLink, Syncthing,
|
||||
Target Merge/Copy, QB Verification, and Staging Cleanup. Route setup and
|
||||
preflight are named, unnumbered pre-steps. Eviction displays Verify Archive
|
||||
Coverage, QB Remove Entry, and Safe File Unlink.
|
||||
|
||||
A stalled job keeps its last overall percentage:
|
||||
|
||||
```text
|
||||
🔵 Archive: Resource A — cache-1 → archive-2
|
||||
[2/5 Syncthing: 63.42%, overall 32.68%, stalled, last progress 38m ago]
|
||||
Waiting indefinitely; retry occurs when progress/service connectivity returns.
|
||||
```
|
||||
|
||||
Failure messages include stable phase and clear reason, while hiding secrets
|
||||
and host-only paths.
|
||||
|
||||
## Clear and remove behavior
|
||||
|
||||
`Clear Finished` hides only green successful jobs and leaves a 30-day default
|
||||
tombstone. Orange Cleanup Required is not clearable as finished.
|
||||
|
||||
`Remove One` first replaces buttons with current-page job numbers and Cancel.
|
||||
A successful green row hides immediately. A queued job confirmation deletes
|
||||
only its queue record/reservation. An active, failed-with-residue, or cancelling
|
||||
job requires a second confirmation and begins/continues compensation; its row
|
||||
remains until safe cleanup is known. A committed job can only retry staging
|
||||
cleanup. No history action deletes a cache placement; that is exclusively Evict
|
||||
Cache.
|
||||
|
||||
## Restart and rate-limit behavior
|
||||
|
||||
Sessions, pages, filters, chosen entries, confirmation bindings, and live status
|
||||
views are persisted. On bot restart, active status messages resume updates when
|
||||
the stored chat/message remains editable; otherwise users can reopen Job Status
|
||||
without losing job state. “message is not modified” is treated as success.
|
||||
Telegram retry-after values pause/coalesce edits without affecting orchestration.
|
||||
|
||||
## Interactive test entry
|
||||
|
||||
An opt-in `Archive Control (Test)` entry reuses this same handler with an
|
||||
independent callback namespace and SQLite database. Its scripted backend
|
||||
provides comprehensive, empty, and slow-route scenarios without contacting
|
||||
control/client daemons, qBittorrent, or Syncthing. The production entry,
|
||||
callback namespace, database, jobs, placements, and other bot features are
|
||||
unchanged.
|
||||
|
||||
The test entry exposes administrator-only `/actest` controls to reset a
|
||||
scenario, move a synthetic job through displayed states, fail the next selected
|
||||
backend call, complete a slow route, inspect recent trace events, or export the
|
||||
complete JSONL trace. The trace contains callback and filter inputs, session
|
||||
state transitions, exact message/keyboard renders, backend results/errors, and
|
||||
job transitions, so a UI report can be reproduced without live storage nodes.
|
||||
+245
@@ -0,0 +1,245 @@
|
||||
# Testing strategy
|
||||
|
||||
Tests run locally in containers and never inspect or mutate live x1, x2,
|
||||
lithium, or Titan services. Host tooling follows `playground/ENV.md`: Docker and
|
||||
Compose provide runtimes/build tools and write generated files as UID/GID
|
||||
`1001:1001`.
|
||||
|
||||
## Current happy-path coverage
|
||||
|
||||
The initial happy-path pass is tracked by executable capability rather than by
|
||||
declaring an unfinished phase complete:
|
||||
|
||||
As of 2026-07-23, the full control/bot suite passes 102 tests, the full client
|
||||
suite passes 60 tests, and a clean five-project Compose run passes automatic
|
||||
2×2 route creation plus archive → covered eviction → unarchive with SHA-256
|
||||
verification at both materialized targets.
|
||||
|
||||
| Capability | Unit/contract | Component/E2E |
|
||||
| --- | --- | --- |
|
||||
| Registration, heartbeat, replay, snapshots | Covered | Control/client integration covered |
|
||||
| On-demand inventory and content trees | Covered | WebSocket integration covered |
|
||||
| Automatic pairwise route provisioning | Covered | Local 2×2 topology covered |
|
||||
| Manifest/ready-marker publication | Covered | Receiver filesystem round trip covered |
|
||||
| Hardlink, reflink, sparse-copy fallback | Covered | Local filesystem round trip covered |
|
||||
| Existing same-size target reuse | Covered | Local filesystem round trip covered |
|
||||
| qB stopped add, selection, guarded recheck, start/resume, entry-only delete | Covered for 4.x/5.x semantics | Current pinned service covered |
|
||||
| Syncthing scoped rescan and completion proof | Covered | Real route transfer covered |
|
||||
| Transfer command state machine and qB verification | Covered | Archive and unarchive covered |
|
||||
| Archive/unarchive placement commit | Covered | Both directions covered |
|
||||
| Cache eviction | Coverage union, inode/path safety, idempotency covered | Real retained-source eviction covered |
|
||||
| Telegram archive-control flows | Persistence, nonce, stale callback, stalled rendering, dummy controls/trace covered | Real handler with isolated dummy backend covered; live Telegram remains release smoke |
|
||||
|
||||
Cancellation, rollback provenance, shared/unknown eviction paths,
|
||||
case-collisions, and stale preview guards now have focused tests. The broader
|
||||
chaos matrix remains a release gate rather than part of the initial happy-path
|
||||
run.
|
||||
|
||||
The executable complex E2E matrix additionally covers complementary partial
|
||||
archive placements, selective cache-placement merging, repeated placement
|
||||
generation after eviction, two-archive coverage union,
|
||||
uncovered/no-op/stale-preview rejection, exact-path sharing and unknown file
|
||||
retention, queued record-only cancellation, precommit rollback after target
|
||||
materialization, control restart after source staging, durable command replay
|
||||
after a client disconnect, and a no-downloaded-bytes assertion at the target.
|
||||
It also mutates the source selection and creates an absent target after job
|
||||
confirmation; both hostile races must fail before commit with the precise
|
||||
precondition reason while retaining the source. A wrong-size pre-existing
|
||||
target and an unwritable target root likewise fail without commit; the latter
|
||||
is wrapped in fixture cleanup that restores the original directory mode.
|
||||
|
||||
The second adversarial matrix also passes from clean state. It covers pure-v2
|
||||
and hybrid archive/evict/unarchive round trips, including qBittorrent's
|
||||
truncated-v2 primary key and hybrid v1/v2 aliases; a disconnected target that
|
||||
becomes `STALLED` at 20% and resumes successfully; bounded disk exhaustion with
|
||||
an exact precommit failure and intact source; and duplicate/reordered protocol
|
||||
delivery from a separate WebSocket probe process.
|
||||
|
||||
The remaining Compose expansion includes supported
|
||||
partfiles, sparse-capability mismatch, case/type collisions, broader
|
||||
service-level network partitions, and database backup/restore during active
|
||||
recovery. Focused unit/component tests already cover several of these
|
||||
invariants, but they remain unchecked in the full five-project topology.
|
||||
|
||||
## Test layers
|
||||
|
||||
### Proto contract
|
||||
|
||||
- Buf format, lint, descriptor build, and breaking checks.
|
||||
- Decode every canonical protobuf-JSON fixture through the descriptor.
|
||||
- Golden round trips for envelopes and on-disk manifests.
|
||||
- Reject duplicate JSON keys, unknown major versions, invalid hashes/ranges,
|
||||
malformed UUIDs/timestamps, over-limit frames, token outside registration,
|
||||
and unknown required commands.
|
||||
- Consumer tests assert the generated binding header records the exact proto
|
||||
release/commit.
|
||||
|
||||
### Unit and property tests
|
||||
|
||||
Control tests cover state transitions, hybrid-identity alias reservations,
|
||||
FIFO scheduling,
|
||||
placement unions, coverage-set calculation, command retries, event gap
|
||||
reconciliation, UI nonce idempotency, retention, and backup selection.
|
||||
|
||||
Client tests cover path normalization/confinement, file-index range algebra,
|
||||
tree expansion, root mapping, capability probes, version adapters, manifest
|
||||
digesting, journal replay, progress aggregation, and safe directory removal.
|
||||
Property/fuzz inputs emphasize path traversal, Unicode/case collisions,
|
||||
overlapping selections, reordered/duplicated events, and partial journals.
|
||||
|
||||
### Adapter contract tests
|
||||
|
||||
Pinned qBittorrent and Syncthing container versions cover every supported API
|
||||
family. Assertions include stopped add, exact selection, export, full recheck,
|
||||
download-attempt detection, entry-only delete, folder/device idempotency,
|
||||
events fallback, need/completion proof, and redacted errors. Partfile adapters
|
||||
are tested only against explicitly supported qBittorrent/libtorrent fixtures.
|
||||
|
||||
### Component integration tests
|
||||
|
||||
- Control plus fake clients: reconnect, duplicate ID replacement, lost ack,
|
||||
replay, sequence gaps, old connection fencing, queue leases, and restart.
|
||||
- One client plus qB/Syncthing: inventory, staging, sparse/copy fallbacks,
|
||||
service restart, permission failure, and database recovery.
|
||||
- Two clients plus two Syncthing services: automatic device/folder creation,
|
||||
bidirectional nonce verification, timeout, and existing-route discovery.
|
||||
|
||||
## Cross-project E2E topology
|
||||
|
||||
The coordination repository owns uniquely named Compose projects:
|
||||
|
||||
```text
|
||||
e2e/
|
||||
├── control/ # mogic archive-control core + test HTTP adapter
|
||||
├── cache-1/ # qBittorrent + Syncthing + cache client
|
||||
├── cache-2/ # qBittorrent + Syncthing + cache client
|
||||
├── archive-1/ # qBittorrent + Syncthing + archive client
|
||||
├── archive-2/ # qBittorrent + Syncthing + archive client
|
||||
├── fixtures/
|
||||
├── scenarios/
|
||||
└── scripts/
|
||||
```
|
||||
|
||||
Each data-node project has isolated config, state, backup, qB data, sync data,
|
||||
and service volumes. All join one dedicated, uniquely labeled test network.
|
||||
Syncthing advertises static Compose service addresses in E2E; production
|
||||
defaults remain `dynamic`. The topology deliberately starts without explicit
|
||||
route folders so tests exercise automatic 2×2 pairing.
|
||||
|
||||
Teardown resolves exact project names/labels and stops only those resources.
|
||||
Volume deletion is a separate explicit test action, never an unscoped prune.
|
||||
|
||||
## Thin test HTTP adapter
|
||||
|
||||
The adapter is enabled only in the control E2E configuration and delegates to
|
||||
`ArchiveControlService`; it contains no orchestration logic. Initial endpoints:
|
||||
|
||||
| Method/path | Purpose |
|
||||
| --- | --- |
|
||||
| `GET /test/v1/clients` | connection/health/capability view |
|
||||
| `GET /test/v1/routes` | route discovery/provisioning state |
|
||||
| `GET /test/v1/resources?operation=...` | on-demand eligible source listing |
|
||||
| `GET /test/v1/resources/{id}/tree?...` | scoped complete tree |
|
||||
| `POST /test/v1/jobs/preview` | validate and calculate exact delta/coverage |
|
||||
| `POST /test/v1/jobs` | create using preview revision and idempotency key |
|
||||
| `GET /test/v1/jobs/{id}` | projected state and progress |
|
||||
| `POST /test/v1/jobs/{id}/cancel` | queued removal or active compensation |
|
||||
| `POST /test/v1/jobs/{id}/hide` | UI/history-only removal |
|
||||
| `POST /test/v1/scheduler/pause` | stop automatic advancement at a durable boundary |
|
||||
| `POST /test/v1/scheduler/advance` | dispatch exactly the next eligible command |
|
||||
| `POST /test/v1/scheduler/resume` | restore automatic advancement |
|
||||
| `POST /test/v1/protocol/order-probe` | create and lease one synthetic ordering-test job |
|
||||
| `POST /test/v1/maintenance/backup` | request and observe a test backup |
|
||||
|
||||
JSON objects are adapter DTOs, not a second daemon protocol. Test scripts use
|
||||
curl, poll bounded state conditions, and dump control/client/service logs on
|
||||
failure. Scheduler gating is available only through the loopback, opt-in test
|
||||
adapter and is absent from normal daemon and Telegram workflows.
|
||||
|
||||
## Interactive Telegram UX harness
|
||||
|
||||
The integrated bot has an opt-in `Archive Control (Test)` entry backed by a
|
||||
scripted in-process service. It runs the production conversation handler and
|
||||
planner, but never starts a control listener and never contacts a client
|
||||
daemon, qBittorrent, or Syncthing. Its UI sessions, scenario state, synthetic
|
||||
jobs, and append-only trace share one dedicated SQLite test database that
|
||||
configuration validation forbids from being the production control database.
|
||||
|
||||
The `comprehensive` scenario provides enough resources, targets, and retained
|
||||
jobs to exercise resource/job pagination, filtering, Unicode and long names,
|
||||
full and partial placements, selective entry previews, archive, unarchive,
|
||||
covered eviction, job removal, every significant displayed job state, and
|
||||
single-use confirmations. `empty` covers empty lists. `route_slow` keeps a
|
||||
newly requested route pending until an administrator explicitly marks it ready.
|
||||
|
||||
The existing Telegram manager allowlist and optional numeric-ID pin apply.
|
||||
Private-chat controls are:
|
||||
|
||||
```text
|
||||
/actest status
|
||||
/actest reset [comprehensive|empty|route_slow]
|
||||
/actest advance <job-id> <state>
|
||||
/actest fail-next <backend-method> [message]
|
||||
/actest route-ready
|
||||
/actest trace [limit]
|
||||
/actest export
|
||||
```
|
||||
|
||||
`advance` accepts queued, preparing, running, waiting, stalled, cancelling,
|
||||
cleanup_required, succeeded, failed, and cancelled. `fail-next` injects one
|
||||
deterministic planner-style backend failure. Reset clears only the test
|
||||
sessions, jobs, scenario state, and trace. State otherwise survives bot
|
||||
restarts.
|
||||
|
||||
The SQLite trace records callback receipt and validated callback state before
|
||||
and after handling, text filters, complete rendered text and inline keyboards,
|
||||
backend calls/results/errors, test-control commands, and job transitions.
|
||||
`trace` shows a bounded recent view in chat; `export` sends the complete trace
|
||||
as JSON Lines for offline diagnosis.
|
||||
|
||||
## Fixture matrix
|
||||
|
||||
Fixtures include small deterministic v1, v2, and hybrid torrents with:
|
||||
|
||||
- full, complementary, overlapping, and one-directory selections;
|
||||
- same-size corrupt pre-existing targets and type/size collisions;
|
||||
- nested shared directories and exact file paths shared by multiple torrents;
|
||||
- unknown files/directories inside a resource-owned directory;
|
||||
- supported, unsupported, and ambiguous partfiles;
|
||||
- sparse files plus a simulated non-sparse target capability;
|
||||
- canonical and renamed/noncanonical qBittorrent paths;
|
||||
- hardlink, reflink, and copy-only filesystem conditions;
|
||||
- permissions and space-reserve failures.
|
||||
|
||||
## Acceptance scenarios
|
||||
|
||||
| Area | Required scenarios |
|
||||
| --- | --- |
|
||||
| Archive | new target, selective target, source retained, second archive replica |
|
||||
| Merge | complementary sources build union, overlap transfers delta only, no-op rejected, selected-incomplete target rejected |
|
||||
| Unarchive | new cache, merge existing cache, archive always retained, repeat after prior eviction |
|
||||
| Eviction | one-archive coverage, union coverage, missing coverage disabled, shared/unknown content retained |
|
||||
| Routes | on-demand 2×2 creation, eager mesh, existing match, slow success, timeout, restart during provisioning |
|
||||
| Recovery | restart every step on source/target/control, lost DB with proof, ambiguous loss fails closed |
|
||||
| Messaging | duplicate command/event, lost ack, sequence gap, stale revision, duplicate client ID |
|
||||
| Safety | attempted download, corrupt same-size file, symlink/path escape, special file, partfile mismatch |
|
||||
| Operations | stall retains percent, offline wait, cancellation each phase, postcommit cleanup retry |
|
||||
| Database | online backup under load, pre/post migration, retention, corrupt backup rejection, offline restore |
|
||||
| UI | pagination/filter chain, stale/double callback, persisted session, clear vs evict separation |
|
||||
|
||||
Every scenario asserts final qB selections/states, placement generations,
|
||||
manifests/journals, filesystem ownership/provenance, job event order, absence of
|
||||
unexpected received bytes, and preservation of unrelated files.
|
||||
|
||||
## Fault injection and pass criteria
|
||||
|
||||
Scenario hooks pause or restart each container, disconnect network edges,
|
||||
duplicate/reorder protocol frames, fill a bounded test filesystem, change an
|
||||
external qB selection, and kill processes between intent and completion journal
|
||||
writes. Tests use eventual assertions with explicit deadlines; long production
|
||||
waits are shortened by test-only configuration without altering state logic.
|
||||
|
||||
A release candidate passes only when the whole matrix succeeds repeatedly from
|
||||
clean volumes and from retained/restarted volumes, existing mogic-bot regression
|
||||
tests remain green, Buf reports no unintended break, and both amd64 and arm64
|
||||
client images start and pass smoke/config checks.
|
||||
@@ -0,0 +1,180 @@
|
||||
# Resource and workflow model
|
||||
|
||||
## Resource identity and file selection
|
||||
|
||||
A resource is identified by its BitTorrent metainfo identity:
|
||||
|
||||
- v1: lowercase 40-character SHA-1 info hash;
|
||||
- v2: lowercase 64-character SHA-256 info hash;
|
||||
- hybrid: both values.
|
||||
|
||||
The qBittorrent-local torrent hash is retained only as an adapter identifier.
|
||||
Identity matching never relies on display name or filesystem path. Control
|
||||
maintains aliases for both hashes of a hybrid torrent: one matching hash joins
|
||||
records only when any other supplied hash does not conflict, and the normalized
|
||||
identity is then enriched with both values. Reservations cover every known
|
||||
alias so v1-only and hybrid observations cannot schedule concurrently.
|
||||
|
||||
A selection is a normalized set of torrent file indices. Ranges are sorted,
|
||||
coalesced, and immutable after job creation. Only selected/skipped state is
|
||||
portable; qBittorrent priority levels are deliberately ignored. A directory
|
||||
choice expands to all currently available selected-complete descendant file
|
||||
indices before confirmation.
|
||||
|
||||
For every transfer, control fetches complete source and target content trees
|
||||
immediately before staging. It validates the source fingerprint, expands the
|
||||
requested selection, reads the target's verified baseline, and computes:
|
||||
|
||||
```text
|
||||
transfer delta = requested source files - verified target files
|
||||
target union = verified target files + transfer delta
|
||||
```
|
||||
|
||||
An empty delta is a no-op and is rejected rather than recorded as a successful
|
||||
transfer. Separate one-source/one-target jobs can merge complementary
|
||||
selections into the same placement over time.
|
||||
|
||||
## Job creation
|
||||
|
||||
All creation paths follow the same service-layer transaction:
|
||||
|
||||
1. Resolve a fresh source summary and suitable online targets.
|
||||
2. Optionally resolve one file or directory; otherwise use all
|
||||
selected-complete source files.
|
||||
3. Fetch full source and target trees and compute the exact delta.
|
||||
4. Validate role, route feasibility, canonical paths, generation, completion,
|
||||
disk capacity, and advertised capabilities. Partfile compatibility is not an
|
||||
eligibility filter at this stage.
|
||||
5. Show a final confirmation containing operation, resource, source, target,
|
||||
requested size/count, and whether this extends an existing placement.
|
||||
6. Consume a single-use confirmation nonce and atomically insert the immutable
|
||||
job, resource reservation, and initial event.
|
||||
|
||||
The previewed requested selection, verified target baseline, exact transfer
|
||||
delta, and their logical byte counts become immutable fields in the job
|
||||
definition. Execution recomputes and compares them rather than silently
|
||||
adapting to external changes.
|
||||
|
||||
Partfile handling is deliberately checked again immediately before the first
|
||||
data step. A resource is not hidden as ineligible merely because its partfile
|
||||
format may be unsafe; the accepted job fails with an explicit reason when the
|
||||
live preflight proves it cannot be handled.
|
||||
|
||||
## Route readiness
|
||||
|
||||
Each data route is one `sendreceive` Syncthing folder shared by exactly two
|
||||
devices: one cache client and one archive client. The Syncthing folder ID is the
|
||||
route ID.
|
||||
|
||||
At registration, clients report compatible folders below their configured sync
|
||||
root. Control pairs matching route IDs. It ignores and never modifies
|
||||
multi-device or wrong-type folders.
|
||||
|
||||
If a needed pair has no route, the default `on_demand` policy performs:
|
||||
|
||||
1. Generate a route/folder ID and deterministic relative folder path.
|
||||
2. Send each client the peer device ID and advertised addresses.
|
||||
3. Have each client idempotently add the peer device and its side of the same
|
||||
folder.
|
||||
4. Wait for both Syncthing instances to report a healthy shared folder.
|
||||
5. Exchange unique nonce files in both directions and require both observations.
|
||||
6. Mark the route ready and refresh registrations/routes.
|
||||
|
||||
The default timeout is 30 minutes and is configurable. A timeout leaves a
|
||||
diagnosable provisioning state. Rollback may remove only configuration created
|
||||
by that attempt and only while ownership is unambiguous. A previously verified
|
||||
route is never automatically deleted. `eager_mesh` runs the same process for
|
||||
all missing online cache/archive pairs.
|
||||
|
||||
## Archive and unarchive transfers
|
||||
|
||||
Archive and unarchive share one transfer state machine; only source/target roles
|
||||
and placement policy differ.
|
||||
|
||||
### Preflight
|
||||
|
||||
- Revalidate both resource fingerprints and placement generations.
|
||||
- Require source-selected files to be complete and regular, canonical entries.
|
||||
- Reject noncanonical qBittorrent file/folder renames in v1.
|
||||
- Confirm target path types/sizes do not conflict.
|
||||
- For an existing target, require every currently selected baseline file to be
|
||||
complete and verified. Unexpected selected-incomplete files fail as stale
|
||||
external state rather than being downloaded or silently deselected.
|
||||
- Confirm free space/reserve, permissions, sparse capability, supported
|
||||
partfile format, route health, and negotiated protocol features.
|
||||
- Export the source `.torrent` metainfo.
|
||||
- Persist the exact source, target baseline, requested selection, and delta.
|
||||
|
||||
### Five visible steps
|
||||
|
||||
1. **Source Stage/HardLink.** Create the isolated job directory and materialize
|
||||
only delta files plus required metainfo/auxiliary data. Try hard link,
|
||||
reflink, then copy, journaling every result. Write the manifest atomically
|
||||
and the ready marker last.
|
||||
2. **Syncthing.** Wait for the destination to receive the complete job prefix.
|
||||
Report job-scoped percentage and approximate speed, but do not infer success
|
||||
from a progress percentage alone.
|
||||
3. **Target Merge/Copy.** Materialize the delta beneath the configured qB root.
|
||||
Reuse same-size pre-existing regular files without overwriting them; record
|
||||
provenance so verification or rollback can distinguish them.
|
||||
4. **QB Verification.** Add a missing torrent stopped with manual management
|
||||
and explicit save path, or apply a transactional delta to an existing
|
||||
torrent. Set selected/skipped state, run a full recheck of the target union,
|
||||
and fail immediately if qBittorrent attempts to download. Successful recheck
|
||||
atomically advances the placement generation and is the commit point.
|
||||
5. **Staging Cleanup.** Remove only job-owned staging artifacts from both
|
||||
endpoints. A post-commit cleanup failure produces `CLEANUP_REQUIRED`; it
|
||||
never rolls back or deletes the committed placement.
|
||||
|
||||
Archive keeps the cache source. Unarchive always keeps the archive source. A
|
||||
new target is started only after successful verification so it can seed; an
|
||||
existing target returns to its captured stopped/running state. Archive Control
|
||||
does not copy tags, categories, rate limits, queue position, or other
|
||||
source-local metadata.
|
||||
|
||||
### Existing target merge
|
||||
|
||||
Before changing an existing target, the client durably records its selection,
|
||||
run state, qB identifier, content revision, file provenance, and partfile
|
||||
baseline. Failure before commit restores that baseline and removes only files
|
||||
created by the current job that remain unreferenced. Pre-existing paths are
|
||||
never removed. Success stores the verified union, allowing later jobs from
|
||||
other sources to add complementary file indices.
|
||||
|
||||
## Evict cache
|
||||
|
||||
Eviction is independent of transfer history and operates on one whole cache
|
||||
placement in v1.
|
||||
|
||||
1. Query the cache placement and all relevant online archive placements fresh.
|
||||
2. Build per-file archive coverage; the union may span multiple archive nodes.
|
||||
3. Disable confirmation and explain missing files if coverage is incomplete.
|
||||
4. On confirmation, reserve the resource and persist the exact cache
|
||||
fingerprint/generation plus archive proofs.
|
||||
5. Revalidate those proofs immediately before deletion.
|
||||
6. Ask qBittorrent to remove the cache entry with data deletion disabled.
|
||||
7. Snapshot exact paths referenced by all remaining local torrents.
|
||||
8. Unlink only manifest-owned files not exactly referenced by another torrent.
|
||||
9. Remove a partfile only when ownership by the evicted torrent is
|
||||
unambiguous.
|
||||
10. Attempt non-recursive `rmdir` from deepest known owned directory upward;
|
||||
retain directories containing unknown files, subdirectories, or other
|
||||
application data.
|
||||
|
||||
There is no force override in v1. Retrying any step observes prior effects and
|
||||
continues safely. Eviction never changes archive placements.
|
||||
|
||||
## Cancellation, removal, and display retention
|
||||
|
||||
- Cancelling a queued job deletes the queue record and releases its reservation
|
||||
only; no client command or file operation occurs.
|
||||
- Cancelling an active pre-commit job requests cooperative stop and runs
|
||||
attributable compensation. The source placement remains intact.
|
||||
- After commit, cancellation can only drive staging cleanup. It cannot undo the
|
||||
committed target.
|
||||
- A failed or cancelling job stays visible until its cleanup/compensation state
|
||||
is known.
|
||||
- Clear Finished and removal of a successful record affect only UI visibility.
|
||||
They never touch qBittorrent, files, routes, or placements.
|
||||
- Hidden jobs retain a minimal idempotency tombstone for 30 days by default;
|
||||
detailed events may be purged sooner under configured retention.
|
||||
Reference in New Issue
Block a user