docs: publish archive control design set
This commit is contained in:
@@ -0,0 +1,137 @@
|
||||
# Agreed design decisions
|
||||
|
||||
This is the concise decision ledger from the design interview. Later documents
|
||||
expand these rules but must not contradict them.
|
||||
|
||||
## Roles and authority
|
||||
|
||||
- Client roles are `archive` and `cache`. `archive` and `unarchive` are job
|
||||
operations, not daemon roles.
|
||||
- Titan's control daemon is the sole cross-node orchestrator. Clients never
|
||||
command one another and do not advance to the next workflow step without a
|
||||
persisted Titan command.
|
||||
- Archive nodes retain committed resources. Cache eviction is a distinct,
|
||||
explicit operation.
|
||||
- Transfer and eviction jobs are durable, idempotent, recoverable, and
|
||||
serialized globally per torrent resource.
|
||||
|
||||
## Connection and protocol
|
||||
|
||||
- Every WebSocket connection starts with a registration containing client ID,
|
||||
role, protocol/capabilities, and the single pre-shared token. The token occurs
|
||||
in no later message. Invalid registration closes the connection.
|
||||
- A new valid registration for a duplicate client ID disconnects the old
|
||||
connection.
|
||||
- Clients reconnect forever with configurable capped exponential backoff.
|
||||
- Commands are at-least-once, durably deduplicated, and separate acceptance
|
||||
acknowledgements from completion events.
|
||||
- Protobuf JSON uses a versioned `archive_control.v1` envelope. Per-job event
|
||||
sequences support gap detection and snapshot reconciliation.
|
||||
|
||||
## State, recovery, and scheduling
|
||||
|
||||
- Titan and every client have SQLite state. They reconcile active jobs on every
|
||||
reconnect and fail closed when lost state cannot be proven from journals,
|
||||
manifests, and observed external state.
|
||||
- SQLite backups use the online backup API. Defaults are every six hours, 12
|
||||
recent, 14 daily, and eight weekly copies, plus pre/post-migration backups.
|
||||
- Default concurrency is one active data-moving job per client and per route.
|
||||
Disjoint node pairs may run concurrently. Queueing is durable FIFO.
|
||||
- A queued job owns a per-resource reservation. Cancelling it removes only the
|
||||
queued record/reservation and never sends cleanup commands.
|
||||
- Connectivity or transfer stalls wait indefinitely. A configurable 30-minute
|
||||
no-progress threshold marks the job `STALLED` without losing its percentage.
|
||||
|
||||
## Resources and selections
|
||||
|
||||
- Resource identity supports BitTorrent v1, v2, and hybrid info hashes. A
|
||||
qBittorrent-local hash is a separate adapter identifier. Hybrid hash aliases
|
||||
share reservations and are joined only when no supplied hash conflicts.
|
||||
- Only selected versus skipped matters; qBittorrent priority levels are not
|
||||
preserved.
|
||||
- Placements are keyed by resource and node and contain a verified set of
|
||||
selected torrent file indices. Multiple cache and archive placements are
|
||||
allowed.
|
||||
- Transfer jobs have one source and one target. A requested selection is
|
||||
expanded to immutable file indices. Targets merge the missing delta into
|
||||
their existing selection transactionally.
|
||||
- Initial Telegram selection supports either all source-selected complete
|
||||
entries or one file/directory. The protocol supports multiple entries from
|
||||
v1.
|
||||
- Full source and target file inventories are fetched immediately before
|
||||
staging, even for the normal `Confirm` path. The previewed baseline and delta
|
||||
are persisted immutably and must still match at execution.
|
||||
|
||||
## Transfer safety
|
||||
|
||||
- Each transfer uses `<sync-folder>/.archive-control/jobs/<job-id>/` with a
|
||||
payload, exported torrent, protobuf-JSON manifest, auxiliary artifacts, and a
|
||||
ready marker.
|
||||
- File materialization tries hard link, then reflink, then safe copy. It never
|
||||
follows symlinks or special files and never escapes configured roots.
|
||||
- Pre-existing target paths are never overwritten. Same-size files may be
|
||||
reused only after qBittorrent's stopped recheck proves them valid.
|
||||
- New target torrents are explicitly placed under the configured qB root,
|
||||
added stopped with manual torrent management, assigned selected/skipped
|
||||
states, fully rechecked, and started only after successful verification.
|
||||
- Existing target merges preserve a durable baseline. Failure restores the
|
||||
baseline selection/state and removes only attributable delta data.
|
||||
- qBittorrent/libtorrent partfiles are transferred or merged only through a
|
||||
tested version-aware adapter. An unsafe format fails job preflight with a
|
||||
clear reason.
|
||||
- Sparse support is probed once at client startup for configured roots. Jobs
|
||||
involving sparse files require semantic sparse support on every leg; exact
|
||||
physical extent layouts are not required.
|
||||
- Permission handling is fail-fast. The daemon never chmods/chowns source data.
|
||||
|
||||
## Commit, cleanup, and deletion
|
||||
|
||||
- A transfer commits after the target's selected union passes a stopped full
|
||||
recheck without downloading. After commit, the new placement is retained.
|
||||
- Staging cleanup is post-commit. Its failure is `CLEANUP_REQUIRED`, not a
|
||||
failed transfer, and cannot cause committed archive data to be deleted.
|
||||
- Archive jobs no longer delete their source cache placement.
|
||||
- Eviction is a durable job. It requires fresh, complete archive coverage for
|
||||
every selected cache file; coverage may be the union of multiple online
|
||||
archive nodes. There is no force override initially.
|
||||
- Eviction removes the qBittorrent entry without data deletion, checks exact
|
||||
path overlap with remaining torrents, unlinks eligible known files, and uses
|
||||
non-recursive bottom-up directory removal. It never blindly removes a tree.
|
||||
|
||||
## Syncthing routes
|
||||
|
||||
- Client config supplies Syncthing endpoint/API key and separate API-visible
|
||||
and daemon-local sync roots. Compatible `sendreceive` folders beneath the
|
||||
root are discovered automatically.
|
||||
- Syncthing's native folder ID is the route ID. Routes are dedicated to exactly
|
||||
two devices: one archive and one cache client.
|
||||
- Missing routes can be created `on_demand` (default) or by `eager_mesh`.
|
||||
Clients automatically pair missing Syncthing devices and create/verify
|
||||
folders. Optional advertised addresses default to `dynamic`.
|
||||
- Route setup timeout defaults to 30 minutes. Archive Control never removes a
|
||||
verified route merely because a node is absent.
|
||||
|
||||
## User interface and retention
|
||||
|
||||
- Main actions are Archive, Unarchive, Evict Cache, and Job Status.
|
||||
- Telegram state is shared across authorized admins. Jobs/events store no
|
||||
Telegram user owner.
|
||||
- Existing private-chat `@username` authorization remains compatible; optional
|
||||
numeric user-ID pins harden configured entries when available.
|
||||
- Archive Control's Telegram view/filter/confirmation sessions persist in
|
||||
SQLite. Destructive confirmations use single-use nonces.
|
||||
- Telegram progress edits are coalesced to at most once every five seconds by
|
||||
default, with immediate important transitions.
|
||||
- Clear/Remove hides completed records without touching placements. Minimal
|
||||
idempotency tombstones remain for 30 days by default.
|
||||
|
||||
## Delivery and testing
|
||||
|
||||
- `archive-control-proto` is drafted and reviewed before application code.
|
||||
- Host tooling stays light. Buf, protoc, tests, and builds run in Docker and
|
||||
preserve host UID/GID `1001:1001`.
|
||||
- The client image is `sodium/archive-clients`, published for linux/amd64 and
|
||||
linux/arm64 with immutable release tags.
|
||||
- Cross-project E2E lives in this coordination repository and simulates two
|
||||
cache nodes, two archive nodes, and a thin control node. It never mutates the
|
||||
live x1/x2/lithium services.
|
||||
Reference in New Issue
Block a user