docs: publish archive control design set

This commit is contained in:
2026-07-23 13:39:09 +00:00
parent 6ca94a546e
commit 7e295cff3a
12 changed files with 1960 additions and 7 deletions
+137
View File
@@ -0,0 +1,137 @@
# Agreed design decisions
This is the concise decision ledger from the design interview. Later documents
expand these rules but must not contradict them.
## Roles and authority
- Client roles are `archive` and `cache`. `archive` and `unarchive` are job
operations, not daemon roles.
- Titan's control daemon is the sole cross-node orchestrator. Clients never
command one another and do not advance to the next workflow step without a
persisted Titan command.
- Archive nodes retain committed resources. Cache eviction is a distinct,
explicit operation.
- Transfer and eviction jobs are durable, idempotent, recoverable, and
serialized globally per torrent resource.
## Connection and protocol
- Every WebSocket connection starts with a registration containing client ID,
role, protocol/capabilities, and the single pre-shared token. The token occurs
in no later message. Invalid registration closes the connection.
- A new valid registration for a duplicate client ID disconnects the old
connection.
- Clients reconnect forever with configurable capped exponential backoff.
- Commands are at-least-once, durably deduplicated, and separate acceptance
acknowledgements from completion events.
- Protobuf JSON uses a versioned `archive_control.v1` envelope. Per-job event
sequences support gap detection and snapshot reconciliation.
## State, recovery, and scheduling
- Titan and every client have SQLite state. They reconcile active jobs on every
reconnect and fail closed when lost state cannot be proven from journals,
manifests, and observed external state.
- SQLite backups use the online backup API. Defaults are every six hours, 12
recent, 14 daily, and eight weekly copies, plus pre/post-migration backups.
- Default concurrency is one active data-moving job per client and per route.
Disjoint node pairs may run concurrently. Queueing is durable FIFO.
- A queued job owns a per-resource reservation. Cancelling it removes only the
queued record/reservation and never sends cleanup commands.
- Connectivity or transfer stalls wait indefinitely. A configurable 30-minute
no-progress threshold marks the job `STALLED` without losing its percentage.
## Resources and selections
- Resource identity supports BitTorrent v1, v2, and hybrid info hashes. A
qBittorrent-local hash is a separate adapter identifier. Hybrid hash aliases
share reservations and are joined only when no supplied hash conflicts.
- Only selected versus skipped matters; qBittorrent priority levels are not
preserved.
- Placements are keyed by resource and node and contain a verified set of
selected torrent file indices. Multiple cache and archive placements are
allowed.
- Transfer jobs have one source and one target. A requested selection is
expanded to immutable file indices. Targets merge the missing delta into
their existing selection transactionally.
- Initial Telegram selection supports either all source-selected complete
entries or one file/directory. The protocol supports multiple entries from
v1.
- Full source and target file inventories are fetched immediately before
staging, even for the normal `Confirm` path. The previewed baseline and delta
are persisted immutably and must still match at execution.
## Transfer safety
- Each transfer uses `<sync-folder>/.archive-control/jobs/<job-id>/` with a
payload, exported torrent, protobuf-JSON manifest, auxiliary artifacts, and a
ready marker.
- File materialization tries hard link, then reflink, then safe copy. It never
follows symlinks or special files and never escapes configured roots.
- Pre-existing target paths are never overwritten. Same-size files may be
reused only after qBittorrent's stopped recheck proves them valid.
- New target torrents are explicitly placed under the configured qB root,
added stopped with manual torrent management, assigned selected/skipped
states, fully rechecked, and started only after successful verification.
- Existing target merges preserve a durable baseline. Failure restores the
baseline selection/state and removes only attributable delta data.
- qBittorrent/libtorrent partfiles are transferred or merged only through a
tested version-aware adapter. An unsafe format fails job preflight with a
clear reason.
- Sparse support is probed once at client startup for configured roots. Jobs
involving sparse files require semantic sparse support on every leg; exact
physical extent layouts are not required.
- Permission handling is fail-fast. The daemon never chmods/chowns source data.
## Commit, cleanup, and deletion
- A transfer commits after the target's selected union passes a stopped full
recheck without downloading. After commit, the new placement is retained.
- Staging cleanup is post-commit. Its failure is `CLEANUP_REQUIRED`, not a
failed transfer, and cannot cause committed archive data to be deleted.
- Archive jobs no longer delete their source cache placement.
- Eviction is a durable job. It requires fresh, complete archive coverage for
every selected cache file; coverage may be the union of multiple online
archive nodes. There is no force override initially.
- Eviction removes the qBittorrent entry without data deletion, checks exact
path overlap with remaining torrents, unlinks eligible known files, and uses
non-recursive bottom-up directory removal. It never blindly removes a tree.
## Syncthing routes
- Client config supplies Syncthing endpoint/API key and separate API-visible
and daemon-local sync roots. Compatible `sendreceive` folders beneath the
root are discovered automatically.
- Syncthing's native folder ID is the route ID. Routes are dedicated to exactly
two devices: one archive and one cache client.
- Missing routes can be created `on_demand` (default) or by `eager_mesh`.
Clients automatically pair missing Syncthing devices and create/verify
folders. Optional advertised addresses default to `dynamic`.
- Route setup timeout defaults to 30 minutes. Archive Control never removes a
verified route merely because a node is absent.
## User interface and retention
- Main actions are Archive, Unarchive, Evict Cache, and Job Status.
- Telegram state is shared across authorized admins. Jobs/events store no
Telegram user owner.
- Existing private-chat `@username` authorization remains compatible; optional
numeric user-ID pins harden configured entries when available.
- Archive Control's Telegram view/filter/confirmation sessions persist in
SQLite. Destructive confirmations use single-use nonces.
- Telegram progress edits are coalesced to at most once every five seconds by
default, with immediate important transitions.
- Clear/Remove hides completed records without touching placements. Minimal
idempotency tombstones remain for 30 days by default.
## Delivery and testing
- `archive-control-proto` is drafted and reviewed before application code.
- Host tooling stays light. Buf, protoc, tests, and builds run in Docker and
preserve host UID/GID `1001:1001`.
- The client image is `sodium/archive-clients`, published for linux/amd64 and
linux/arm64 with immutable release tags.
- Cross-project E2E lives in this coordination repository and simulates two
cache nodes, two archive nodes, and a thin control node. It never mutates the
live x1/x2/lithium services.