Files

7.4 KiB

Agreed design decisions

This is the concise decision ledger from the design interview. Later documents expand these rules but must not contradict them.

Roles and authority

  • Client roles are archive and cache. archive and unarchive are job operations, not daemon roles.
  • Titan's control daemon is the sole cross-node orchestrator. Clients never command one another and do not advance to the next workflow step without a persisted Titan command.
  • Archive nodes retain committed resources. Cache eviction is a distinct, explicit operation.
  • Transfer and eviction jobs are durable, idempotent, recoverable, and serialized globally per torrent resource.

Connection and protocol

  • Every WebSocket connection starts with a registration containing client ID, role, protocol/capabilities, and the single pre-shared token. The token occurs in no later message. Invalid registration closes the connection.
  • A new valid registration for a duplicate client ID disconnects the old connection.
  • Clients reconnect forever with configurable capped exponential backoff.
  • Commands are at-least-once, durably deduplicated, and separate acceptance acknowledgements from completion events.
  • Protobuf JSON uses a versioned archive_control.v1 envelope. Per-job event sequences support gap detection and snapshot reconciliation.

State, recovery, and scheduling

  • Titan and every client have SQLite state. They reconcile active jobs on every reconnect and fail closed when lost state cannot be proven from journals, manifests, and observed external state.
  • SQLite backups use the online backup API. Defaults are every six hours, 12 recent, 14 daily, and eight weekly copies, plus pre/post-migration backups.
  • Default concurrency is unrestricted: every eligible queued job is admitted in one scheduler pass and independent jobs execute concurrently on a client. Physical qBittorrent, Syncthing, disk, and network capacity are therefore the natural limit. enforce_concurrency_limits=true restores the optional legacy per-client/per-route gates. Commands for one job remain serialized.
  • A queued job owns a per-resource reservation. Cancelling it removes only the queued record/reservation and never sends cleanup commands.
  • Connectivity or transfer stalls wait indefinitely. A configurable 30-minute no-progress threshold marks the job STALLED without losing its percentage.

Resources and selections

  • Resource identity supports BitTorrent v1, v2, and hybrid info hashes. A qBittorrent-local hash is a separate adapter identifier. Hybrid hash aliases share reservations and are joined only when no supplied hash conflicts.
  • Only selected versus skipped matters; qBittorrent priority levels are not preserved.
  • Placements are keyed by resource and node and contain a verified set of selected torrent file indices. Multiple cache and archive placements are allowed.
  • Transfer jobs have one source and one target. A requested selection is expanded to immutable file indices. Targets merge the missing delta into their existing selection transactionally.
  • Initial Telegram selection supports either all source-selected complete entries or one file/directory. The protocol supports multiple entries from v1.
  • Full source and target file inventories are fetched immediately before staging, even for the normal Confirm path. The previewed baseline and delta are persisted immutably and must still match at execution.

Transfer safety

  • Each transfer uses <sync-folder>/.archive-control/jobs/<job-id>/ with a payload, exported torrent, protobuf-JSON manifest, auxiliary artifacts, and a ready marker.
  • File materialization tries hard link, then reflink, then safe copy. It never follows symlinks or special files and never escapes configured roots.
  • Pre-existing target paths are never overwritten. Same-size files may be reused only after qBittorrent's stopped recheck proves them valid.
  • New target torrents are explicitly placed under the configured qB root, added stopped with manual torrent management, assigned selected/skipped states, fully rechecked, and started only after successful verification.
  • Existing target merges preserve a durable baseline. Failure restores the baseline selection/state and removes only attributable delta data.
  • qBittorrent/libtorrent partfiles are transferred or merged only through a tested version-aware adapter. An unsafe format fails job preflight with a clear reason.
  • Sparse support is probed once at client startup for configured roots. Jobs involving sparse files require semantic sparse support on every leg; exact physical extent layouts are not required.
  • Permission handling is fail-fast. The daemon never chmods/chowns source data.

Commit, cleanup, and deletion

  • A transfer commits after the target's selected union passes a stopped full recheck without downloading. After commit, the new placement is retained.
  • Staging cleanup is post-commit. Its failure is CLEANUP_REQUIRED, not a failed transfer, and cannot cause committed archive data to be deleted.
  • Archive jobs no longer delete their source cache placement.
  • Eviction is a durable job. It requires fresh, complete archive coverage for every selected cache file; coverage may be the union of multiple online archive nodes. There is no force override initially.
  • Eviction removes the qBittorrent entry without data deletion, checks exact path overlap with remaining torrents, unlinks eligible known files, and uses non-recursive bottom-up directory removal. It never blindly removes a tree.

Syncthing routes

  • Client config supplies Syncthing endpoint/API key and separate API-visible and daemon-local sync roots. Compatible sendreceive folders beneath the root are discovered automatically.
  • Syncthing's native folder ID is the route ID. Routes are dedicated to exactly two devices: one archive and one cache client.
  • Missing routes can be created on_demand (default) or by eager_mesh. Clients automatically pair missing Syncthing devices and create/verify folders. Optional advertised addresses default to dynamic.
  • Route setup timeout defaults to 30 minutes. Archive Control never removes a verified route merely because a node is absent.

User interface and retention

  • Main actions are Archive, Unarchive, Evict Cache, and Job Status.
  • Telegram state is shared across authorized admins. Jobs/events store no Telegram user owner.
  • Existing private-chat @username authorization remains compatible; optional numeric user-ID pins harden configured entries when available.
  • Archive Control's Telegram view/filter/confirmation sessions persist in SQLite. Destructive confirmations use single-use nonces.
  • Telegram progress edits are coalesced to at most once every five seconds by default, with immediate important transitions.
  • Clear/Remove hides completed records without touching placements. Minimal idempotency tombstones remain for 30 days by default.

Delivery and testing

  • archive-control-proto is drafted and reviewed before application code.
  • Host tooling stays light. Buf, protoc, tests, and builds run in Docker and preserve host UID/GID 1001:1001.
  • The client image is sodium/archive-clients, published for linux/amd64 and linux/arm64 with immutable release tags.
  • Cross-project E2E lives in this coordination repository and simulates two cache nodes, two archive nodes, and a thin control node. It never mutates the live x1/x2/lithium services.