docs: publish archive control design set
This commit is contained in:
@@ -0,0 +1,153 @@
|
||||
# Storage safety, rollback, and recovery
|
||||
|
||||
## Path trust boundary
|
||||
|
||||
qBittorrent metadata, Syncthing configuration, transfer manifests, and
|
||||
filesystem directory entries are all untrusted inputs. Before any access, the
|
||||
client:
|
||||
|
||||
1. validates the canonical torrent-relative path;
|
||||
2. rejects absolute paths, empty/`.`/`..` components, NULs, platform separators,
|
||||
duplicate normalized names, and configured case-fold collisions;
|
||||
3. resolves the API-visible path through one validated root mapping;
|
||||
4. traverses without following symlinks and checks the resulting descriptor is
|
||||
a regular file or an explicitly created directory;
|
||||
5. verifies every operation remains beneath the local configured root.
|
||||
|
||||
Sockets, devices, FIFOs, symlinks, and other special entries fail preflight.
|
||||
Permission or ownership mismatch is fail-fast. Archive Control never changes
|
||||
source ownership or mode to make a job pass.
|
||||
|
||||
## Isolated staging
|
||||
|
||||
Every route stores transient jobs under:
|
||||
|
||||
```text
|
||||
<sync-root>/.archive-control/jobs/<job-id>/
|
||||
├── payload/<original torrent hierarchy>
|
||||
├── metainfo/source.torrent
|
||||
├── artifacts/...
|
||||
├── manifest.json
|
||||
└── ready.json
|
||||
```
|
||||
|
||||
Temporary writes use an attempt-specific name, fsync where durability matters,
|
||||
and atomic rename. The manifest records every expected file and method. The
|
||||
ready marker is the final publish action. Job IDs and known subpaths are used
|
||||
directly; cleanup never expands a glob or recursively targets the sync root.
|
||||
|
||||
## Materialization fallback
|
||||
|
||||
For each file, source staging and target materialization try:
|
||||
|
||||
1. hard link when the same filesystem and permissions allow it;
|
||||
2. reflink/clone when supported;
|
||||
3. bounded buffered copy into a temporary file followed by atomic placement.
|
||||
|
||||
The journal records intent before mutation and completion afterward, including
|
||||
source identity, destination, method, logical/allocated sizes, and whether the
|
||||
target pre-existed. Restarts reconcile the journal against file descriptors and
|
||||
the manifest instead of blindly repeating operations.
|
||||
|
||||
Free-space checks account for methods that allocate data plus a configurable
|
||||
reserve. A mid-copy space failure leaves only an attributable temporary file.
|
||||
No operation overwrites an existing target. A same-type, same-size regular file
|
||||
may be provisionally reused, but qBittorrent's full recheck is the integrity
|
||||
authority; any other collision fails before mutation.
|
||||
|
||||
## Sparse files
|
||||
|
||||
At startup, each client probes semantic sparse-file behavior once for every
|
||||
configured qB and Syncthing local root. Capability is advertised with the
|
||||
connection. A job with any sparse source file requires sparse support for every
|
||||
staging and destination root involved. If a target leg lacks it, the job fails
|
||||
clearly before transfer.
|
||||
|
||||
Sparse files are retained as sparse, without attempting to reproduce an exact
|
||||
physical hole map. Hardlink/reflink naturally preserve semantics; the copy
|
||||
implementation uses a sparse-aware strategy and verifies logical length and
|
||||
allocation behavior. Exact allocated-block counts need not match between
|
||||
filesystems.
|
||||
|
||||
## qBittorrent transaction boundary
|
||||
|
||||
For a new target, the client exports source metainfo, places candidate data,
|
||||
adds the torrent stopped with manual management and explicit save path, applies
|
||||
the exact union selection, and starts a full recheck. It monitors both state and
|
||||
transfer counters. Any download transition or received payload byte is a hard
|
||||
failure and the torrent is stopped immediately.
|
||||
|
||||
For an existing target, the durable baseline includes its selected indices,
|
||||
run state, local identifier, save-path fingerprint, content revision, and
|
||||
partfile state. The job adds only the delta. Before commit, rollback restores
|
||||
that baseline and removes only current-job-created, unshared files. Source
|
||||
torrent state is never changed by transfer logic.
|
||||
|
||||
The target placement commits only after the selected union verifies completely
|
||||
while stopped. The placement generation and commit event are persisted in the
|
||||
same control transaction. Once committed, target content is outside rollback
|
||||
scope.
|
||||
|
||||
## Partfiles
|
||||
|
||||
Libtorrent partfiles may contain data for skipped or non-materialized torrent
|
||||
files and their layout varies by version. Archive Control therefore uses a
|
||||
version-aware adapter keyed by observed qBittorrent/libtorrent version and a
|
||||
tested format signature. The adapter must support export/import and, for an
|
||||
existing target, merge with a recoverable baseline.
|
||||
|
||||
Unknown, ambiguous, or incompatible partfile state fails the job at live
|
||||
preflight with `UNSUPPORTED` or `PRECONDITION_FAILED` and a precise reason. It
|
||||
is never copied over an existing partfile blindly. Rollback restores the
|
||||
baseline. Eviction removes a partfile only when its ownership by the removed
|
||||
torrent is unambiguous.
|
||||
|
||||
## Safe eviction deletion
|
||||
|
||||
Before unlinking, the client persists:
|
||||
|
||||
- the confirmed cache placement generation and file set;
|
||||
- fresh archive coverage proofs;
|
||||
- the exact torrent manifest and local paths;
|
||||
- qBittorrent entry-removal result;
|
||||
- exact paths referenced by all remaining torrents;
|
||||
- per-file unlink and per-directory `rmdir` results.
|
||||
|
||||
The qBittorrent entry is removed with `deleteFiles=false`. For each owned file,
|
||||
the client refuses removal if another remaining torrent references the exact
|
||||
path. Eligible known files are unlinked individually. It then attempts
|
||||
non-recursive `rmdir` deepest-first for known owned directories. `ENOTEMPTY`
|
||||
means retain and continue; unknown files or directories are never traversed or
|
||||
deleted. There is no recursive delete fallback.
|
||||
|
||||
## Compensation by phase
|
||||
|
||||
| Point | Safe outcome |
|
||||
| --- | --- |
|
||||
| Queued, no command | Delete queue row/reservation only |
|
||||
| Source staging | Stop; unlink only journaled staging artifacts |
|
||||
| Syncthing transfer | Stop publishing; clean only isolated job namespace |
|
||||
| Target materialization | Restore target baseline; remove attributable delta |
|
||||
| qB recheck before commit | Stop/remove newly added entry without data deletion, then compensate delta |
|
||||
| After commit | Keep placement; retry staging cleanup only |
|
||||
| Eviction after qB removal | Resume safe per-file unlink; never re-add or delete archives automatically |
|
||||
|
||||
If attribution is uncertain, retain data and require manual intervention.
|
||||
|
||||
## SQLite durability and backup
|
||||
|
||||
Both daemons enable foreign keys, WAL, a bounded busy timeout, and explicit
|
||||
transactions. Schema migrations create an online backup before and after the
|
||||
migration and fail atomically.
|
||||
|
||||
The default recurring policy runs SQLite's online backup API every six hours,
|
||||
verifies checksum and SQLite integrity, writes via temporary file plus atomic
|
||||
rename, and retains 12 recent, 14 daily, and eight weekly copies. Backups live
|
||||
outside qBittorrent and Syncthing roots. Configuration warns if the backup and
|
||||
database share a filesystem, but may permit it for small installations.
|
||||
|
||||
Offline tooling provides list, verify, and restore. Restore requires stopped
|
||||
daemon access, verifies the chosen backup, preserves the suspect database under
|
||||
a timestamped name, installs the replacement atomically, and runs integrity and
|
||||
schema checks before normal startup. Backups contain no configured secrets.
|
||||
|
||||
Reference in New Issue
Block a user