docs: publish archive control design set
This commit is contained in:
@@ -0,0 +1,180 @@
|
||||
# Resource and workflow model
|
||||
|
||||
## Resource identity and file selection
|
||||
|
||||
A resource is identified by its BitTorrent metainfo identity:
|
||||
|
||||
- v1: lowercase 40-character SHA-1 info hash;
|
||||
- v2: lowercase 64-character SHA-256 info hash;
|
||||
- hybrid: both values.
|
||||
|
||||
The qBittorrent-local torrent hash is retained only as an adapter identifier.
|
||||
Identity matching never relies on display name or filesystem path. Control
|
||||
maintains aliases for both hashes of a hybrid torrent: one matching hash joins
|
||||
records only when any other supplied hash does not conflict, and the normalized
|
||||
identity is then enriched with both values. Reservations cover every known
|
||||
alias so v1-only and hybrid observations cannot schedule concurrently.
|
||||
|
||||
A selection is a normalized set of torrent file indices. Ranges are sorted,
|
||||
coalesced, and immutable after job creation. Only selected/skipped state is
|
||||
portable; qBittorrent priority levels are deliberately ignored. A directory
|
||||
choice expands to all currently available selected-complete descendant file
|
||||
indices before confirmation.
|
||||
|
||||
For every transfer, control fetches complete source and target content trees
|
||||
immediately before staging. It validates the source fingerprint, expands the
|
||||
requested selection, reads the target's verified baseline, and computes:
|
||||
|
||||
```text
|
||||
transfer delta = requested source files - verified target files
|
||||
target union = verified target files + transfer delta
|
||||
```
|
||||
|
||||
An empty delta is a no-op and is rejected rather than recorded as a successful
|
||||
transfer. Separate one-source/one-target jobs can merge complementary
|
||||
selections into the same placement over time.
|
||||
|
||||
## Job creation
|
||||
|
||||
All creation paths follow the same service-layer transaction:
|
||||
|
||||
1. Resolve a fresh source summary and suitable online targets.
|
||||
2. Optionally resolve one file or directory; otherwise use all
|
||||
selected-complete source files.
|
||||
3. Fetch full source and target trees and compute the exact delta.
|
||||
4. Validate role, route feasibility, canonical paths, generation, completion,
|
||||
disk capacity, and advertised capabilities. Partfile compatibility is not an
|
||||
eligibility filter at this stage.
|
||||
5. Show a final confirmation containing operation, resource, source, target,
|
||||
requested size/count, and whether this extends an existing placement.
|
||||
6. Consume a single-use confirmation nonce and atomically insert the immutable
|
||||
job, resource reservation, and initial event.
|
||||
|
||||
The previewed requested selection, verified target baseline, exact transfer
|
||||
delta, and their logical byte counts become immutable fields in the job
|
||||
definition. Execution recomputes and compares them rather than silently
|
||||
adapting to external changes.
|
||||
|
||||
Partfile handling is deliberately checked again immediately before the first
|
||||
data step. A resource is not hidden as ineligible merely because its partfile
|
||||
format may be unsafe; the accepted job fails with an explicit reason when the
|
||||
live preflight proves it cannot be handled.
|
||||
|
||||
## Route readiness
|
||||
|
||||
Each data route is one `sendreceive` Syncthing folder shared by exactly two
|
||||
devices: one cache client and one archive client. The Syncthing folder ID is the
|
||||
route ID.
|
||||
|
||||
At registration, clients report compatible folders below their configured sync
|
||||
root. Control pairs matching route IDs. It ignores and never modifies
|
||||
multi-device or wrong-type folders.
|
||||
|
||||
If a needed pair has no route, the default `on_demand` policy performs:
|
||||
|
||||
1. Generate a route/folder ID and deterministic relative folder path.
|
||||
2. Send each client the peer device ID and advertised addresses.
|
||||
3. Have each client idempotently add the peer device and its side of the same
|
||||
folder.
|
||||
4. Wait for both Syncthing instances to report a healthy shared folder.
|
||||
5. Exchange unique nonce files in both directions and require both observations.
|
||||
6. Mark the route ready and refresh registrations/routes.
|
||||
|
||||
The default timeout is 30 minutes and is configurable. A timeout leaves a
|
||||
diagnosable provisioning state. Rollback may remove only configuration created
|
||||
by that attempt and only while ownership is unambiguous. A previously verified
|
||||
route is never automatically deleted. `eager_mesh` runs the same process for
|
||||
all missing online cache/archive pairs.
|
||||
|
||||
## Archive and unarchive transfers
|
||||
|
||||
Archive and unarchive share one transfer state machine; only source/target roles
|
||||
and placement policy differ.
|
||||
|
||||
### Preflight
|
||||
|
||||
- Revalidate both resource fingerprints and placement generations.
|
||||
- Require source-selected files to be complete and regular, canonical entries.
|
||||
- Reject noncanonical qBittorrent file/folder renames in v1.
|
||||
- Confirm target path types/sizes do not conflict.
|
||||
- For an existing target, require every currently selected baseline file to be
|
||||
complete and verified. Unexpected selected-incomplete files fail as stale
|
||||
external state rather than being downloaded or silently deselected.
|
||||
- Confirm free space/reserve, permissions, sparse capability, supported
|
||||
partfile format, route health, and negotiated protocol features.
|
||||
- Export the source `.torrent` metainfo.
|
||||
- Persist the exact source, target baseline, requested selection, and delta.
|
||||
|
||||
### Five visible steps
|
||||
|
||||
1. **Source Stage/HardLink.** Create the isolated job directory and materialize
|
||||
only delta files plus required metainfo/auxiliary data. Try hard link,
|
||||
reflink, then copy, journaling every result. Write the manifest atomically
|
||||
and the ready marker last.
|
||||
2. **Syncthing.** Wait for the destination to receive the complete job prefix.
|
||||
Report job-scoped percentage and approximate speed, but do not infer success
|
||||
from a progress percentage alone.
|
||||
3. **Target Merge/Copy.** Materialize the delta beneath the configured qB root.
|
||||
Reuse same-size pre-existing regular files without overwriting them; record
|
||||
provenance so verification or rollback can distinguish them.
|
||||
4. **QB Verification.** Add a missing torrent stopped with manual management
|
||||
and explicit save path, or apply a transactional delta to an existing
|
||||
torrent. Set selected/skipped state, run a full recheck of the target union,
|
||||
and fail immediately if qBittorrent attempts to download. Successful recheck
|
||||
atomically advances the placement generation and is the commit point.
|
||||
5. **Staging Cleanup.** Remove only job-owned staging artifacts from both
|
||||
endpoints. A post-commit cleanup failure produces `CLEANUP_REQUIRED`; it
|
||||
never rolls back or deletes the committed placement.
|
||||
|
||||
Archive keeps the cache source. Unarchive always keeps the archive source. A
|
||||
new target is started only after successful verification so it can seed; an
|
||||
existing target returns to its captured stopped/running state. Archive Control
|
||||
does not copy tags, categories, rate limits, queue position, or other
|
||||
source-local metadata.
|
||||
|
||||
### Existing target merge
|
||||
|
||||
Before changing an existing target, the client durably records its selection,
|
||||
run state, qB identifier, content revision, file provenance, and partfile
|
||||
baseline. Failure before commit restores that baseline and removes only files
|
||||
created by the current job that remain unreferenced. Pre-existing paths are
|
||||
never removed. Success stores the verified union, allowing later jobs from
|
||||
other sources to add complementary file indices.
|
||||
|
||||
## Evict cache
|
||||
|
||||
Eviction is independent of transfer history and operates on one whole cache
|
||||
placement in v1.
|
||||
|
||||
1. Query the cache placement and all relevant online archive placements fresh.
|
||||
2. Build per-file archive coverage; the union may span multiple archive nodes.
|
||||
3. Disable confirmation and explain missing files if coverage is incomplete.
|
||||
4. On confirmation, reserve the resource and persist the exact cache
|
||||
fingerprint/generation plus archive proofs.
|
||||
5. Revalidate those proofs immediately before deletion.
|
||||
6. Ask qBittorrent to remove the cache entry with data deletion disabled.
|
||||
7. Snapshot exact paths referenced by all remaining local torrents.
|
||||
8. Unlink only manifest-owned files not exactly referenced by another torrent.
|
||||
9. Remove a partfile only when ownership by the evicted torrent is
|
||||
unambiguous.
|
||||
10. Attempt non-recursive `rmdir` from deepest known owned directory upward;
|
||||
retain directories containing unknown files, subdirectories, or other
|
||||
application data.
|
||||
|
||||
There is no force override in v1. Retrying any step observes prior effects and
|
||||
continues safely. Eviction never changes archive placements.
|
||||
|
||||
## Cancellation, removal, and display retention
|
||||
|
||||
- Cancelling a queued job deletes the queue record and releases its reservation
|
||||
only; no client command or file operation occurs.
|
||||
- Cancelling an active pre-commit job requests cooperative stop and runs
|
||||
attributable compensation. The source placement remains intact.
|
||||
- After commit, cancellation can only drive staging cleanup. It cannot undo the
|
||||
committed target.
|
||||
- A failed or cancelling job stays visible until its cleanup/compensation state
|
||||
is known.
|
||||
- Clear Finished and removal of a successful record affect only UI visibility.
|
||||
They never touch qBittorrent, files, routes, or placements.
|
||||
- Hidden jobs retain a minimal idempotency tombstone for 30 days by default;
|
||||
detailed events may be purged sooner under configured retention.
|
||||
Reference in New Issue
Block a user