archive-clients/docs/testing.md

Testing strategy

Tests run locally in containers and never inspect or mutate live x1, x2, lithium, or Titan services. Host tooling follows playground/ENV.md: Docker and Compose provide runtimes/build tools and write generated files as UID/GID 1001:1001.

Current happy-path coverage

The initial happy-path pass is tracked by executable capability rather than by declaring an unfinished phase complete:

As of 2026-07-23, the full control/bot suite passes 103 tests, the full client suite passes 60 tests, and a clean five-project Compose run passes automatic 2×2 route creation plus archive → covered eviction → unarchive with SHA-256 verification at both materialized targets.

Capability	Unit/contract	Component/E2E
Registration, heartbeat, replay, snapshots	Covered	Control/client integration covered
On-demand inventory and content trees	Covered	WebSocket integration covered
Automatic pairwise route provisioning	Covered	Local 2×2 topology covered
Manifest/ready-marker publication	Covered	Receiver filesystem round trip covered
Hardlink, reflink, sparse-copy fallback	Covered	Local filesystem round trip covered
Existing same-size target reuse	Covered	Local filesystem round trip covered
qB stopped add, selection, guarded recheck, start/resume, entry-only delete	Covered for 4.x/5.x semantics	Current pinned service covered
Syncthing scoped rescan and completion proof	Covered	Real route transfer covered
Transfer command state machine and qB verification	Covered	Archive and unarchive covered
Archive/unarchive placement commit	Covered	Both directions covered
Cache eviction	Coverage union, inode/path safety, idempotency covered	Real retained-source eviction covered
Telegram archive-control flows	Persistence, nonce, stale callback, stalled rendering, dummy controls/trace covered	Real handler with isolated dummy backend covered; live Telegram remains release smoke

Cancellation, rollback provenance, shared/unknown eviction paths, case-collisions, and stale preview guards now have focused tests. The broader chaos matrix remains a release gate rather than part of the initial happy-path run.

The executable complex E2E matrix additionally covers complementary partial archive placements, selective cache-placement merging, repeated placement generation after eviction, two-archive coverage union, uncovered/no-op/stale-preview rejection, exact-path sharing and unknown file retention, queued record-only cancellation, precommit rollback after target materialization, control restart after source staging, durable command replay after a client disconnect, and a no-downloaded-bytes assertion at the target. It also mutates the source selection and creates an absent target after job confirmation; both hostile races must fail before commit with the precise precondition reason while retaining the source. A wrong-size pre-existing target and an unwritable target root likewise fail without commit; the latter is wrapped in fixture cleanup that restores the original directory mode.

The second adversarial matrix also passes from clean state. It covers pure-v2 and hybrid archive/evict/unarchive round trips, including qBittorrent's truncated-v2 primary key and hybrid v1/v2 aliases; a disconnected target that becomes STALLED at 20% and resumes successfully; bounded disk exhaustion with an exact precommit failure and intact source; and duplicate/reordered protocol delivery from a separate WebSocket probe process.

The remaining Compose expansion includes supported partfiles, sparse-capability mismatch, case/type collisions, broader service-level network partitions, and database backup/restore during active recovery. Focused unit/component tests already cover several of these invariants, but they remain unchecked in the full five-project topology.

Test layers

Proto contract

• Buf format, lint, descriptor build, and breaking checks.
• Decode every canonical protobuf-JSON fixture through the descriptor.
• Golden round trips for envelopes and on-disk manifests.
• Reject duplicate JSON keys, unknown major versions, invalid hashes/ranges, malformed UUIDs/timestamps, over-limit frames, token outside registration, and unknown required commands.
• Consumer tests assert the generated binding header records the exact proto release/commit.

Unit and property tests

Control tests cover state transitions, hybrid-identity alias reservations, FIFO scheduling, placement unions, coverage-set calculation, command retries, event gap reconciliation, UI nonce idempotency, retention, and backup selection.

Client tests cover path normalization/confinement, file-index range algebra, tree expansion, root mapping, capability probes, version adapters, manifest digesting, journal replay, progress aggregation, and safe directory removal. Property/fuzz inputs emphasize path traversal, Unicode/case collisions, overlapping selections, reordered/duplicated events, and partial journals.

Adapter contract tests

Pinned qBittorrent 4.5–4.6 and 5.x and Syncthing container versions cover every supported API family. Assertions include stopped add, exact selection, export, full recheck, download-attempt detection, entry-only delete, folder/device idempotency, events fallback, need/completion proof, and redacted errors. Partfile adapters are tested only against explicitly supported qBittorrent/libtorrent fixtures.

Component integration tests

• Control plus fake clients: reconnect, duplicate ID replacement, lost ack, replay, sequence gaps, old connection fencing, queue leases, and restart.
• One client plus qB/Syncthing: inventory, staging, sparse/copy fallbacks, service restart, permission failure, and database recovery.
• Two clients plus two Syncthing services: automatic device/folder creation, bidirectional nonce verification, timeout, and existing-route discovery.

Cross-project E2E topology

The coordination repository owns uniquely named Compose projects:

e2e/
├── control/    # mogic archive-control core + test HTTP adapter
├── cache-1/    # qBittorrent + Syncthing + cache client
├── cache-2/    # qBittorrent + Syncthing + cache client
├── archive-1/  # qBittorrent + Syncthing + archive client
├── archive-2/  # qBittorrent + Syncthing + archive client
├── fixtures/
├── scenarios/
└── scripts/

Each data-node project has isolated config, state, backup, qB data, sync data, and service volumes. All join one dedicated, uniquely labeled test network. Syncthing advertises static Compose service addresses in E2E; production defaults remain dynamic. The topology deliberately starts without explicit route folders so tests exercise automatic 2×2 pairing.

Teardown resolves exact project names/labels and stops only those resources. Volume deletion is a separate explicit test action, never an unscoped prune.

Thin test HTTP adapter

The adapter is enabled only in the control E2E configuration and delegates to ArchiveControlService; it contains no orchestration logic. Initial endpoints:

Method/path	Purpose
GET /test/v1/clients	connection/health/capability view
GET /test/v1/routes	route discovery/provisioning state
GET /test/v1/resources?operation=...	on-demand eligible source listing
GET /test/v1/resources/{id}/tree?...	scoped complete tree
POST /test/v1/jobs/preview	validate and calculate exact delta/coverage
POST /test/v1/jobs	create using preview revision and idempotency key
GET /test/v1/jobs/{id}	projected state and progress
POST /test/v1/jobs/{id}/cancel	queued removal or active compensation
POST /test/v1/jobs/{id}/hide	UI/history-only removal
POST /test/v1/scheduler/pause	stop automatic advancement at a durable boundary
POST /test/v1/scheduler/advance	dispatch exactly the next eligible command
POST /test/v1/scheduler/resume	restore automatic advancement
POST /test/v1/protocol/order-probe	create and lease one synthetic ordering-test job
POST /test/v1/maintenance/backup	request and observe a test backup

JSON objects are adapter DTOs, not a second daemon protocol. Test scripts use curl, poll bounded state conditions, and dump control/client/service logs on failure. Scheduler gating is available only through the loopback, opt-in test adapter and is absent from normal daemon and Telegram workflows.

Interactive Telegram UX harness

The integrated bot has an opt-in Archive Control (Test) entry backed by a scripted in-process service. It runs the production conversation handler and planner, but never starts a control listener and never contacts a client daemon, qBittorrent, or Syncthing. Its UI sessions, scenario state, synthetic jobs, and append-only trace share one dedicated SQLite test database that configuration validation forbids from being the production control database.

The comprehensive scenario provides enough resources, targets, and retained jobs to exercise resource/job pagination, filtering, Unicode and long names, full and partial placements, selective entry previews, archive, unarchive, covered eviction, job removal, every significant displayed job state, and single-use confirmations. empty covers empty lists. route_slow keeps a newly requested route pending until an administrator explicitly marks it ready.

The existing Telegram manager allowlist and optional numeric-ID pin apply. Private-chat controls are:

/actest status
/actest reset [comprehensive|empty|route_slow]
/actest advance <job-id> <state>
/actest fail-next <backend-method> [message]
/actest route-ready
/actest trace [limit]
/actest export

advance accepts queued, preparing, running, waiting, stalled, cancelling, cleanup_required, succeeded, failed, and cancelled. fail-next injects one deterministic planner-style backend failure. Reset clears only the test sessions, jobs, scenario state, and trace. State otherwise survives bot restarts.

The SQLite trace records callback receipt and validated callback state before and after handling, text filters, complete rendered text and inline keyboards, backend calls/results/errors, test-control commands, and job transitions. trace shows a bounded recent view in chat; export sends the complete trace as JSON Lines for offline diagnosis. See the /actest operator guide for complete syntax, valid states/failure points, and a recommended review sequence.

Fixture matrix

Fixtures include small deterministic v1, v2, and hybrid torrents with:

• full, complementary, overlapping, and one-directory selections;
• same-size corrupt pre-existing targets and type/size collisions;
• nested shared directories and exact file paths shared by multiple torrents;
• unknown files/directories inside a resource-owned directory;
• supported, unsupported, and ambiguous partfiles;
• sparse files plus a simulated non-sparse target capability;
• canonical and renamed/noncanonical qBittorrent paths;
• hardlink, reflink, and copy-only filesystem conditions;
• permissions and space-reserve failures.

Acceptance scenarios

Area	Required scenarios
Archive	new target, selective target, source retained, second archive replica
Merge	complementary sources build union, overlap transfers delta only, no-op rejected, selected-incomplete target rejected
Unarchive	new cache, merge existing cache, archive always retained, repeat after prior eviction
Eviction	one-archive coverage, union coverage, missing coverage disabled, shared/unknown content retained
Routes	on-demand 2×2 creation, eager mesh, existing match, slow success, timeout, restart during provisioning
Recovery	restart every step on source/target/control, lost DB with proof, ambiguous loss fails closed
Messaging	duplicate command/event, lost ack, sequence gap, stale revision, duplicate client ID
Safety	attempted download, corrupt same-size file, symlink/path escape, special file, partfile mismatch
Operations	stall retains percent, offline wait, cancellation each phase, postcommit cleanup retry, disconnect/reconnect during every transfer step with exactly-once replay
Database	online backup under load, pre/post migration, retention, corrupt backup rejection, offline restore
UI	pagination/filter chain, stale/double callback, persisted session, clear vs evict separation

Every scenario asserts final qB selections/states, placement generations, manifests/journals, filesystem ownership/provenance, job event order, absence of unexpected received bytes, and preservation of unrelated files.

Fault injection and pass criteria

Scenario hooks pause or restart each container, disconnect network edges, duplicate/reorder protocol frames, fill a bounded test filesystem, change an external qB selection, and kill processes between intent and completion journal writes. Tests use eventual assertions with explicit deadlines; long production waits are shortened by test-only configuration without altering state logic.

A release candidate passes only when the whole matrix succeeds repeatedly from clean volumes and from retained/restarted volumes, existing mogic-bot regression tests remain green, Buf reports no unintended break, and both amd64 and arm64 client images start and pass smoke/config checks.