archive-clients/docs/testing.md

# Testing strategy

Tests run locally in containers and never inspect or mutate live x1, x2,
lithium, or Titan services. Host tooling follows `playground/ENV.md`: Docker and
Compose provide runtimes/build tools and write generated files as UID/GID
`1001:1001`.

## Current happy-path coverage

The initial happy-path pass is tracked by executable capability rather than by
declaring an unfinished phase complete:

As of 2026-07-23, the full control/bot suite passes 103 tests, the full client
suite passes 60 tests, and a clean five-project Compose run passes automatic
2×2 route creation plus archive → covered eviction → unarchive with SHA-256
verification at both materialized targets.

| Capability | Unit/contract | Component/E2E |
| --- | --- | --- |
| Registration, heartbeat, replay, snapshots | Covered | Control/client integration covered |
| On-demand inventory and content trees | Covered | WebSocket integration covered |
| Automatic pairwise route provisioning | Covered | Local 2×2 topology covered |
| Manifest/ready-marker publication | Covered | Receiver filesystem round trip covered |
| Hardlink, reflink, sparse-copy fallback | Covered | Local filesystem round trip covered |
| Existing same-size target reuse | Covered | Local filesystem round trip covered |
| qB stopped add, selection, guarded recheck, start/resume, entry-only delete | Covered for 4.x/5.x semantics | Current pinned service covered |
| Syncthing scoped rescan and completion proof | Covered | Real route transfer covered |
| Transfer command state machine and qB verification | Covered | Archive and unarchive covered |
| Archive/unarchive placement commit | Covered | Both directions covered |
| Cache eviction | Coverage union, inode/path safety, idempotency covered | Real retained-source eviction covered |
| Telegram archive-control flows | Persistence, nonce, stale callback, stalled rendering, dummy controls/trace covered | Real handler with isolated dummy backend covered; live Telegram remains release smoke |

Cancellation, rollback provenance, shared/unknown eviction paths,
case-collisions, and stale preview guards now have focused tests. The broader
chaos matrix remains a release gate rather than part of the initial happy-path
run.

The executable complex E2E matrix additionally covers complementary partial
archive placements, selective cache-placement merging, repeated placement
generation after eviction, two-archive coverage union,
uncovered/no-op/stale-preview rejection, exact-path sharing and unknown file
retention, queued record-only cancellation, precommit rollback after target
materialization, control restart after source staging, durable command replay
after a client disconnect, and a no-downloaded-bytes assertion at the target.
It also mutates the source selection and creates an absent target after job
confirmation; both hostile races must fail before commit with the precise
precondition reason while retaining the source. A wrong-size pre-existing
target and an unwritable target root likewise fail without commit; the latter
is wrapped in fixture cleanup that restores the original directory mode.

The second adversarial matrix also passes from clean state. It covers pure-v2
and hybrid archive/evict/unarchive round trips, including qBittorrent's
truncated-v2 primary key and hybrid v1/v2 aliases; a disconnected target that
becomes `STALLED` at 20% and resumes successfully; bounded disk exhaustion with
an exact precommit failure and intact source; and duplicate/reordered protocol
delivery from a separate WebSocket probe process.

The remaining Compose expansion includes supported
partfiles, sparse-capability mismatch, case/type collisions, broader
service-level network partitions, and database backup/restore during active
recovery. Focused unit/component tests already cover several of these
invariants, but they remain unchecked in the full five-project topology.

## Test layers

### Proto contract

- Buf format, lint, descriptor build, and breaking checks.
- Decode every canonical protobuf-JSON fixture through the descriptor.
- Golden round trips for envelopes and on-disk manifests.
- Reject duplicate JSON keys, unknown major versions, invalid hashes/ranges,
  malformed UUIDs/timestamps, over-limit frames, token outside registration,
  and unknown required commands.
- Consumer tests assert the generated binding header records the exact proto
  release/commit.

### Unit and property tests

Control tests cover state transitions, hybrid-identity alias reservations,
FIFO scheduling,
placement unions, coverage-set calculation, command retries, event gap
reconciliation, UI nonce idempotency, retention, and backup selection.

Client tests cover path normalization/confinement, file-index range algebra,
tree expansion, root mapping, capability probes, version adapters, manifest
digesting, journal replay, progress aggregation, and safe directory removal.
Property/fuzz inputs emphasize path traversal, Unicode/case collisions,
overlapping selections, reordered/duplicated events, and partial journals.

### Adapter contract tests

Pinned qBittorrent 4.5–4.6 and 5.x and Syncthing container versions cover every
supported API family. Assertions include stopped add, exact selection, export, full recheck,
download-attempt detection, entry-only delete, folder/device idempotency,
events fallback, need/completion proof, and redacted errors. Partfile adapters
are tested only against explicitly supported qBittorrent/libtorrent fixtures.

### Component integration tests

- Control plus fake clients: reconnect, duplicate ID replacement, lost ack,
  replay, sequence gaps, old connection fencing, queue leases, and restart.
- One client plus qB/Syncthing: inventory, staging, sparse/copy fallbacks,
  service restart, permission failure, and database recovery.
- Two clients plus two Syncthing services: automatic device/folder creation,
  bidirectional nonce verification, timeout, and existing-route discovery.

## Cross-project E2E topology

The coordination repository owns uniquely named Compose projects:

```text
e2e/
├── control/    # mogic archive-control core + test HTTP adapter
├── cache-1/    # qBittorrent + Syncthing + cache client
├── cache-2/    # qBittorrent + Syncthing + cache client
├── archive-1/  # qBittorrent + Syncthing + archive client
├── archive-2/  # qBittorrent + Syncthing + archive client
├── fixtures/
├── scenarios/
└── scripts/
```

Each data-node project has isolated config, state, backup, qB data, sync data,
and service volumes. All join one dedicated, uniquely labeled test network.
Syncthing advertises static Compose service addresses in E2E; production
defaults remain `dynamic`. The topology deliberately starts without explicit
route folders so tests exercise automatic 2×2 pairing.

Teardown resolves exact project names/labels and stops only those resources.
Volume deletion is a separate explicit test action, never an unscoped prune.

## Thin test HTTP adapter

The adapter is enabled only in the control E2E configuration and delegates to
`ArchiveControlService`; it contains no orchestration logic. Initial endpoints:

| Method/path | Purpose |
| --- | --- |
| `GET /test/v1/clients` | connection/health/capability view |
| `GET /test/v1/routes` | route discovery/provisioning state |
| `GET /test/v1/resources?operation=...` | on-demand eligible source listing |
| `GET /test/v1/resources/{id}/tree?...` | scoped complete tree |
| `POST /test/v1/jobs/preview` | validate and calculate exact delta/coverage |
| `POST /test/v1/jobs` | create using preview revision and idempotency key |
| `GET /test/v1/jobs/{id}` | projected state and progress |
| `POST /test/v1/jobs/{id}/cancel` | queued removal or active compensation |
| `POST /test/v1/jobs/{id}/hide` | UI/history-only removal |
| `POST /test/v1/scheduler/pause` | stop automatic advancement at a durable boundary |
| `POST /test/v1/scheduler/advance` | dispatch exactly the next eligible command |
| `POST /test/v1/scheduler/resume` | restore automatic advancement |
| `POST /test/v1/protocol/order-probe` | create and lease one synthetic ordering-test job |
| `POST /test/v1/maintenance/backup` | request and observe a test backup |

JSON objects are adapter DTOs, not a second daemon protocol. Test scripts use
curl, poll bounded state conditions, and dump control/client/service logs on
failure. Scheduler gating is available only through the loopback, opt-in test
adapter and is absent from normal daemon and Telegram workflows.

## Interactive Telegram UX harness

The integrated bot has an opt-in `Archive Control (Test)` entry backed by a
scripted in-process service. It runs the production conversation handler and
planner, but never starts a control listener and never contacts a client
daemon, qBittorrent, or Syncthing. Its UI sessions, scenario state, synthetic
jobs, and append-only trace share one dedicated SQLite test database that
configuration validation forbids from being the production control database.

The `comprehensive` scenario provides enough resources, targets, and retained
jobs to exercise resource/job pagination, filtering, Unicode and long names,
full and partial placements, selective entry previews, archive, unarchive,
covered eviction, job removal, every significant displayed job state, and
single-use confirmations. `empty` covers empty lists. `route_slow` keeps a
newly requested route pending until an administrator explicitly marks it ready.

The existing Telegram manager allowlist and optional numeric-ID pin apply.
Private-chat controls are:

```text
/actest status
/actest reset [comprehensive|empty|route_slow]
/actest advance <job-id> <state>
/actest fail-next <backend-method> [message]
/actest route-ready
/actest trace [limit]
/actest export
```

`advance` accepts queued, preparing, running, waiting, stalled, cancelling,
cleanup_required, succeeded, failed, and cancelled. `fail-next` injects one
deterministic planner-style backend failure. Reset clears only the test
sessions, jobs, scenario state, and trace. State otherwise survives bot
restarts.

The SQLite trace records callback receipt and validated callback state before
and after handling, text filters, complete rendered text and inline keyboards,
backend calls/results/errors, test-control commands, and job transitions.
`trace` shows a bounded recent view in chat; `export` sends the complete trace
as JSON Lines for offline diagnosis. See the
[`/actest` operator guide](telegram-ux.md#actest-operator-guide) for complete
syntax, valid states/failure points, and a recommended review sequence.

## Fixture matrix

Fixtures include small deterministic v1, v2, and hybrid torrents with:

- full, complementary, overlapping, and one-directory selections;
- same-size corrupt pre-existing targets and type/size collisions;
- nested shared directories and exact file paths shared by multiple torrents;
- unknown files/directories inside a resource-owned directory;
- supported, unsupported, and ambiguous partfiles;
- sparse files plus a simulated non-sparse target capability;
- canonical and renamed/noncanonical qBittorrent paths;
- hardlink, reflink, and copy-only filesystem conditions;
- permissions and space-reserve failures.

## Acceptance scenarios

| Area | Required scenarios |
| --- | --- |
| Archive | new target, selective target, source retained, second archive replica |
| Merge | complementary sources build union, overlap transfers delta only, no-op rejected, selected-incomplete target rejected |
| Unarchive | new cache, merge existing cache, archive always retained, repeat after prior eviction |
| Eviction | one-archive coverage, union coverage, missing coverage disabled, shared/unknown content retained |
| Routes | on-demand 2×2 creation, eager mesh, existing match, slow success, timeout, restart during provisioning |
| Recovery | restart every step on source/target/control, lost DB with proof, ambiguous loss fails closed |
| Messaging | duplicate command/event, lost ack, sequence gap, stale revision, duplicate client ID |
| Safety | attempted download, corrupt same-size file, symlink/path escape, special file, partfile mismatch |
| Operations | stall retains percent, offline wait, cancellation each phase, postcommit cleanup retry |
| Database | online backup under load, pre/post migration, retention, corrupt backup rejection, offline restore |
| UI | pagination/filter chain, stale/double callback, persisted session, clear vs evict separation |

Every scenario asserts final qB selections/states, placement generations,
manifests/journals, filesystem ownership/provenance, job event order, absence of
unexpected received bytes, and preservation of unrelated files.

## Fault injection and pass criteria

Scenario hooks pause or restart each container, disconnect network edges,
duplicate/reorder protocol frames, fill a bounded test filesystem, change an
external qB selection, and kill processes between intent and completion journal
writes. Tests use eventual assertions with explicit deadlines; long production
waits are shortened by test-only configuration without altering state logic.

A release candidate passes only when the whole matrix succeeds repeatedly from
clean volumes and from retained/restarted volumes, existing mogic-bot regression
tests remain green, Buf reports no unintended break, and both amd64 and arm64
client images start and pass smoke/config checks.