cabbage/archive-clients
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
v0.1.2
archive-clients/docs/deployment-and-usage.md
T

8.9 KiB
Raw Permalink Blame History

Configuration, deployment, and usage

Configuration rules

Both daemons parse strict TOML: unknown keys, duplicate keys, invalid types, unsafe relative paths, and missing required secrets fail startup. Effective precedence is command line, environment, TOML, then documented default. Secret fields support a direct environment value or a *_file path, never both. Environment interpolation is explicit ${NAME} syntax; an unset value fails.

--print-effective-config emits a validated, fully resolved configuration with all secrets redacted. Durations accept a documented unit string. Paths shown to the APIs and paths visible inside the daemon are configured as pairs and tested for consistent mapping before registration.

Important defaults:

  • registration deadline 10s;
  • heartbeat interval 15s, offline timeout 45s;
  • unacknowledged command retry timeout 15s, three total attempts;
  • reconnect 1s initial, 60s cap, jitter enabled;
  • route policy on_demand, setup timeout 30m;
  • stall warning 30m, with no automatic failure timeout;
  • one active data job per client and route;
  • Telegram coalesce window 5s, UI session TTL 24h;
  • tombstone retention 30d;
  • database backup interval 6h, retention 12/14/8.

Control configuration example

[archive_control]
enabled = true
listen = "127.0.0.1:8765"
websocket_path = "/archive_control"
shared_token_file = "/run/secrets/archive_control_token"
database = "/var/lib/mogic/archive-control.db"
backup_dir = "/var/backups/mogic/archive-control"
registration_timeout = "10s"
heartbeat_interval = "15s"
offline_timeout = "45s"
command_ack_timeout = "15s" # Delay before resending the same command ID.
command_max_attempts = 3 # Total sends, including the initial attempt.
stall_after = "30m" # Warning state only; jobs continue waiting.
route_policy = "on_demand" # Alternative: eager_mesh.
route_setup_timeout = "30m"
max_active_per_client = 1
max_active_per_route = 1
max_envelope_bytes = 1048576
inventory_chunk_bytes = 524288

[archive_control.backup]
interval = "6h"
recent = 12
daily = 14
weekly = 8

[archive_control.telegram]
progress_coalesce = "5s"
session_ttl = "24h"
tombstone_retention = "30d"
# Existing username allowlist remains authoritative when no ID is pinned.
admin_user_ids = { "@alice" = 123456789 }

The test-only HTTP adapter has a separate test_http.enabled flag that is false by default and omitted from production Compose configuration.

[archive_control.test_http]
enabled = false # Enable only in the isolated local E2E control stack.
listen = "127.0.0.1:0" # Non-loopback binds are rejected.
max_body_bytes = 1048576

Client configuration examples

All nodes use the same image. The examples show two active cache and two active archive nodes. Values are illustrative; credentials are mounted secrets.

# cache-1.toml
client_id = "cache-1"
display_name = "Cache 1"
role = "cache"
control_endpoint = "ws://control:8765/archive_control"
shared_token_file = "/run/secrets/archive_control_token"
state_db = "/var/lib/archive-control/client.db"
backup_dir = "/var/backups/archive-control"

[connection]
registration_timeout = "10s" # First response must arrive within this window.
heartbeat_interval = "15s" # Server may negotiate a different effective value.
offline_timeout = "45s"
reconnect_initial = "1s"
reconnect_max = "60s" # Retry forever, never wait longer than this.
reconnect_reset_after = "60s"
reconnect_jitter = true

[jobs]
stall_after = "30m" # Warning only; no automatic job failure.
verification_timeout = "30m" # Stopped qB full-recheck deadline.
poll_interval = "1s" # Active qB/Syncthing observation interval.
# Worst-case copy fallback must leave this many bytes free.
free_space_reserve_bytes = 1073741824

[backup]
interval = "6h"
recent = 12
daily = 14
weekly = 8

[qbittorrent]
endpoint = "http://qbittorrent:8080"
username = "${QB_USER}"
password_file = "/run/secrets/qb_password"
api_root = "/downloads"
local_root = "/data/qb"

[syncthing]
endpoint = "http://syncthing:8384"
api_key_file = "/run/secrets/syncthing_api_key"
api_root = "/sync"
local_root = "/data/sync"
advertised_addresses = ["dynamic"]

The remaining node examples omit optional [connection], [jobs], and [backup] tables and therefore use these same defaults; deployments may override them per node.

# cache-2.toml: same service layout, distinct stable identity/state volumes.
client_id = "cache-2"
display_name = "Cache 2"
role = "cache"
control_endpoint = "ws://control:8765/archive_control"
shared_token_file = "/run/secrets/archive_control_token"
state_db = "/var/lib/archive-control/client.db"
backup_dir = "/var/backups/archive-control"

[qbittorrent]
endpoint = "http://qbittorrent:8080"
username = "${QB_USER}"
password_file = "/run/secrets/qb_password"
api_root = "/downloads"
local_root = "/data/qb"

[syncthing]
endpoint = "http://syncthing:8384"
api_key_file = "/run/secrets/syncthing_api_key"
api_root = "/sync"
local_root = "/data/sync"
advertised_addresses = ["dynamic"]

# archive-1.toml
client_id = "archive-1"
display_name = "Archive 1"
role = "archive"
control_endpoint = "ws://control:8765/archive_control"
shared_token_file = "/run/secrets/archive_control_token"
state_db = "/var/lib/archive-control/client.db"
backup_dir = "/var/backups/archive-control"

[qbittorrent]
endpoint = "http://qbittorrent:8080"
username = "${QB_USER}"
password_file = "/run/secrets/qb_password"
api_root = "/archive"
local_root = "/data/archive"

[syncthing]
endpoint = "http://syncthing:8384"
api_key_file = "/run/secrets/syncthing_api_key"
api_root = "/sync"
local_root = "/data/sync"
advertised_addresses = ["dynamic"]

# archive-2.toml
client_id = "archive-2"
display_name = "Archive 2"
role = "archive"
control_endpoint = "ws://control:8765/archive_control"
shared_token_file = "/run/secrets/archive_control_token"
state_db = "/var/lib/archive-control/client.db"
backup_dir = "/var/backups/archive-control"

[qbittorrent]
endpoint = "http://qbittorrent:8080"
username = "${QB_USER}"
password_file = "/run/secrets/qb_password"
api_root = "/archive"
local_root = "/data/archive"

[syncthing]
endpoint = "http://syncthing:8384"
api_key_file = "/run/secrets/syncthing_api_key"
api_root = "/sync"
local_root = "/data/sync"
advertised_addresses = ["dynamic"]

No explicit route matrix is configured. Clients discover existing compatible folders, report their Syncthing device ID, and control creates missing cache/archive routes according to policy.

Client Compose example

services:
  archive-client:
    image: sodium/archive-clients:v0.1.2
    user: "1001:1001"
    restart: unless-stopped
    command: ["archive-client", "--config", "/etc/archive-control/client.toml"]
    environment:
      QB_USER: admin
    volumes:
      - ./client.toml:/etc/archive-control/client.toml:ro
      - ./secrets:/run/secrets:ro
      - ./state:/var/lib/archive-control
      - ./backups:/var/backups/archive-control
      - ./qb-data:/data/qb
      - ./sync-data:/data/sync

Archive nodes mount /data/archive instead of /data/qb. Mounts must line up with the configured local roots and the service containers' API-visible roots. The daemon runs unprivileged; directory ownership is prepared by the operator, and mismatch causes fail-fast startup.

Image and release policy

The production client image contains application source and dependencies; it does not bind-mount source code. A multi-stage Dockerfile uses a slim Python base and removes build-only packages/caches. Buildx publishes sodium/archive-clients:<semver> for linux/amd64 and linux/arm64, records the multi-platform digest, and may update latest. Deployments pin the immutable semantic tag or digest.

Generated protobuf Python bindings are committed in each consumer with the exact archive-control-proto tag/commit recorded. Release order is proto, control consumer, client consumer, E2E, then image publication.

Operator usage

  1. Prepare local qB, sync, state, backup, config, and secret mounts with the container UID/GID (default example 1001:1001).
  2. Run a config validation/effective-config command before starting.
  3. Start clients. Confirm service health, sparse/root probes, and registration.
  4. Allow route discovery/provisioning to complete when first needed.
  5. Use Telegram Archive/Unarchive; review the exact source, target, and selection before confirming.
  6. Use Evict Cache separately only when its fresh coverage check is enabled.
  7. Treat orange Cleanup Required as a committed transfer needing staging cleanup; do not manually remove the target placement.
  8. Verify database backups periodically with the offline tooling.

The relevant runtime must be stopped before restore. The control maintenance entry point and the client's archive-client-backup command support list, verify <backup>, and restore <backup>; restore verifies the copy, preserves the replaced database and SQLite sidecars with a timestamped suspect name, and installs the selected database atomically.

Reference in New Issue View Git Blame Copy Permalink