How Recovery Works

BanditDB survives crashes and restarts automatically. No manual intervention required.

The Tuple Lifecycle

Every decision moves through five stages, from predict() to a folded-in matrix update:

closed tuples → Parquet export · replayable · causal analysis via logged propensities (LinUCB)

The Two Files

Phase 1 — Load the Checkpoint

If checkpoint.json exists, BanditDB reads it and restores all campaign matrices directly into memory — no replaying, just deserialisation. The checkpoint records the WAL byte offset at which it was taken.

If no checkpoint exists, BanditDB starts from an empty state and replays the entire WAL from byte 0.

If a checkpoint exists but cannot be read, BanditDB falls back to checkpoint.prev. If neither is readable it refuses to start rather than coming up empty — an empty start would serve a blank database and then overwrite the evidence at the next checkpoint. Set BANDITDB_ALLOW_CORRUPT_CHECKPOINT=true to override and accept the loss.

Phase 2 — Replay the WAL Tail

BanditDB opens bandit_wal.jsonl, seeks to the checkpoint's byte offset, and replays every event written after that point. A completed checkpoint ends with WAL rotation, which rewrites the log to begin exactly at the checkpoint boundary — so the recorded offset is 0 and replay starts from the beginning of the rotated file.

Durability Guarantees

Durability is split by event type, deliberately. What you get depends on what you wrote.

RPO for acknowledged writes is zero. This is not bounded by the checkpoint interval — checkpointing controls WAL size and replay time, not durability. Only in-flight requests that never received a response are lost, which is ordinary for any database.

Predictions are weaker on purpose, and it costs less than it sounds. A lost prediction record costs the ability to match one late reward; it never affects model state. The predictions still awaiting a reward travel inside checkpoint.json, so a reward arriving after a restart still matches the prediction that preceded it.

What POST /checkpoint Does

1. Flush barrier — drains all pending events and fsyncs to disk, responds with confirmed byte offset.
2. Snapshot — serialises all campaign matrices and the predictions still awaiting a reward to checkpoint.tmp. The tmp file is fsynced, the previous generation is retained as checkpoint.prev, then the rename is made durable by fsyncing the directory.
3. Parquet export — joins Predicted + Rewarded events, writes matched pairs as a timestamped shard per campaign. Shards older than BANDITDB_EXPORT_RETAIN_SHARDS (default 50) are pruned.
4. Neural retrain (NeuralLinUCB campaigns only) — if retrain_every rewards have accumulated since the last retrain, runs Algorithm 2, re-accumulates arm matrices in the new embedding space, and saves .safetensors weights to {DATA_DIR}/neural/.
5. WAL rotation — truncates WAL to only the tail. Pre-checkpoint history is no longer needed for recovery.

Recommended Production Setup

# Auto-checkpoint every 10,000 rewards
BANDITDB_CHECKPOINT_INTERVAL=10000

# Or cap WAL size (useful on edge deployments)
BANDITDB_MAX_WAL_SIZE_MB=50

# Back up the two recovery files on a schedule
cp /data/checkpoint.json  /backup/checkpoint-$(date +%s).json
cp /data/bandit_wal.jsonl /backup/wal-$(date +%s).jsonl

To move BanditDB to a new host: copy checkpoint.json and bandit_wal.jsonl to the same DATA_DIR on the new machine and start. Recovery is automatic.