Observability & Production Monitoring

Know when the system is working, not just when it's running.

The Silent Failure Problem

A contextual bandit has a failure mode that produces no errors, no latency spikes, and no obvious operational signal: exploration collapse. The engine is healthy. Every prediction succeeds. Every reward matches. The WAL is flushing. But the model stopped learning days ago — one arm has absorbed all traffic and the system is silently serving a static policy.

For enterprise deployments this is a contractual and audit risk. You are paying for an adaptive system. Without explicit observability, you have no way to distinguish "the bandit has correctly converged to a winning arm" from "the bandit collapsed to a wrong arm two weeks ago and nobody noticed."

BanditDB surfaces this through three layered signals: live selection entropy per campaign, an aggregated health endpoint, and a causal validation pipeline.

Selection Entropy

Every call to GET /campaign/:id/diagnostics computes the normalised Shannon entropy of the arm selection distribution:

H = 1.0 means perfectly uniform selection across all arms. H = 0.0 means one arm receives all traffic. The normalisation by log(n_arms) makes the scale consistent regardless of how many arms a campaign has.

Entropy Status and Guards

Raw entropy is not sufficient for alerting. A campaign with H = 0.12 could be healthy (one arm has genuinely won) or broken (the model collapsed before accumulating enough data). Two guard conditions prevent false positives before a status is computed:

When neither guard fires, the status is set by thresholds on H:

Entropy Trend

BanditDB stores an entropy snapshot at every checkpoint and compares it to the current value. This distinguishes a campaign that has been collapsed for months (data quality issue) from one that collapsed last night (operational incident).

Full Diagnostics Response

A campaign with critical entropy returns a self-explanatory triage payload alongside all existing diagnostics:

curl http://localhost:8080/campaign/prices/diagnostics \
  -H "X-Api-Key: your-key"

{
  "campaign_id": "prices",
  "selection_entropy": 0.09,
  "entropy_status": "critical",
  "entropy_trend": "falling",
  "converged": false,
  "likely_cause": "recent_collapse",
  "suggested_action": "Entropy dropped since last checkpoint. Check reward pipeline for bugs or recent config changes.",
  "total_predictions": 4821,
  "total_rewards": 312,
  "arm_stats": {
    "price_10": { "predictions": 4698, "rewards": 301, "avg_reward": 0.61 },
    "price_15": { "predictions": 89,   "rewards": 8,   "avg_reward": 0.58 },
    "price_20": { "predictions": 34,   "rewards": 3,   "avg_reward": 0.55 }
  }
}

Operational Recipes

When an alert fires, the first step is always to look at the diagnostics: per-arm predictions, rewards, avg_reward, converged, and entropy_trend together identify which of five scenarios is occurring.

Health Endpoint Integration

GET /health is the public probe, designed for load balancers, uptime monitors, and Kubernetes readiness checks. It requires no authentication and returns overall status only:

curl http://localhost:8080/health

{ "status": "ok", "version": "2.0.0", "features": ["neural"] }

Per-campaign entropy moved to GET /health/detail in 2.0.0 and requires a reader key. The campaign identifiers it returns carry the tenant prefix in multi-tenant deployments, so serving them from an unauthenticated endpoint would have exposed the tenant list. Results are scoped to the caller's tenant.

curl -H "X-Api-Key: $KEY" http://localhost:8080/health/detail

{
  "status": "degraded",
  "version": "2.0.0",
  "campaigns": {
    "prices":        { "entropy": 0.09, "status": "critical" },
    "recommendations": { "entropy": 0.71, "status": "ok" },
    "onboarding":    { "entropy": 0.34, "status": "warning" }
  }
}

The deliberate choice of HTTP 200 for entropy degradation — rather than 503 — means a k8s readiness probe or load balancer health check will not remove the instance from rotation just because a campaign's model quality has degraded. Service availability and model quality are separate failure modes and should be monitored separately.

Kubernetes

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

The k8s probe will mark the pod not-ready only when the WAL writer fails (503). Campaign-level entropy degradation returns 200 and does not affect routing — which is the correct behaviour: a degraded model is still serving predictions, and removing the pod from rotation does not fix the underlying cause.

Structured logging

When entropy_status is warning or critical, BanditDB emits a structured WARN log line at diagnostics time. Set LOG_FORMAT=json to receive machine-parseable output compatible with CloudWatch, Datadog, Splunk, and most log aggregation pipelines:

{
  "level": "WARN",
  "campaign": "prices",
  "entropy": "0.091",
  "trend": "Falling",
  "status": "Critical",
  "likely_cause": "recent_collapse",
  "message": "entropy: low selection entropy detected"
}

An alert rule on message = "entropy: low selection entropy detected" and level = WARN is sufficient to wire this into any log-based alerting system without code changes.

Prometheus Metrics

GET /metrics returns Prometheus text format. Since 2.0.0 it requires an API key by default — the output names campaigns and arms, which is tenant-identifying. Set BANDITDB_METRICS_PUBLIC=true to expose it anonymously.

Four of these exist specifically to surface failures that are otherwise silent — the service keeps returning 200s while something degrades:

Two things deliberately have no alert. The fsync interval is measured to be nearly irrelevant across a 20× range, because the writer also syncs whenever it goes idle. And checkpoint frequency controls WAL size and replay time, not durability — acknowledged writes are already on disk.

Causal Validation

Entropy alerting tells you whether the bandit is still exploring. Causal analysis tells you whether the exploration it did was causally correct — whether the arm receiving the most traffic is actually causing better outcomes, or whether it is merely correlated with outcomes that were already good for that user segment.

Standard OPE estimators (IPS, SNIPS, Doubly Robust) answer the question: "what reward would policy π achieve?" Causal analysis answers a different question: "what is the causal effect of each arm, controlling for the fact that the bandit was selecting them non-randomly based on context?"

The distinction matters in production. A bandit can correctly route high-converting users to arm A — not because arm A causes conversions, but because arm A was selected for users who already convert well. IPS estimators will confirm arm A is performing well. Causal analysis will correctly identify that the arm's observed advantage is partially or fully explained by user selection bias.

Running the analysis

# Checkpoint first to flush the latest data to Parquet
curl -X POST http://localhost:8080/checkpoint -H "X-Api-Key: your-key"

# Run causal analysis
pip install econml scikit-learn polars pandas numpy
python scripts/causal_analysis.py \
  --parquet /data/exports/prices.parquet \
  --features price_sensitivity recency_days cart_value_norm segment_score

The script works with LinUCB, Thompson Sampling, and Progressive campaigns. Thompson Sampling campaigns now log per-prediction propensities via adaptive Monte Carlo sampling (N=8–64 draws per arm, driven by posterior spread). The causal estimator (CausalForestDML) does not use these logged propensities — it learns the selection propensity internally from observed (arm, context) pairs using Double Machine Learning, which is more appropriate for aggregated bandit data than using time-varying logged propensities from a non-stationary policy. The logged TS propensities are used by the IPS/SNIPS offline evaluator in banditdb.eval.

Section 0 — Positivity & Confounding Diagnostics

This section runs before the expensive causal forests and checks whether the DML validity assumptions hold. It is the causal analysis equivalent of the entropy alert: a positivity violation means the bandit never selected a given arm for some region of the feature space, making causal effect estimates for that arm in that region extrapolation rather than inference.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
0. POSITIVITY & CONFOUNDING DIAGNOSTICS
   (pre-flight check — run before fitting causal forests)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Arm                               AUC    P<0.05   P>0.95
  ────────────────────────────────  ─────  ───────  ───────
  price_10                          0.812    2.1%     1.8%
  price_15                          0.831    1.4%     0.9%
  price_20                          0.744   34.2%     0.3%   ⚠ positivity violation

  ⚠  Positivity violations often indicate exploration collapse.
     Check GET /campaign/:id/diagnostics for entropy_status.

The AUC of the internal propensity model (model_t) quantifies how strongly context predicts arm selection. AUC ≈ 0.5 means near-random selection — the bandit was exploring freely and causal estimates are highly reliable. AUC > 0.8 means strong confounding — the DML is doing important correction work but estimates carry more uncertainty because few users in the "wrong" arm condition exist.

P<0.05 is the fraction of observations where the model predicts near-zero probability of this arm being selected. When this exceeds 20%, the arm was almost never chosen in that feature region and the CATE estimate there is unreliable.

Section 2 — Causal Assignment vs Bandit Selection

The most operationally important output. The causal forest identifies, for each user in the dataset, which arm would causally produce the best outcome. This is compared to what the bandit actually selected.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
2. CAUSAL ASSIGNMENT vs BANDIT SELECTION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Arm                              Causal   Bandit     Gap
  ────────────────────────────────  ───────  ───────  ───────
  price_10                          42.1%    14.2%   +27.9%  ████████████████████
  price_15                          51.3%    79.4%   -28.1%  █████████████████████████
  price_20                           6.6%     6.4%    +0.2%  ███

Section 5 — Selection Stability Over Time

Splits the campaign timeline into five equal-sized buckets (by prediction timestamp) and shows per-arm selection rate across the full campaign history. This surfaces convergence, collapse, and pipeline events directly from the Parquet export without requiring access to the live diagnostics endpoint.

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
5. SELECTION STABILITY OVER TIME
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  t1(961)  t2(961)  t3(961)  t4(961)  t5(961)

  price_10   ░░▒▒▒  [ 18.2%  22.1%  28.4%  31.7%  35.9%]
  price_15   ████▓  [ 64.1%  59.3%  55.2%  52.8%  47.3%]
  price_20   ▒░░░·  [ 17.7%  18.6%  16.4%  15.5%  16.8%]

  Stable across buckets  → IID assumption holds, DML estimates reliable
  Monotonic increase     → bandit converging or collapsing to this arm
  Sudden jump in t4/t5   → possible reward pipeline event or config change

The example above shows healthy convergence in progress: price_15 is steadily losing traffic to price_10 as the model learns. The DML IID assumption is not perfectly satisfied (the policy is non-stationary) but the gradual shift means causal estimates over the full history are a reasonable approximation of the average policy. A sudden jump in t4 or t5 — rather than a gradual drift — would indicate a pipeline or configuration event and should prompt investigation before trusting the causal estimates.

Interpreting the full output