SentinelAI — Reproducible Drift-Monitoring Reference System

SentinelAI is a multi-service observability prototype for AI systems. Its directly implemented statistical component compares an expected and an observed histogram with Population Stability Index (PSI) and a Kolmogorov–Smirnov (KS) CDF distance, then raises a drift flag when either configured threshold is crossed.

The Go ingestion service can optionally mirror accepted inference telemetry to Amazon Data Firehose, which Terraform configures to deliver GZIP-compressed NDJSON objects into a private, versioned, encrypted Amazon S3 telemetry bucket. This AWS path is disabled by default for local development and is a best-effort observability mirror, not a transactional dual-write guarantee.

The versioned evidence measures a portable Python reference of the drift decision rule on seeded 32-bin synthetic histograms—not native C++ execution, HTTP latency, concurrent service load, Firehose/S3 throughput, or production drift-detection accuracy. The benchmark is a repeatable regression signal, not a claim of real-world model quality.

Let $p=(p_1,...,p_B)$ and $q=(q_1,...,q_B)$ be expected and actual histogram-bin weights. The C++ engine computes

[ \operatorname{PSI}(p,q) = \sum_{i=1}^{B}(q_i-p_i)\log\left(\frac{q_i}{p_i}\right), ]

summing only bins where both values are positive. It also forms normalized cumulative distributions:

[ P_k=\sum_{i=1}^{k}\frac{p_i}{\sum_jp_j},\qquad Q_k=\sum_{i=1}^{k}\frac{q_i}{\sum_jq_j},\qquad \operatorname{KS}(p,q)=\max_{1\leq k\leq B}|P_k-Q_k|. ]

The implementation emits a drift event when

[ \operatorname{drift}(p,q)=[\operatorname{PSI}(p,q)>0.20]\lor[\operatorname{KS}(p,q)>0.10]. ]

This logic is implemented in drift-engine/drift_engine.cpp and mirrored by benchmarks/run_benchmark.py. The benchmark supplies normalized positive bins; the service does not add smoothing to zero-valued PSI bins, so zero handling is an explicit limitation of the present implementation.

For $B$ bins, PSI and KS each make one linear pass: $O(B)$ time and $O(1)$ auxiliary working memory (apart from the input vectors). The calculation is not a hypothesis test: the two thresholds are fixed decision settings, not calibrated p-values.

The full protocol, environment (CPython 3.12.13 on Windows 11), raw confusion matrix, and limitations are versioned in benchmarks/latest.json and explained in benchmarks/benchmark_report.md. Reproduce the artifact with:

python benchmarks/run_benchmark.py --output benchmarks/latest.json

1. Under a leakage-free, labeled production dataset, how well do the fixed PSI/KS thresholds detect meaningful distribution shift?
2. How sensitive are false positives and false negatives to bin count, threshold selection, and zero-bin handling?
3. How does native C++ and end-to-end service latency scale with histogram size and concurrent requests?
4. Which operational metrics best distinguish feature drift from changes in data volume or service latency?

This section is the portfolio audit entry point for SentinelAI. It describes an engineering promotion path; it is not a claim that the repository is already production-authorized.

flowchart LR
    Source --> Build[Release binary] --> Tests[Unit + sanitizer tests] --> Artifact[Versioned artifact]

The repository uses Python services, a Go ingestion service, and a small C++ drift executable. Reproduce the portable evidence or build the statistical engine directly:

python benchmarks/run_benchmark.py --output benchmarks/latest.json
g++ -std=c++17 drift-engine/drift_engine.cpp -o drift-engine/drift_engine

If the project uses external services, model artifacts, cloud credentials, or private data, start them through documented local fixtures or mocks. Never place secrets or identifiable records in the repository.

A benchmark number belongs in a versioned artifact tied to a commit and hardware/runtime description. Engineering benchmarks must not be presented as clinical, financial, safety, or model-quality validation without the appropriate domain evidence.

What is production-ready for this repository?
A reproducible build, tested public contract, controlled configuration, observable runtime, documented security boundary, versioned artifacts, and a tested rollback path.

What must remain explicit?
The intended use, excluded use, data/credential handling, model or algorithm limitations, and which metrics are measured versus aspirational.

What should be completed next?
Use the linked production-readiness issue for this repository as the checklist. Resolve missing tests, deployment instructions, observability, supply-chain controls, and release evidence before attaching a production claim.

SentinelAI separates primary inference paths from telemetry and evaluation layers in its local architecture.

[ Incoming User Query ] ───► [ Async Proxy Gateway ] ───► [ Downstream Application ]
                                      │
                         (Non-Blocking Telemetry Mirror)
                                      ▼
                    ┌──────────────────────────────────────┐
                    │    SentinelAI Asynchronous Engine    │
                    ├──────────────────────────────────────┤
                    │  • Parallelized Guardrail Evaluation │
                    │  • LLM SRE Diagnostics               │
                    │  • Token Cost & Allocation Trackers  │
                    └──────────────────┬───────────────────┘
                                       ▼
                         [ Streamlit Observability Control Plane ]

Prerequisites: Docker 24+ with Compose v2 (docker compose version).

# 1. Copy environment defaults
cp .env.example .env

# 2. Start the full local stack
docker compose up --build

Once running, open:

curl -X POST http://localhost:8080/log \
  -H "Content-Type: application/json" \
  -d '{"model_id":"demo","model_version":"v1","latency_ms":120,"tokens_in":32,"tokens_out":64,"status":"ok"}'

curl -X POST http://localhost:7070/drift \
  -H "Content-Type: application/json" \
  -d '{"model_id":"demo","feature_name":"latency","expected":[0.2,0.3,0.25,0.25],"actual":[0.1,0.35,0.30,0.25]}'

curl -X POST http://localhost:8000/summarize \
  -H "Content-Type: application/json" \
  -d '{"log_data":"PSI 0.35 on latency feature, model demo v1","persist":false}'

All configuration is via environment variables. Copy .env.example to .env and adjust.

Snowflake (optional): set WAREHOUSE_MODE=snowflake and fill in SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_PASSWORD, SNOWFLAKE_DATABASE, SNOWFLAKE_SCHEMA, SNOWFLAKE_WAREHOUSE.

The host-facing ingestion endpoint is an NGINX gateway at port 8080; Go ingestion replicas are internal-only and can be started with docker compose up --build --scale ingestion-service=3. /health is liveness and /ready includes the Postgres dependency; the CI smoke test checks gateway readiness, multi-replica routing, and continued readiness after one replica stops. The optional EXPOSE_INSTANCE_ID=true setting exists only for that test and is disabled by default.

When FIREHOSE_ENABLED=true, a bounded worker mirrors accepted inference events to Amazon Data Firehose after the primary warehouse path succeeds. Firehose queue pressure or AWS delivery errors do not change /ready and do not fail an otherwise accepted primary ingestion request; they are surfaced through Prometheus metrics and logs.

User → NGINX Ingestion Gateway (8080) → Go Ingestion Replicas → Postgres (local) / Snowflake (optional)
                                        │
                                        ├──► bounded Firehose queue ──► Amazon Data Firehose ──► Amazon S3
                                        │                                  │
                                        │                                  └──► CloudWatch delivery logs
                                        ↓
                              Drift Engine C++ (7070)
                                        ↓
                               LLM Guard Python (8000)
                                        ↓
                           Streamlit Dashboard (8501)
                                        ↓
                         Prometheus (9090) + Grafana (3000)

The terraform/ configuration defines an AWS telemetry path that can be provisioned independently of local Docker Compose:

flowchart LR
    App[Model / application] --> Gateway[NGINX]
    Gateway --> Go[Go ingestion replicas]
    Go --> Warehouse[(Postgres / Snowflake path)]
    Go -. bounded fail-open mirror .-> Queue[In-memory queue]
    Queue --> Firehose[Amazon Data Firehose]
    Firehose --> S3[(Private S3 telemetry bucket)]
    Firehose --> CW[CloudWatch delivery logs]

Terraform defines a private S3 bucket with public-access blocking, versioning, SSE-S3 encryption and a configurable retention policy; a Firehose stream with 60-second / 5-MiB buffering, GZIP compression and time-partitioned S3 prefixes; CloudWatch delivery logging; a Firehose service role; and a separate least-privilege writer policy for the SentinelAI workload.

cd terraform
terraform init
terraform fmt -check
terraform validate
terraform plan

Do not commit AWS access keys. The Go producer uses the AWS SDK default credential chain; for an EKS deployment, attach the Terraform firehose_writer_policy_arn output to the ingestion workload identity rather than injecting long-lived credentials.

Producer-side observability is exposed through:

ingestion_firehose_records_total{status="queued"}
ingestion_firehose_records_total{status="delivered"}
ingestion_firehose_records_total{status="error"}
ingestion_firehose_records_total{status="dropped"}

The repository CI validates the AWS integration code without requiring AWS credentials:

• go mod tidy must produce no module-file drift.
• go test ./... validates Firehose configuration and NDJSON PutRecord behavior against a fake client.
• terraform fmt, terraform init -backend=false, and terraform validate verify the IaC syntax and provider graph.
• The normal Docker Compose CI continues to exercise the local multi-replica ingestion path with Firehose disabled by default.

CI does not run terraform apply, create billable AWS resources, or prove live Firehose-to-S3 delivery. Live AWS throughput, durability, retry behavior, IAM attachment, and end-to-end delivery latency remain deployment-level validation work.

This path is implemented as a best-effort telemetry mirror. The queue is bounded and in-memory, process termination can lose queued mirror records, and SDK retries may create duplicates. See docs/aws-firehose-s3.md for deployment, IAM, failure semantics, observability, and cost controls.

The committed baseline is generated by a seeded, dependency-free harness that mirrors the PSI/KS decision rule in the C++ drift engine. These numbers measure the Python reference implementation—not native C++ or end-to-end HTTP latency. See the full methodology, interpretation, and limitations and raw JSON evidence.

python benchmarks/run_benchmark.py --output benchmarks/latest.json

CI reruns the benchmark on every pull request, validates its schema and F1 regression floor, and uploads raw evidence for 30 days. For comparable hosts, median or P95 increases above 15% require investigation and a documented baseline update.

The perfect synthetic classification result is a regression signal for deliberately separated perturbation classes; it is not a production accuracy claim. Native C++, service concurrency, network, warehouse, GPU, Firehose/S3 throughput, and real-world labeled drift benchmarks remain future evaluation layers.

• Add an ephemeral live-AWS integration test that proves Firehose delivery into a disposable S3 prefix without turning normal PR CI into a billable deployment.
• Add PutRecordBatch batching and measured queue/backpressure benchmarks before claiming higher producer throughput.
• Add a durable local spool or explicit replay mechanism for mirror records that cannot be lost on process termination.
• Attach the writer policy through an EKS workload identity path and validate least-privilege IAM end to end.
• Add Athena/Glue-compatible telemetry schemas only after a concrete query workload and reproducible evidence exist.