feat(observability): self-hosted log aggregation (loki + structured logs)

[Pablosinyores]

Summary

PR2 of the observability series for issue #69 (WS-7.5). Stacked on top of #83. Adds self-hosted log aggregation so we can answer "why did it revert?" without ssh'ing in, and swaps Rust and Go logging to structured JSON at the entry points Promtail sees.

What's in

• Loki + Promtail services in deploy/docker/docker-compose.yml, both self-hosted on aether-net. Loki runs single-binary with filesystem storage and 168h retention; Promtail scrapes the Docker socket and attaches service + container_name labels.
• Grafana Loki datasource provisioned alongside Prometheus, pinned UID loki so dashboards can reference it stably.
• Rust tracing-subscriber JSON layer gated by LOG_FORMAT=json (set on the aether-rust container). Unset / any other value keeps the existing pretty output for local cargo run.
• Go log/slog JSON handler installed as the default logger in each binary entry (cmd/executor, cmd/monitor, cmd/pooldiscovery). log.Printf call sites at the module boundaries are converted to slog.Info/Warn/Error with structured key/value fields. internal/risk/manager.go boundary logs go through slog too (using the default logger). Deeper utility loggers in gas_oracle.go, nonce.go, submitter.go, bundle.go, state.go are intentionally left as stdlib log for now to keep the diff scoped.
• Recent errors panel appended to the Overview dashboard, querying {service=~\"aether-.*\"} | json | level=~\"(?i)(error|warn)\" with a 200-line cap, newest first.

What's out (stacked PR3 — WS-7.6 / WS-7.7)

• OpenTelemetry tracing to Tempo
• scripts/canary.py

Test plan

• go build ./... clean
• go vet ./... clean
• go test ./cmd/executor/... ./internal/risk/... ./cmd/monitor/... ./cmd/pooldiscovery/... green
• cargo check -p aether-grpc-server clean
• cargo test -p aether-grpc-server --lib green
• docker-compose.yml, loki-config.yml, promtail-config.yml, loki.yml, overview.json all parse as valid YAML/JSON
• Bring the stack up (docker compose -f deploy/docker/docker-compose.yml up -d), confirm curl http://localhost:3100/ready returns ready, confirm Grafana shows both datasources healthy and the Overview dashboard renders the new panel.
• Synthesize an error in the Go executor and confirm it shows up in the Recent Errors panel within one scrape interval.

Notes

• Targeting feat/observability-pr1 (PR feat(observability): Prometheus alerts, Grafana dashboards, Slack routing #83). Once that merges, GitHub will rebase this onto main automatically.
• The SLACK_WEBHOOK_URL env substitution for Alertmanager introduced in PR1 is left untouched; no new secrets are required for PR2.

[0xfandom]

Review — PR2 of 3 for #69 (WS-7.5 log aggregation)

Scoped to what PR2 adds on top of #83. Config is sound, the LOG_FORMAT=json gate is correct, and the Loki/Promtail wiring is consistent with the Grafana datasource + dashboard panel query.

Acceptance criteria

#	Criterion	Evidence	Verdict
1	Loki 168h retention	deploy/docker/loki/loki-config.yml:31 retention_period: 168h + reject_old_samples_max_age: 168h	Met
2	Loki schema/storage current	loki-config.yml:20-28 TSDB + schema v13 + 24h index period; allow_structured_metadata: true supported in Loki 2.9.4 (pinned at docker-compose.yml:108)	Met
3	Single-host auth off	loki-config.yml:1 auth_enabled: false	Met
4	Loki filesystem paths align with volume	chunks_directory: /loki/chunks, path_prefix: /loki, compactor working_directory: /loki/compactor all under mount loki-data:/loki (docker-compose.yml:115)	Met
5	Promtail labels match dashboard query	promtail-config.yml:21-27 emits service + container_name; dashboard query overview.json:184 is {service=~"aether-.*"} | json | level=~"(?i)(error|warn)" which matches compose service names aether-rust / aether-go	Met
6	Promtail clients → loki DNS	promtail-config.yml:10 http://loki:3100/loki/api/v1/push (not localhost)	Met
7	Promtail positions persisted	promtail-config.yml:7 → /var/lib/promtail/positions.yaml, mounted via promtail-data:/var/lib/promtail in docker-compose.yml:127	Met
8	Docker socket mounted for SD	docker-compose.yml:126 /var/run/docker.sock:/var/run/docker.sock:ro	Met
9	depends_on ordering	docker-compose.yml:128 promtail depends on loki; no cycles	Met
10	Grafana Loki datasource	deploy/docker/grafana/provisioning/datasources/loki.yml uid: loki, access: proxy, url: http://loki:3100 — matches dashboard panel datasource.uid at overview.json:179	Met
11	LOG_FORMAT=json gate correct	crates/grpc-server/src/main.rs:40-52 matches only exact "json"; any other value (including JSON, empty string, unicode, unset) falls through to the human-readable fmt() branch. Wildcard arm is the default	Met
12	tracing-subscriber json feature	Cargo.toml:43 features = ["env-filter", "json"] — added cleanly, no other deps touched	Met
13	Sensible span defaults	main.rs:44-45 with_current_span(true).with_span_list(false) — avoids per-event span-list inflation (important for Loki ingestion volume)	Met
14	slog handler set before any log line	cmd/executor/main.go:93, cmd/monitor/metrics.go:99, cmd/pooldiscovery/main.go:277 — slog.SetDefault(...) is the first statement in each main(). No init-order hazard	Met
15	risk package inherits the JSON handler	internal/risk/manager.go:149,209,318,331,353,402 uses package-level slog (= slog.Default()). Executor sets the default before constructing RiskManager at cmd/executor/main.go:168, so risk lines ship as JSON with the same aether-go Docker label the dashboard query needs	Met
16	Level regex handles slog casing	slog's JSONHandler emits "level":"INFO"/"WARN"/"ERROR" uppercase; panel regex uses (?i) at overview.json:184	Met
17	No Slack-only policy regression	PR2 does not touch alertmanager.yml or reintroduce PagerDuty/Telegram/Discord in any new config. The legacy enum in cmd/monitor/alerter.go is pre-PR2 and only logs — not a submission path	Met
18	No secrets leaked	.env.example adds only LOG_FORMAT doc comment (lines 12-15); SLACK_WEBHOOK_URL was pre-existing from PR1 and is still blank	Met
19	Backward-compatible local dev	Rust: unset LOG_FORMAT → pretty fmt branch (main.rs:49-51). Go: running go test does not invoke any main(), so the JSON handler is not installed in tests — slog.Default() stays stdlib text. Confirmed by reviewer-reported clean test runs	Met

Findings

LOW — dashboard panel coverage gap from deliberately-untouched call sites
The "Recent errors" panel filter is {service=~"aether-.*"} | json | level=~"(?i)(error|warn)". The | json parser silently drops non-JSON lines, so every remaining log.Printf in cmd/executor/submitter.go:138,141, cmd/executor/gas_oracle.go:130,159,163, cmd/executor/nonce.go:55,96,102, cmd/executor/metrics.go:252, and cmd/risk/state.go:16-17 (i.e. the sites the PR body explicitly leaves as stdlib log) will be invisible on the new panel. A few of these are exactly the signals an on-call would want to see — Bundle rejected by %s, eth_getBalance failed, Nonce sync failed. Not a blocker since you called it out as scoped-out, but worth tracking as PR3 scope or a follow-up ticket; otherwise the panel will under-report real incidents.

Also note that slog.SetDefault does not redirect the stdlib log package. A one-liner log.SetOutput(...) bridge or slog.NewLogLogger(...) would at least funnel those lines into the same JSON stream without touching every call site — consider for the next pass.

LOW — fmt.Println banners emit non-JSON after slog.SetDefault
cmd/executor/main.go:95, cmd/monitor/metrics.go:101, cmd/pooldiscovery/main.go:293 print a plain-text banner to stdout immediately after installing the JSON handler. Cosmetically inconsistent and will not parse through | json. Cheap fix: swap for slog.Info("service starting", ...) or delete.

LOW — aether-monitor is built but not started by compose
deploy/docker/go.Dockerfile:8,13 builds the aether-monitor binary but ENTRYPOINT at line 14 is aether-executor, and the aether-go compose service has no sidecar or override to launch monitor. This is pre-PR2, but it means the PR's slog conversion in cmd/monitor/*.go has no effect on Loki ingestion today. Worth noting so expectations are set; ship monitor as a separate compose service in a follow-up.

NIT — PR body lacks a linking line
Per convention the last line should reference the tracking issue. Since PR3 remains, Refs #69 (not Closes) is appropriate.

Verdict

Config is correct, the JSON gate is defensive, the Loki/Promtail/Grafana wiring lines up end-to-end, and no init-order or secret-leak hazards were found. The partial scope is a known tradeoff and properly flagged. Approving — please track the dashboard coverage gap and the monitor-container launch as follow-ups.

[0xfandom]

Re-review round 2 — coverage gap closed

Rebased onto current main (transitively through the updated PR #83). Fix commit: 6edd8b3.

High (the one that actually mattered)

#	Item	Status
1	The new "Recent errors" Grafana panel uses {service=~"aether-.*"} | json | level=~"(?i)(error|warn)" — but several page-worthy log sites in cmd/executor/ utility files were deliberately left on stdlib log.Printf/log.Fatalf. | json can't parse non-JSON lines, so level never gets populated and the panel silently drops them. Worst offender: bundle.go:99 has log.Fatalf("crypto/rand failure: ...") — process-terminating, completely invisible in the panel it's there to surface	Fixed in 6edd8b3. Migrated all stdlib log sites in cmd/executor/bundle.go, cmd/executor/nonce.go, cmd/executor/gas_oracle.go, cmd/executor/submitter.go, and internal/risk/state.go to structured slog.Info/Warn/Error. log.Fatalf("crypto/rand failure", err) → slog.Error("crypto/rand failure", "err", err); os.Exit(1). Every log line now hits the panel.

Medium

#	Item	Status
2	Rust JSON output had nested "fields": {...} instead of root-level fields, making Loki | json expose keys as fields_pool_count / fields_arb_id etc.	Fixed in 6edd8b3. Added .flatten_event(true) to the fmt().json() builder in crates/grpc-server/src/main.rs. Fields now land at the root alongside level / timestamp.
3	loki and promtail had no healthcheck / no resource caps; Promtail's first N pushes could 503 during cold boot because depends_on: [loki] waits only for start, not healthy	Fixed in 6edd8b3. Added wget-based /ready healthcheck to loki service. Converted promtail's depends_on to long form with condition: service_healthy.
4	No Loki ingestion/stream caps — default 5000 streams is room for a future high-cardinality-label regression to OOM the container	Fixed in 6edd8b3. Pinned max_streams_per_user: 500, ingestion_rate_mb: 8, ingestion_burst_size_mb: 16 in loki-config.yml. Belt-and-suspenders for the "someone adds a request_id label in six months" scenario.

Deliberately deferred (noise for this PR)

• Extract Go slog init to internal/logging/slog.go to dedupe the three main()s — follow-up PR.
• cmd/executor/metrics.go stdlib log sites — metrics path, rarely fires, out of scope here.
• .env.example em-dash → hyphen micro-regression.
• fmt.Println startup banner in main.go pre-SetDefault.
• Promtail relabel replacement micro-nit.

Validation

• go test ./... -race -count=1 — pass (4 packages)
• go vet ./... + go build ./... — clean
• cargo test -p aether-grpc-server --release — pass (106 tests)
• cargo clippy --workspace --release --bins --tests -- -D warnings — clean
• loki-config.yml, docker-compose.yml, overview.json all parse cleanly

Note on diff

The rebase onto the updated PR #83 surfaced a few whitespace-only gofmt drifts in unrelated files (struct field alignment). Those are in the diff but are pure cosmetic — they unblock the gofmt -l . check without changing semantics.

Ready for re-review.

[0xfandom]

Summary

PR #84 adds self-hosted Loki + Promtail to the compose stack, wires Grafana to query it with a pinned datasource UID, migrates the Go services to log/slog JSON output, and gates a Rust tracing-subscriber JSON layer behind LOG_FORMAT=json. Round-2 fixes landed cleanly: bundle.go:99 converted to slog.Error + os.Exit(1) (no more invisible log.Fatalf), slog migration completed in bundle/nonce/gas_oracle/submitter/state, Loki has /ready healthcheck with promtail.depends_on.loki.condition: service_healthy, and limits_config is populated with max_streams_per_user, ingestion_rate_mb, and ingestion_burst_size_mb. .flatten_event(true) is correctly in the LOG_FORMAT=json branch. The known-deferred stdlib log.Printf in cmd/executor/metrics.go is the only real gap and is called out explicitly as follow-up.

Issue #69 WS-7.5 AC table

Criterion	Status	Evidence
Self-hosted Loki	Met	deploy/docker/docker-compose.yml:107-123 — grafana/loki:2.9.4, volume-mounted config, /ready healthcheck
Promtail shipping logs	Met	docker-compose.yml:125-138; promtail-config.yml:12-27 docker SD with com.docker.compose.project filter
Rust tracing-subscriber JSON	Met	crates/grpc-server/src/main.rs:39-52 — LOG_FORMAT=json branch, .json().flatten_event(true).with_current_span(true).with_span_list(false)
Cargo feature flag	Met	Cargo.toml:43 — tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
Go slog JSON output	Met (1 known gap)	cmd/executor/main.go:91, cmd/monitor/metrics.go:99, cmd/pooldiscovery/main.go:277 — slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stderr, ...))) as first line of each main
Grafana Loki datasource	Met	grafana/provisioning/datasources/loki.yml:5 — uid: loki pinned, url: http://loki:3100
Errors panel on Overview	Met	overview.json:174-198 — type: "logs", {service=~"aether-.*"} | json | level=~"(?i)(error|warn)", maxLines: 200
Compose integration	Met	docker-compose.yml:107-138 — Loki and Promtail on aether-net, named volumes loki-data/promtail-data declared at :159-163

Round-2 fix verification

Item	Status	Evidence
slog migration complete in bundle/nonce/gas_oracle/submitter/state	Verified	bundle.go:7,9, nonce.go:5, gas_oracle.go:5, submitter.go:10, internal/risk/state.go:5 all import log/slog only; no "log" imports remain
bundle.go:99 log.Fatalf → slog.Error + os.Exit(1)	Verified	cmd/executor/bundle.go:99-101 — slog.Error("crypto/rand failure", "err", err); os.Exit(1); os imported at :9
.flatten_event(true) on JSON builder	Verified	crates/grpc-server/src/main.rs:43 — inside Ok("json") match arm, chained on .json()
Loki healthcheck	Verified	docker-compose.yml:116-120 — wget -qO- http://localhost:3100/ready | grep -q ready, 10s interval, 10 retries
Promtail depends_on long-form map	Verified	docker-compose.yml:133-135 — depends_on: loki: condition: service_healthy (map form)
Loki limits	Verified	loki-config.yml:36-38 — max_streams_per_user: 500, ingestion_rate_mb: 8, ingestion_burst_size_mb: 16

Original-scope fresh pass

• LogQL panel (overview.json:175-197): panel type: "logs" (not deprecated), datasource {type: loki, uid: loki} matches provisioning, maxLines: 200, sortOrder: "Descending", showTime: true. Query shape correct for slog + flatten_event(true) output (both emit level at root).
• Promtail labels (promtail-config.yml:20-27): only service, container_name, compose_project. All bounded by container count (~7-8). No json/regex pipeline stage. No cardinality bomb.
• Loki config: auth_enabled: false (single-tenant OK), filesystem paths under /loki mounted volume, retention_period: 168h + compactor.retention_enabled: true + retention_delete_delay: 2h + delete_request_store: filesystem — retention actually enforced, not just declared. reject_old_samples: true matches window — no silent drops.
• Rust JSON output: with flatten_event(true), emits {"timestamp", "level", "target", "message", <event fields at root>, "span":{"name"}}. Matches panel query.
• Stdlib log.* in runtime path: only cmd/executor/metrics.go:5,108,110,168,299 remain (the deliberately-deferred scope). :168 fires on big.Int→float64 precision loss in every addBigIntCounter, :299 fires on every eth_getBalance RPC failure. These bypass slog and will NOT appear in the "Recent errors" panel — known gap, acknowledged.

Must-fix blockers

None.

Should-fix nits

• S1 — cmd/monitor/alerter.go:79-102 severity mismatch. Critical/Error-class alerts are still logged at slog.Info regardless of alert.Severity. The Recent Errors panel filters level=~(error|warn) and will silently miss every dispatched critical alert. Fix:

  switch alert.Severity {
  case SeverityCritical, SeverityError:
      slog.Error("alert dispatched", ...)
  case SeverityWarning:
      slog.Warn("alert dispatched", ...)
  default:
      slog.Info("alert dispatched", ...)
  }

• S2 — docker-compose.yml:16 still has version: "3.8". Compose v2 warns on this; harmless noise on every docker compose up. Cheap delete.
• S3 — cmd/monitor/metrics.go:132-133 uses fmt.Sprintf to build a URL and pass as single url key. Preserves structure better to log host/port/endpoint separately.
• S4 — loki-config.yml:48 ruler.alertmanager_url wires Loki's ruler to shared Alertmanager despite no Loki rules existing yet. If someone later adds Loki alert rules, they'll fire to the same Alertmanager as Prom rules without any receiver routing to distinguish. Leave ruler unset or add a comment.

Can-defer (follow-up)

• D1 — Migrate cmd/executor/metrics.go off stdlib log (L108, L110, L168, L299). Precision-loss warning at L168 is especially worth capturing in Loki: silent big.Int→float64 truncation on profit/gas counters is money-adjacent signal that should be queryable.
• D2 — Remove obsolete version: "3.8" (see S2).
• D3 — Flashbots signer auth-key redaction audit. submitter.go:143 logs err from rejected bundles — if a builder echoes auth headers back in an error body, those could reach Loki.
• D4 — Consider pipeline_stages: - match: in promtail to drop debug-level lines when slog.Debug/tracing::debug! fires in prod.

Verdict

APPROVE — ship it.

Round-2 blockers are fully addressed. The one remaining stdlib log site in cmd/executor/metrics.go is scoped as a known follow-up and does not block production use of the Loki pipeline for 95%+ of runtime logs. Loki retention is actually enforced, Promtail labels are bounded, datasource UID is pinned, and the Recent Errors panel will correctly parse output from both the Go slog JSON handler and the Rust flatten_event(true) layer.