chore(observability): land PR #84 content on main (cherry-pick recovery) - #103
Merged
Conversation
Self-hosted log aggregation stack wired into the existing docker-compose so container stdout/stderr is scraped by promtail and pushed to loki with 7 day retention. Labels attached per stream: service (from the compose service name) and container_name. Rust service gets LOG_FORMAT=json set on the container so its tracing-subscriber emits structured JSON.
Mirror the prometheus datasource file so grafana auto-provisions loki alongside it, pointing at the in-network loki service. Pinned uid so dashboards can reference it stably.
Enable the tracing-subscriber json feature and branch on LOG_FORMAT at startup: LOG_FORMAT=json installs a structured json layer (current span included, span list omitted to keep lines compact) while any other value keeps the existing pretty text output. Production docker runs with LOG_FORMAT=json so promtail can ship structured records to loki; local cargo run stays human readable.
Install a json slog handler as the default logger in each binary entry point (executor, monitor, pooldiscovery) and convert log.Printf call sites at the module boundaries to slog.Info/Warn/Error with structured key/value fields. Boundary scope only: cmd/executor/main.go, cmd/monitor/*, cmd/pooldiscovery/main.go, and internal/risk/manager.go. Deeper utility loggers in gas_oracle, nonce, submitter, bundle, and state files keep their stdlib log for now and can be migrated later. Fatal exits are converted to slog.Error + os.Exit(1) since slog has no Fatal equivalent. Context-aware InfoContext/ErrorContext is used on the arb processing hot path so future tracing work can attach span ids.
Append a loki-backed logs panel below the existing 7 panels. Filters to aether-* services and level=(error|warn) after parsing the structured log lines, sorted newest first, with details expanded on click. The query works against both the rust tracing json layer and the go slog json handler because both emit a level field.
- migrate bundle/nonce/gas_oracle/submitter/state.go to slog so loki errors-panel stops silently dropping log.Printf / log.Fatalf - flatten_event(true) on rust json tracing so fields land at root, not under fields.* - loki healthcheck + promtail service_healthy dep to avoid cold-boot 503s - loki limits: max_streams_per_user 500, ingestion_rate 8MB, burst 16MB
This was referenced Apr 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recovery PR for PR #84. PR #84 was marked merged but landed in
feat/observability-pr1(its stacked base) instead of main because that parent branch wasn't auto-deleted after PR #83 merged. This PR brings PR #84's Loki + Promtail + structured-log content to main via clean cherry-picks onto current main.feat/observability-pr1— zero conflicts.6edd8b3, round-2 fix1a16248).tracesToLogsV2.datasourceUid: lokiwiring depends on this.Files Changed
deploy/docker/loki/loki-config.yml,deploy/docker/promtail/promtail-config.yml,deploy/docker/docker-compose.yml(loki + promtail services + healthcheck + service_healthy dep + ingestion/stream caps)deploy/docker/grafana/provisioning/datasources/loki.yml(pinneduid: loki)crates/grpc-server/src/main.rs(LOG_FORMAT=jsongate +.flatten_event(true)),Cargo.toml(tracing-subscriber json feature)cmd/executor/main.go,bundle.go,nonce.go,gas_oracle.go,submitter.go,cmd/monitor/alerter.go,dashboard.go,metrics.go,cmd/pooldiscovery/main.go,internal/risk/state.go,manager.godeploy/docker/grafana/dashboards/overview.json(Recent errors panel).env.example(documentsLOG_FORMAT)Acceptance Criteria
go test ./... -race -count=1— passgo vet ./...+go build ./...— cleancargo clippy --workspace --release --bins --tests -- -D warnings— cleanfeat/observability-pr3becomes rebaseable onto fresh mainNotes
feat/observability-pr1,feat/observability-pr2, andrecovery/pr84-to-mainto keep the branch list tidy.