Executive Summary
Overall gh-aw telemetry is flowing and mostly healthy over the last 24h (project gh-aw, Sentry org github). The dominant signal is a recurring partial failure in the Smoke Copilot smoke test (3 separate runs) plus one isolated PR Sous Chef failed run. Alongside these confirmed run-level failures, there is a large cluster of long-running agent-invocation spans that end in error (up to ~17.4 min) — but those spans are orphaned from workflow identity, so their user impact is inconclusive. Several instrumentation gaps (no finish_reasons, orphaned invoke_agent traces, empty errors/logs datasets, missing release on gh-aw spans) limit confident triage and are reported explicitly rather than treated as failures.
Telemetry source: Sentry spans dataset via MCP list_events (build has no search_events/get_trace_details; validated one full trace per problem class by trace:<id> filtering). Companion errors and logs datasets were both queried and returned no results.
Top Reliability Findings
| Priority |
Workflow |
Problem |
Evidence |
Next Action |
| P1 |
Smoke Copilot (engine copilot) |
Recurring run-level failure (partial matrix: gh-aw.run.status:failure alongside success in same run) across 3 runs |
15 failure conclusion spans / 3 distinct traces, 01:10–04:36 UTC. Trace 825e04b1... = 2 failure + 3 success run.status. Runs §28993903866, §28988682885, §28986876483 |
Inspect the failing matrix cell(s) in Smoke Copilot; this is a CI health signal that repeats every run |
| P2 |
PR Sous Chef (engine pi) |
Single failed run (23 successes same day → isolated, not systemic) |
5 failure conclusion spans / 1 trace db0910e5..., run §29040669587 |
Triage the one failed run; monitor — no pattern yet |
| P2 |
[Filtered] (name scrubbed) |
Failed run whose workflow name was PII-filtered by Sentry |
5 failure spans; gh-aw.workflow.name = [Filtered] |
Confirm name in Actions; review Sentry scrubbing rule so run identity survives |
| P3 |
Unattributable (invoke_agent, model copilot/claude-sonnet-4.6) |
Many agent invocations end in error after very long durations (consistent with timeout/cancellation, not confirmed) |
≥20 error spans >60s; max 1,045,510 ms (~17.4 min), many 10–14 min. Traces c633c745..., 10a02bd3..., bc81b6b3... contain only invoke_agent spans |
Add workflow/run correlation to agent-CLI spans, then confirm whether these map to failed runs |
| P4 |
All engines |
Truncation / runaway-token detection impossible — gen_ai.response.finish_reasons never set |
has:gen_ai.response.finish_reasons → 0 results in 24h |
Emit finish_reasons unconditionally (see Notes → emit-side) |
| P5 |
Platform |
errors + logs datasets empty; release absent on gh-aw spans |
Both companion datasets → No results; has:release only matches gateway spans |
Verify error/log export path; map service.version → release |
Representative Traces
View representative traces
P1 — Smoke Copilot recurring failure (validated end-to-end, continuity intact):
- Trace
825e04b1909d944ba787ba46ba3a7578 — 40+ spans, gh-aw.pre_activation.setup + gateway.request share the trace ID (continuity OK); mixed gh-aw.run.status: 2 failure / 3 success → partial-matrix failure. Run §28993903866.
- Trace
d9f77c5824cf77ae1af5f746bc8c8569 — run §28986876483.
- Trace
de0515ea235e7abc6045c55a5a2593c7 — run §28988682885.
P2 — PR Sous Chef isolated failure:
- Trace
db0910e5a8fdec2af6ca7abc15426839, engine pi, run §29040669587.
P3 — Long agent-invocation error (orphaned trace):
- Trace
c633c7458227248514eede38b4e821b7 — a single invoke_agent span, duration 1,045,510 ms, span.status:error, model claude-sonnet-4.6; trace contains no gh-aw.*.setup/conclusion span → cannot attribute to a workflow or run outcome.
- Related:
10a02bd3721025da98ea310a6b0af09f (1,042,102 ms), bc81b6b3af758dbac245336e31faf304 (870k+ ms, repeated).
Recommendations
- Fix the recurring
Smoke Copilot matrix failure first — it fails a subset of jobs on every run (3/3 runs in the window). Smallest useful step: open the failing job in §28993903866 and identify which matrix cell reports gh-aw.run.status:failure.
- Correlate agent-CLI (
invoke_agent) spans to gh-aw runs. These orphaned traces make it impossible to say whether the ~17-min error spans caused any user-visible failure. Attach gh-aw.workflow.name / gh-aw.run.id (or a shared trace/parent) to agent spans — see actions/setup/js/send_otlp_span.cjs and aw_context.cjs.
- Always emit
gen_ai.response.finish_reasons. Today it is only set when runtime metrics include stopReason, so truncation/length outliers are undetectable. Emit a default (e.g. stop/unknown) in send_otlp_span.cjs so finish_reasons:length becomes queryable.
- Verify error/log export and
release mapping. Empty errors/logs datasets + release missing on gh-aw spans weaken incident response; map resource service.version → Sentry release.
Notes
View notes
Confirmed vs inconclusive
- Confirmed failures:
gh-aw.run.status:failure on conclusion spans — 25 spans / 5 distinct runs (Smoke Copilot 15/3 runs, PR Sous Chef 5/1, [Filtered] 5/1). Successes: 100+ spans across ~14 workflows (PR Sous Chef 23, Agentic Workflow Audit 10, etc.).
- Inconclusive: the long
invoke_agent error spans — no status.message, no finish_reasons, and no workflow correlation — so "timeout" is a hypothesis, not a confirmed outcome. Reported as inconclusive runtime outcome + confirmed instrumentation gap, not a confirmed timeout.
Tooling / backend limitations
- Sentry MCP build exposes
list_events only — no search_events or get_trace_details. Trace validation done via trace:<id> filtering (skill fallback). No server-side aggregation, so counts are from capped result pages (limit 100); some categories may be undercounted.
sort field must be included in fields (HTTP 400 otherwise). Quoted multi-word query values must be passed via stdin JSON, not shell-escaped flags.
Instrumentation gaps (cross-checked against emit-side actions/setup/js/send_otlp_span.cjs)
gen_ai.response.finish_reasons: never present (0/24h) — emit-side only sends it when runtime metrics carry stopReason.invoke_agent traces carry no gh-aw.workflow.name / gh-aw.run.status — workflow identity (gh-aw.workflow.name) and run outcome (gh-aw.run.status) are emitted on gh-aw setup/conclusion spans, which live in different traces from the agent-CLI spans → correlation break.release: not present on gh-aw workflow spans (only on gateway spans, i.e. the gateway's own release). Emit-side sends identity as resource attr service.version (2.1.201 locally); Sentry release mapping appears backend-dependent and is not applied here.errors and logs datasets returned No results for 24h — explicit observability finding, not evidence of health.- One workflow name surfaced as
[Filtered] (Sentry PII scrubbing).
Regression assessment: No historical baseline was queried; the Smoke Copilot failure is treated as a recurring current-day pattern (3 runs), not a proven regression.
References:
Generated by 🚨 Daily Reliability Review · 231.9 AIC · ⌖ 46.2 AIC · ⊞ 5.4K · ◷
Executive Summary
Overall gh-aw telemetry is flowing and mostly healthy over the last 24h (project
gh-aw, Sentry orggithub). The dominant signal is a recurring partial failure in theSmoke Copilotsmoke test (3 separate runs) plus one isolatedPR Sous Cheffailed run. Alongside these confirmed run-level failures, there is a large cluster of long-running agent-invocation spans that end inerror(up to ~17.4 min) — but those spans are orphaned from workflow identity, so their user impact is inconclusive. Several instrumentation gaps (nofinish_reasons, orphanedinvoke_agenttraces, emptyerrors/logsdatasets, missingreleaseon gh-aw spans) limit confident triage and are reported explicitly rather than treated as failures.Telemetry source: Sentry
spansdataset via MCPlist_events(build has nosearch_events/get_trace_details; validated one full trace per problem class bytrace:<id>filtering). Companionerrorsandlogsdatasets were both queried and returned no results.Top Reliability Findings
Smoke Copilot(enginecopilot)gh-aw.run.status:failurealongsidesuccessin same run) across 3 runsfailureconclusion spans / 3 distinct traces, 01:10–04:36 UTC. Trace825e04b1...= 2 failure + 3 success run.status. Runs §28993903866, §28988682885, §28986876483PR Sous Chef(enginepi)failureconclusion spans / 1 tracedb0910e5..., run §29040669587[Filtered](name scrubbed)failurespans;gh-aw.workflow.name=[Filtered]invoke_agent, modelcopilot/claude-sonnet-4.6)errorafter very long durations (consistent with timeout/cancellation, not confirmed)c633c745...,10a02bd3...,bc81b6b3...contain onlyinvoke_agentspansgen_ai.response.finish_reasonsnever sethas:gen_ai.response.finish_reasons→ 0 results in 24hfinish_reasonsunconditionally (see Notes → emit-side)errors+logsdatasets empty;releaseabsent on gh-aw spanshas:releaseonly matches gateway spansservice.version→releaseRepresentative Traces
View representative traces
P1 — Smoke Copilot recurring failure (validated end-to-end, continuity intact):
825e04b1909d944ba787ba46ba3a7578— 40+ spans,gh-aw.pre_activation.setup+gateway.requestshare the trace ID (continuity OK); mixedgh-aw.run.status: 2 failure / 3 success → partial-matrix failure. Run §28993903866.d9f77c5824cf77ae1af5f746bc8c8569— run §28986876483.de0515ea235e7abc6045c55a5a2593c7— run §28988682885.P2 — PR Sous Chef isolated failure:
db0910e5a8fdec2af6ca7abc15426839, enginepi, run §29040669587.P3 — Long agent-invocation error (orphaned trace):
c633c7458227248514eede38b4e821b7— a singleinvoke_agentspan, duration 1,045,510 ms,span.status:error, modelclaude-sonnet-4.6; trace contains nogh-aw.*.setup/conclusion span → cannot attribute to a workflow or run outcome.10a02bd3721025da98ea310a6b0af09f(1,042,102 ms),bc81b6b3af758dbac245336e31faf304(870k+ ms, repeated).Recommendations
Smoke Copilotmatrix failure first — it fails a subset of jobs on every run (3/3 runs in the window). Smallest useful step: open the failing job in §28993903866 and identify which matrix cell reportsgh-aw.run.status:failure.invoke_agent) spans to gh-aw runs. These orphaned traces make it impossible to say whether the ~17-min error spans caused any user-visible failure. Attachgh-aw.workflow.name/gh-aw.run.id(or a shared trace/parent) to agent spans — seeactions/setup/js/send_otlp_span.cjsandaw_context.cjs.gen_ai.response.finish_reasons. Today it is only set when runtime metrics includestopReason, so truncation/lengthoutliers are undetectable. Emit a default (e.g.stop/unknown) insend_otlp_span.cjssofinish_reasons:lengthbecomes queryable.releasemapping. Emptyerrors/logsdatasets +releasemissing on gh-aw spans weaken incident response; map resourceservice.version→ Sentryrelease.Notes
View notes
Confirmed vs inconclusive
gh-aw.run.status:failureon conclusion spans — 25 spans / 5 distinct runs (Smoke Copilot 15/3 runs, PR Sous Chef 5/1,[Filtered]5/1). Successes: 100+ spans across ~14 workflows (PR Sous Chef 23, Agentic Workflow Audit 10, etc.).invoke_agenterrorspans — nostatus.message, nofinish_reasons, and no workflow correlation — so "timeout" is a hypothesis, not a confirmed outcome. Reported as inconclusive runtime outcome + confirmed instrumentation gap, not a confirmed timeout.Tooling / backend limitations
list_eventsonly — nosearch_eventsorget_trace_details. Trace validation done viatrace:<id>filtering (skill fallback). No server-side aggregation, so counts are from capped result pages (limit 100); some categories may be undercounted.sortfield must be included infields(HTTP 400 otherwise). Quoted multi-wordqueryvalues must be passed via stdin JSON, not shell-escaped flags.Instrumentation gaps (cross-checked against emit-side
actions/setup/js/send_otlp_span.cjs)gen_ai.response.finish_reasons: never present (0/24h) — emit-side only sends it when runtime metrics carrystopReason.invoke_agenttraces carry nogh-aw.workflow.name/gh-aw.run.status— workflow identity (gh-aw.workflow.name) and run outcome (gh-aw.run.status) are emitted on gh-aw setup/conclusion spans, which live in different traces from the agent-CLI spans → correlation break.release: not present on gh-aw workflow spans (only on gateway spans, i.e. the gateway's own release). Emit-side sends identity as resource attrservice.version(2.1.201locally); Sentryreleasemapping appears backend-dependent and is not applied here.errorsandlogsdatasets returned No results for 24h — explicit observability finding, not evidence of health.[Filtered](Sentry PII scrubbing).Regression assessment: No historical baseline was queried; the Smoke Copilot failure is treated as a recurring current-day pattern (3 runs), not a proven regression.
References: