Skip to content

fix(tests): scale daemon runtime sanitizer timeout - #1742

Open
bmcnaboe wants to merge 2 commits into
DeusData:mainfrom
bmcnaboe:codex/fix-daemon-runtime-sanitizer-timeout-only
Open

fix(tests): scale daemon runtime sanitizer timeout#1742
bmcnaboe wants to merge 2 commits into
DeusData:mainfrom
bmcnaboe:codex/fix-daemon-runtime-sanitizer-timeout-only

Conversation

@bmcnaboe

Copy link
Copy Markdown

What does this PR do?

Uses sanitizer-aware timing budgets for the blocking-Git daemon cancellation regression while preserving the existing native timing budgets and cancellation/cleanup behavior.

Root cause

The ASan/UBSan runner is about 421 MB and can take longer than five seconds to reach main after being copied. The test therefore failed at its readiness backstop before exercising cancellation.

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally (make -f Makefile.cbm test)
  • Lint passes (make -f Makefile.cbm lint-ci)
  • New behavior is covered by the existing regression

Validation

  • scripts/test.sh --suites daemon_runtime: 45 passed
  • real pre-commit hook: 7,555 passed, 8 skipped
  • scripts/test.sh: 7,555 passed, 0 failed, 8 skipped
  • production watchdog and security guards passed

Fixes #1738

Signed-off-by: Brian McNaboe <bmcnaboe@gmail.com>
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@bmcnaboe
bmcnaboe marked this pull request as ready for review August 24, 2026 19:54
@bmcnaboe
bmcnaboe requested a review from DeusData as a code owner August 24, 2026 19:54
pull Bot pushed a commit to Mu-L/codebase-memory-mcp that referenced this pull request Sep 1, 2026
The gate waited 90 x 30s = 45 min for CodeQL to finish on the PR head. That
is shorter than CodeQL actually takes on this repository, so the gate has
been failing runs that had not failed.

Measured on PR DeusData#1426, head 7b72652: the CodeQL SAST workflow completed with
conclusion=success at 17:46:05, having started at 15:41:44 -- 124 minutes.
The gate step ran 16:52:58 to 17:38:44 and reported "BLOCKED: CodeQL timeout"
7 minutes and 21 seconds before the scan it was waiting for succeeded.

Two open contributor pull requests are red from exactly this: DeusData#1426 and
DeusData#1769, both with CodeQL completed=success on their head and every other
check green.

Three further PRs (DeusData#1703, DeusData#1741, DeusData#1742) are also red on codeql-gate alone,
but from a different cause: the CodeQL run on their head is
completed=cancelled, so the gate saw a non-success conclusion and correctly
exited 1 without waiting. This change does not help those and is not
intended to; they need a fresh scan, most likely having been superseded by
concurrency cancel-in-progress in codeql.yml.

300 x 30s = 150 min covers the measured 124 min with margin. The job already
declares timeout-minutes: 240, so the wait still cannot outlive its own job.
No trigger, permission or gating change: codeql-gate blocks exactly what it
blocked before, and a genuine CodeQL failure still exits 1 immediately rather
than waiting out the budget. Only the absence of a verdict waits longer.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData DeusData added maintainer-notes Internal observations and improvement notes stability/performance Server crashes, OOM, hangs, high CPU/memory priority/normal Standard review queue; useful PR with ordinary maintainer urgency. labels Sep 1, 2026
@DeusData

DeusData commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Thank you for isolating the sanitizer-specific startup budget while preserving the native timing and cancellation contracts. The contribution queue is quite full, so review may take a little time. We have the PR routed and will come back with grounded feedback as soon as possible.

@DeusData

DeusData commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Approved. Timing-budget changes to tests get read carefully here, because the usual version of this change is tuning a racy assertion until it passes. Yours is not that, and the diff shows why in one glance.

You scaled the precondition and left the measurement alone. CHILD_CANCEL_BOUND_MS (3000) and CHILD_CLEANUP_BOUND_MS (5000) are untouched — those bound the behaviour actually under test, and they stay exactly as strict as they were. Only CHILD_READY_BOUND_MS and REQUEST_TIMEOUT_MS move, and only under #if CBM_SANITIZED. That split is the whole difference between a legitimate budget and a weakened test: if the cancellation bound had moved too, this would be a different conversation.

And the comment says which is which, right where the next person will meet it: "This readiness wait is a liveness backstop, not the behavior under test." That sentence is what stops someone in six months treating all four constants as interchangeable knobs.

The root cause is a slow deterministic thing, not a racy one. A 421 MB instrumented runner being copied and then taking more than five seconds to reach main is latency, not a race — the process gets there, it just takes longer under instrumentation and more still while the parallel gate is loaded. Waiting longer for something that will definitely happen is fine; waiting longer for something that might not is what we do not do.

Using the compile-time CBM_SANITIZED discriminator rather than sniffing at runtime is right too — the budget follows the build that actually needs it.

One thing worth naming rather than changing: a 60-second readiness backstop means a genuine hang now costs 60 seconds to surface instead of 5. That is the correct trade for a liveness backstop, but it is a real cost if this ever starts failing for a different reason, and worth remembering then.

Your CI is the same story as #1741

All five reds are one cascade, and none of them is yours. security / codeql-gate ran 16:26:30 → 17:12:17 — 45 minutes 47 seconds — which is the gate's old wait budget expiring rather than a finding. lint / lint and lint / lint-mem both record The operation was canceled with zero warnings of their own, and analyze and ci-ok fall out of the same run. That is the identical signature to your other PR.

The budget was raised from 45 to 150 minutes in #1960, merged yesterday. I will update this branch and re-run before merging — main also moved three times yesterday. Nothing needed from you.

Two small, well-argued infrastructure fixes sitting two weeks behind a gate timeout that had nothing to do with either of them. Sorry about that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer-notes Internal observations and improvement notes priority/normal Standard review queue; useful PR with ordinary maintainer urgency. stability/performance Server crashes, OOM, hangs, high CPU/memory

Projects

None yet

Development

Successfully merging this pull request may close these issues.

daemon_runtime blocking-git probe times out before sanitized runner reaches main

2 participants