REGRESSION: POST /amicode/connections/credential (pasqal-cloud) segfaults Bun — bisected to 467fb15cf #82

Activity

1. jack-champagne

   changed the title [-]unit suite dies with exit 137 — one test segfaults Bun 1.3.14 (not OOM)[/-] [+]REGRESSION: POST /amicode/connections/credential (pasqal-cloud) segfaults Bun — bisected to 467fb15cf[/+] on Jul 28, 2026
2. jack-champagne commented on Jul 28, 2026

   

   jack-champagne

   on Jul 28, 2026

   MemberAuthor

   More actions

   Update: reproduces with a REAL python3 — this is production-relevant

   The earlier version of this issue left open whether the crash was an artifact of the test harness substituting bun for the Python interpreter (AMICO_PYTHON = process.execPath). It is not.

   I rewrote stageStubValidator()'s stub from a bun-executed .mjs into an equivalent python3-executed .py (same record/scenario contract) and set AMICO_PYTHON = "python3" — i.e. the real interpreter, as production uses:

   Elapsed: 1408ms | RSS: 109.38MB | Peak: 0.36GB | Faults: 1 | Machine: 32.38GB
   panic(main thread): Segmentation fault at address 0x0
   

   Still crashes. The interpreter was never the variable.

   Control — the python3 stub is sound

   Four other tests in the file, run against the same new python3 stub, all pass:

   test	result
   needs-key → submit → connected → revalidate → disconnect …	1 pass
   null token over the route → session-only connected (AC4)	1 pass
   offline boot over the route …	1 pass
   expiry over the route: a past-expiry claim …	1 pass

   So the stub spawns, records, and exits correctly. The fault is specific to the one test.

   Narrowed: it is the SUCCESS path of the rewritten submit

   Three tests post a full pasqal-cloud credential ({ id: "pasqal-cloud", ...PASQAL }):

   line	validator scenario	result
   342 (full lifecycle)	scenario(0, validatorLine()) — exit 0, full valid line: devices + mintable token + expiry	CRASH
   401 (failure classes)	exit 2 / 4 / 1 — failure classes	pass
   422 (null token)	exit 0, null token	pass

   The only crashing case is the one where the validator succeeds with a complete payload — devices list plus a token to mint. That is precisely the path 467fb15cf rewrote (project selection → token extraction → device list).

   Also ruled out

   • process.env.AMICODE_ROUTE_CANARY — removing it does not stop the crash.
   • Interpreter choice (bun vs python3) — see above.
   • return inside try/finally — hoisting it out does not stop the crash.

   Consequence

   v1.17.3-amicode.10 should not be cut until this is understood. The crash is on the production submit path with the production interpreter, reachable by any user who successfully authenticates a Pasqal account. Whether a real pasqal_validate.py against the live SDK also trips it is untested, but there is no longer any reason to assume the harness is what makes it crash.

   The failing test is doing its job and should stay unskipped.
3. jack-champagne commented on Jul 28, 2026

   

   jack-champagne

   on Jul 28, 2026

   MemberAuthor

   More actions

   Confirmed against Pasqal production with real credentials — network ruled out

   Ran the route against a real Pasqal login, the real pasqal_validate.py, and the real pasqal-cloud SDK. It segfaults.

   3/3 consecutive runs, tightly clustered:

   attempt	elapsed	peak RSS	result
   1	3884 ms	0.38 GB	panic: Segmentation fault at address 0x0
   2	3703 ms	0.37 GB	panic: Segmentation fault at address 0x0
   3	3596 ms	0.36 GB	panic: Segmentation fault at address 0x0

   Machine has 32 GB. Not memory, not intermittent.

   The network hypothesis is dead

   Suspected a connectivity flake. Ruled out three ways:

   1. Connectivity is healthy — apis.pasqal.cloud (the endpoint _account_api_url() resolves to) answers in 0.51 s.
   2. A network failure takes a different, working path — the validator has _is_network_error → EXIT_NETWORK_FAILURE, and the route's failure classes are covered and pass. A run with deliberately-bad credentials cleanly returns state=invalid in 2.8 s without crashing.
   3. The validator standalone succeeds. Invoked directly with the same real credentials, outside the route entirely:

   exit code: 0
     /ok: True
     /project_id: 308cfe99-…
     /devices: list[2] = ["FRESNEL", "FRESNEL_CAN1"]
     /token: <redacted len=1278>
     /expires_at: 2026-07-29T19:17:25…
   

   So the interpreter, the SDK, the network and the validator are all healthy and produce a complete valid payload. The crash is in this repo's handling of that payload.

   Note the timing gradient: a rejected login survives at 2.8 s; a successful one dies at ~3.7 s. It crashes after auth succeeds, devices are listed and the token is minted.

   The stub was faithful after all

   The stub's validatorLine() emits devices: ["EMU_FREE", "FRESNEL"] — a list of strings; the real validator emits ["FRESNEL", "FRESNEL_CAN1"], also a list of strings. No shape mismatch. Consistent with the crash signature being byte-identical between the real validator and the python3 stub (lt10d9b296inEukogC4j+hvE+1vRA2AA). The only unfaithful part was the interpreter (AMICO_PYTHON = process.execPath), and that produced a slightly different signature.

   Bottom line

   467fb15cf introduced a crash on the production Pasqal submit success path. Any user who successfully authenticates a Pasqal account takes down the opencode server process. This is not a test artifact and not environmental.

   • v1.17.3-amicode.10 must not be cut until this is fixed.
   • The failing test must not be skipped — it is accurate.

   Side effect worth checking

   Because the process dies mid-submit, cleanup never runs and a freshly minted Pasqal token is left written to disk while the connection state is never reconciled. Worth checking what the connections panel reports after a crashed submit, and whether a real user ends up with a persisted credential and no connected state.

   cc @kate
4. jack-champagne commented on Jul 28, 2026

   

   jack-champagne

   on Jul 28, 2026

   MemberAuthor

   More actions

   Root cause: the native keychain write

   The segfault is at connections.ts:1005:

   pasqalSecretStore().write(PASQAL_SECRET_ACCOUNT, { username, password })

   which reaches @napi-rs/keyring's native Entry.setPassword() via keychainSecretStore.write().

   Evidence

   Instrumented the success path and ran the hermetic test:

   [T2] before writeCredential
   [T3] after writeCredential, before keychain
   panic(main thread): Segmentation fault at address 0x0
   

   [T4] after keychain write never prints.

   Installing the in-memory store removes the crash entirely — the test then fails on an ordinary assertion instead of segfaulting:

   restoreSecretStore = setPasqalSecretStore(inMemorySecretStore())

   0 pass / 1 fail / 0 crash   (was: segfault)
   

   Why this explains the exact pass/fail pattern

   pasqalSecretStore().write() sits only on the outcome.kind === "valid" && outcome.token !== null branch:

   test	branch taken	keychain write?	result
   full lifecycle (valid + token)	valid + token	yes	CRASH
   null token → session-only	valid, null token	no	pass
   failure classes (exit 2/4/1)	config / invalid	no	pass

   That is the whole mystery: only one branch touches the native keyring.

   467fb15cf is the first bad commit because it introduced that write — the ADR 0001 addendum for #194, putting the password in the OS keychain so the ~24 h token can be silently re-minted.

   It needs the route context

   The keyring call is not crash-prone on its own. Both of these pass:

   • @napi-rs/keyring setPassword under bun run — fine
   • the same call as a standalone bun test case — fine

   It only segfaults inside the Effect HttpApi route handler, after writeCredential. So this is a narrow Bun/NAPI-under-Effect interaction, not "keyring is broken."

   Correction to my earlier comment

   I previously wrote that "any user who successfully authenticates a Pasqal account takes down the opencode server process." That is not established and I withdraw it.

   I tried to confirm it on the real server path — bun run … serve on port 14099, real credentials, POST to the route. The server survived, but returned state=invalid: authentication failed, so execution never reached the keychain write. Inconclusive, not a pass.

   So the confirmed scope today is: the crash reproduces in the test runner, at the keychain write. Whether the compiled production binary crashes at the same site is still unverified. It needs a re-run once credentials authenticate again (see below).

   Two separate test defects worth fixing regardless

   1. amicode-connections-routes.test.ts has no keychain isolation. Its sibling amicode-connections.test.ts does setPasqalSecretStore(inMemorySecretStore()) in beforeEach; the routes test does not. So it drives the real OS keychain on the developer's machine. (The real pasqal-cloud/default slot reads as absent, so nothing was clobbered — the crash happens during the write, before it persists.)
   2. AMICO_PASQAL_KEYCHAIN_SERVICE is not in that test's ENV_KEYS. pasqal-secret.ts documents it as existing precisely "so an isolated sandbox / test run gets its own slot and never shares the real connection's secret." Unused here.

   Either mechanism fixes the crash. The seam is the better choice — it keeps the test hermetic with no keyring daemon dependency at all.

   Note on an artifact in my reproduction

   After installing the seam, the test fails on:

   record.keys
   +   "LC_CTYPE",
       "PASQAL_PASSWORD",
   

   That is mine, not a product bug — I had swapped the stub validator from a bun-executed .mjs to a python3-executed .py, and CPython synthesizes LC_CTYPE into os.environ. Any real fix should assert against the interpreter's actual env surface, or filter locale vars.
5. jack-champagne commented on Jul 28, 2026

   

   jack-champagne

   on Jul 28, 2026

   MemberAuthor

   More actions

   Production path does NOT crash — retracting the release block

   I previously said v1.17.3-amicode.10 must not be cut. That was wrong. I retract it.

   Ran the real route on a real server with working credentials:

   bun run … packages/opencode/src/index.ts serve --port 14101
   POST /amicode/connections/credential   (real login, real validator, real keychain)
   
     ok= True  error= None
     state= connected  identity_set= True
     devices= [{'name': 'FRESNEL'}, {'name': 'FRESNEL_CAN1'}]
   
   server: ALIVE
   

   And it demonstrably reached the crash site. Two independent confirmations that execution passed through connections.ts:1005:

   1. The keychain slot was written — username len: 17, password len: 15 (an isolated slot via AMICO_PASQAL_KEYCHAIN_SERVICE, since deleted).
   2. The token was persisted — pasqal.json with ['expires_at','project_id','token'], token_len: 1278.

   So pasqalSecretStore().write() executed against the real native keyring, on the real success branch, and the server survived.

   Corrected conclusion

   The segfault is confined to the bun test runner. It is not reachable in production.

   context	keychain write at connections.ts:1005	result
   bun test + Effect route	executes	segfault
   bun run … serve + Effect route	executes	survives
   bun test, keyring call standalone	executes	survives
   bun run, keyring call standalone	executes	survives

   Only the first combination crashes. So this is a Bun-test-runner/NAPI interaction, not a product fault — 467fb15cf is not shipping a crash to users.

   What the actual defect is

   amicode-connections-routes.test.ts has no keychain isolation, so it drives the developer's real OS keyring inside bun test — the one context where that segfaults.

   Its sibling amicode-connections.test.ts does this in beforeEach and consequently passes:

   restoreSecretStore = setPasqalSecretStore(inMemorySecretStore())

   Adding the same to the routes test removes the segfault; the test then runs normally. AMICO_PASQAL_KEYCHAIN_SERVICE is also absent from that file's ENV_KEYS, despite pasqal-secret.ts documenting it as existing so "an isolated sandbox / test run gets its own slot and never shares the real connection's secret."

   Revised actions

   • v1.17.3-amicode.10 is not blocked by this. The remaining blocker for a green unit (linux) is this test, not the product.
   • Install the in-memory seam in amicode-connections-routes.test.ts (and add AMICO_PASQAL_KEYCHAIN_SERVICE to ENV_KEYS for belt and braces). Fixes the crash and stops unit tests touching a developer's real keyring.
   • Worth a defensive look regardless: the try/catch around the keychain write in keychainSecretStore.write() assumes a JS throw. A native fault bypasses it entirely. Not reachable in production per the above, but the comment "a keychain that cannot be reached never throws into the caller" is stronger than what try/catch can actually guarantee around a NAPI call.

   Apologies for the churn on the release call — the earlier claim was made before I could get a working login onto the production path.
6. jack-champagne

   added a commit that references this issue on Jul 28, 2026

   test(server): isolate the keychain in the connections-route suite

   ...

   1e83e32