The chatgpt.com/backend-api/codex endpoint has an intermittent failure mode
where it accepts the connection but never emits a single stream event — the
socket just hangs. Direct sequential probing reproduces it (0 events, no HTTP
status), and a fresh reconnect then succeeds in ~2s. Today the only guard is
the wall-clock stale timeout in interruptible_api_call, so a dead-on-arrival
connection is held for the full stale window (90-900s depending on context /
config) before the retry loop can reconnect — minutes of wasted wall time per
stall, at a rate of ~20% of calls during affected windows.
Add a TTFB watchdog scoped to the codex_responses path:
- codex_runtime.run_codex_stream stamps agent._codex_stream_last_event_ts on
*every* stream event (not just output-text deltas), so reasoning-only and
tool-call-only turns are not mistaken for a stall.
- interruptible_api_call resets that marker before the worker starts and, while
it is still None, kills the connection once elapsed exceeds the TTFB cutoff
(default 45s, tunable via HERMES_CODEX_TTFB_TIMEOUT_SECONDS, 0 disables). The
raised TimeoutError flows through the existing retry path unchanged.
Once any event has arrived the stream is healthy and only the existing
wall-clock stale timeout applies, so legitimate long generations are never
interrupted. Gated to codex_responses; the chat_completions non-stream,
anthropic and bedrock branches have no first-event signal and are untouched.
Adds tests/agent/test_codex_ttfb_watchdog.py covering the stall kill, the
events-flowing pass-through, and the env-disable path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8601c4d44c