Two caching holes made MoA re-bill essentially its entire input stream:
1. AGGREGATOR: anthropic_prompt_cache_policy() judged the agent's own
model/provider — on the MoA path those are the virtual preset name and
'moa', which match no caching branch, so _use_prompt_caching was False
and the acting aggregator (Claude on OpenRouter) ran with ZERO
cache_control breakpoints. Measured on identical opus-4.8 sessions:
85% cache share solo vs 2% via MoA — ~30M re-billed input tokens on one
132-task benchmark run. Fix: when provider == 'moa', resolve the policy
from the preset's real aggregator slot (provider/model/base_url/api_mode
via resolve_runtime_provider).
2. ADVISORS: _run_reference never applied cache_control at all, and
Anthropic caching is opt-in per request — Claude advisors served 0
cache reads across 1,227 benchmark calls (11.5M re-billed input tokens)
even though the advisory view is append-only across iterations (stable
prefix; the synthetic end marker is last so it never pollutes it). Fix:
_maybe_apply_advisor_cache_control() reuses the SAME policy function and
SAME system_and_3 layout as the main loop, judged on the advisor slot's
own resolved runtime — advisor requests are now decorated exactly like
an acting agent on that provider. Auto-caching routes (OpenAI-family)
are left untouched by policy.
Live-verified on the wire (per-iteration opus+gpt5.5 preset, 4 fan-outs):
claude advisor fan-out 2-3 cache_write=2161/2344, fan-out 4
cache_read=2206 / fresh_in=2; aggregator session cache share 84%/77%
(vs 2%/0% before). Sub-1024-token prompts correctly stay uncached
(Anthropic minimum).
Follow-up on the salvaged #56392 guard. The cherry-picked change matched
custom:<name> pool entries against the primary by raw base_url string
equality, which (a) can't disambiguate two named custom providers sharing
one gateway base_url and (b) left a latent bare-"custom" entry bypass.
Route the match through get_custom_provider_pool_key(rt[base_url]) compared
against the entry's custom:<name> key, mirroring the sibling guard in
recover_with_credential_pool. Use CUSTOM_POOL_PREFIX instead of the literal.
Add regression tests for the custom same-endpoint (swap) and cross-endpoint
(skip) branches, plus the plain-provider fallback-pool case from #56885.
The salvaged fix wired per-provider ssl_ca_cert / ssl_verify (and
HERMES_CA_BUNDLE) into the MAIN OpenAI client. This follow-up:
- Auxiliary client parity: process_bootstrap.build_keepalive_http_client
accepts and forwards verify; auxiliary_client._resolve_aux_verify mirrors
the main-client TLS resolution (via load_config_readonly, the read-only
fast path) so compression/vision/web_extract/title-gen/session_search
honor the same per-provider CA. Without this, chat worked against a
private-CA endpoint but every auxiliary call still failed APIConnectionError.
- switch_model now reads custom_providers from live config (load_config_readonly)
instead of the init-time agent._custom_providers snapshot, so ssl_ca_cert /
ssl_verify edits are honored on mid-session model switch — matching the
context-length reload (#15779).
- Drop the dead client-level verify= where a custom httpx transport is used
(httpx ignores it there); verify lives on the transport. Fix docstrings.
Applies to both run_agent._build_keepalive_http_client and process_bootstrap.
- resolve_httpx_verify: add CURL_CA_BUNDLE to the env chain (consistency with
agent/ssl_guard._CA_BUNDLE_ENV_VARS) and emit a loud logger.warning naming
the endpoint whenever ssl_verify:false disables verification.
- get_custom_provider_tls_settings: case-insensitive base_url match (config
dedup already lowercases; scheme/host are case-insensitive) so a mixed-case
entry doesn't silently drop its CA. Exact match preserved — no prefix bypass.
- Demote best-effort except Exception: pass in agent_init/switch_model to
logger.debug(exc_info=True).
- Tests for aux verify forwarding, _resolve_aux_verify, case-insensitive
match, and prefix-bypass rejection.
Wire ssl_ca_cert and ssl_verify through custom_providers config and env
vars into the keepalive httpx client, fixing APIConnectionError against
mkcert/self-signed Ollama proxies behind HTTPS.
@janrenz's PR #35862 added prompt_caching.enabled=false at init only. But
_anthropic_prompt_cache_policy re-derives _use_prompt_caching on every /model
switch (agent_runtime_helpers) and fallback-model swap (chat_completion_helpers),
which re-enabled markers and re-broke the strict proxy the toggle was meant to fix.
Move the kill switch into anthropic_prompt_cache_policy so it returns (False, False)
on every path. Drop the now-redundant init-time override (kept @janrenz's isinstance
hardening on the cache_ttl read). Add policy-level tests + docs for the toggle.
Follow-up to salvaged PR #35862.
* fix(agent): drop tool_calls with empty function.name to prevent orphan 400
Salvage of #12807 by @melonboy312 — rebased onto current main (sanitizer
moved to agent_runtime_helpers), scoped to the sanitizer fix, with a
regression test that fails without it.
* fix(agent): repair (not drop) empty-name tool_calls to preserve anti-priming + prevent 400
Dropping empty-name tool_calls in the pre-call sanitizer collided with #47967,
which intentionally keeps an empty-name call paired with a synthesized
'tool name was empty' anti-priming result so weak models self-correct without
a full catalog dump. Dropping the call orphaned that result and stripped the
signal (breaking tests/agent/test_empty_tool_name_loop_dampening.py).
The actual HTTP 400 cause is an ORPHANED function_call_output (adapter drops
the empty-name function_call but keeps its output). Rename the blank name to a
non-empty sentinel instead: the call and its result stay paired, the adapter
no longer drops the function_call, no orphan, no 400 — and the anti-priming
result content the model needs is preserved.
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
* fix(agent): merge consecutive assistant messages in repair_message_sequence
Strict OpenAI-compatible providers (DeepSeek v4, Moonshot/Kimi) reject a
replayed history where an assistant message carrying tool_calls is
immediately followed by another assistant message instead of its tool
results — HTTP 400 'An assistant message with tool_calls must be
followed by tool messages...'.
repair_message_sequence (the defensive belt run before every API call)
fixed orphan-tool and consecutive-user shapes but never merged
consecutive assistant messages. Adds a Pass 0 that collapses adjacent
assistant turns into one — union of tool_calls, concatenated content,
carried reasoning_content — covering both reported shapes:
- parallel tool calls split across two assistant turns (#29148)
- content-only assistant followed by tool_calls-only assistant (#49147)
A tool result or user turn between two assistants blocks the merge
(distinct, valid rounds). Runs before Pass 1 so the merged union of
tool_call ids is known to the orphan-tool filter.
Closes#29148, #49147.
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
Co-authored-by: woaini30050 <woaini30050@users.noreply.github.com>
Co-authored-by: weidzhou <weidzhou@users.noreply.github.com>
* fix(agent): exempt codex Responses interim turns from assistant merge
The Pass 0 consecutive-assistant merge collapsed codex_responses interim
turns, which legitimately stay separate — each carries its own encrypted
continuation state (codex_reasoning_items / codex_message_items) that
must replay verbatim. Skip the merge when either side is a codex interim
(has codex_reasoning_items / codex_message_items / finish_reason=='incomplete').
Fixes the slice-2 regression in test_run_agent_codex_responses.py
(test_duplicate_detection_distinguishes_different_codex_{reasoning,message_items}).
---------
Co-authored-by: Bartok9 <danielrpike9@gmail.com>
Co-authored-by: woaini30050 <woaini30050@users.noreply.github.com>
Co-authored-by: weidzhou <weidzhou@users.noreply.github.com>
OpenRouter returns 429 in two shapes: an account-level throttle on the
user's key, and an upstream-provider throttle (DeepSeek/Anthropic/etc.
rate-limiting OpenRouter's aggregate traffic). The classifier treated
both identically and rotated/exhausted OPENROUTER_API_KEY on every 429 —
burning the key for ~24min and silently disabling auxiliary features
(compression, summarization, vision) on an upstream throttle where the
key was healthy.
Add a FailoverReason.upstream_rate_limit classified from OpenRouter's
unambiguous wrapper message "Provider returned error" (the same signal
the metadata-raw parser already trusts). Recovery skips credential
rotation and defers to the fallback chain to switch models instead.
Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>
The gateway/CLI /model switch path (switch_model in agent_runtime_helpers)
built the MoAClient facade but left agent.api_mode at the value
determine_api_mode / the resolved aggregator transport produced (e.g.
codex_responses or anthropic_messages). The conversation loop dispatches on
agent.api_mode, so a non-chat_completions value made the primary/acting call
go through client.responses.create — which the MoAClient facade has no
.responses for — and fall through to the moa://local placeholder, 404 three
times, then fall back to a reference model (issues #54259, #54669).
agent_init.py already pins api_mode=chat_completions for provider==moa; mirror
that in the live switch so the primary call always routes through
MoAClient.chat.completions. The aggregator's real transport is resolved and
applied inside the reference/aggregator fan-out, not on the outer call.
Vision requests routed through the OpenAI-compat API server forward the
raw multi-part content list ([{type:"text"}, {type:"image_url"}, ...])
straight through as user_message. The codex intermediate-ack detector
flattened it with (user_message or "").strip(), so a truthy list survived
and .strip() raised AttributeError — killing any Codex-routed vision turn
that took the require_workspace path.
Route through the existing _summarize_user_message_for_log helper (which
already backs the logging/banner previews on main), and widen the param
type hint from str to Any to match how the function is actually called.
The two logging-preview sites the original PR also touched were fixed
independently on main by the conversation-loop refactor.
Co-authored-by: Hermes Agent <agent@nousresearch.com>
_sanitize_api_messages() compared raw tool_call_id strings without
stripping whitespace. When assistant-side IDs and tool-result IDs
diverged due to surrounding whitespace, valid tool results were treated
as orphaned and replaced with [Result unavailable] stub placeholders.
Strip whitespace in _get_tool_call_id_static() (both call_id/id paths,
dict and object) and at the two result_call_id comparison sites in
sanitize_api_messages(). Adds regression tests for preserved-whitespace
results and orphaned-whitespace removal.
Closes#9999
Defense-in-depth on top of _safe_session_filename_component (#5958):
Sink (makes the bad write impossible regardless of entry point):
- run_agent._save_session_log: sanitize session_id before building the
session_{sid}.json snapshot path.
- agent_runtime_helpers.dump_api_request_debug: sanitize before building
the request_dump_{sid}_{ts}.json path.
Boundary (clean 400 instead of a silently-hashed filename):
- api_server rejects path-traversal-shaped X-Hermes-Session-Id on the
session-continuation path and the explicit /api/sessions create path,
reusing gateway.session._is_path_unsafe (mirrors the native gateway's
entry-boundary guard). Also enforces the session-header length cap on
the continuation path.
Tests: traversal session_id stays contained at the write site; sanitizer
always yields a traversal-free segment; the API header rejects
../, absolute, and Windows-traversal IDs with 400.
* fix(agent): config-driven intent-ack continuation for all api_modes (#27881)
The agent could end a turn after only stating intent ('I will run a health
check...') without executing the announced tool call, forcing the user to
re-prompt. A continuation guard that catches this and nudges the model to
proceed already existed but was hard-gated to the codex_responses api_mode,
so Gemini/Claude/OpenRouter turns never benefited.
- New agent.intent_ack_continuation config (default 'auto' = codex-only,
byte-stable for existing conversations). 'true'/model-list opts every
api_mode in; 'false' disables. Mirrors agent.tool_use_enforcement's shape.
- looks_like_codex_intermediate_ack gains require_workspace (default True).
The opted-in path drops the codebase/filesystem requirement so general
autonomous workflows (server ops, deploys, API calls) are caught, not just
coding tasks. Future-ack + action-verb + short-content + no-prior-tool
guards still apply; the 2-nudge-per-turn cap is unchanged.
- Resolution centralized in intent_ack_continuation_mode (off/codex_only/all).
* docs(infographic): intent-ack continuation (#27881)
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.
After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the #33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.
Salvaged from #25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.
Fixes#25205
Same bug class as the Anthropic fix (#26293): the OpenAI/aggregator client is
built without max_retries, so the SDK default of 2 applies. The SDK's own 1-2s
backoff ignores Retry-After and retries inside hermes's outer conversation loop,
burning request slots against a rate-limited bucket. Set max_retries=0 at the
single create_openai_client chokepoint (covers init, switch_model, recovery,
restore, request-scoped). auxiliary_client builds its own clients and is not
wrapped by the loop, so it keeps SDK retries.
A persistent upstream 401 on a single-entry OAuth pool (common for Claude
Max subscribers) made the credential-pool recovery spin forever:
try_refresh_current() re-mints a fresh token and reports success on every
401, so recover_with_credential_pool returned True and the retry loop
continue'd without ever incrementing retry_count or reaching the
auth-failover block. The configured fallback_model never activated and the
agent appeared to hang.
Cap consecutive successful same-entry refreshes (keyed by provider +
pool-entry id) at 2; once exceeded, treat the credential as unrecoverable
and return not-recovered so the loop falls through to
_try_activate_fallback. The 429/billing paths already rotate-or-fall-through
correctly (mark_exhausted_and_rotate returns None on a single entry), so
only the auth-refresh branch needed the cap.
Co-authored-by: Hermes Agent <hermes@nousresearch.com>
switch_model() swapped model/provider/base_url/api_key but never
refreshed agent._credential_pool, which stays bound to the original
provider. recover_with_credential_pool() then sees a pool.provider !=
agent.provider mismatch and short-circuits — so a 429/401 on the new
provider gets no rotation and falls through to fallback instead.
Reload load_pool(new_provider) inside switch_model when the provider
changes (or the pool is missing). The reload is inside the protected
swap block and the pool is added to the rollback snapshot, so a failed
client rebuild restores the original pool.
Fixes#16678, #52727.
The session database records billing_provider and billing_base_url using
COALESCE(column, ?) in update_token_counts(), making them write-once.
When a user switches models mid-session via /model, the runtime (agent.provider,
agent.base_url) updates correctly, but the session row never reflects the new
provider. This causes the dashboard Models page to display a stale provider
badge and misattributes token usage / cost analytics.
Fix: add update_session_billing_route() that unconditionally sets
billing_provider, billing_base_url, and billing_mode (no COALESCE), and call
it from switch_model() in agent_runtime_helpers.py after the swap succeeds.
This follows the same pattern as update_session_model() which already
unconditionally updates the model column (added for the identical COALESCE
problem on the model field).
Closes#48248
The success/staged gating and op-expansion for mirroring built-in memory
writes to external providers lived in a standalone agent/memory_write_bridge.py
helper called inline from two core call sites (tool_executor.py,
agent_runtime_helpers.py). That left the mirror decision-making in the agent
loop, outside the memory-provider interface.
Fold it into a new MemoryManager.notify_memory_tool_write() entry point: the
loop now hands over the raw tool result + args and a metadata callback, and the
manager decides whether/what to mirror. Both core call sites collapse to a
single call; the orphan module is removed. No MemoryProvider ABC change.
Tests rewritten as behavior tests against the manager method.
Mirror built-in memory writes to external providers only after the native memory tool succeeds and is not staged for approval. Keep OpenViking's built-in memory mirroring add-only, since Hermes native memory entries do not yet have stable OpenViking file URIs for replace/remove.
Add a narrow viking_forget tool for exact user memory file deletion and document the current OpenViking write/delete behavior.
* feat(providers): remove google-gemini-cli + google-antigravity OAuth providers
Google now actively bans accounts for third-party tools that piggyback on
Gemini CLI / Antigravity / Code Assist OAuth, and because abuse prevention
sits at a backend layer the ban can extend to the entire Google account
(Gmail/Drive), with a second violation being permanent.
Ref: https://github.com/google-gemini/gemini-cli/discussions/20632
Removes both OAuth inference providers entirely (modules, provider profiles,
auth/runtime/config/models wiring, the /gquota Code Assist quota command,
the antigravity-cli optional skill, desktop + docs surface in en + zh-Hans).
The API-key 'gemini' provider (GOOGLE_API_KEY/GEMINI_API_KEY against
generativelanguage.googleapis.com) is unaffected and stays fully supported.
* fix(skills): keep the antigravity-cli skill — only the OAuth provider is removed
The antigravity-cli optional skill orchestrates the external `agy` binary as
a coding-agent tool via the terminal tool — it does NOT wrap Hermes inference
through the banned google-antigravity OAuth provider, so it carries none of
the account-ban risk that motivated removing that provider. Restore the skill,
its docs page, the sidebar entry, and the optional-skills catalog row. The
google-antigravity / google-gemini-cli inference providers stay fully removed.
* fix(agent): strip stale reasoning_content when falling back to a strict provider
A reasoning primary (DeepSeek/Kimi/MiMo thinking mode) pins reasoning_content
on every assistant tool-call turn (a single space " " pad). api_messages is
built once under the primary; on a mid-session fallback to a strict
OpenAI-compatible provider (Mistral, Cerebras, Groq, SambaNova), those stale
pads were replayed verbatim and rejected with HTTP 400/422:
body.messages.2.assistant.reasoning_content: Extra inputs are not
permitted (input: ' ')
reapply_reasoning_echo_for_provider() only ever ADDED pads, so it never
reconciled history built under a reasoning primary against a strict fallback.
copy_reasoning_content_for_api() also leaked empty-string and 'reasoning'-only
shapes to non-pad providers.
Fix both sites: when the active provider does not enforce echo-back, strip
reasoning_content (empty, space-pad, or non-empty) entirely. Re-padding when
switching TO a reasoning provider is preserved. Covers the Cerebras 400 from
#45655 and the DeepSeek->Mistral 422 fallback report.
Refs #45655.
* test: update reasoning-replay tests for strict-provider stripping
test_explicit_reasoning_content_beats_normalized_reasoning_on_replay was
implicitly running on the OpenRouter fixture (non-pad); pin it to a reasoning
provider so the precedence it checks is observable. Add a positive
strict-provider test asserting reasoning_content is stripped on replay.
The session-stable system prompt embeds Model:/Provider: identity lines,
but mid-turn failover (try_activate_fallback) swaps the runtime without
touching them, so a fallback model misreports itself as the primary when
asked "what model are you?".
rewrite_prompt_model_identity() rewrites the last occurrence of each line
on _cached_system_prompt when a fallback activates (and back on restore,
byte-identical so the primary's prefix cache still hits). The rewrite is
never persisted to the session DB. _sync_failover_system_message() patches
the in-flight api_messages[0] at all 8 failover sites so the current turn
ships the corrected identity. Cache-safe: the fallback's prefix cache is
cold on a model switch anyway.
Co-authored-by: Hermes Agent <noreply@nousresearch.com>
The memory tool was strictly one-op-per-call. With the store running near
its char limit by design, a new add that would overflow gets rejected with
'consolidate now, then retry' -- but the model could not consolidate and add
in one call. It had to remove/replace across several turns, then retry the
add, each turn re-sending the whole conversation context. Expensive thrash.
Add an 'operations' array: a list of add/replace/remove ops applied
atomically against the FINAL char budget. The model frees space and adds new
entries in ONE call, even when an add alone would overflow. All-or-nothing:
any bad op aborts the whole batch, nothing written.
Root-cause note: the two agent-level memory interception sites
(agent_runtime_helpers.py, tool_executor.py) silently dropped any param not
in their explicit kwarg list, so 'operations' never reached the handler and
batch calls failed with 'Unknown action None'. Both now pass it through and
bridge each add/replace op to external memory providers.
Also: success response is now terminal (done=true + 'do not repeat' note,
no full-entries echo that invited re-edits); schema rewritten to lead with
the batch mechanism and an explicit one-shot stop rule (2138 -> 1476 chars).
Live-verified: near-full consolidate-and-add went 7 calls -> 1 call,
stable across 3 reps. 103 memory/approval tests + 398 background-review/
run_agent tests green; 6 new batch tests added.
Keep request dump writes on the shared atomic JSON path, add regression coverage for request body/error/stdout redaction, and map the salvaged contributor email for release attribution.
dump_api_request_debug() masks the provider Authorization header but writes
the request `body` (system prompt, tool defs, context-embedded values) and the
error message raw via atomic_json_write. This path also fires unconditionally
on API errors (not only under HERMES_DUMP_REQUESTS), so any secret surfaced
into context (e.g. an integration token) lands in cleartext at
request_dump_*.json on every failed call.
Run the serialized dump through the existing redact_sensitive_text() scrubber
(already used for logs/tool output) before persisting and before the
HERMES_DUMP_REQUEST_STDOUT print; preserve atomicity via temp-file +
Path.replace. Also add the Notion internal-integration prefix (ntn_) to
_PREFIX_PATTERNS so bare values are caught.
Per SECURITY.md §3.2 this is a redaction (in-process heuristic) hardening, not
a §3.1 vulnerability. Refs #46583.
Custom endpoints carry two naming conventions for the same provider: the
agent's provider attribute is the generic 'custom' label while the pool
is keyed 'custom:<normalized-name>'. The defensive guard in
recover_with_credential_pool compared them literally, logged
'Credential pool provider mismatch: pool=custom:<name>, agent=custom',
and skipped recovery — so 401 refresh and 429 rotation never ran for
ANY custom-provider user (seen in the field on a Fireworks setup whose
dead key burned full retry cycles every turn with the skip warning on
each one).
Accept the pair only when the agent's CURRENT base_url resolves to the
same pool key via get_custom_provider_pool_key, preserving the guard's
original purpose (#33088/#33163): a fallback provider or a different
custom endpoint still skips pool mutation.
Follow-up to the #44837 clamp: a min() clamp only fixes cursor overshoot
past the new end of the list. When repair_message_sequence drops/merges
messages at indexes below the cursor, the clamp leaves the cursor pointing
past unflushed rows and the turn-end flush silently skips them.
Extract repair_message_sequence_with_cursor(): snapshot the flushed prefix
by object identity before repair, then recompute the cursor as the count
of surviving flushed messages. Falls back to the clamp when no snapshot is
available. Keeps the safety guard in _flush_messages_to_session_db.
Adds targeted tests for overshoot, before-cursor compaction, no-repair,
bare-agent, and the flush guard.
* fix(gateway): gate oversized Telegram voice/audio before download
Adds a pre-download size check to the Telegram voice and audio inbound
paths. Files that exceed _max_doc_bytes (default 20 MB) are rejected
before get_file() is called, preventing silent OOM-style stalls on large
uploads. A human-readable note is appended to the event text so the
model can explain the limit to the user.
Also extends 403 entitlement detection in recover_with_credential_pool
to cover two additional cases: 'oauth authentication is currently not
allowed for this organization' and Anthropic anthropic_messages-mode 403s,
both of which should be treated as entitlement failures rather than
transient errors.
Tests: 7 new cases in test_telegram_voice_v0_regressions.py covering
the size gate (accept, reject, note text) and the STT-failure notice path.
Salvaged from #40487 (cryptopafi) — cherry-picked the Telegram voice
policy and 403 entitlement fixes; LiveKit/Discord/uv.lock workstreams
left for separate PRs.
* test(gateway): drop orphaned voice tests not backed by this PR
The cherry-picked test file from #40487 included 3 tests for STT-failure
notice and voice-mode (_handle_voice_command 'on' -> voice_only) behavior
that this PR intentionally does NOT salvage (those belong to the LiveKit/
voice-policy workstreams left in #40487). They fail on both this branch
and clean main because the feature code isn't present.
Keep only the 2 tests backed by code actually in this PR:
- test_telegram_audio_size_gate_rejects_oversized_media_before_download
(covers the _telegram_media_size_allowed guard this PR adds)
- test_voice_tts_is_explicit_audio_reply_opt_in (matches current main)
Removed now-unused imports (MessageEvent, MessageType, AsyncMock).
* refactor(desktop): dock terminal under chat and simplify file rail
Keep the right rail focused on file browsing while moving the persistent terminal into the chat column bottom slot, and make terminal colors follow the active light/dark mode instead of a fixed Solarized palette.
* fix(desktop): make the terminal a resizable, themed side pane
- Move the terminal into a resizable pane (viewport-% widths) that shares
<main>'s stacking context, so its drag handle no longer sits under the
fixed terminal overlay; works on either rail side.
- Restore +x on node-pty's spawn-helper before the first spawn to fix
"posix_spawnp failed" on macOS prebuilds (real cause; drop the redundant
shell-candidate retry loop).
- Gate terminal open/fit/start on document.fonts.ready and strip leading
blank rows (re-armed before the resize Ctrl-L redraw) so the prompt sits
flush at the top with no starship add_newline gap.
- Inherit the app editor-surface color as the terminal background.
- Bind Ctrl+` (⌃` on macOS) to toggle the terminal; add a palette entry.
* feat(desktop): show platform hotkey hints in the command palette
- Render each palette item's live binding as a <KbdGroup> hint via a new
comboTokens() helper (mac shows ⌘/⌃/⌥/⇧, every other platform shows
Ctrl/Alt/Shift — never a ⌘ on PC).
- Default the terminal toggle to ⌘` / Ctrl+` (the ~ key) on both platforms.
- Drop the hardcoded (⌘⏎) baked into the composer steer tooltip; render it
platform-aware with formatCombo instead.
* fix(desktop): drop the active check on the command-palette terminal item
* fix(desktop): remove active/check states from the command palette
* fix(desktop): allow ⌥/Shift-drag selection over mouse-mode TUIs
Full-screen apps (hermes --tui, vim) enable mouse reporting, so a plain
drag can't select text and ⌘/Ctrl+L (add-selection-to-chat) had nothing
to send. Enable macOptionClickForcesSelection so ⌥-drag on macOS (Shift
elsewhere) forces a native selection over mouse-mode apps.
* feat(desktop): tell the in-pane agent it's embedded in the GUI
Set HERMES_DESKTOP_TERMINAL=1 on the terminal pane's shell env and surface
it in build_environment_hints, so a hermes/--tui launched inside the pane
knows it's next to the GUI chat and that ⌥/Shift-drag + ⌘/Ctrl+L sends a
selection to the composer. Distinct from HERMES_DESKTOP (agent backend).
* refactor(desktop): drop the redundant Ctrl+` terminal-toggle fallback
The toggle now ships as mod+` on both platforms, so the standard combo
index handles it — the bespoke fallback (and its stale 'old default'
comment) is dead weight.
* fix(desktop): read live terminal selection for ⌘/Ctrl+L
A redraw-heavy TUI (spinners/clocks) outruns onSelectionChange, leaving the
React selection state empty so the state-gated shortcut listener never
attached and ⌘L no-op'd. Always listen and read xterm's live selection (with
a native fallback) at press time; only swallow the key when there's text to
send. Drops the now-redundant custom key handler.
* feat(desktop): make any agent aware it's in the Hermes desktop GUI
Generalize the runtime-surface hint: fire for HERMES_DESKTOP (the backend
powering the GUI chat) as well as HERMES_DESKTOP_TERMINAL (a hermes in the
embedded terminal pane), so it's about being inside the desktop GUI, not
about being a TUI. The terminal-pane selection note stays pane-specific.
* feat(desktop): give the GUI agent a read_terminal tool
The in-app terminal buffer lives in the renderer (xterm), so expose it to the
chat agent over the same blocking bridge clarify uses: read_terminal emits
terminal.read.request, the renderer serializes the buffer (visible screen by
default, or a start_line/count range against total_lines) and answers
terminal.read.respond. Gated to the GUI via HERMES_DESKTOP.
Also restores the flipped-layout titlebar inset fix (app-shell +
desktop-controller) for terminal/preview rails at the window's left edge.
* chore(desktop): trim read_terminal comments
* feat(desktop): add a terminal toggle to the statusbar
The file rail lost its terminal icon, leaving ⌘` and the command palette
as the only ways in. Add a one-click toggle to the statusbar's left
cluster, mirroring the command-center item: it reads $terminalTakeover so
it lights up while the pane is open and stays in sync with the hotkey, and
is gated to chat view (the only place the pane can show).
* fix(desktop): relabel the terminal header button to what it does
The in-pane button claimed a focus/split fullscreen toggle ("Focus
terminal view" / "Return to split view", screen-full/normal icons), but
the terminal is just a resizable side pane — there's no fullscreen. The
button only mounts while the pane is open, so the focus branch was dead
and clicking it merely closed the terminal. Relabel to "Hide terminal"
with a close icon, drop the dead conditional and the now-unused takeover
read.
* fix(desktop): move the terminal toggle next to the version item
Relocate it from the left cluster to the right of the statusbar, just
left of the client version item.
* feat(desktop): default the terminal to PowerShell on Windows
Prefer pwsh (7+) then Windows PowerShell 5.1 over cmd.exe, falling back to
comspec only when neither is present. -NoLogo drops the startup banner so
the prompt sits flush like the POSIX shells.
* feat(desktop): show a persistent divider on the terminal pane
The resize sash only painted on hover, so the terminal/chat boundary was
invisible at rest. Add an opt-in `divider` prop to Pane that paints a thin
resting hairline on the resize edge (side-aware, so it tracks the rail when
the layout flips) and enable it on the terminal pane.
* refactor(desktop): resolve the terminal shell instead of hardcoding it
Make shell selection a real resolver: an explicit override wins
(HERMES_DESKTOP_SHELL on both platforms, $SHELL on POSIX), otherwise
auto-detect the best installed shell — pwsh > Windows PowerShell 5.1 > cmd
on Windows, zsh > bash > sh on POSIX. A shared shellSpecFor() picks the
interactive flags by family, so an overridden bash/pwsh/cmd all launch
correctly.
* fix(desktop): repaint the terminal on light/dark switch
Setting term.options.theme updated colors for the DOM renderer but not the
WebGL one, which caches glyph colors in a texture atlas — so already-drawn
cells kept their old palette after a mode switch. Hold the WebglAddon in a
ref and clear its atlas when the theme changes.
* fix(desktop): match the terminal palette to VS Code Light+/Dark+
Adopt VS Code's exact default ANSI palette (the terminalColorRegistry
defaults), enable minimumContrastRatio: 4.5 so foregrounds are clamped
against the background the way the integrated terminal does, and key the
light/dark choice off renderedMode (the painted surface) instead of
resolvedMode so it can't invert. The canvas + inset paint the live skin
surface (--ui-editor-surface-background) so the terminal blends with the
app and follows light/dark, while the contrast clamp keeps colors crisp.
* fix(desktop): tighten command palette search to substring matching
cmdk's default fuzzy scorer matched anything with the query letters
scattered across an item, so e.g. "color" never narrowed to color
entries. Add a substring filter: every typed word must literally appear
in an item's value/keywords, keeping results tight and predictable.
* fix(desktop): blend the terminal header into the skin surface
The persistent-terminal overlay painted the static palette background
(#1e1e1e/#ffffff), so the transparent header strip revealed a near-black
slab above the surface-colored body. Paint the overlay with the live
--ui-editor-surface-background so header and body read as one pane.
* fix(desktop): re-resolve the terminal surface on skin switch
The canvas surface only re-resolved on light/dark change, so switching
skins at the same mode left the WebGL canvas painted with the old tint
until reload. Key the resolve off themeName too. Also trim the palette
comments.
* chore(desktop): drop redundant terminal theming header comment
VolcEngine's api/plan endpoint occasionally leaks raw XML attribute
fragments into tool_use.name when its protocol-translation layer
converts the model's native XML-style tool emission to Anthropic
Messages tool_use blocks, producing names like:
terminal" parameter="command" string="true
execute_code" parameter="code" string="true
session_search" parameter="session_id" string="true
The corruption happens server-side at the provider, but it breaks
every tool call for affected users — no normalization rule in
repair_tool_call can rescue them, so each request runs through three
retries and then aborts as partial.
Add an early sanitizer in agent_runtime_helpers.repair_tool_call that
trims at the first ' " ', " ' ", '<', or '>' character (idx > 0
only) so the rest of the existing repair pipeline (lowercase /
snake_case / fuzzy match) can resolve the cleaned name normally.
Whitespace is deliberately NOT a separator — the legitimate
"write file" -> write_file repair path (covered by
test_space_to_underscore) must keep working.
Tests: 11 new regression cases in TestVolcEngineXmlPollution
covering all three observed polluted names, CamelCase + pollution
mix, single-quote variants, angle-bracket variants, clean-name
passthrough, and the whitespace-preservation guard. All 18 pre-
existing repair tests still pass (29 total in the file).
A steer rides inside a tool result (the only role-alternation-safe slot
mid-turn), so a bare "User guidance:" line reads as untrusted tool content —
well-behaved models refuse it as suspected prompt injection (observed live:
"I only follow instructions from you directly, not ones injected through
command results").
- Wrap steers in a bounded, self-describing [OUT-OF-BAND USER MESSAGE] marker
(prompt_builder.format_steer_marker), shared by both drain sites.
- Add STEER_CHANNEL_NOTE to the core system prompt so the model expects this
exact marker and trusts it as a genuine user message — while still ignoring
lookalikes buried in tool/web/file output. Static text → byte-stable prompt,
no prompt-cache regression; gated on the agent having tools.
- Desktop: steer ack is now an inline transcript note (⏩ steered · …) instead
of a toast.
Marker is intentionally static (not a per-session nonce) to honor the
byte-stable system-prompt caching policy; nonce hardening noted as follow-up.
The salvaged observer contract gated the API-request hot path on has_hook()
but left the per-tool emit ungated: every tool call ran result-field
derivation + payload dict build + invoke_hook dispatch even with zero
plugins registered.
- _emit_post_tool_call_hook now short-circuits on has_hook("post_tool_call")
and derives status/error fields lazily (after the gate, only when a
listener will consume them). status defaults to None -> derived; explicit
blocked/cancelled callers still pass status through.
- transform_tool_result emit (pre-existing hook) likewise gated on
has_hook(); skips _tool_result_observer_fields when no listener.
- Removed the now-redundant _tool_result_observer_fields pre-computation at
the three ok-path call sites (model_tools, agent_runtime_helpers,
tool_executor) — the helper derives them, so the no-listener path costs
one dict lookup and the call sites shrink.
- Tests: stub has_hook=True where payload correctness is asserted; add a
no-listener regression proving post_tool_call/transform_tool_result emit
is skipped when nothing is registered.
Adds backend-neutral observer hooks for plugins: session, turn, API
request, tool, approval, and subagent lifecycle events with stable
correlation IDs (session_id, task_id, turn_id, api_request_id,
tool_call_id, parent/child subagent ids). Extends VALID_HOOKS with
api_request_error and subagent_start.
Hot path is zero-cost when no plugin subscribes: has_hook()/presence
checks gate all payload construction, request payloads are returned
by reference when no middleware rewrites, and the sanitized response
payload no longer embeds raw response objects.
Bundles the optional NeMo-Relay observability plugin
(plugins/observability/nemo_relay) as an in-repo consumer of the new
hooks, peer to the existing langfuse plugin. Fails open when the
optional nemo-relay package is not installed.
Authored-by: Bryan Bednarski <bbednarski@nvidia.com>
Salvaged from #29722 onto current main.
Tool Search read its catalog from the global registry (get_tool_definitions
with no toolset scope = 'start with everything'), so a restricted-toolset
session — subagent, kanban worker, curated gateway session — could:
1. tool_search the entire process registry, not just its granted tools, and
2. tool_call any registered plugin/MCP tool it was never given, because
registry.dispatch() has no enabled_tools gate for non-execute_code tools.
A scoped session (enabled_toolsets=['mcp-github']) reported total_available=26
and successfully invoked an out-of-scope plugin tool via tool_call.
Fix:
- handle_function_call gains enabled_toolsets/disabled_toolsets; the bridge
dispatch scopes get_tool_definitions to them (also stops polluting the
process-global _last_resolved_tool_names with out-of-scope tools, which
leaked into execute_code's sandbox-tool fallback).
- A defense-in-depth gate rejects any tool_call'd name not in the scoped
deferrable catalog.
- tool_executor's unwrap (both concurrent + sequential paths) enforces the
same scope before dispatch, since it unwraps tool_call -> underlying name
and bypasses the bridge branch. New _tool_search_scoped_names() helper,
cached per-agent on registry generation + toolset scope.
- New scoped_deferrable_names() helper in tool_search.py shared by both sites.
Tests: 4 new regression tests in TestRegression_ToolsetScoping (scoped
catalog, out-of-scope tool_call rejection, no global pollution, helper).
Remove unused imports (F401) and duplicate/shadowed import
redefinitions (F811) across the codebase using ruff's safe
autofixes. No behavioral changes -- imports only.
- ~1400 safe autofixes applied across 644 files (net -1072 lines)
- __init__.py re-exports preserved (excluded from F401 removal so
public re-export surfaces stay intact)
- Re-exports that are imported or monkeypatched by tests but look
unused in their defining module are kept with explicit # noqa:
F401 (gateway/run.py load_dotenv; run_agent re-exports from
agent.message_sanitization, agent.context_compressor,
agent.retry_utils, agent.prompt_builder, agent.process_bootstrap,
agent.codex_responses_adapter)
- Unsafe F841 (unused-variable) fixes deliberately skipped -- those
can change behavior when the RHS has side effects
- ruff lints remain disabled in pyproject.toml (only PLW1514 is
selected); this is a one-time cleanup, not a config change
Verification:
- python -m compileall: clean
- pytest --collect-only: all 27161 tests collect (zero import errors)
- core entry points import clean (run_agent, model_tools, cli,
toolsets, hermes_state, batch_runner, gateway)
- static scan: every name any test imports directly from an edited
module still resolves
api_messages is built once before the retry loop while the primary provider
is active. When a mid-conversation fallback switches to a require-side thinking
provider (DeepSeek/Kimi/MiMo), assistant turns built under a non-require primary
(e.g. Codex) go out without reasoning_content and the new provider rejects the
request with HTTP 400 ("reasoning_content must be passed back").
Re-apply the echo-back pad against the current provider immediately before
building the request kwargs. Idempotent and a no-op unless the active provider
enforces echo-back, so it covers all fallback paths without affecting normal or
reject-side operation.
Drafted by Claude (Opus 4.7) under human review while fixing a personal deployment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes#33163.
When _try_activate_fallback() switches from one provider to another (e.g.
openai-codex → openrouter), the credential pool still belongs to the
primary provider. This causes two compounding bugs:
1. The pool retains the primary's base_url. Downstream pool recovery
(rate_limit / billing / auth) calls _swap_credential() with a primary
entry which overwrites the agent's base_url back to the primary's
endpoint. Every fallback request then 404s against the wrong host.
2. Pool recovery acting on errors from the FALLBACK provider mutates the
PRIMARY's pool state (#33088 reported a related corruption pattern),
exhausting/rotating entries that have nothing to do with the failure.
Two layered fixes:
a) try_activate_fallback (agent/chat_completion_helpers.py): on fallback
activation, clear agent._credential_pool when the fallback provider
doesn't match the pool's provider. Pool is preserved when the fallback
shares the pool's provider (e.g. multiple openrouter entries).
b) recover_with_credential_pool (agent/agent_runtime_helpers.py):
defensive guard rejects any pool mutation when agent.provider doesn't
match pool.provider. Defense-in-depth — should never fire after (a)
is in place, but covers any future path that attaches a stale pool.
Salvaged from @zccyman's PR #33217. The original PR was written against
the pre-refactor monolithic run_agent.py; both target functions have
since been extracted to module-level helpers. Behavior is identical —
the guards live in the canonical extracted locations.
Tests
- New tests/run_agent/test_fallback_credential_isolation.py (7 tests
covering: fallback clears mismatched pool, fallback preserves matching
pool, recovery rejects mismatched pool, recovery accepts matching
pool, 429-from-z.ai-doesn't-exhaust-codex-pool, _client_kwargs
base_url survives pool clear, _swap_credential doesn't restore
primary URL after fallback).
- Cross-verified: 77/77 passing across fallback isolation tests +
agent/test_credential_pool.py — no regression.
Co-authored-by: zccyman <16263913+zccyman@users.noreply.github.com>
Closes#33175.
switch_model() in agent/agent_runtime_helpers.py mutated agent.model and
agent.provider before rebuilding the client, with no try/except to restore
them on failure. If the rebuild raised (bad API key, network error,
build_anthropic_client failure, etc.) the agent was left with the new
model+provider name paired with the OLD client — producing HTTP 400s like
"claude-sonnet-4-6 is not supported on openai-codex" on the next turn.
Callers in cli.py, gateway/run.py, and tui_gateway/server.py already catch
the exception and warn the user, but the warning was misleading because
the swap had partially succeeded; the agent's state was torn.
Snapshot every mutated field before the swap, wrap the swap+rebuild block
in try/except, and restore the snapshot on failure before re-raising so
the caller's warning surfaces.
Reported by @amirariff91. Tests cover both branches (chat_completions and
anthropic_messages) and the cross-branch case (anthropic -> openai).
After key #1 is marked exhausted the retry still called the API with key #1
due to env-var bias in _get_cached_client / resolve_api_key_provider_credentials.
Fix: peek the pool and pass the active entry's key as explicit_api_key.
Secondary: api_key_hint in mark_exhausted_and_rotate pins the correct entry
under concurrent CLI+gateway calls; _is_payment_error matches GoUsageLimitError;
extract_api_error_context parses "Resets in Xhr Ymin".
Closes#26145.
When the user interrupts the retry loop between two 429s (Ctrl-C in
interactive mode, /new, gateway disconnect), the local has_retried_429
flag dies with the recovery function. On the next user prompt the agent
restarts with has_retried_429=False, hits 429 on the exhausted credential,
sets the flag, returns 'retry once'. Repeat forever — the second 429 that
would trigger rotation is never reached, and healthy entries (priority>0
free/paid accounts) are never tried.
Fix: in recover_with_credential_pool's rate_limit branch, pre-check
pool.current().last_status before running the retry-once dance. If the
current entry is already STATUS_EXHAUSTED, rotate immediately. Uses
getattr() for the attribute read so existing tests with SimpleNamespace
mocks (which only set 'label') keep working.
Co-authored-by: zccyman <16263913+zccyman@users.noreply.github.com>
_recover_with_credential_pool had a second classification site that blanket-
treated any 403 against xai-oauth as entitlement (defense-in-depth for
#26847). That override defeated the new _is_entitlement_failure
disambiguator from the parent commit — bad-credentials 403s still
short-circuited the refresh path.
Apply the same WKE-unauthenticated / OAuth2-validation-phrase guard at
the override site so xAI's authoritative 'this is auth, not entitlement'
signal wins there too. The #26847 catch-all still triggers for genuine
entitlement bodies that don't carry the disambiguator.
Closes the end-to-end gap exposed by
test_recover_with_credential_pool_refreshes_on_xai_bad_credentials_403.
