Add a shared PAGE_MAX_W (1200px) and center OverlayMain within its pane
so settings and command center bodies stay readable instead of sprawling
on wide/ultrawide displays.
hermes debug share reads os.getenv — the invoking terminal's environment — but
launchd/systemd and the desktop-spawned `serve` backend load credentials from
~/.hermes/.env, not the login shell. A key exported in the shell but absent
from .env is invisible to the backend, yet the dump printed a bare "set",
sending support down a phantom "the key is configured" path.
This was the actual trap behind a "Desktop has no web_search / no tools"
report: FIRECRAWL_API_KEY was a shell export (so `debug share` in a terminal
read "firecrawl set") but not in .env, so the launchd backend's
check_web_api_key returned False and web_search was gated off — which a
contributor then misdiagnosed as a missing `desktop` platform registration.
The dump now annotates any key set in-process but missing from ~/.hermes/.env
with "(shell only — not in .env; managed/desktop backend may not see it)" so
the mismatch is obvious instead of hidden behind "set".
The merged webhook session-close fix (#57370, salvaging #57322) wrapped
handle_message in a try/finally — but BasePlatformAdapter.handle_message
is fire-and-forget: it spawns _process_message_background and returns
before the agent run starts. The finally-close therefore ran BEFORE
get_or_create_session created the session row, found no session_id, and
silently no-op'd — the ghost-session leak persisted on the real path.
(The shipped test masked this by stubbing handle_message with a fake
that created the row synchronously.)
Move the close to an on_processing_complete override — the lifecycle
hook the base class fires at the TRUE end of the run, on the success,
failure, and cancellation paths alike. Empirically verified through the
real fire-and-forget pipeline: before, ended_at stayed NULL; after,
ended_at is set with end_reason=webhook_complete and the row is
prunable.
Tests now stub only the runner-side _message_handler (the seam the live
gateway injects) so handle_message / _process_message_background /
on_processing_complete all run for real; adds an AsyncSessionDB-facade
coverage test for the coroutine-await branch.
Broadens Tranquil-Flow's profile-startup timeout fix (#48518) from getProfiles
+ refreshActiveProfile to the rest of the calls the desktop fires during
connect: /api/config, /api/config/defaults, /api/model/info, /api/model/options,
/api/cron/jobs. On a profile-heavy or remote install any of these can exceed
the 15s DEFAULT_FETCH_TIMEOUT_MS while the backend is alive-but-busy (e.g.
list_profiles walks the skill tree per profile), surfacing as the spurious
"Timed out connecting to Hermes backend after 15000ms" that hangs the UI
(#48504).
Uses the surgical per-call mechanism (renamed STARTUP_PROFILE_REQUEST_TIMEOUT_MS
→ STARTUP_REQUEST_TIMEOUT_MS) rather than raising the global default (the
alternative in #48526): the liveness poll /api/status and all interactive/
runtime calls keep the short default, so a genuinely-dead backend is still
detected fast and the boot readiness probe (waitForHermes) is untouched.
Supersedes #48518 (carried as the base commit) and #48526 (global-default
raise). Fixes#48504.
Co-authored-by: YapBi <129007007+HeLLGURD@users.noreply.github.com>
Co-authored-by: Tranquil-Flow <66773372+Tranquil-Flow@users.noreply.github.com>
Follow-up to @helix4u's #57336 salvage. Two review findings:
- W1: model-picker grouped custom-provider rows by
(api_url, credential, api_mode) but NOT extra_headers. Entries sharing a
URL+credential+api_mode yet declaring different headers (e.g. per-tenant
routing behind one proxy) collapsed into one row and probed /models with
whichever header set was seen first (order-dependent). Fold a canonical
header identity into group_key so distinct header-authed endpoints stay
separate; drops the now-dead first-non-empty merge branch.
- W2: the extra_headers stringify+None-filter comprehension existed in 5
copies (config.py x2, runtime_provider.py, model_switch.py, models.py).
Extract one shared hermes_cli.config.normalize_extra_headers primitive;
all sites now call it.
Tests: +normalize_extra_headers unit tests, +regression test proving two
same-endpoint entries with different headers stay distinct and each probes
with its own headers. 223 targeted tests pass; ruff clean.
Follow-up to the cherry-picked #31856 fix. The contributor's guard defers
idle-TTL eviction until the session store reports the session expired, so the
expiry watcher can tear the agent down and fire MemoryProvider.on_session_end()
with the live transcript. Two gaps remained:
1. Memory-leak regression for mode='none' sessions. _is_session_expired()
returns False forever for the 'none' reset policy, so the naive guard would
never idle-evict those agents — reopening the unbounded-cache leak the idle
sweep (#11565) exists to relieve. Added SessionStore.is_session_finalizable()
(a public predicate: will the expiry watcher EVER finalize this session?) and
gate the deferral on it. mode='none' agents fall through to soft eviction as
before.
2. on_session_end still dropped on the LRU-cap path. Both cache-pressure paths
(_enforce_agent_cache_cap and _sweep_idle_cached_agents) soft-evict via
_release_evicted_agent_soft, which by design does NOT fire on_session_end.
If cache pressure evicts a finalizable-but-not-yet-expired agent before it
expires, the watcher later finds no cached agent and the hook is skipped.
Added _commit_memory_before_soft_evict(): at LRU eviction, if the session is
finalizable and not yet expired, commit end-of-session extraction via the
live agent's own (fully-scoped) memory manager using commit_memory_session()
— extraction WITHOUT provider teardown, so the eviction stays soft and a
resumed turn keeps working. Skipped for mode='none' (no missed boundary to
compensate) and expired sessions (the watcher tears those down directly).
This closes#11205 for ALL eviction paths and reset policies, not just the
idle-sweep + finite-policy case, while preserving the soft-eviction
resumability contract (never calls close() on a live session).
Tests: 5 new cases in test_agent_cache.py (mode='none' still reaped, LRU-cap
commits for finalizable / skips for none, real is_session_finalizable
predicate); all mutation-checked. Contributor's original 2 tests updated to
assert the finalizable path explicitly.
The idle-TTL sweep (_sweep_idle_cached_agents) was evicting agents
as soon as they passed _AGENT_CACHE_IDLE_TTL_SECS, even when the
session hadn't expired yet. In daily-reset mode the reset can fire
hours after the last user message — evicting the agent early means
the session-expiry watcher has no agent in cache to call
on_session_end() with, so memory providers miss the live transcript.
Now the sweep checks the session store before evicting: if the
session still exists and hasn't expired, the agent stays in cache
so the expiry watcher can tear it down properly later.
When the session store is unavailable or throws, falls back to the
original eviction behavior (safe default).
Fixes: #11205
Self-review (ruff+ty lint diff = 0 net-new; 2-agent deep review) surfaced one
Warning + comment-accuracy nits; no Critical:
- W1: the local-probe TTL cache memoized None (probe failure) for 30s, so a
probe that failed during a startup race would suppress a legit retry once
the server came up. Cache only positive results — still fully bounds the
hot-path probe rate (reachable servers cache their value) while an
unreachable one re-probes on the next call. Add a regression test asserting
a None result is NOT cached (retry re-probes); mutation-verified.
- Tighten the platform-guard comment: gateway/TUI/cron already construct with
quiet_mode=True (gated by `not agent.quiet_mode`), so the guard's active job
is CLI dedup vs show_banner, not "filling the gateway/TUI gap" as originally
worded.
Verified not-issues (per review): positive-value 30s cache does not break the
reconcile-after-restart freshness contract (restart = fresh process, empty
cache); cache key is collision-safe; platform guard is correct in both
directions (no runtime path leaves platform None on a non-CLI surface).
Tests: 149 passed. ruff clean; ty 0 net-new vs base.
Desktop/dashboard WebSocket connections drop during long agent operations
(delegate_task subagents, large model outputs) when the uvicorn event loop is
GIL-starved for minutes. Root cause: uvicorn's ws keepalive ping runs on the
SAME event loop as agent turns. A single synchronous GIL-holding call on a
worker thread (a regex/scrub over a large output, or a long subagent turn)
freezes the loop, so it cannot process the incoming pong within ws_ping_timeout
and uvicorn closes an otherwise-healthy connection (#53773: 'event loop stalled
226.3s'; #48445/#50005). Loosening the timeout only raises the threshold — a
multi-minute stall sails past any finite window.
The keepalive ping exists to detect half-open connections (reverse-proxy 524,
dropped tunnels), which cannot happen on loopback: there is no network or proxy
in the path, and a dead local client tears the socket down with a real FIN/RST
that starlette surfaces as WebSocketDisconnect regardless of the ping. So on
loopback the ping provides ~no liveness value while actively killing
recoverable stalls — disable it entirely (ws_ping_interval/timeout=None).
Non-loopback (public) binds sit behind a Cloudflare Tunnel where half-open IS a
real failure mode, so the ping stays at 20/20 to detect it.
Empirically verified (real uvicorn + websockets peer): with ws_ping=None the
server never closes a silent peer during an 8s window; with the pre-fix 2s/2s
window uvicorn closes it. A genuinely-dead client still fires the
WebSocketDisconnect reap path regardless of the ping.
Note: this fixes the local Desktop case (the OP's scenario). A remote Desktop
over an authenticated public dashboard route (McCalebTheSecond's comment) keeps
the ping and needs the deeper GIL-hotspot fix — tracked separately.
Closes#53773
Salvage review of #56431 surfaced one Critical + two Warning issues; fix
them on top of the contributor's cherry-picked commits:
1. Critical — duplicate non-agentic warning on the interactive CLI. The new
agent_init warning fires on every platform, but cli.py show_banner()
already warns on CLI (richer output + /model hint), so a CLI user saw the
warning twice per startup. Guard the agent_init emit to skip platform=="cli"
— it now fills exactly the gateway/TUI gap the PR intended, no duplication.
2. Warning — vLLM error-parse regex under-matched. The patterns required a
literal space before the number, so "max_model_len: 32768", "=32768",
"(32768)", and "... is 32768" all returned None. Broaden both patterns to
accept :/=/(/ 'is' delimiters. Add a parametrized test over all delimiter
variants.
3. Warning — per-call live probe latency on local endpoints. The new
reconcile-on-hit + pre-defaults step-7 probe made every local resolution
fire a synchronous network probe (banner + /model switch + compressor
update_model each within one startup). Add a 30s in-process TTL cache
keyed by (model, base_url) around _query_local_context_length so back-to-
back resolutions reuse one round-trip; not persisted to disk, so the
reconcile freshness contract (re-probe after restart) is preserved. Add an
autouse fixture clearing the cache between tests + TTL coverage.
Tests: 148 passed (was 138). ruff clean.
Replace the webhook delivery-close path's direct reach into private
SessionStore._entries (which also bypassed the store lock) with a public,
lock-held peek_session_id(session_key) accessor. Mirrors the existing
lookup_by_session_id inverse helper. Keeps a getattr fallback for older
stores / test doubles. Adds a unit test for the accessor.
Webhook deliveries created a unique one-shot session (delivery_id baked into
the session key at gateway/platforms/webhook.py:668) but the adapter fired
handle_message via asyncio.create_task WITHOUT ever ending the session
(webhook.py:713, pre-fix). Nothing else closes it: the gateway caches/expires
the agent per session_key but never calls end_session for the webhook path,
and _end_session_on_close teardown doesn't run for these fire-and-forget tasks.
SessionDB.prune_sessions (hermes_state.py:4965) only deletes rows WHERE
ended_at IS NOT NULL. So every webhook session stayed with ended_at NULL ->
unprunable -> unbounded state.db growth. This was the primary driver of the
SQLite lock-contention gateway outage.
Fix: wrap the delivery in _run_delivery_and_close, which awaits
handle_message and then (in finally, so failures still reap) calls
_end_webhook_session -> SessionDB.end_session(session_id, 'webhook_complete').
This mirrors how cron closes its session with 'cron_complete'
(cron/scheduler.py:3065). end_session is first-reason-wins and no-ops on an
already-ended row, so it never clobbers a compression/agent_close reason.
Adds tests/gateway/test_webhook_session_close.py asserting the invariant
(a completed webhook session has ended_at set + is prunable), including the
error-path case, against a real SessionStore + SessionDB.
Add regression tests for vLLM max_model_len error parsing, stale local
cache reconciliation, live probes over llama defaults, and the 64K minimum
guard on persistent cache writes.
(cherry picked from commit 1cb47ef437de7ce289cb358e8d6b89e9194b43ed)
Reconcile stale local disk cache against live vLLM/Ollama max_model_len
probes, probe local servers before the llama hardcoded default, parse
vLLM max_model_len overflow errors, and surface the non-agentic Hermes 3/4
warning at agent init on gateway/TUI.
Sub-64K live probes are returned for startup rejection but are not
persisted to the context cache — preserving the 64K minimum-context
contract instead of normalizing undersized windows as valid config.
(cherry picked from commit c3a02db4fd9d57b7b0eb2732de91f8334d311aa5)
Deduplicate the '_BULLET_RE.match or _ORDERED_RE.match' idiom used at the
list-run entry guard and the blank-line lookahead into a single helper, so
adding future marker types is a one-point change. Pure refactor, no
behavior change (22 block_kit tests still pass).
Refine the blank-line handling so a blank line only continues a list run
when the next non-blank line is another list item. This keeps a list ->
paragraph -> list sequence as three separate blocks and matches the
contiguous-list layout for mixed/nested lists (one rich_text block, split
into sub-lists by (indent, ordered)), rather than emitting a separate
block per item.
Adds regression tests for the mixed blank-separated layout and the
list->paragraph->list boundary.
When a Markdown ordered list has blank lines between items (common in
LLM-authored content), the list run loop breaks on each blank line.
Slack numbers each rich_text_list independently, so N items produce N
lists each starting at 1.
Skip blank lines inside the list run as soft separators instead of
breaking, so ordered items stay in one rich_text_list and Slack renders
the correct numbering.
Fixes#57076
normalize_usage only read output_tokens_details.reasoning_tokens (the
Responses API shape). Chat Completions providers — OpenAI, OpenRouter,
DeepSeek, and every OpenAI-compatible proxy — report it under
completion_tokens_details.reasoning_tokens, so reasoning_tokens was 0 for
every chat_completions reasoning model: hidden thinking was invisible in
session accounting, MoA traces, and the eval's per-task token columns.
Measured impact (HermesBench MoA run on deepseek-v4-flash, 4,828 advisor
calls): reasoning_tokens showed 0 everywhere while individual calls burned
up to 21.5K hidden thinking tokens to emit ~500 visible tokens. Verified
live against OpenRouter: deepseek-v4-flash returns
completion_tokens_details.reasoning_tokens=61 for a 74-completion-token
call; the field was simply never read.
Responses-shape reads are unchanged; the new read only fires when the
Responses shape yielded nothing.
setup.runtime_check and setup.status are polled by the Desktop frontend on
connect and periodically (use-status-snapshot → evaluateRuntimeReadiness), but
neither was in _LONG_HANDLERS — so dispatch() ran both inline on the WS reader
thread. Under GIL pressure from concurrent agent turns (terminal I/O, large
output, background-process completions) either can block for seconds:
- setup.runtime_check → resolve_runtime_provider() (config read, auth check,
may probe the provider endpoint)
- setup.status → _has_any_provider_configured() (provider config + credential
scan)
While either blocks the reader thread the WS read loop can't service later
requests; the frontend RPC timeout fires, the client drops the socket, and the
lost setup.runtime_check response reads as ready=false — a false "needs setup"
/ "Settings failed to load" even though the provider is configured.
Route both to the RPC pool (same precedent as #55545's session.list/pet.info/
process.list). The handlers are read-only and pool writes go through the
lock-guarded write_json, so there's no ordering or safety concern.
Test asserts all 5 frontend-polled RPCs are pool-routed.
Co-authored-by: izumi0uu <izumi0uu@gmail.com>
delete_profile stopped only the process named in gateway.pid, but a Desktop
app spawns a headless `serve`/`dashboard` backend per profile that holds the
profile's SQLite connection open and keeps writing sessions/WAL/sandbox files.
That backend is never in gateway.pid, so a CLI `hermes profile delete` run
while the Desktop app is up left it writing into the tree — rmtree's final
rmdir then failed with ENOTEMPTY (#47368 "Bug 2"), and pre-guard it also
resurrected the directory.
- _profile_bound_backend_pids(): find running Hermes backends bound to this
profile via a `--profile <name>` selector or a HERMES_HOME env resolving to
the profile dir. Tightly scoped — current-user only, backend subcommands
(serve/dashboard/gateway) only so an interactive chat is never killed, and
never this process or its ancestors.
- _stop_profile_backends(): terminate them (graceful, then force), best-effort
so it can never make delete worse.
- _rmtree_with_retry(): a few spaced retries absorb the ENOTEMPTY / Windows
file-lock race from a just-terminated writer's in-flight -wal/-shm/sandbox
writes instead of failing the whole delete on a race the next attempt wins.
Complements the recreation guard (deleted profiles no longer reappear) and the
Desktop teardown-before-delete flow; this is the CLI-side convergence fix for a
delete run while a Desktop-managed backend is live.
Part of #47368.
A Z.ai desktop user reported thinking reverting to medium after one turn,
burning ~200% of a week's credits in 4 days despite reasoning_effort: false
in config.yaml. Four compounding bugs:
- _session_info reported reasoning_effort "" for disabled reasoning,
indistinguishable from unset — the desktop adopted it after the first
turn, wiping its sticky "thinking off" pick so every later chat
reverted to the default effort.
- config.set key=reasoning always wrote agent.reasoning_effort to global
config.yaml, so every desktop model-menu selection (preset.effort ??
'medium') clobbered the user's configured value. Now session-scoped
like the messaging gateway's /reasoning, landing on
create_reasoning_override so lazily-built sessions keep it too.
- YAML `reasoning_effort: false`/`off`/`no` (boolean False) was coerced
to "" by every loader's `str(x or "")`, silently re-enabling thinking.
parse_reasoning_effort now treats False/"false"/"disabled" as
{"enabled": False}; loaders (tui gateway, gateway, cli, cron,
delegate) pass the raw value through. The desktop config reader also
crashed on the boolean (false.trim()), aborting voice/STT settings.
- The zai provider profile never sent thinking on the wire, and GLM-4.5+
defaults to thinking ON server-side — so disabling reasoning was a
silent no-op on direct Z.ai, the actual token burner. The profile now
emits extra_body.thinking {"type": "enabled"|"disabled"} for
thinking-capable GLM models, mirroring the DeepSeek profile.
Also: /new (session reset) now carries reasoning_config across the
rebuild like model_override; config.get reasoning prefers the session's
live value and maps a config False to "none"; Settings shows "Off"
instead of a blank select for hand-written false.
When the renderer sends a DELETE /api/profiles/{name} request, the IPC
handler tears down the profile's pool backend (or primary backend) via
prepareProfileDeleteRequest. However, the very next line calls
ensureBackend(profile), which spawns a fresh pool backend for the just-
deleted profile. The new backend's startup path calls ensure_hermes_home(),
which recreates the profile directory — defeating the deletion and leaving
the process as a zombie.
On the next Desktop restart the cycle repeats: the profile directory exists,
the Desktop spawns a backend, the backend recreates the directory after
deletion, and PIDs accumulate indefinitely.
Fix: make prepareProfileDeleteRequest return the torn-down profile name.
The IPC handler uses this to route the DELETE to the primary backend
instead of spawning a new pool backend for the deleted profile.
Fixes#52279
The terminal-refresh quarantine filtered in-memory entries on
source == "device_code" but built removed_ids from the deleted
"loopback_pkce" source name, so the revoked device-code entry was
never pruned from the persisted pool in auth.json. Also restores the
_print_loopback_ssh_hint test suite scoped to Spotify (the helper's
remaining caller) instead of deleting it wholesale.
Replace the loopback/PKCE-callback server and manual-paste fallback with
the RFC 8628 device-code flow as the only xAI Grok OAuth login path. The
flow works in headless/SSH/container sessions with no 127.0.0.1 listener,
shrinking the local attack surface.
- Poll the token endpoint with server-provided interval, honoring
slow_down and expires_in; store tokens with auth_mode
oauth_device_code.
- Adaptive proactive refresh skew for short-lived device-code JWTs;
rotated tokens sync back to auth.json, the global root store, and the
credential pool (no refresh-token replay).
- Clear source suppression on successful re-login (CLI + dashboard) and
drop the duplicate dashboard pool entry so exactly one seeded
device_code entry exists.
- Use the shared device_code source name for consistency with the
nous/codex device-code providers.
- Desktop: remove the loopback OAuth flow states and dead type variants;
pkce providers' sign-in URL selection is unchanged.
- Docs (EN + zh-Hans) rewritten for device-code login; drop the deleted
--manual-paste flag from documented commands.
The colored-square rail stops scaling once a user racks up many profiles:
tiny drag targets and an endless horizontal scroll strip. Past a threshold
(13) the rail swaps the squares for a compact select dropdown — same active
tint + initial glyph, minus the drag-reorder / long-press-recolor / per-row
context menu that only make sense at small counts. Two render paths behind
one flag; the left default↔all toggle, the "+" create button, and Manage
stay put in both. Rename/delete/color remain reachable via Manage.
MSYS_NO_PATHCONV is honored by Git for Windows bash only. _find_bash's
final shutil.which fallback can return MSYS2-proper or Cygwin bash,
which ignore it and honor MSYS2_ARG_CONV_EXCL instead. Set both so argv
path conversion stays disabled regardless of which bash flavor spawns.
Also subsumes the cmd /c mangling in #56147.
Git Bash mangles native Windows command flags (/FO, /TN, /Create) into
bogus paths. Hermes terminal and background spawns now opt out by default
so tasklist, schtasks, and wmic work without manual prefixes.
Fixes#56700.
Both Desktop picker surfaces (status-bar model menu, settings/onboarding
dialog) only asked the connected gateway's model.options once a session
existed; before that they fell back to the Desktop REST/global options, which
can't see virtual providers a remote gateway exposes — including the MoA
presets from #53817. Centralize the fetch rule in requestModelOptions(): prefer
the connected gateway whenever one exists (no session_id needed — the RPC
resolves disk config), REST only when no gateway is connected.
The status-bar MoA preset section now renders from the same model.options
payload (the virtual `moa` provider row) instead of the local /api/model/moa
REST config, so remote presets appear correctly; the row is filtered out of
the main provider groups so presets don't list twice. Preset selection keeps
the persistent switchTo path from #56417 and drops the vestigial session gate —
like regular model rows, a pre-session pick ships on the next session.create.
Fixes#53817.
Rebased and reconciled with #56417 (persistent MoA selection), which landed
after this PR was opened and covered its one-shot-/moa half.
attachImagePath fetched its thumbnail through readDesktopFileDataUrl, which in
remote mode routes every read to the gateway fs bridge. Paperclip picks,
clipboard saves, and OS drops always produce paths on the LOCAL machine, so the
gateway read 404s — toasting "image preview failed" and dropping the thumbnail
even though the attach itself works (upload reads local bytes via the Electron
bridge). Read the local bridge first and fall back to the remote facade, which
still serves in-app drags from the remote project tree. Local mode is
unchanged (the facade already reads locally there).
Follow-up to #56572, which restored the remote paperclip picker and made this
path reachable from the picker as well.
startUpdatePoller() only called checkBackendUpdates() — never checkUpdates().
The statusbar version pill reads $updateStatus (set by checkUpdates()), so the
commit-behind counter stayed null after restart. It only appeared when the user
manually clicked the pill, which triggered checkUpdates() via openUpdateOverlayFor.
Added void checkUpdates() in three places alongside the existing
checkBackendUpdates() calls:
- On startup in startUpdatePoller()
- In the 30-minute setInterval callback
- In the onFocus handler
checkUpdates() uses the Electron IPC bridge (local git check), not the gateway,
so no mode gating is needed. The existing $updateChecking atom guard prevents
double-fire on overlap.
Fixes#53079
