feat(desktop+gateway): remote media relay — attach images/PDFs and display gateway images over the network

Desktop connected to a remote gateway can now attach images and PDFs and
display agent-written images. Previously the desktop passed a LOCAL file path
to image.attach; on a remote gateway that path doesn't exist, so the image was
silently dropped ("skipped unreadable path") and the vision model never saw it.
The reverse direction was also broken — images the agent wrote on the gateway
rendered as dead links in the remote client.

Gateway (tui_gateway/server.py):
- image.attach_bytes: base64 byte upload written into the gateway's own images
  dir and queued via the existing native-image-attach pipeline. Magic-byte
  extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap,
  structured error codes. Accepts content_base64/filename (canonical) and
  data/ext (older-desktop aliases).
- pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI
  and queues the pages as images; 50 MB / 25-page caps. Accepts host path or
  base64 upload.
- Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image)
  so the two methods and the existing image.attach don't duplicate logic.

Gateway (hermes_cli/web_server.py):
- GET /api/media: returns a gateway-local image as a base64 data URL so remote
  clients can display it. Auth-gated like every /api route, extension
  allowlist + size cap, AND confined to the gateway's own media roots
  (images/screenshots/cache, resolved symlink-safe) so an authed caller can't
  read image-extension files anywhere on disk.

Desktop (apps/desktop):
- syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the
  connection mode is 'remote'; the local fast path is unchanged.
- media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and
  markdown-text fetch images over /api/media in remote mode.

Consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437)
into one coherent implementation, taking the strongest parts of each and adding
shared-helper cleanup plus the /api/media root-confinement hardening on top.
The per-profile gateway switching from #38876 is intentionally left out as a
separable feature. TUI file uploads (#40492) remain a separate surface.

Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop
media.remote unit tests; full tui_gateway + web_server suites green (472
passed); tsc -b clean; E2E verified the full attach→disk→queue and
gateway-path→data-URL display round-trip plus the out-of-root security block.

Co-authored-by: Max Mitcham <maxmitcham@mac.home>
Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com>
Co-authored-by: Chris Cook <ccook@nvms.com>
Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>
This commit is contained in:
teknium1 2026-06-07 04:37:38 -07:00 committed by Teknium
parent 20fd0bde5d
commit 16786f3bb3
11 changed files with 759 additions and 11 deletions

View file

@ -796,6 +796,74 @@ def _probe_gateway_health() -> tuple[bool, dict | None]:
return False, None
# Image MIME types this endpoint will serve. Extension-allowlisted so an
# authenticated caller can't pull non-image files through it.
_MEDIA_CONTENT_TYPES = {
".png": "image/png",
".jpg": "image/jpeg",
".jpeg": "image/jpeg",
".gif": "image/gif",
".webp": "image/webp",
".svg": "image/svg+xml",
".bmp": "image/bmp",
".ico": "image/x-icon",
}
_MEDIA_MAX_BYTES = 25 * 1024 * 1024
def _media_serve_roots() -> list[Path]:
"""Directories ``GET /api/media`` is allowed to read from.
Confined to where the agent and attach pipeline actually write media on the
gateway host its images dir and cache subtree. This stops an authenticated
client from reading image-extension files anywhere on disk (e.g. a renamed
key or a screenshot outside the cache) merely because the suffix passes the
allowlist.
"""
home = get_hermes_home()
roots = [home / "images", home / "screenshots", home / "cache"]
out: list[Path] = []
for root in roots:
try:
out.append(root.resolve())
except (OSError, RuntimeError):
continue
return out
@app.get("/api/media")
async def get_media(path: str):
"""Return a gateway-local image file as a base64 data URL.
Lets remote clients (the desktop app over the network, or the web dashboard
in a browser) display images the agent wrote to *this* machine's filesystem
they can't read the gateway's local disk directly.
Auth-gated by the session token like every other /api route. Restricted to
an image-extension allowlist, a size cap, AND the gateway's own media roots
(resolved, symlink-safe) so it can't be used to read arbitrary files.
"""
try:
target = Path(path).expanduser().resolve()
except (OSError, RuntimeError):
raise HTTPException(status_code=400, detail="Invalid path")
if target.suffix.lower() not in _MEDIA_CONTENT_TYPES:
raise HTTPException(status_code=415, detail="Unsupported media type")
roots = _media_serve_roots()
if not any(target == root or root in target.parents for root in roots):
raise HTTPException(status_code=403, detail="Path outside media roots")
if not target.is_file():
raise HTTPException(status_code=404, detail="File not found")
if target.stat().st_size > _MEDIA_MAX_BYTES:
raise HTTPException(status_code=413, detail="File too large")
encoded = base64.b64encode(target.read_bytes()).decode("ascii")
return {"data_url": f"data:{_MEDIA_CONTENT_TYPES[target.suffix.lower()]};base64,{encoded}"}
@app.get("/api/status")
async def get_status():
current_ver, latest_ver = check_config_version()