feat(desktop+gateway): remote media relay — attach images/PDFs and display gateway images over the network
Desktop connected to a remote gateway can now attach images and PDFs and
display agent-written images. Previously the desktop passed a LOCAL file path
to image.attach; on a remote gateway that path doesn't exist, so the image was
silently dropped ("skipped unreadable path") and the vision model never saw it.
The reverse direction was also broken — images the agent wrote on the gateway
rendered as dead links in the remote client.
Gateway (tui_gateway/server.py):
- image.attach_bytes: base64 byte upload written into the gateway's own images
dir and queued via the existing native-image-attach pipeline. Magic-byte
extension sniffing, data-URL prefix + whitespace tolerance, 25 MB cap,
structured error codes. Accepts content_base64/filename (canonical) and
data/ext (older-desktop aliases).
- pdf.attach: renders each page to PNG via pdftoppm (poppler-utils) at 150 DPI
and queues the pages as images; 50 MB / 25-page caps. Accepts host path or
base64 upload.
- Shared helpers (_decode_attach_base64, _sniff_image_ext, _queue_attached_image)
so the two methods and the existing image.attach don't duplicate logic.
Gateway (hermes_cli/web_server.py):
- GET /api/media: returns a gateway-local image as a base64 data URL so remote
clients can display it. Auth-gated like every /api route, extension
allowlist + size cap, AND confined to the gateway's own media roots
(images/screenshots/cache, resolved symlink-safe) so an authed caller can't
read image-extension files anywhere on disk.
Desktop (apps/desktop):
- syncImageAttachmentsForSubmit uploads bytes via image.attach_bytes when the
connection mode is 'remote'; the local fast path is unchanged.
- media.ts gains isRemoteGateway() + gatewayMediaDataUrl(); directive-text and
markdown-text fetch images over /api/media in remote mode.
Consolidates the competing remote-media PRs (#38876, #40317, #21908, #39437)
into one coherent implementation, taking the strongest parts of each and adding
shared-helper cleanup plus the /api/media root-confinement hardening on top.
The per-profile gateway switching from #38876 is intentionally left out as a
separable feature. TUI file uploads (#40492) remain a separate surface.
Tested: 11 new tui_gateway tests + 5 /api/media endpoint tests + desktop
media.remote unit tests; full tui_gateway + web_server suites green (472
passed); tsc -b clean; E2E verified the full attach→disk→queue and
gateway-path→data-URL display round-trip plus the out-of-root security block.
Co-authored-by: Max Mitcham <maxmitcham@mac.home>
Co-authored-by: Justlrnal4 <Justlrnal4@users.noreply.github.com>
Co-authored-by: Chris Cook <ccook@nvms.com>
Co-authored-by: Thomas Paquette <thomas.paquette@gmail.com>
This commit is contained in:
parent
20fd0bde5d
commit
16786f3bb3
11 changed files with 759 additions and 11 deletions
|
|
@ -796,6 +796,74 @@ def _probe_gateway_health() -> tuple[bool, dict | None]:
|
|||
return False, None
|
||||
|
||||
|
||||
# Image MIME types this endpoint will serve. Extension-allowlisted so an
|
||||
# authenticated caller can't pull non-image files through it.
|
||||
_MEDIA_CONTENT_TYPES = {
|
||||
".png": "image/png",
|
||||
".jpg": "image/jpeg",
|
||||
".jpeg": "image/jpeg",
|
||||
".gif": "image/gif",
|
||||
".webp": "image/webp",
|
||||
".svg": "image/svg+xml",
|
||||
".bmp": "image/bmp",
|
||||
".ico": "image/x-icon",
|
||||
}
|
||||
_MEDIA_MAX_BYTES = 25 * 1024 * 1024
|
||||
|
||||
|
||||
def _media_serve_roots() -> list[Path]:
|
||||
"""Directories ``GET /api/media`` is allowed to read from.
|
||||
|
||||
Confined to where the agent and attach pipeline actually write media on the
|
||||
gateway host — its images dir and cache subtree. This stops an authenticated
|
||||
client from reading image-extension files anywhere on disk (e.g. a renamed
|
||||
key or a screenshot outside the cache) merely because the suffix passes the
|
||||
allowlist.
|
||||
"""
|
||||
home = get_hermes_home()
|
||||
roots = [home / "images", home / "screenshots", home / "cache"]
|
||||
out: list[Path] = []
|
||||
for root in roots:
|
||||
try:
|
||||
out.append(root.resolve())
|
||||
except (OSError, RuntimeError):
|
||||
continue
|
||||
return out
|
||||
|
||||
|
||||
@app.get("/api/media")
|
||||
async def get_media(path: str):
|
||||
"""Return a gateway-local image file as a base64 data URL.
|
||||
|
||||
Lets remote clients (the desktop app over the network, or the web dashboard
|
||||
in a browser) display images the agent wrote to *this* machine's filesystem
|
||||
— they can't read the gateway's local disk directly.
|
||||
|
||||
Auth-gated by the session token like every other /api route. Restricted to
|
||||
an image-extension allowlist, a size cap, AND the gateway's own media roots
|
||||
(resolved, symlink-safe) so it can't be used to read arbitrary files.
|
||||
"""
|
||||
try:
|
||||
target = Path(path).expanduser().resolve()
|
||||
except (OSError, RuntimeError):
|
||||
raise HTTPException(status_code=400, detail="Invalid path")
|
||||
|
||||
if target.suffix.lower() not in _MEDIA_CONTENT_TYPES:
|
||||
raise HTTPException(status_code=415, detail="Unsupported media type")
|
||||
|
||||
roots = _media_serve_roots()
|
||||
if not any(target == root or root in target.parents for root in roots):
|
||||
raise HTTPException(status_code=403, detail="Path outside media roots")
|
||||
|
||||
if not target.is_file():
|
||||
raise HTTPException(status_code=404, detail="File not found")
|
||||
if target.stat().st_size > _MEDIA_MAX_BYTES:
|
||||
raise HTTPException(status_code=413, detail="File too large")
|
||||
|
||||
encoded = base64.b64encode(target.read_bytes()).decode("ascii")
|
||||
return {"data_url": f"data:{_MEDIA_CONTENT_TYPES[target.suffix.lower()]};base64,{encoded}"}
|
||||
|
||||
|
||||
@app.get("/api/status")
|
||||
async def get_status():
|
||||
current_ver, latest_ver = check_config_version()
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue