feat(sessions): trace export + HF upload via 'sessions export --format trace' (#60507)
* feat(trace): upload sessions to HF Agent Trace Viewer Salvage trace upload as a smaller CLI-first feature: deterministic Claude Code JSONL export, fail-closed redaction, lazy Hugging Face dependency, and no gateway slash-command wiring. * chore(trace): drop external porting references from docstrings Describe the trace-upload design in Hermes' own terms. * feat(sessions): fold trace upload into 'sessions export --format trace' Integrates the HF Agent Trace Viewer exporter (PR #36145) onto the unified export surface instead of a separate 'hermes trace' subcommand: - --format trace: Claude Code JSONL to stdout/file, or one <id>.trace.jsonl per session for filtered bulk export; defaults to the most recent session when no --session-id/filters given. - --upload pushes to the user's private HF traces dataset (--public to opt out of private); reads HF_TOKEN with guided setup when missing. - traces are secret-redacted by default (force mode); --no-redact opts out after review; redaction failure blocks export (fail closed). - hermes_cli/trace.py + subcommands/trace.py removed; agent/trace_upload.py is the single engine. Docs EN + zh-Hans; 4 new CLI tests.
This commit is contained in:
Teknium
GitHub
parent
5e51b123f3
commit
0e04d14209
7 changed files with 955 additions and 2 deletions
398
agent/trace_upload.py
Normal file
398
agent/trace_upload.py
Normal file
|
|
@ -0,0 +1,398 @@
|
|||
"""Upload a Hermes session transcript to Hugging Face as an agent trace.
|
||||
|
||||
Hermes stores sessions in its own SQLite store (``hermes_state.SessionDB``),
|
||||
so we reconstruct the conversation and emit it in the **Claude Code JSONL**
|
||||
shape — one of the three formats the Hugging Face Agent Trace Viewer
|
||||
auto-detects (Claude Code / Codex / Pi). No dataset-side preprocessing is
|
||||
needed; the Hub tags the dataset ``agent-traces`` and opens it in the viewer.
|
||||
|
||||
Docs: https://huggingface.co/docs/hub/agent-traces
|
||||
|
||||
Design notes
|
||||
------------
|
||||
* **Zero LLM turn.** This is a deterministic export — it never spends a
|
||||
model call. The ``hermes trace upload`` subcommand calls
|
||||
:func:`upload_session_trace` directly.
|
||||
* **Private by default.** Traces can contain prompts, tool output, local
|
||||
paths, and secrets. The dataset is created private and every text body
|
||||
is passed through Hermes' secret redactor (``force=True``) unless the
|
||||
caller explicitly opts out with ``redact=False``.
|
||||
* **Never raises.** Returns a user-facing status string so command
|
||||
handlers can echo it straight back to the user. Programmatic callers
|
||||
that need the URL can use :func:`build_trace_jsonl` + :func:`_do_upload`
|
||||
directly.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict, List, Optional, Tuple
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_DATASET_NAME = "hermes-traces"
|
||||
_HERMES_VERSION = "hermes-agent"
|
||||
_REDACTION_BLOCKED_MESSAGE = (
|
||||
"Trace upload blocked: secret redaction failed, so the transcript may "
|
||||
"still contain credentials or other sensitive data. Fix the redactor or "
|
||||
"rerun with --no-redact only after manually reviewing the transcript."
|
||||
)
|
||||
|
||||
|
||||
class TraceRedactionError(RuntimeError):
|
||||
"""Raised when a trace cannot be safely redacted before upload."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Conversion: Hermes OpenAI-format messages -> Claude Code JSONL
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _now_iso() -> str:
|
||||
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z"
|
||||
|
||||
|
||||
def _redact(text: Any, enabled: bool) -> Any:
|
||||
"""Redact secrets from a string body when redaction is enabled.
|
||||
|
||||
Non-strings pass through untouched. Uses Hermes' shared redactor with
|
||||
``force=True`` so an upload always scrubs known secret shapes even if
|
||||
the user disabled log redaction globally.
|
||||
"""
|
||||
if not enabled or not isinstance(text, str) or not text:
|
||||
return text
|
||||
try:
|
||||
from agent.redact import redact_sensitive_text
|
||||
return redact_sensitive_text(text, force=True)
|
||||
except Exception as exc:
|
||||
logger.warning("Trace upload redaction failed; refusing upload", exc_info=True)
|
||||
raise TraceRedactionError(_REDACTION_BLOCKED_MESSAGE) from exc
|
||||
|
||||
|
||||
def _content_to_blocks(content: Any, redact: bool) -> List[Dict[str, Any]]:
|
||||
"""Normalize a message ``content`` field into Anthropic content blocks."""
|
||||
if content is None:
|
||||
return []
|
||||
if isinstance(content, str):
|
||||
return [{"type": "text", "text": _redact(content, redact)}]
|
||||
if isinstance(content, list):
|
||||
blocks: List[Dict[str, Any]] = []
|
||||
for part in content:
|
||||
if isinstance(part, dict):
|
||||
ptype = part.get("type")
|
||||
if ptype == "text":
|
||||
blocks.append({"type": "text", "text": _redact(part.get("text", ""), redact)})
|
||||
elif ptype in ("image_url", "image"):
|
||||
# Keep a placeholder; the viewer renders text turns and we
|
||||
# don't want to inline base64 blobs into a trace.
|
||||
blocks.append({"type": "text", "text": "[image omitted]"})
|
||||
else:
|
||||
blocks.append({"type": "text", "text": _redact(json.dumps(part), redact)})
|
||||
else:
|
||||
blocks.append({"type": "text", "text": _redact(str(part), redact)})
|
||||
return blocks
|
||||
return [{"type": "text", "text": _redact(json.dumps(content), redact)}]
|
||||
|
||||
|
||||
def _tool_calls_to_blocks(tool_calls: Any, redact: bool) -> List[Dict[str, Any]]:
|
||||
"""Convert OpenAI tool_calls into Anthropic ``tool_use`` content blocks."""
|
||||
blocks: List[Dict[str, Any]] = []
|
||||
if not isinstance(tool_calls, list):
|
||||
return blocks
|
||||
for tc in tool_calls:
|
||||
if not isinstance(tc, dict):
|
||||
continue
|
||||
fn = tc.get("function") or {}
|
||||
name = fn.get("name") or tc.get("name") or "tool"
|
||||
raw_args = fn.get("arguments")
|
||||
if isinstance(raw_args, str):
|
||||
try:
|
||||
parsed = json.loads(raw_args) if raw_args.strip() else {}
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
parsed = {"_raw": raw_args}
|
||||
elif isinstance(raw_args, dict):
|
||||
parsed = raw_args
|
||||
else:
|
||||
parsed = {}
|
||||
if redact:
|
||||
try:
|
||||
parsed = json.loads(_redact(json.dumps(parsed), redact))
|
||||
except (json.JSONDecodeError, ValueError):
|
||||
logger.warning("Trace upload redacted tool arguments are not valid JSON; refusing upload")
|
||||
raise TraceRedactionError(_REDACTION_BLOCKED_MESSAGE)
|
||||
blocks.append({
|
||||
"type": "tool_use",
|
||||
"id": tc.get("id") or f"toolu_{uuid.uuid4().hex[:16]}",
|
||||
"name": name,
|
||||
"input": parsed,
|
||||
})
|
||||
return blocks
|
||||
|
||||
|
||||
def build_trace_jsonl(
|
||||
messages: List[Dict[str, Any]],
|
||||
*,
|
||||
session_id: str,
|
||||
model: str = "",
|
||||
cwd: str = "",
|
||||
redact: bool = True,
|
||||
) -> str:
|
||||
"""Render Hermes conversation messages as Claude Code JSONL text.
|
||||
|
||||
Each non-system message becomes one JSONL line in the Claude Code
|
||||
transcript shape the HF Agent Trace Viewer auto-detects:
|
||||
|
||||
* ``user`` / ``tool`` -> ``{"type": "user", "message": {...}}``
|
||||
* ``assistant`` -> ``{"type": "assistant", "message": {...}}``
|
||||
with ``content`` blocks (text + ``tool_use``).
|
||||
|
||||
Tool results are emitted as user turns carrying a ``tool_result``
|
||||
block keyed by ``tool_call_id`` — the same way Claude Code records
|
||||
them. Turns are linked via ``uuid`` / ``parentUuid``.
|
||||
"""
|
||||
lines: List[str] = []
|
||||
parent: Optional[str] = None
|
||||
base_ts = _now_iso()
|
||||
git_branch = ""
|
||||
try:
|
||||
import subprocess
|
||||
if cwd:
|
||||
r = subprocess.run(
|
||||
["git", "rev-parse", "--abbrev-ref", "HEAD"],
|
||||
capture_output=True, text=True, timeout=3, cwd=cwd,
|
||||
)
|
||||
if r.returncode == 0:
|
||||
git_branch = r.stdout.strip()
|
||||
except Exception:
|
||||
git_branch = ""
|
||||
|
||||
def _common(turn_uuid: str) -> Dict[str, Any]:
|
||||
return {
|
||||
"parentUuid": parent,
|
||||
"isSidechain": False,
|
||||
"userType": "external",
|
||||
"cwd": cwd or os.getcwd(),
|
||||
"sessionId": session_id,
|
||||
"version": _HERMES_VERSION,
|
||||
"gitBranch": git_branch,
|
||||
"uuid": turn_uuid,
|
||||
"timestamp": base_ts,
|
||||
}
|
||||
|
||||
for msg in messages:
|
||||
role = msg.get("role")
|
||||
if role == "system":
|
||||
continue
|
||||
turn_uuid = str(uuid.uuid4())
|
||||
|
||||
if role == "assistant":
|
||||
blocks = _content_to_blocks(msg.get("content"), redact)
|
||||
blocks.extend(_tool_calls_to_blocks(msg.get("tool_calls"), redact))
|
||||
if not blocks:
|
||||
blocks = [{"type": "text", "text": ""}]
|
||||
entry = _common(turn_uuid)
|
||||
entry["type"] = "assistant"
|
||||
entry["message"] = {
|
||||
"role": "assistant",
|
||||
"model": model or "unknown",
|
||||
"content": blocks,
|
||||
}
|
||||
lines.append(json.dumps(entry, ensure_ascii=False))
|
||||
parent = turn_uuid
|
||||
continue
|
||||
|
||||
if role == "tool":
|
||||
tool_use_id = msg.get("tool_call_id") or msg.get("tool_name") or "tool"
|
||||
result_content = _redact(
|
||||
msg.get("content") if isinstance(msg.get("content"), str)
|
||||
else json.dumps(msg.get("content")),
|
||||
redact,
|
||||
)
|
||||
entry = _common(turn_uuid)
|
||||
entry["type"] = "user"
|
||||
entry["message"] = {
|
||||
"role": "user",
|
||||
"content": [{
|
||||
"type": "tool_result",
|
||||
"tool_use_id": tool_use_id,
|
||||
"content": result_content,
|
||||
}],
|
||||
}
|
||||
lines.append(json.dumps(entry, ensure_ascii=False))
|
||||
parent = turn_uuid
|
||||
continue
|
||||
|
||||
# Default: user (and any unknown role) -> user turn.
|
||||
content = msg.get("content")
|
||||
if isinstance(content, str):
|
||||
message_content: Any = _redact(content, redact)
|
||||
else:
|
||||
message_content = _content_to_blocks(content, redact)
|
||||
entry = _common(turn_uuid)
|
||||
entry["type"] = "user"
|
||||
entry["message"] = {"role": "user", "content": message_content}
|
||||
lines.append(json.dumps(entry, ensure_ascii=False))
|
||||
parent = turn_uuid
|
||||
|
||||
return "\n".join(lines) + ("\n" if lines else "")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Upload
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _resolve_hf_token() -> Optional[str]:
|
||||
"""Return the user's Hugging Face token from the usual env vars."""
|
||||
for var in ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HUGGING_FACE_HUB_TOKEN", "HUGGINGFACE_TOKEN"):
|
||||
val = os.getenv(var)
|
||||
if val and val.strip():
|
||||
return val.strip()
|
||||
return None
|
||||
|
||||
|
||||
_NO_TOKEN_MESSAGE = (
|
||||
"Can't upload — no Hugging Face token is available. To set it up:\n"
|
||||
"\n"
|
||||
"1. Create a token with WRITE access at https://huggingface.co/settings/tokens\n"
|
||||
" (New token -> type \"Write\" -> copy it).\n"
|
||||
"2. Add it to your environment as HF_TOKEN (e.g. in ~/.hermes/.env):\n"
|
||||
" HF_TOKEN=hf_xxxxxxxxxxxxxxxxxxxx\n"
|
||||
"3. Run /upload-trace again (or `hermes trace upload`)."
|
||||
)
|
||||
|
||||
|
||||
def _do_upload(
|
||||
jsonl: str,
|
||||
*,
|
||||
token: str,
|
||||
session_id: str,
|
||||
dataset_name: str = DEFAULT_DATASET_NAME,
|
||||
private: bool = True,
|
||||
) -> str:
|
||||
"""Create (idempotently) the private dataset and push the trace file.
|
||||
|
||||
Returns a user-facing status string. Never raises.
|
||||
"""
|
||||
try:
|
||||
from tools import lazy_deps
|
||||
lazy_deps.ensure("tool.trace_upload", prompt=False)
|
||||
except Exception:
|
||||
# lazy-install unavailable/declined — fall through to the import,
|
||||
# which surfaces the install hint below if the package is missing.
|
||||
pass
|
||||
try:
|
||||
from huggingface_hub import HfApi
|
||||
except ImportError:
|
||||
return ("Hugging Face upload needs the `huggingface_hub` package "
|
||||
"(`pip install huggingface_hub`).")
|
||||
|
||||
api = HfApi(token=token)
|
||||
try:
|
||||
who = api.whoami()
|
||||
user = who.get("name") if isinstance(who, dict) else None
|
||||
except Exception as e:
|
||||
logger.warning("HF whoami failed: %s", e)
|
||||
return ("Your Hugging Face token was rejected (whoami failed). "
|
||||
"Make sure it has WRITE access and isn't expired.")
|
||||
if not user:
|
||||
return "Could not resolve your Hugging Face username from the token."
|
||||
|
||||
repo_id = f"{user}/{dataset_name}"
|
||||
try:
|
||||
api.create_repo(
|
||||
repo_id=repo_id, repo_type="dataset", private=private, exist_ok=True,
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("HF create_repo failed for %s: %s", repo_id, e)
|
||||
return f"Could not create/access dataset {repo_id}: {e}"
|
||||
|
||||
path_in_repo = f"sessions/{session_id}.jsonl"
|
||||
try:
|
||||
api.upload_file(
|
||||
path_or_fileobj=jsonl.encode("utf-8"),
|
||||
path_in_repo=path_in_repo,
|
||||
repo_id=repo_id,
|
||||
repo_type="dataset",
|
||||
commit_message=f"add session trace {session_id}",
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning("HF upload_file failed for %s: %s", repo_id, e)
|
||||
return f"Upload to Hugging Face failed: {e}"
|
||||
|
||||
return (f"Uploaded -> https://huggingface.co/datasets/{repo_id}/blob/main/{path_in_repo}\n"
|
||||
f"View in the trace viewer: https://huggingface.co/datasets/{repo_id}")
|
||||
|
||||
|
||||
def load_session_messages(
|
||||
session_id: str, db_path=None
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
|
||||
"""Load a session's conversation + metadata from the SQLite store.
|
||||
|
||||
Returns ``(messages, meta)``. ``meta`` is ``{}`` when the session row is
|
||||
missing (messages may still be present for a live, untitled session).
|
||||
"""
|
||||
from hermes_state import SessionDB
|
||||
db = SessionDB(db_path=db_path) if db_path else SessionDB()
|
||||
resolved = db.resolve_session_id(session_id) or session_id
|
||||
meta = db.get_session(resolved) or {}
|
||||
messages = db.get_messages_as_conversation(resolved)
|
||||
return messages, meta
|
||||
|
||||
|
||||
def upload_session_trace(
|
||||
session_id: str,
|
||||
*,
|
||||
model: str = "",
|
||||
cwd: str = "",
|
||||
redact: bool = True,
|
||||
private: bool = True,
|
||||
dataset_name: str = DEFAULT_DATASET_NAME,
|
||||
db_path=None,
|
||||
token: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Top-level entry point used by the CLI/gateway/subcommand.
|
||||
|
||||
Loads the session, converts it to Claude Code JSONL, and uploads it to
|
||||
the user's private ``{user}/hermes-traces`` dataset. Returns a
|
||||
user-facing status string and never raises.
|
||||
"""
|
||||
if not session_id:
|
||||
return "No active session to upload."
|
||||
|
||||
token = token or _resolve_hf_token()
|
||||
if not token:
|
||||
return _NO_TOKEN_MESSAGE
|
||||
|
||||
try:
|
||||
messages, meta = load_session_messages(session_id, db_path=db_path)
|
||||
except Exception as e:
|
||||
logger.warning("Failed to load session %s for trace upload: %s", session_id, e)
|
||||
return f"Could not load session {session_id}: {e}"
|
||||
|
||||
if not messages:
|
||||
return "No transcript to upload for this session yet."
|
||||
|
||||
resolved_model = model or meta.get("model") or ""
|
||||
try:
|
||||
jsonl = build_trace_jsonl(
|
||||
messages,
|
||||
session_id=session_id,
|
||||
model=resolved_model,
|
||||
cwd=cwd,
|
||||
redact=redact,
|
||||
)
|
||||
except TraceRedactionError:
|
||||
return _REDACTION_BLOCKED_MESSAGE
|
||||
if not jsonl.strip():
|
||||
return "No transcript content to upload for this session."
|
||||
|
||||
return _do_upload(
|
||||
jsonl,
|
||||
token=token,
|
||||
session_id=session_id,
|
||||
dataset_name=dataset_name,
|
||||
private=private,
|
||||
)
|
||||
|
|
@ -13543,9 +13543,33 @@ def main():
|
|||
)
|
||||
sessions_export.add_argument(
|
||||
"--format",
|
||||
choices=["jsonl", "md", "qmd", "html"],
|
||||
choices=["jsonl", "md", "qmd", "html", "trace"],
|
||||
default="jsonl",
|
||||
help="Export format (default: jsonl)",
|
||||
help=(
|
||||
"Export format (default: jsonl). 'trace' emits Claude Code JSONL "
|
||||
"for the Hugging Face Agent Trace Viewer"
|
||||
),
|
||||
)
|
||||
sessions_export.add_argument(
|
||||
"--upload",
|
||||
action="store_true",
|
||||
help=(
|
||||
"trace only: upload to your Hugging Face traces dataset instead "
|
||||
"of writing a local file (needs HF_TOKEN)"
|
||||
),
|
||||
)
|
||||
sessions_export.add_argument(
|
||||
"--public",
|
||||
action="store_true",
|
||||
help="trace --upload only: create/update a public dataset instead of private",
|
||||
)
|
||||
sessions_export.add_argument(
|
||||
"--no-redact",
|
||||
action="store_true",
|
||||
help=(
|
||||
"trace only: skip the forced secret redaction; "
|
||||
"only use after manual review"
|
||||
),
|
||||
)
|
||||
sessions_export.add_argument(
|
||||
"--only",
|
||||
|
|
@ -13889,6 +13913,122 @@ def main():
|
|||
db.close()
|
||||
return
|
||||
|
||||
# Claude Code JSONL trace export — local file or HF upload.
|
||||
# Redaction is ON by default for traces (they leave the machine
|
||||
# when --upload is used); --no-redact opts out after review.
|
||||
if args.format == "trace":
|
||||
if getattr(args, "only", None):
|
||||
print("--only user-prompts supports --format jsonl or md.")
|
||||
db.close()
|
||||
return
|
||||
session_id = args.session_id
|
||||
if not session_id and not filters:
|
||||
# Match the shell's common intent: "the last thing I did".
|
||||
rows = db.list_sessions_rich(limit=1, order_by_last_active=True)
|
||||
session_id = rows[0].get("id") if rows else None
|
||||
if not session_id:
|
||||
print("No session found to export. Pass --session-id.")
|
||||
db.close()
|
||||
return
|
||||
if session_id and not db.resolve_session_id(session_id):
|
||||
print(f"Session '{session_id}' not found.")
|
||||
db.close()
|
||||
return
|
||||
|
||||
from agent.trace_upload import (
|
||||
TraceRedactionError,
|
||||
build_trace_jsonl,
|
||||
upload_session_trace,
|
||||
)
|
||||
|
||||
redact_trace = not getattr(args, "no_redact", False)
|
||||
|
||||
if getattr(args, "upload", False):
|
||||
if not session_id:
|
||||
print("--upload exports one session: pass --session-id (or drop filters to use the most recent).")
|
||||
db.close()
|
||||
return
|
||||
resolved = db.resolve_session_id(session_id)
|
||||
db.close()
|
||||
status = upload_session_trace(
|
||||
resolved,
|
||||
cwd="",
|
||||
redact=redact_trace,
|
||||
private=not getattr(args, "public", False),
|
||||
)
|
||||
print(status)
|
||||
return
|
||||
|
||||
# Local trace file(s)
|
||||
def _trace_ids():
|
||||
if session_id:
|
||||
return [db.resolve_session_id(session_id)]
|
||||
candidates = db.list_prune_candidates(**filters)
|
||||
if args.dry_run:
|
||||
print(
|
||||
f"Would export {len(candidates)} session(s) "
|
||||
f"({describe_filters(filters)})."
|
||||
)
|
||||
for row in candidates[:100]:
|
||||
print(f" {row.get('id')} {row.get('source', '')}")
|
||||
if len(candidates) > 100:
|
||||
print(f" ... {len(candidates) - 100} more")
|
||||
return None
|
||||
return [row["id"] for row in candidates]
|
||||
|
||||
ids = _trace_ids()
|
||||
if ids is None:
|
||||
db.close()
|
||||
return
|
||||
|
||||
def _render_trace(sid):
|
||||
meta = db.get_session(sid) or {}
|
||||
messages = db.get_messages_as_conversation(sid)
|
||||
if not messages:
|
||||
return None
|
||||
return build_trace_jsonl(
|
||||
messages,
|
||||
session_id=sid,
|
||||
model=meta.get("model") or "",
|
||||
cwd="",
|
||||
redact=redact_trace,
|
||||
)
|
||||
|
||||
try:
|
||||
if len(ids) == 1:
|
||||
jsonl = _render_trace(ids[0])
|
||||
if not jsonl:
|
||||
print(f"No transcript to export for session '{ids[0]}'.")
|
||||
db.close()
|
||||
return
|
||||
if not args.output or args.output == "-":
|
||||
sys.stdout.write(jsonl)
|
||||
else:
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
f.write(jsonl)
|
||||
print(f"Exported 1 session trace to {args.output}")
|
||||
else:
|
||||
out_dir = (
|
||||
Path(args.output).expanduser()
|
||||
if args.output and args.output != "-"
|
||||
else get_hermes_home() / "session-exports"
|
||||
)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
exported = 0
|
||||
for sid in ids:
|
||||
jsonl = _render_trace(sid)
|
||||
if not jsonl:
|
||||
continue
|
||||
(out_dir / f"{sid}.trace.jsonl").write_text(
|
||||
jsonl, encoding="utf-8"
|
||||
)
|
||||
exported += 1
|
||||
print(f"Exported {exported} session trace(s) to {out_dir}")
|
||||
except TraceRedactionError:
|
||||
print("Redaction failed; refusing to export unredacted trace content.")
|
||||
db.close()
|
||||
return
|
||||
|
||||
if args.format == "jsonl":
|
||||
if not args.output:
|
||||
print("JSONL export requires an output path (use - for stdout).")
|
||||
|
|
|
|||
261
tests/agent/test_trace_upload.py
Normal file
261
tests/agent/test_trace_upload.py
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
"""Tests for agent.trace_upload — Hugging Face session-trace upload.
|
||||
|
||||
Covers the Claude Code JSONL converter, HF token resolution, the no-token
|
||||
message path, and the upload path with a mocked ``HfApi`` (verifying repo
|
||||
id, file path, and content without touching the network).
|
||||
"""
|
||||
|
||||
import json
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from agent import trace_upload
|
||||
from agent.trace_upload import (
|
||||
build_trace_jsonl,
|
||||
upload_session_trace,
|
||||
_resolve_hf_token,
|
||||
_do_upload,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Converter
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _sample_messages():
|
||||
return [
|
||||
{"role": "system", "content": "you are hermes"},
|
||||
{"role": "user", "content": "list files"},
|
||||
{"role": "assistant", "content": "Listing.", "tool_calls": [
|
||||
{"id": "call_1", "function": {"name": "terminal", "arguments": '{"command": "ls"}'}},
|
||||
]},
|
||||
{"role": "tool", "tool_call_id": "call_1", "tool_name": "terminal", "content": "a.txt\nb.txt"},
|
||||
{"role": "assistant", "content": "Two files."},
|
||||
]
|
||||
|
||||
|
||||
def test_converter_skips_system_and_counts_lines():
|
||||
jsonl = build_trace_jsonl(_sample_messages(), session_id="s1", model="m")
|
||||
lines = [json.loads(x) for x in jsonl.strip().split("\n")]
|
||||
assert len(lines) == 4 # system dropped
|
||||
assert all(o["sessionId"] == "s1" for o in lines)
|
||||
|
||||
|
||||
def test_converter_links_turns_as_linked_list():
|
||||
jsonl = build_trace_jsonl(_sample_messages(), session_id="s1")
|
||||
lines = [json.loads(x) for x in jsonl.strip().split("\n")]
|
||||
prev = None
|
||||
for o in lines:
|
||||
assert o["parentUuid"] == prev
|
||||
prev = o["uuid"]
|
||||
|
||||
|
||||
def test_converter_emits_tool_use_and_tool_result():
|
||||
jsonl = build_trace_jsonl(_sample_messages(), session_id="s1", model="m")
|
||||
lines = [json.loads(x) for x in jsonl.strip().split("\n")]
|
||||
# line 0 user, line 1 assistant (text + tool_use), line 2 tool_result, line 3 assistant
|
||||
assert lines[0]["type"] == "user"
|
||||
assert lines[1]["type"] == "assistant"
|
||||
blocks = lines[1]["message"]["content"]
|
||||
assert any(b.get("type") == "text" for b in blocks)
|
||||
tool_use = [b for b in blocks if b.get("type") == "tool_use"]
|
||||
assert tool_use and tool_use[0]["name"] == "terminal"
|
||||
assert tool_use[0]["input"] == {"command": "ls"}
|
||||
# tool result rides on a user turn
|
||||
assert lines[2]["type"] == "user"
|
||||
tr = lines[2]["message"]["content"][0]
|
||||
assert tr["type"] == "tool_result"
|
||||
assert tr["tool_use_id"] == "call_1"
|
||||
assert "a.txt" in tr["content"]
|
||||
|
||||
|
||||
def test_converter_redacts_secrets_by_default():
|
||||
msgs = [{"role": "user", "content": "key OPENAI_API_KEY=sk-abc123def456ghi789jklmno end"}]
|
||||
jsonl = build_trace_jsonl(msgs, session_id="s1", redact=True)
|
||||
assert "sk-abc123def456ghi789jklmno" not in jsonl
|
||||
|
||||
|
||||
def test_converter_refuses_unredacted_passthrough_when_redactor_fails(monkeypatch):
|
||||
def boom(_text, *, force=False):
|
||||
raise RuntimeError("redactor unavailable")
|
||||
|
||||
monkeypatch.setattr("agent.redact.redact_sensitive_text", boom)
|
||||
msgs = [{"role": "user", "content": "OPENAI_API_KEY=sk-abc123def456ghi789jklmno"}]
|
||||
|
||||
with pytest.raises(trace_upload.TraceRedactionError):
|
||||
build_trace_jsonl(msgs, session_id="s1", redact=True)
|
||||
|
||||
|
||||
def test_upload_blocks_when_redactor_fails(monkeypatch):
|
||||
monkeypatch.setenv("HF_TOKEN", "hf_test")
|
||||
|
||||
def boom(_text, *, force=False):
|
||||
raise RuntimeError("redactor unavailable")
|
||||
|
||||
monkeypatch.setattr("agent.redact.redact_sensitive_text", boom)
|
||||
with patch.object(trace_upload, "load_session_messages", return_value=(_sample_messages(), {})), \
|
||||
patch.object(trace_upload, "_do_upload") as upload_mock:
|
||||
msg = upload_session_trace("s1")
|
||||
|
||||
assert "Trace upload blocked" in msg
|
||||
upload_mock.assert_not_called()
|
||||
|
||||
|
||||
def test_converter_keeps_secrets_when_redact_disabled():
|
||||
secret = "sk-abc123def456ghi789jklmno"
|
||||
msgs = [{"role": "user", "content": f"key OPENAI_API_KEY={secret} end"}]
|
||||
jsonl = build_trace_jsonl(msgs, session_id="s1", redact=False)
|
||||
assert secret in jsonl
|
||||
|
||||
|
||||
def test_converter_image_placeholder():
|
||||
msgs = [{"role": "user", "content": [
|
||||
{"type": "text", "text": "look"},
|
||||
{"type": "image_url", "image_url": {"url": "data:image/png;base64,AAAA"}},
|
||||
]}]
|
||||
jsonl = build_trace_jsonl(msgs, session_id="s1")
|
||||
line = json.loads(jsonl.strip())
|
||||
assert any("image omitted" in b.get("text", "") for b in line["message"]["content"])
|
||||
assert "AAAA" not in jsonl
|
||||
|
||||
|
||||
def test_converter_empty_messages_returns_empty():
|
||||
assert build_trace_jsonl([], session_id="s1") == ""
|
||||
|
||||
|
||||
def test_converter_handles_dict_tool_arguments():
|
||||
msgs = [{"role": "assistant", "content": "", "tool_calls": [
|
||||
{"id": "c", "function": {"name": "f", "arguments": {"already": "dict"}}},
|
||||
]}]
|
||||
jsonl = build_trace_jsonl(msgs, session_id="s1")
|
||||
line = json.loads(jsonl.strip())
|
||||
tu = [b for b in line["message"]["content"] if b.get("type") == "tool_use"][0]
|
||||
assert tu["input"] == {"already": "dict"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Token resolution
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_resolve_token_prefers_hf_token(monkeypatch):
|
||||
monkeypatch.setenv("HF_TOKEN", "hf_primary")
|
||||
monkeypatch.setenv("HUGGINGFACE_TOKEN", "hf_secondary")
|
||||
assert _resolve_hf_token() == "hf_primary"
|
||||
|
||||
|
||||
def test_resolve_token_falls_back(monkeypatch):
|
||||
monkeypatch.delenv("HF_TOKEN", raising=False)
|
||||
monkeypatch.delenv("HUGGINGFACE_HUB_TOKEN", raising=False)
|
||||
monkeypatch.delenv("HUGGING_FACE_HUB_TOKEN", raising=False)
|
||||
monkeypatch.setenv("HUGGINGFACE_TOKEN", "hf_fallback")
|
||||
assert _resolve_hf_token() == "hf_fallback"
|
||||
|
||||
|
||||
def test_resolve_token_none(monkeypatch):
|
||||
for v in ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HUGGING_FACE_HUB_TOKEN", "HUGGINGFACE_TOKEN"):
|
||||
monkeypatch.delenv(v, raising=False)
|
||||
assert _resolve_hf_token() is None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Top-level upload entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_upload_no_session_id():
|
||||
assert "No active session" in upload_session_trace("")
|
||||
|
||||
|
||||
def test_upload_no_token(monkeypatch):
|
||||
for v in ("HF_TOKEN", "HUGGINGFACE_HUB_TOKEN", "HUGGING_FACE_HUB_TOKEN", "HUGGINGFACE_TOKEN"):
|
||||
monkeypatch.delenv(v, raising=False)
|
||||
msg = upload_session_trace("some_session")
|
||||
assert "no Hugging Face token" in msg
|
||||
|
||||
|
||||
def test_upload_empty_transcript(monkeypatch):
|
||||
monkeypatch.setenv("HF_TOKEN", "hf_test")
|
||||
with patch.object(trace_upload, "load_session_messages", return_value=([], {})):
|
||||
msg = upload_session_trace("s1")
|
||||
assert "No transcript" in msg
|
||||
|
||||
|
||||
def test_upload_happy_path_mocked(monkeypatch):
|
||||
"""Full upload path with a mocked HfApi — verifies repo id / path / content."""
|
||||
pytest.importorskip("huggingface_hub") # optional dep; runtime degrades gracefully
|
||||
monkeypatch.setenv("HF_TOKEN", "hf_test")
|
||||
messages = _sample_messages()
|
||||
|
||||
fake_api = MagicMock()
|
||||
fake_api.whoami.return_value = {"name": "alice"}
|
||||
|
||||
with patch.object(trace_upload, "load_session_messages",
|
||||
return_value=(messages, {"model": "claude-x"})), \
|
||||
patch("huggingface_hub.HfApi", return_value=fake_api):
|
||||
msg = upload_session_trace("20260531_abc", cwd="/tmp")
|
||||
|
||||
# Returned a viewer URL
|
||||
assert "huggingface.co/datasets/alice/hermes-traces" in msg
|
||||
|
||||
# Created private dataset repo
|
||||
fake_api.create_repo.assert_called_once()
|
||||
_, kwargs = fake_api.create_repo.call_args
|
||||
assert kwargs["repo_id"] == "alice/hermes-traces"
|
||||
assert kwargs["repo_type"] == "dataset"
|
||||
assert kwargs["private"] is True
|
||||
|
||||
# Uploaded the JSONL to sessions/<id>.jsonl
|
||||
fake_api.upload_file.assert_called_once()
|
||||
_, ukwargs = fake_api.upload_file.call_args
|
||||
assert ukwargs["path_in_repo"] == "sessions/20260531_abc.jsonl"
|
||||
assert ukwargs["repo_id"] == "alice/hermes-traces"
|
||||
body = ukwargs["path_or_fileobj"]
|
||||
if isinstance(body, bytes):
|
||||
body = body.decode("utf-8")
|
||||
# Content is valid Claude Code JSONL
|
||||
first = json.loads(body.strip().split("\n")[0])
|
||||
assert first["type"] in ("user", "assistant")
|
||||
assert first["sessionId"] == "20260531_abc"
|
||||
|
||||
|
||||
def test_upload_public_flag(monkeypatch):
|
||||
pytest.importorskip("huggingface_hub") # optional dep
|
||||
monkeypatch.setenv("HF_TOKEN", "hf_test")
|
||||
fake_api = MagicMock()
|
||||
fake_api.whoami.return_value = {"name": "bob"}
|
||||
with patch.object(trace_upload, "load_session_messages",
|
||||
return_value=(_sample_messages(), {})), \
|
||||
patch("huggingface_hub.HfApi", return_value=fake_api):
|
||||
upload_session_trace("s1", private=False)
|
||||
_, kwargs = fake_api.create_repo.call_args
|
||||
assert kwargs["private"] is False
|
||||
|
||||
|
||||
def test_upload_whoami_failure(monkeypatch):
|
||||
pytest.importorskip("huggingface_hub") # optional dep
|
||||
monkeypatch.setenv("HF_TOKEN", "hf_bad")
|
||||
fake_api = MagicMock()
|
||||
fake_api.whoami.side_effect = Exception("401 unauthorized")
|
||||
with patch.object(trace_upload, "load_session_messages",
|
||||
return_value=(_sample_messages(), {})), \
|
||||
patch("huggingface_hub.HfApi", return_value=fake_api):
|
||||
msg = upload_session_trace("s1")
|
||||
assert "token was rejected" in msg
|
||||
|
||||
|
||||
def test_do_upload_missing_huggingface_hub(monkeypatch):
|
||||
"""If huggingface_hub import fails, return a clear install hint."""
|
||||
# Disable lazy-install so the import path deterministically fails here
|
||||
# instead of attempting a real pip install in CI.
|
||||
monkeypatch.setenv("HERMES_DISABLE_LAZY_INSTALLS", "1")
|
||||
import builtins
|
||||
real_import = builtins.__import__
|
||||
|
||||
def fake_import(name, *args, **kwargs):
|
||||
if name == "huggingface_hub":
|
||||
raise ImportError("no module")
|
||||
return real_import(name, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(builtins, "__import__", fake_import)
|
||||
msg = _do_upload("{}\n", token="t", session_id="s1")
|
||||
assert "huggingface_hub" in msg
|
||||
|
|
@ -504,3 +504,121 @@ def test_sessions_export_redact_scrubs_secrets(monkeypatch, tmp_path):
|
|||
text = next(tmp_path.glob("*.md")).read_text(encoding="utf-8")
|
||||
assert secret not in text
|
||||
assert "api key:" in text
|
||||
|
||||
|
||||
def _trace_fake_db(captured):
|
||||
class FakeDB:
|
||||
def resolve_session_id(self, session_id):
|
||||
return "s1"
|
||||
|
||||
def get_session(self, session_id):
|
||||
return {"id": session_id, "model": "test-model"}
|
||||
|
||||
def get_messages_as_conversation(self, session_id):
|
||||
captured["conv"] = session_id
|
||||
return [
|
||||
{"role": "user", "content": "hello trace"},
|
||||
{"role": "assistant", "content": "hi"},
|
||||
]
|
||||
|
||||
def close(self):
|
||||
captured["closed"] = True
|
||||
|
||||
return FakeDB()
|
||||
|
||||
|
||||
def test_sessions_export_trace_writes_claude_jsonl(monkeypatch, tmp_path, capsys):
|
||||
import json
|
||||
|
||||
import hermes_cli.main as main_mod
|
||||
import hermes_state
|
||||
|
||||
captured = {}
|
||||
out = tmp_path / "trace.jsonl"
|
||||
monkeypatch.setattr(hermes_state, "SessionDB", lambda: _trace_fake_db(captured))
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
["hermes", "sessions", "export", "--format", "trace", "--session-id", "s1", str(out)],
|
||||
)
|
||||
|
||||
main_mod.main()
|
||||
|
||||
assert "Exported 1 session trace" in capsys.readouterr().out
|
||||
lines = [json.loads(line) for line in out.read_text(encoding="utf-8").splitlines()]
|
||||
assert {rec["type"] for rec in lines} == {"user", "assistant"}
|
||||
assert all("uuid" in rec for rec in lines)
|
||||
assert captured["conv"] == "s1"
|
||||
assert captured["closed"] is True
|
||||
|
||||
|
||||
def test_sessions_export_trace_stdout(monkeypatch, capsys):
|
||||
import json
|
||||
|
||||
import hermes_cli.main as main_mod
|
||||
import hermes_state
|
||||
|
||||
captured = {}
|
||||
monkeypatch.setattr(hermes_state, "SessionDB", lambda: _trace_fake_db(captured))
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
["hermes", "sessions", "export", "--format", "trace", "--session-id", "s1", "-"],
|
||||
)
|
||||
|
||||
main_mod.main()
|
||||
|
||||
lines = [json.loads(line) for line in capsys.readouterr().out.splitlines()]
|
||||
assert len(lines) == 2
|
||||
assert lines[0]["type"] == "user"
|
||||
|
||||
|
||||
def test_sessions_export_trace_upload_routes_to_uploader(monkeypatch, capsys):
|
||||
import hermes_cli.main as main_mod
|
||||
import hermes_state
|
||||
from agent import trace_upload as trace_mod
|
||||
|
||||
captured = {}
|
||||
monkeypatch.setattr(hermes_state, "SessionDB", lambda: _trace_fake_db(captured))
|
||||
|
||||
def fake_upload(session_id, **kwargs):
|
||||
captured["uploaded"] = session_id
|
||||
captured["kwargs"] = kwargs
|
||||
return "Uploaded -> https://example.test/dataset"
|
||||
|
||||
monkeypatch.setattr(trace_mod, "upload_session_trace", fake_upload)
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"hermes", "sessions", "export", "--format", "trace",
|
||||
"--session-id", "s1", "--upload", "--public",
|
||||
],
|
||||
)
|
||||
|
||||
main_mod.main()
|
||||
|
||||
assert captured["uploaded"] == "s1"
|
||||
assert captured["kwargs"]["private"] is False
|
||||
assert captured["kwargs"]["redact"] is True
|
||||
assert "Uploaded ->" in capsys.readouterr().out
|
||||
|
||||
|
||||
def test_sessions_export_trace_only_flag_rejected(monkeypatch, capsys):
|
||||
import hermes_cli.main as main_mod
|
||||
import hermes_state
|
||||
|
||||
captured = {}
|
||||
monkeypatch.setattr(hermes_state, "SessionDB", lambda: _trace_fake_db(captured))
|
||||
monkeypatch.setattr(
|
||||
sys,
|
||||
"argv",
|
||||
[
|
||||
"hermes", "sessions", "export", "--format", "trace",
|
||||
"--session-id", "s1", "--only", "user-prompts", "-",
|
||||
],
|
||||
)
|
||||
|
||||
main_mod.main()
|
||||
|
||||
assert "--only user-prompts supports --format jsonl or md." in capsys.readouterr().out
|
||||
|
|
|
|||
|
|
@ -238,6 +238,8 @@ LAZY_DEPS: dict[str, tuple[str, ...]] = {
|
|||
"mcp==1.26.0",
|
||||
"starlette==1.0.1", # CVE-2026-48710 — keep in sync with pyproject [computer-use]
|
||||
),
|
||||
# HF Agent Trace Viewer upload (hermes trace upload / /upload-trace).
|
||||
"tool.trace_upload": ("huggingface-hub==1.2.3",),
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -338,6 +338,23 @@ hermes sessions export - --session-id 20250305_091523_a1b2c3d4 --only user-promp
|
|||
|
||||
Works with `--format jsonl` (default) or `md`, honors the same filters for bulk export, and combines with `--redact`.
|
||||
|
||||
### Export Traces to the HF Agent Trace Viewer
|
||||
|
||||
`--format trace` emits Claude Code JSONL — the transcript shape the Hugging Face Hub auto-detects for its [Agent Trace Viewer](https://huggingface.co/docs/hub/agent-traces). Write it locally, or add `--upload` to push it to your own private `hermes-traces` dataset (reads `HF_TOKEN`):
|
||||
|
||||
```bash
|
||||
# Trace of the most recent session, to stdout
|
||||
hermes sessions export --format trace
|
||||
|
||||
# One session to a local trace file
|
||||
hermes sessions export --format trace --session-id 20250305_091523_a1b2c3d4 trace.jsonl
|
||||
|
||||
# Upload straight to your private HF traces dataset
|
||||
hermes sessions export --format trace --session-id 20250305_091523_a1b2c3d4 --upload
|
||||
```
|
||||
|
||||
Trace exports are secret-redacted by default (they're meant to leave the machine); `--no-redact` opts out after manual review. `--upload` is private unless `--public`. Bulk trace export with filters writes one `<id>.trace.jsonl` per session.
|
||||
|
||||
### Export Sessions to Markdown/QMD
|
||||
|
||||
Pass `--format md` or `--format qmd` when you want a readable, file-based archive before hiding or deleting old sessions. Markdown/QMD exports write one file per session into a directory (default: `~/.hermes/session-exports`).
|
||||
|
|
|
|||
|
|
@ -315,6 +315,23 @@ hermes sessions export - --session-id 20250305_091523_a1b2c3d4 --only user-promp
|
|||
|
||||
支持 `--format jsonl`（默认）或 `md`，批量导出时同样支持全部过滤器，也可与 `--redact` 组合。
|
||||
|
||||
### 导出 Trace 到 HF Agent Trace Viewer
|
||||
|
||||
`--format trace` 生成 Claude Code JSONL — Hugging Face Hub 的 [Agent Trace Viewer](https://huggingface.co/docs/hub/agent-traces) 可自动识别的转录格式。可以写入本地文件，或加 `--upload` 推送到你自己的私有 `hermes-traces` 数据集（读取 `HF_TOKEN`）：
|
||||
|
||||
```bash
|
||||
# 最近一个 session 的 trace，输出到 stdout
|
||||
hermes sessions export --format trace
|
||||
|
||||
# 将一个 session 导出为本地 trace 文件
|
||||
hermes sessions export --format trace --session-id 20250305_091523_a1b2c3d4 trace.jsonl
|
||||
|
||||
# 直接上传到你的私有 HF traces 数据集
|
||||
hermes sessions export --format trace --session-id 20250305_091523_a1b2c3d4 --upload
|
||||
```
|
||||
|
||||
Trace 导出默认强制脱敏（它们本来就是要离开本机的）；`--no-redact` 需人工审查后才建议使用。`--upload` 默认私有，除非加 `--public`。带过滤器的批量 trace 导出会为每个 session 写一个 `<id>.trace.jsonl`。
|
||||
|
||||
### 导出 Session 为 Markdown/QMD
|
||||
|
||||
当你想在隐藏或删除旧 session 之前保留一份可读的文件归档时，传入 `--format md` 或 `--format qmd`。Markdown/QMD 导出会为每个 session 写入一个文件到目录中（默认：`~/.hermes/session-exports`）。
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue