feat(unreal-mcp): live-verify skill against a running UE 5.8 server; encode e2e test findings
Ran the full loop against a real editor (blank project, ModelContextProtocol + ToolsetRegistry + AllToolsets enabled): raw MCP handshake, discovery walk, environment relight for golden hour, primitive monument build, virtual-camera captures with vision judgment, exposure debugging, annotated spatial capture. 67 toolsets advertised; every dispatch semantic below observed, not inferred. Corrections and additions from the live run: - Qualified toolset names (editor_toolset.toolsets.scene.SceneTools) with SHORT tool_name; TOptional params must be explicit null; find_actors requires ''/[] for its schema-required optionals; ObjectTools values is a JSON *string*; refPath object references; returnValue wrapping; per-property failure lists with schema-in-error - HTTP wire contract: initialize=JSON + session header, tools/call=SSE frame after game-thread completion (plain-JSON clients read empty body) - CaptureViewport as virtual camera (captureTransform, meter-unit annotation grid + actor callouts) verified with pixel evidence; recipes rewritten to use it instead of viewport piloting - New pitfalls from real failures: template-level environment-actor duplication compounding into whiteouts (find-first/spawn-if-missing rule), template exposure calibration vs physical lux (12b), objective exposure check via ffprobe YAVG (12c), untitled-level Save-As modal deadlock, macOS full-Xcode + Metal Toolchain requirement (xcodebuild -downloadComponent MetalToolchain) - Live toolset census (67 on blank project), LogsToolset/ConfigSettings/ SemanticSearch highlights, UE EULA 6(e) licensing note
This commit is contained in:
parent
a8b81c56a0
commit
d24ab20404
5 changed files with 254 additions and 80 deletions
|
|
@ -43,7 +43,9 @@ Two halves, in this order: the editor side must be up before Hermes connects.
|
|||
|
||||
### One-time, editor side
|
||||
|
||||
1. Unreal Editor **5.8+** with a project open.
|
||||
1. Unreal Editor **5.8+** with a project open. (macOS: full Xcode must be
|
||||
installed and its license accepted — the editor exits on first launch
|
||||
without it; see pitfalls.)
|
||||
2. **Edit > Plugins** — enable **Unreal MCP** (its Toolset Registry
|
||||
dependency auto-enables). Restart the editor when prompted.
|
||||
3. The typed toolsets ship separately from the server: also enable the
|
||||
|
|
@ -98,10 +100,13 @@ The discovery walk, always in this order:
|
|||
|
||||
1. `list_toolsets` → see what capability groups this project actually has
|
||||
(the surface is project-dependent: enabled plugins, Game Feature Plugins,
|
||||
and any custom toolsets all contribute).
|
||||
and any custom toolsets all contribute). Names come back FULLY QUALIFIED
|
||||
(`editor_toolset.toolsets.scene.SceneTools`,
|
||||
`EditorToolset.EditorAppToolset`) — use them verbatim as `toolset_name`.
|
||||
2. `describe_toolset` on the group you need → read the real parameter
|
||||
schemas. Never guess parameter names — schemas are the contract.
|
||||
3. `call_tool` with exact toolset/tool name and arguments.
|
||||
3. `call_tool` with the qualified toolset name, the SHORT tool name
|
||||
(`find_actors`, not the dotted form), and arguments matching the schema.
|
||||
|
||||
Cache what you learn for the session; re-list only after the editor side
|
||||
changes (new plugin enabled, toolset authored, `RefreshTools` run).
|
||||
|
|
@ -162,8 +167,11 @@ Rules of the world while you work:
|
|||
not actor **names** (internal, unique). Prefer resolving actors by
|
||||
label/class queries, then hold on to whatever handle the tool returns.
|
||||
- Prefer physically-plausible lighting values (lux/candela/Kelvin) over
|
||||
arbitrary brightness numbers — `references/scene-craft.md` has the
|
||||
numeric recipes (sun intensities, exposure, fog densities, moods).
|
||||
arbitrary brightness numbers — but FIRST read the existing sun's
|
||||
intensity to learn the scene's calibration convention; template worlds
|
||||
are often calibrated around `intensity: 10`, and physical values blow
|
||||
them out (`references/scene-craft.md` has the numbers,
|
||||
`references/pitfalls.md` #12b has the calibration rule).
|
||||
|
||||
## From Plain English to a Scene
|
||||
|
||||
|
|
@ -224,6 +232,10 @@ Load on demand; keep SKILL.md-level rules in mind throughout.
|
|||
examples. When docs and the live schema disagree, the live schema wins.
|
||||
- **Don't expose the server beyond localhost.** Loopback-only, no auth, by
|
||||
design. Never suggest binding it wider.
|
||||
- **Licensing note.** The server logs on start: data transmitted via the
|
||||
plugin to a connected LLM service is Licensed Technology under the UE
|
||||
EULA (§6(e)) — the user is responsible for ensuring their LLM provider
|
||||
doesn't train on it. Surface this if the user asks about data handling.
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
|
|
|
|||
|
|
@ -21,7 +21,20 @@ returns nothing/near-nothing, the toolset provider plugin (AllToolsets) or
|
|||
Toolset Registry isn't enabled in this project. Fix in Edit > Plugins,
|
||||
restart the editor, restart the Hermes session.
|
||||
|
||||
### 3. Port 8000 conflicts
|
||||
### 3. macOS: full Xcode is required, not just Command Line Tools
|
||||
|
||||
On a Mac, the editor needs Xcode to compile shaders for Metal. Without it,
|
||||
first launch dies with a modal "Xcode Not Found" dialog and the editor
|
||||
exits as soon as it's dismissed (verified UE 5.8 behavior — the log shows
|
||||
`RequestExit` right after the dialog). Fix: install full Xcode from the App
|
||||
Store, open it once to accept the license / install components, and if it
|
||||
lives anywhere other than `/Applications/Xcode.app`, point the toolchain at
|
||||
it: `sudo xcode-select -s /path/to/Xcode.app`. Verify with
|
||||
`xcode-select -p` (should print an Xcode path, not the bare CLT path).
|
||||
Expect the first successful editor launch after that to spend a long time
|
||||
compiling shaders.
|
||||
|
||||
### 4. Port 8000 conflicts
|
||||
|
||||
Common collisions: local dev servers, Jupyter, other MCP hosts. Symptom: the
|
||||
server fails to bind (Output Log) or Hermes' probe times out. Fix: change
|
||||
|
|
@ -99,19 +112,54 @@ this skill afterward.
|
|||
|
||||
## Editor & Scene State
|
||||
|
||||
### 12. Never assume a fresh level
|
||||
### 12. Never assume a fresh level — and NEVER double-spawn environment actors
|
||||
|
||||
Query the scene before the first edit. The user's level may have existing
|
||||
actors, a non-default sun, post-process volumes with exposure overrides —
|
||||
your lighting changes can look wrong because of a pre-existing volume, not
|
||||
your values.
|
||||
Query the scene before the first edit. Template levels (Open World, etc.)
|
||||
ALREADY contain a DirectionalLight, SkyAtmosphere, SkyLight,
|
||||
ExponentialHeightFog, and often VolumetricCloud. Spawning your own creates
|
||||
duplicates that compound (double fog = whiteout, double sky = wrong
|
||||
exposure) and are invisible in a screenshot until things look
|
||||
inexplicably wrong. Live-verified failure: spawning a "golden hour kit"
|
||||
into the default Open World template produced two of everything and a
|
||||
254/255-luminance whiteout. Rule: `find_actors` for each environment class
|
||||
FIRST; configure what exists; spawn only what's missing.
|
||||
|
||||
### 12b. Read the existing sun before imposing physical light values
|
||||
|
||||
Scene-craft tables give physical values (golden hour ≈ 10k lux), but a
|
||||
template's world is calibrated as a SYSTEM — the default Open World sun is
|
||||
`intensity: 10` (lux-ish units under default exposure), not 100,000. Setting
|
||||
12,000 lux into that world blows the frame to pure white regardless of
|
||||
small exposure tweaks. Live-verified rule: `get_properties` the existing
|
||||
sun's intensity first. If the scene is calibrated low (single-digit sun),
|
||||
work RELATIVE to it (e.g. golden hour ≈ 0.5–1× the template's noon value,
|
||||
warm temperature, low pitch) and let auto-exposure adapt, or rebuild the
|
||||
whole exposure chain deliberately (manual EV100 + physical values
|
||||
everywhere). Mixing the two conventions is the #1 whiteout cause.
|
||||
`ObjectTools.reset_properties` is the escape hatch — it restores per-project
|
||||
defaults when you've painted into a corner.
|
||||
|
||||
### 12c. Verify exposure objectively, not just by eye
|
||||
|
||||
A capture can look "bright" in vision judgment while being unrecoverable.
|
||||
Cheap objective check on any capture (editor-host filesystem):
|
||||
`ffprobe -f lavfi -i "movie=<png>,signalstats" -show_entries
|
||||
frame_tags=lavfi.signalstats.YAVG -of json` — YAVG > 250 means blown
|
||||
white, < 5 means black. Use it whenever a lighting change should have
|
||||
moved the histogram; it distinguishes "fog whiteout" from "exposure
|
||||
whiteout" faster than iterating blind.
|
||||
|
||||
### 13. In-memory edits are lost on crash — save per milestone
|
||||
|
||||
Everything you do lives in unsaved packages until a save happens. The editor
|
||||
is an application that can crash, especially mid-experimental-feature. Save
|
||||
the level + dirty packages after every milestone. Also: some operations
|
||||
(level streaming, some asset moves) behave differently on unsaved assets.
|
||||
the level + dirty packages after every milestone (`AssetTools.save_assets`,
|
||||
`SceneTools.save_actor`). Caveat: an UNTITLED level (`/Temp/Untitled_*`,
|
||||
the state after File > New or launching without a map argument) may route a
|
||||
save through the Save-As dialog — a modal that deadlocks the MCP loop
|
||||
(pitfall 8). Prefer starting from a saved level (pass the map path on the
|
||||
editor command line, or `SceneTools.load_level` a real `/Game/...` map)
|
||||
before doing hours of work.
|
||||
|
||||
### 14. Label ≠ Name ≠ path — use the full path as the stable identifier
|
||||
|
||||
|
|
|
|||
|
|
@ -25,11 +25,22 @@ dead" cause).
|
|||
|
||||
Session preamble for every recipe (do once):
|
||||
|
||||
1. `list_toolsets` → note actor/scene/material/level capability groups.
|
||||
1. `list_toolsets` → note the qualified names (e.g.
|
||||
`editor_toolset.toolsets.scene.SceneTools`,
|
||||
`EditorToolset.EditorAppToolset`).
|
||||
2. `describe_toolset` on each group you'll touch → cache schemas.
|
||||
3. Query current level state (all actors + classes) → never assume empty.
|
||||
4. Locate the screenshot path for this project:
|
||||
`<Project>/Saved/Screenshots/<Platform>/`.
|
||||
3. Query current level (`SceneTools.get_current_level`) and inventory the
|
||||
environment: `find_actors` for DirectionalLight, SkyAtmosphere,
|
||||
SkyLight, ExponentialHeightFog, PostProcessVolume, VolumetricCloud.
|
||||
**Configure existing environment actors; spawn only what's missing** —
|
||||
template levels ship with most of them, and duplicates compound into
|
||||
whiteouts.
|
||||
4. Read the existing sun's `intensity` — it tells you the scene's exposure
|
||||
calibration (template worlds are often calibrated around `intensity: 10`,
|
||||
not physical lux; see pitfalls 12b before applying scene-craft absolute
|
||||
values).
|
||||
5. Locate your verification path: `EditorAppToolset.CaptureViewport` with a
|
||||
`captureTransform` is the virtual camera — no viewport piloting needed.
|
||||
|
||||
Save the level + dirty packages after every phase marked 💾. One tool call
|
||||
at a time throughout — no batching, ever.
|
||||
|
|
@ -148,16 +159,20 @@ Brief: "make it golden hour and give me a cinematic shot of <subject>"
|
|||
|
||||
**Phase 2 — the camera**
|
||||
|
||||
- INTENT a framed CineCamera, not a viewport eyeball.
|
||||
DISCOVER CineCameraActor spawn + property-set; viewport-pilot or
|
||||
camera-view capability if advertised.
|
||||
VALUES position: subject-distance by lens — 85 mm at 400–600 cm for a
|
||||
prop/character subject; height 120–160 cm; aim so subject sits on a
|
||||
thirds intersection, horizon in upper or lower third (not center).
|
||||
Focal 85 mm, aperture f/2.0, manual focus distance = measured
|
||||
camera→subject distance (compute from the two locations).
|
||||
VERIFY screenshot THROUGH this camera (pilot it / set viewport to its
|
||||
view first — confirm by a cheap `HighResShot 1` before the big one).
|
||||
- INTENT a framed shot WITHOUT touching the user's viewport.
|
||||
DISCOVER `EditorAppToolset.CaptureViewport` with `captureTransform` — a
|
||||
virtual camera; no CineCamera or viewport piloting needed for stills.
|
||||
VALUES position: subject-distance by lens-equivalent framing — for a
|
||||
prop/monument subject ~500–800 cm back, height 120–160 cm; rotation
|
||||
aimed so the subject sits on a thirds intersection, horizon in upper or
|
||||
lower third. Slight upward pitch (+2° to +5°) from below eye height
|
||||
reads heroic and guarantees sky/horizon in frame.
|
||||
For an actual CineCameraActor (user wants a camera in the level, DoF,
|
||||
or a Sequencer shot): spawn `/Script/CinematicCamera.CineCameraActor`,
|
||||
set focal/aperture/focus via ObjectTools on its CineCameraComponent,
|
||||
then capture with `captureTransform` matching its transform.
|
||||
VERIFY capture at viewport res first; iterate framing cheaply, then take
|
||||
the final.
|
||||
|
||||
**Phase 3 — deliver**
|
||||
|
||||
|
|
|
|||
|
|
@ -74,6 +74,15 @@ compose correctly with exposure instead of fighting it.
|
|||
|
||||
### Sun (DirectionalLight, lux)
|
||||
|
||||
**Calibration check first (live-verified):** template levels often ship a
|
||||
sun at `intensity: 10` with auto-exposure tuned around it — physical lux
|
||||
values below will blow such a scene to white. Read the existing sun's
|
||||
intensity; if it's single/double digits, scale moods RELATIVE to it (noon =
|
||||
template value, golden hour ≈ 0.5–0.7×, overcast ≈ 0.3×, night ≈ 0.01×)
|
||||
and rely on temperature + pitch for the mood. The absolute table applies
|
||||
when you own the whole exposure chain (manual EV100 + physical values
|
||||
everywhere):
|
||||
|
||||
| Condition | Intensity (lux) | Pitch | Temperature |
|
||||
|---|---|---|---|
|
||||
| Noon, clear | 75,000–120,000 | −60° to −90° | 5,500–6,000 K |
|
||||
|
|
|
|||
|
|
@ -14,8 +14,15 @@ only — no stdio/WebSocket). It implements the protocol but ships no tools of
|
|||
its own. Tools come from **Toolsets** — classes deriving from
|
||||
`UToolsetDefinition` (C++) or `unreal.ToolsetDefinition` (Python) — collected
|
||||
at startup by the **Toolset Registry** subsystem (sibling plugin,
|
||||
auto-enabled). Epic's shipped toolsets are delivered by a separate toolset
|
||||
provider plugin (**AllToolsets**); project plugins and Game Feature Plugins
|
||||
auto-enabled; the registry itself ships no toolsets either). The shipped
|
||||
tools live in per-domain plugins under `Engine/Plugins/Experimental/
|
||||
Toolsets/` — the workhorse is **EditorToolset** (core editor toolsets,
|
||||
Python + C++) — and **AllToolsets** is a one-checkbox aggregator plugin
|
||||
that depends on ~21 of them (verified from `AllToolsets.uplugin`, 5.8):
|
||||
AIModule, AnimationAssistant, AutomationTest, ConfigSettings, Conversation,
|
||||
DataRegistry, DataflowAgent, Editor, GameFeatures, GameplayTags, GAS,
|
||||
MCPClient, Niagara, PCG, Physics, Plugin, SemanticSearch, SlateInspector,
|
||||
StateTree, UMG, WorldConditions. Project plugins and Game Feature Plugins
|
||||
can contribute more. Unreal MCP wraps every registered tool call as an MCP
|
||||
Tool. Execution is **serialized onto the game thread** — one tool call at a
|
||||
time, editor UI blocked while each runs.
|
||||
|
|
@ -50,57 +57,143 @@ Schema payload grows with every registered toolset, and tool authors are told
|
|||
NOT to rely on eager advertising — stay in tool-search mode unless a very
|
||||
small fixed surface is wanted.
|
||||
|
||||
## call_tool dispatch semantics
|
||||
## call_tool dispatch semantics (live-verified, 5.8)
|
||||
|
||||
Confirmed against Epic's own agent-facing skill for this server:
|
||||
Verified against a running 5.8 server; these details are where naive
|
||||
clients die:
|
||||
|
||||
- `call_tool` takes `toolset_name`, `tool_name`, and an `arguments` object
|
||||
matching the schema from `describe_toolset`. The result returns on the
|
||||
same turn.
|
||||
- Tool identities are effectively dotted: `BlueprintTools.create`,
|
||||
`SequencerTools.create_level_sequence`,
|
||||
`LiveCodingToolset.CompileLiveCoding` — they are dispatched server-side
|
||||
and never appear as native MCP tools while tool search is on.
|
||||
- Top-level dispatch (omitting `toolset_name`) is reserved for tools
|
||||
registered directly on the MCP server — and is rejected for `call_tool`
|
||||
itself.
|
||||
- `list_toolsets` returns **fully-qualified** toolset names — Python:
|
||||
`editor_toolset.toolsets.scene.SceneTools`; C++:
|
||||
`EditorToolset.EditorAppToolset`. Epic's prose says "SceneTools"; the
|
||||
registry speaks qualified names. Use them verbatim in `describe_toolset`
|
||||
and `call_tool`'s `toolset_name`.
|
||||
- `tool_name` must be the **short** name (`get_current_level`,
|
||||
`CaptureViewport`). Passing the fully-qualified tool name fails with
|
||||
"Unknown tool" even though `describe_toolset` displays qualified names.
|
||||
- `call_tool` args: `{"toolset_name": ..., "tool_name": ..., "arguments":
|
||||
{...}}`; result returns on the same turn (the HTTP response blocks until
|
||||
the game thread finishes the call).
|
||||
- **`TOptional` parameters must be passed explicitly as `null`** — omitting
|
||||
them errors with `input param "X" needs a default value`. E.g.
|
||||
`CaptureViewport` minimal call is `{"captureTransform": null,
|
||||
"annotations": null, "bShowUI": false}`.
|
||||
- **Schema `required` is literal.** `find_actors` marks `name`, `tag`,
|
||||
`collision_channels` required even though they're semantically optional —
|
||||
pass `""` / `[]` to mean "any".
|
||||
- **Property names are camelCase with UE's `b` prefix intact** at this
|
||||
reflection layer: `bUseTemperature`, `bAtmosphereSunLight`, `fogDensity`,
|
||||
`bRealTimeCapture`, `mobility`. Writing `useTemperature` does NOT error
|
||||
the whole call — the response names each property that could not be set
|
||||
(schema-in-error style; READ error text, it lists the exact failures and
|
||||
often the full input schema).
|
||||
- **Object references travel as `{"refPath": "<soft object path>"}`**
|
||||
everywhere (actors, classes, components). Class refs use
|
||||
`/Script/Module.Class` (e.g. `/Script/Engine.PointLight`); actor refs are
|
||||
the full path (`/Temp/Untitled_1.Untitled_1:PersistentLevel.DirectionalLight_UAID_...`).
|
||||
Spawn/find tools RETURN refPaths — capture and reuse them.
|
||||
- **`ObjectTools.set_properties` takes `values` as a JSON *string***, not
|
||||
an object: `{"instance": {"refPath": ...}, "values":
|
||||
"{\"intensity\": 10.0}"}`. `get_properties` likewise returns a JSON
|
||||
string inside `returnValue`. Double-encode/decode accordingly.
|
||||
- Primitive results arrive wrapped as `{"returnValue": ...}` inside the
|
||||
text content block.
|
||||
|
||||
### HTTP wire behavior (for raw clients / debugging)
|
||||
|
||||
- `initialize` → plain JSON response + `Mcp-Session-Id` header you must
|
||||
echo on every subsequent request; `notifications/initialized` → 202
|
||||
empty; `tools/call` → **`text/event-stream`**: the result arrives as an
|
||||
`event: message` + `data: <jsonrpc>` frame only when the game thread
|
||||
finishes. A client that treats the response as plain JSON reads an empty
|
||||
body. Send `Accept: application/json, text/event-stream` always.
|
||||
|
||||
## Shipped toolsets
|
||||
|
||||
The registry is project-dependent; treat this as orientation, not contract —
|
||||
`describe_toolset` on the live server is the only source of truth for tool
|
||||
names and schemas. Epic's plugin pack describes the shipped surface as
|
||||
"hundreds of tools across 30+ toolsets": actors, blueprints, materials,
|
||||
Niagara, Control Rigs, Sequencer, State Trees, widgets, Gameplay Ability
|
||||
System, automation testing, Live Coding.
|
||||
The registry is project-dependent; `describe_toolset` on the live server is
|
||||
the only source of truth for schemas. The core surface below is verified
|
||||
against EditorToolset's source in the 5.8 install (Python:
|
||||
`.../EditorToolset/Content/Python/editor_toolset/toolsets/`; C++:
|
||||
`EditorAppToolset.h`).
|
||||
|
||||
Toolset names confirmed in Epic's docs and Epic's own Claude Code skill pack:
|
||||
**EditorToolset plugin (the core), Python toolsets** (live-verified on 5.8;
|
||||
qualified prefix `editor_toolset.toolsets.<module>.<Class>`):
|
||||
|
||||
| Toolset | Scope |
|
||||
| Toolset | Verified tools (subset) |
|
||||
|---|---|
|
||||
| `SceneTools` | Scene-level queries and edits |
|
||||
| `ActorTools` | Inspect/modify actors: transforms, labels, parent-child relationships, components |
|
||||
| `MaterialInstanceTools` | Create/configure material instances |
|
||||
| `MaterialTools` | Material assets |
|
||||
| `ObjectTools` | Generic UObject property inspection/editing |
|
||||
| `BlueprintTools` | Blueprint creation/editing (e.g. `BlueprintTools.create`) |
|
||||
| `StaticMeshTools` | Static mesh asset operations |
|
||||
| `LevelTools` | Level operations |
|
||||
| `SequencerTools` | Level Sequences (e.g. `create_level_sequence`) |
|
||||
| `LiveCodingToolset` | C++ Live Coding recompile (`CompileLiveCoding` blocks until the compile finishes and surfaces compiler diagnostics) |
|
||||
| `AgentSkillToolset` | Project-registered Agent Skills (see below) |
|
||||
| `GASToolsets` (plugin, C++) | Gameplay Ability System attributes — ships disabled; enabling prompts an experimental-feature warning |
|
||||
| `scene.SceneTools` | `load_level`, `get_current_level`, `find_actors` (by name/type/tag/bounds), `add_to_scene_from_class`, `add_to_scene_from_asset`, `remove_from_scene`, `save_actor`, `create_level_instance`, folders |
|
||||
| `actor.ActorTools` | `get_label`/`set_label`, tags, `get_actor_transform`/`set_actor_transform` (`xform` fields optional = "don't change"), parenting, components |
|
||||
| `primitive.PrimitiveTools` | `add_cube` (dimensions), `add_sphere` (radius), `add_cylinder`/`add_cone` (radius+height) — adds StaticMeshComponents with `local_transform` to a host actor: spawn `/Script/Engine.Actor`, then compose. The fastest blocking path, zero asset dependencies |
|
||||
| `object.ObjectTools` | `list_properties` (returns full JSON schema of every property), `get_properties`/`set_properties` (JSON-string `values`), `reset_properties` (restore defaults — also your rollback), `get_class`, `search_subclasses` |
|
||||
| `material_instance.MaterialInstanceTools` | `create`, `list_parameters`, `get/set_scalar_parameter`, `get/set_vector_parameter` |
|
||||
| `asset.AssetTools` | `find_assets`, `load_asset`, `exists`, `save_assets`, `is_dirty`, `get_dependencies`/`get_referencers` (check before delete!), `delete`, `move`, `duplicate`, folders, `read_file`/`write_file` (project-scoped) |
|
||||
| `blueprint.BlueprintTools` (+ dsl/layout/node) | Blueprint authoring |
|
||||
| `material.MaterialTools`, `static_mesh.StaticMeshTools`, `texture.TextureTools`, `data_table.DataTableTools`, … | per-asset-type operations |
|
||||
| `programmatic.ProgrammaticToolset` | **the batching escape hatch** — see below |
|
||||
|
||||
Known gap: the shipped toolsets contain **no mesh-modelling tools** — you
|
||||
can spawn/place/instance existing meshes but not author new geometry. The
|
||||
supported route to parametric geometry is a custom Python toolset wrapping
|
||||
**Geometry Script** (`UDynamicMesh`: append box/cylinder/sphere, booleans,
|
||||
then `Create New Static Mesh Asset from Mesh` to bake an `SM_` asset). For
|
||||
organic/sculpted meshes, model in Blender (`blender-mcp` skill) and import.
|
||||
**`EditorToolset.EditorAppToolset` (C++, same plugin) — the agent's eyes
|
||||
(full live list):** `CaptureViewport`, `CaptureEditorImage`,
|
||||
`CaptureAssetImage`, `GetCameraTransform`/`SetCameraTransform`,
|
||||
`GetSelectedActors`/`SelectActors`/`FocusOnActors`/`GetVisibleActors`,
|
||||
`WorldPosToScreenCoords`/`ScreenCoordsToWorld`,
|
||||
`GetSelectedAssets`/`SelectAssets`,
|
||||
`GetContentBrowserPath`/`SetContentBrowserPath`, `OpenEditorForAsset`,
|
||||
`GetOpenAssets`, `SearchCVars`, `StartPIE`/`StopPIE`/`IsPIERunning`.
|
||||
|
||||
`CaptureViewport` specifics (live-verified): args `{"captureTransform":
|
||||
<transform-or-null>, "annotations": <config-or-null>, "bShowUI": false}`.
|
||||
Returns base64 PNG (decode + save it yourself) plus camera
|
||||
location/rotation/FOV. `captureTransform` captures from any pose WITHOUT
|
||||
moving the user's viewport — use it as a virtual camera. Annotation config
|
||||
`{"gridSpacingCm": 500, "gridExtentCm": 3000, "gridHeight": <ground Z>,
|
||||
"labelActors": true}` overlays a projected ground grid and actor callouts;
|
||||
**grid coordinate labels are in METERS** (world cm ÷ 100). Use annotated
|
||||
captures for placement work, clean ones for beauty checks.
|
||||
|
||||
Also confirmed live: `ToolsetRegistry.AgentSkillToolset`,
|
||||
`EditorToolset.LogsToolset` (read Output Log + set verbosity — useful for
|
||||
self-debugging), `SemanticSearchToolset` (hybrid vector+BM25 asset search),
|
||||
five `NiagaraToolsets.NiagaraToolset_*` groups, `PCGToolset` (+Spatial),
|
||||
`UMGToolSet`, three `GASToolsets.*`, `AutomationTestToolset`,
|
||||
`ConfigSettingsToolset` (read/write Project Settings & Editor Preferences
|
||||
sections by schema — the remote path to exposure defaults, rendering
|
||||
settings, etc.), `SlateInspectorToolset`, `PluginToolset`,
|
||||
`animation_toolset.toolsets.sequencer.SequencerTools` + keyframing/
|
||||
controlrig/outliner siblings, `aimodule_toolset` BehaviorTreeTools,
|
||||
`state_tree_toolset` StateTreeTools, and more — 67 toolsets on a blank
|
||||
project with AllToolsets enabled.
|
||||
|
||||
Known gap: no mesh-modelling tools — spawn/place/instance existing meshes,
|
||||
yes; author new geometry, no. The supported route to parametric geometry is
|
||||
a custom Python toolset wrapping **Geometry Script** (`UDynamicMesh`:
|
||||
append box/cylinder/sphere, booleans, then `Create New Static Mesh Asset
|
||||
from Mesh` to bake an `SM_` asset). For organic/sculpted meshes, model in
|
||||
Blender (`blender-mcp` skill) and import.
|
||||
|
||||
First-session move: `list_toolsets`, then `describe_toolset` each group you
|
||||
plan to use, and keep those schemas in working memory for the session.
|
||||
|
||||
## ProgrammaticToolset — sanctioned batching
|
||||
|
||||
The serial-call rule makes N-step edits slow over the wire. The shipped
|
||||
answer is `ProgrammaticToolset` (verified in `programmatic.py`):
|
||||
|
||||
1. `get_execution_environment` — **mandatory first call** (the tool's own
|
||||
docstring requires it); returns the allowed modules, script constraints,
|
||||
and usage instructions.
|
||||
2. `execute_tool_script(script)` — runs a **sandboxed** Python script that
|
||||
defines `run() -> dict`. Inside, you call other registered tools
|
||||
programmatically and glue them with logic — one MCP round-trip for a
|
||||
whole loop (e.g. spawn 20 actors with computed transforms).
|
||||
|
||||
Sandbox facts (from source): allowed imports are `json`, `math`,
|
||||
`datetime`, `copy`, `re`, `time` only; `open()` is restricted to
|
||||
project-contained paths; scripts run inside an editor **transaction scope**
|
||||
(undo-friendly); it is tool orchestration, NOT general Python — arbitrary
|
||||
`unreal.*` calls are not the contract. Data returns via `run()`'s dict.
|
||||
|
||||
Use it whenever a recipe loop exceeds ~5 homogeneous calls; keep one-off
|
||||
edits as plain `call_tool`.
|
||||
|
||||
## Project Agent Skills (AgentSkillToolset)
|
||||
|
||||
Projects and plugins can register **Agent Skills** — named instruction
|
||||
|
|
@ -121,20 +214,17 @@ Check at the start of unfamiliar work in any project, not just once ever.
|
|||
|
||||
An agent that can't see the viewport is flying blind. In order of preference:
|
||||
|
||||
1. **A shipped screenshot/viewport tool, if the registry advertises one** —
|
||||
check `list_toolsets`/`describe_toolset` output for viewport, screenshot,
|
||||
or thumbnail capture tools and use those (they return the image through
|
||||
MCP directly).
|
||||
2. **Console command via any shipped console/exec tool**: `HighResShot 1`
|
||||
writes the current viewport to
|
||||
1. **`EditorAppToolset.CaptureViewport`** (confirmed shipped) — returns the
|
||||
image through MCP as base64 PNG with camera metadata; supports capturing
|
||||
from an arbitrary transform without disturbing the user's viewport, and
|
||||
an optional annotation overlay (world-space meter grid + actor callouts)
|
||||
for spatial-placement work. This is the default verification tool.
|
||||
2. **Console `HighResShot` via any console/exec tool** when you need
|
||||
resolutions beyond the viewport: `HighResShot 3840x2160` writes to
|
||||
`<Project>/Saved/Screenshots/<Platform>/` on the EDITOR host's
|
||||
filesystem. `HighResShot 3840x2160` for fixed resolution,
|
||||
`HighResShot 2` for 2× viewport. Read the file back with `read_file`/
|
||||
`vision_analyze` (same machine) — remember paths resolve on the editor
|
||||
host.
|
||||
3. **Custom toolset escape hatch** (below) exposing
|
||||
`unreal.AutomationLibrary.take_high_res_screenshot()` or viewport
|
||||
capture, when nothing shipped covers it.
|
||||
filesystem; read the file back (same machine) with `vision_analyze`.
|
||||
3. **Custom toolset escape hatch** for anything else (e.g. camera-actor
|
||||
framed captures with MRQ-quality settings).
|
||||
|
||||
Always `vision_analyze` the capture and art-direct against the brief before
|
||||
declaring a milestone done.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue