fix(gateway): auto-start after container restart via planned-stop marker (#42675) (#43236)

* fix(gateway): auto-start after container restart via planned-stop marker

On Docker (s6-overlay), the gateway runs as a dynamically-registered s6
service. When the container stops/restarts/upgrades, s6 sends the gateway
a plain SIGTERM. The shutdown path (_stop_impl) ended with an
unconditional _update_runtime_status("stopped"), persisting
gateway_state=stopped to the volume. container_boot.py reads that on the
next boot and only auto-starts gateways whose last state was "running"
(_AUTOSTART_STATES) — so after a routine `docker compose up
--force-recreate` the gateway stays down and messaging channels silently
go dark, with no error surfaced (issue #42675).

The codebase already distinguishes intentional stops from unexpected
signals via the planned-stop marker (write_planned_stop_marker /
consume_planned_stop_marker_for_self): `hermes gateway stop`,
systemd/launchd ExecStop, and Ctrl+C write a marker before signalling,
so the handler classifies them as planned. An unmarked SIGTERM
(container/s6 restart, OOM, bare kill) is signal-initiated.

This wires that existing classification through to the state persist,
rather than adding unreliable signal-source inference:

- run.py: GatewayRunner._signal_initiated_shutdown, set in
  shutdown_signal_handler's unmarked-signal branch. In _stop_impl, a
  signal-initiated (non-restart) teardown now persists "running" instead
  of "stopped" — preserving the operator's run-intent and overwriting the
  mid-shutdown "draining" marker so _AUTOSTART_STATES matches on reboot.
  Operator stops and restarts persist "stopped" as before.

- service_manager.py: S6ServiceManager.stop() now writes the planned-stop
  marker for the supervised PID (read from s6-svstat) before `s6-svc -d`,
  so an in-container `hermes gateway stop` is correctly classified as
  intentional (parity with the systemd/launchd/host stop paths, which
  already mark). Best-effort: a marker-write failure falls back to the
  safe signal-initiated path.

Tests: shutdown persist-decision table (signal→running, operator→stopped,
restart→stopped), s6 stop marker write + svstat PID parse + failure
tolerance. The signal→running and s6-marker tests fail without the
respective source change. Verified end-to-end against a container built
from this branch: an unmarked SIGTERM to the live gateway leaves
gateway_state=running (shutdown-context log confirms signal path);
existing real container-restart suite still green.

* docs(docker): clarify gateway autostart distinguishes operator-stop from container-kill

The per-profile-supervision section described the autostart-across-restart
contract as "running gateways come back, stopped stay stopped" without
spelling out what records 'stopped'. That contract was the source of
#42675 confusion: users expected a restart to bring the gateway back and
it didn't. With the write-side fix, only an explicit `hermes gateway stop`
records 'stopped'; container/s6 restart SIGTERMs (incl. image upgrades and
unexpected exits) leave the state 'running' so the gateway auto-starts.
Make that distinction explicit in both the multi-profile and
per-profile-supervision sections.

* test(docker): real-restart autostart E2E for #42675

Adds test_live_gateway_autostarts_after_real_restart_without_manual_state_stamp:
a live s6-supervised gateway is killed by an actual `docker restart`
SIGTERM (no manual gateway_state stamp, no planned-stop marker) and must
auto-start on the next boot. Exercises the WRITE side of the fix that the
existing stamp-based tests bypass.

Verified to FAIL against an origin/main image (reconciler logs
prior_state=stopped action=registered — the #42675 bug) and PASS against
the fixed image (prior_state=running action=started).

tests/hermes_cli/test_service_manager.py

@@ -799,3 +799,111 @@ def test_s6_is_running_parses_svstat(
         return _sp.CompletedProcess(cmd, 0, "", "")
     monkeypatch.setattr("subprocess.run", _svstat_down)
     assert S6ServiceManager(scandir=s6_scandir).is_running("gateway-coder") is False
+
+
+# ---------------------------------------------------------------------------
+# S6 stop writes a planned-stop marker (issue #42675)
+#
+# `hermes gateway stop` inside a container dispatches through
+# S6ServiceManager.stop() -> `s6-svc -d`, which SIGTERMs the gateway.
+# That SIGTERM is indistinguishable from the one s6/Docker sends on a
+# container restart unless we mark the intentional stop first. Without
+# the marker, the gateway's shutdown handler can't tell an operator
+# stop from a restart kill, and the gateway_state=stopped suppression
+# (run.py) would never engage for explicit stops.
+# ---------------------------------------------------------------------------
+
+
+def test_s6_supervised_pid_parses_svstat(monkeypatch, s6_scandir):
+    """_supervised_pid extracts the PID from `up (pid NNNN) ...`."""
+    import subprocess as _sp
+
+    def _fake(cmd, **kw):
+        return _sp.CompletedProcess(cmd, 0, "up (pid 4242) 17 seconds\n", "")
+
+    monkeypatch.setattr("subprocess.run", _fake)
+    mgr = S6ServiceManager(scandir=s6_scandir)
+    assert mgr._supervised_pid("gateway-coder") == 4242
+
+
+def test_s6_supervised_pid_none_when_down(monkeypatch, s6_scandir):
+    """A down service (`s6-svstat` rc!=0 or no pid) yields None."""
+    import subprocess as _sp
+
+    def _fake(cmd, **kw):
+        return _sp.CompletedProcess(cmd, 0, "down (exitcode 0) 3 seconds\n", "")
+
+    monkeypatch.setattr("subprocess.run", _fake)
+    mgr = S6ServiceManager(scandir=s6_scandir)
+    assert mgr._supervised_pid("gateway-coder") is None
+
+
+def test_s6_stop_writes_planned_stop_marker(monkeypatch, s6_scandir):
+    """stop() must mark the supervised PID before `s6-svc -d` so the
+    gateway recognises the SIGTERM as an intentional stop (#42675)."""
+    import subprocess as _sp
+
+    svc_dir = s6_scandir / "gateway-coder"
+    svc_dir.mkdir()  # so _run_svc doesn't raise GatewayNotRegisteredError
+
+    svc_calls: list[list[str]] = []
+
+    def _fake(cmd, **kw):
+        seq = list(cmd) if isinstance(cmd, (list, tuple)) else [str(cmd)]
+        if seq and seq[0].startswith("/command/"):
+            seq[0] = seq[0][len("/command/"):]
+        svc_calls.append(seq)
+        if seq and seq[0] == "s6-svstat":
+            return _sp.CompletedProcess(cmd, 0, "up (pid 9090) 5 seconds\n", "")
+        return _sp.CompletedProcess(cmd, 0, "", "")
+
+    monkeypatch.setattr("subprocess.run", _fake)
+
+    marked: list[int] = []
+    monkeypatch.setattr(
+        "gateway.status.write_planned_stop_marker",
+        lambda pid: marked.append(pid) or True,
+    )
+
+    mgr = S6ServiceManager(scandir=s6_scandir)
+    mgr.stop("gateway-coder")
+
+    assert marked == [9090], (
+        f"stop() must write the planned-stop marker for the supervised PID; "
+        f"marked={marked}"
+    )
+    # And it must still issue the down command.
+    assert any(
+        cmd[0] == "s6-svc" and "-d" in cmd for cmd in svc_calls
+    ), f"s6-svc -d not invoked; saw: {svc_calls}"
+
+
+def test_s6_stop_tolerates_marker_write_failure(monkeypatch, s6_scandir):
+    """A marker-write failure must not block the stop (best-effort)."""
+    import subprocess as _sp
+
+    svc_dir = s6_scandir / "gateway-coder"
+    svc_dir.mkdir()
+
+    svc_calls: list[list[str]] = []
+
+    def _fake(cmd, **kw):
+        seq = list(cmd) if isinstance(cmd, (list, tuple)) else [str(cmd)]
+        if seq and seq[0].startswith("/command/"):
+            seq[0] = seq[0][len("/command/"):]
+        svc_calls.append(seq)
+        if seq and seq[0] == "s6-svstat":
+            return _sp.CompletedProcess(cmd, 0, "up (pid 9090) 5 seconds\n", "")
+        return _sp.CompletedProcess(cmd, 0, "", "")
+
+    monkeypatch.setattr("subprocess.run", _fake)
+
+    def _boom(pid):
+        raise OSError("disk full")
+
+    monkeypatch.setattr("gateway.status.write_planned_stop_marker", _boom)
+
+    mgr = S6ServiceManager(scandir=s6_scandir)
+    mgr.stop("gateway-coder")  # must not raise
+
+    assert any(cmd[0] == "s6-svc" and "-d" in cmd for cmd in svc_calls)