← Back to PRs

#23764: feat(tui): render inline images via MEDIA: protocol and pi-tui Image

by ademczuk open 2026-02-22 17:29 View on GitHub →
size: M
## Summary The TUI (`openclaw tui`) currently shows `[image/png 3kb (omitted)]` placeholders instead of rendering images inline in terminals that support the Kitty or iTerm2 graphics protocols. This PR fixes that by leveraging the existing `MEDIA:/path` text protocol that `imageResult()` already produces alongside sanitized image data. **4 files changed, 138 insertions, 9 deletions** -- all within `src/tui/`. ### Why the naive approach fails There are three independent reasons the obvious "just use base64 data" approach doesn't work: 1. **`sanitizeToolResult()` strips image data** before it reaches the TUI -- but `imageResult()` also creates a `MEDIA:/path/to/file` text block that survives sanitization 2. **Pi-tui's `Text` component breaks escape sequences** -- `wrapTextWithAnsi()` destroys binary image data. Using pi-tui's `Image` class as a `Container` child via `addChild()` bypasses text processing entirely 3. **The verbose filter discards tool results** -- default verbose level "off" drops ALL tool events before `ToolExecutionComponent` is ever created Plus: **agent-stream tool events are never sent to TUI WebSocket clients** for live sessions, so the only working path for tool result display is history loading -- which has its own bug for local runs. ### Changes | File | Change | |------|--------| | `tui-formatters.ts` | Add `resultHasMedia()` utility to detect MEDIA: content in tool results | | `tool-execution.ts` | Import pi-tui `Image`, add `getMediaPaths()` + capability detection, render images as `addChild()` children, filter MEDIA: lines from text output | | `tui-event-handlers.ts` | Bypass verbose filter for MEDIA: tool results (live events), fix `maybeRefreshHistoryForRun` early return that skipped `loadHistory()` for local runs | | `tui-session-actions.ts` | Bypass `showTools` filter for MEDIA: tool results (history loading) | ### How it works 1. When a tool result contains `MEDIA:/path/to/file` text, `ToolExecutionComponent.refresh()` reads the file with `readFileSync`, base64-encodes it, and creates a pi-tui `Image` child 2. Terminal capability is detected via `TERM_PROGRAM` / `ITERM_SESSION_ID` env vars (WezTerm, Kitty, iTerm2, Ghostty) 3. `extractText()` filters out MEDIA: lines and suppresses `[image (omitted)]` placeholders when the actual image will be rendered 4. Tool results with MEDIA: content bypass the verbose filter in both live events and history loading ### Graceful degradation - **Terminals without image support**: standard text placeholder (`[image/png 3kb (omitted)]`) is shown - **Remote clients**: `readFileSync` fails silently for paths not on the local filesystem; text placeholder remains - **No MEDIA: paths**: behavior is identical to current code ### Tested with - OpenClaw 2026.2.22 in Docker, WezTerm on Windows via iTerm2 protocol - All 129 TUI tests pass, 0 lint warnings - Verified inline rendering of PNG/JPG images from tool results ![OpenClaw TUI rendering inline images via iTerm2 protocol in WezTerm](https://raw.githubusercontent.com/ademczuk/MenuVision/master/docs/openclaw-tui-inline-images.png) *Screenshot: OpenClaw TUI in WezTerm showing inline-rendered restaurant menu photos via the iTerm2 image protocol.* ### Discussion This work was documented and discussed in #23618 before this PR was created. ## Test plan - [x] `pnpm build` succeeds (287 files bundled) - [x] `pnpm check` passes (format + lint clean; pre-existing tsgo errors in telegram/monitor.ts are unrelated) - [x] `pnpm test` -- all 129 TUI tests pass (16 test files) - [x] Manual test: inline images render in WezTerm via iTerm2 protocol - [x] Manual test: text placeholder shown when terminal doesn't support images - [ ] Test in Kitty terminal - [ ] Test in iTerm2 on macOS 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- greptile_comment --> <h3>Greptile Summary</h3> Enables inline image rendering in the TUI for terminals supporting Kitty/iTerm2 graphics protocols by reading local files referenced via the `MEDIA:/path` text protocol that survives sanitization. **Key changes:** - Added `resultHasMedia()` utility to detect `MEDIA:` content in tool results - Modified `ToolExecutionComponent` to read image files via `readFileSync`, base64-encode them, and render using pi-tui's `Image` class - Bypassed verbose filter for tool results containing `MEDIA:` paths in both live events and history loading - Terminal capability detection via env vars (`TERM_PROGRAM`, `ITERM_SESSION_ID`) for WezTerm, Kitty, iTerm2, Ghostty - Fixed early return in `maybeRefreshHistoryForRun` that prevented tool results from appearing for local runs **Issue found:** - The MEDIA: path extraction regex in `getMediaPaths()` doesn't match the canonical pattern used elsewhere in the codebase and won't handle backtick-wrapped paths <h3>Confidence Score: 4/5</h3> - Safe to merge with minor regex pattern inconsistency that may prevent some edge cases from rendering - Implementation is well-structured with proper error handling and graceful degradation. The regex pattern issue is unlikely to affect common use cases since most paths won't be backtick-wrapped, but it's an inconsistency worth addressing for robustness. All tests pass and the feature is additive (won't break existing functionality). - src/tui/components/tool-execution.ts - check the MEDIA: path extraction regex pattern <sub>Last reviewed commit: 92ba3d7</sub> <!-- greptile_other_comments_section --> <sub>(2/5) Greptile learns from your feedback when you react with thumbs up/down!</sub> <!-- /greptile_comment -->

Most Similar PRs