Skip to main content

Ghost media companion

The optional default plugin plays the original Ghost APNGs inside the human message composer. It does not create an agent or a separate tiled window. Other plugins can replace it through the shared composer/raster UI contract.

Image output and terminal support​

The renderer probes actual capabilities, independently of the file format:

  • Kitty graphics: full RGBA source pixels, transparency and cached frame uploads.
  • SIXEL: an adaptive palette of up to 256 colours, queried physical cell size, and alpha composited against the queried terminal background. The renderer erases stale placements before text/layout changes; normal frame updates replace their own rectangle without a blank flash.
  • Other terminals: a compact status, with images unavailable in the composer title. The default does not turn a GIF into a low-resolution text character.

AGENT_GRAPHICS=auto is the default. kitty explicitly selects Kitty output; sixel selects SIXEL (cell metrics/background replies are still required for safe placement); off disables image output. In OpenTUI, the composer also supports Kitty graphics through tmux when allow-passthrough on is configured and the terminal confirms virtual image placement. Unanswered or unsupported probes fall back to status; screen and other multiplexer paths remain text-only. SIXEL terminals can answer the direct cell-size query, or text-area pixel/cell queries (as VTE does). Font-size changes re-query geometry. Direct native image updates bypass React/Ink when text and layout are unchanged; tmux composer frames use the negotiated virtual-image path. Actual terminal throughput still determines how many frames are visible. No FFmpeg executable is needed at runtime.

A terminal is a rendering client, not a universally addressable framebuffer. Full pixel playback requires a graphics protocol. FFmpeg's libcaca output is text rendering too; it cannot add image support to an arbitrary terminal. See the Kitty protocol, VTE SIXEL property and FFmpeg CACA documentation.

GNOME Terminal can enable SIXEL through its profile when the VTE build includes it; application probing must still confirm support. If the setting exists in your installation, enable it for a chosen profile in the terminal settings or with its enable-sixel GSettings key. This project does not modify user terminal profiles automatically. Unsupported builds/settings stay in status mode.

Playback and commands​

  • /mascot on|off: attach/restart or remove the companion; persisted.
  • /mascot pause|resume: freeze source time or continue it; persisted.
  • /mascot pet: briefly select the happy animation.
  • /mascot status: show frame count, average source fps and image-slot state.
  • /mascot reload: reread configuration and the pack.

These are immediate user controls. They run while an agent is working without queueing a prompt, interrupting its turn or making a model request. The prompt handler is also retained for the legacy dispatch path. Controls preserve their order for the same companion. Cancellation revokes later UI and timer work; an already completed settings write is not rolled back. Pause freezes source time when its control starts, before settings are saved; activity changes and transient expiry continue while saving is in progress.

GIF and APNG retain source frame timing and loop counts. The player uses elapsed source time, skips late frames instead of accumulating rendering delay, freezes when hidden/compact, and disposes timers on unload. Protected human dialogs and slash completion keep their input space. NO_COLOR, TERM=dumb and pause stop motion; activity/status changes and transient expiry still work. The text layer permits up to 60 redraws/s, while pixel-only updates use the native image path. Frame deadlines round up to the next millisecond, and publication that crosses a frame boundary schedules a prompt catch-up using the current source time. This is not a guarantee of 60 visible fps in every terminal or connection.

The configured OpenTUI frontend selects Kitty or SIXEL only after actual terminal capability detection. SIXEL also requires queried pixel geometry, which is refreshed after resize. Kitty composer images through tmux require passthrough and a successful capability reply. Unsupported protocols, screen, constrained panes, AGENT_GRAPHICS=off, reduced motion and NO_COLOR retain the selected producer's compact status instead of substituting block artwork. Pixel subscriptions stop in compact mode; activity labels and transient expiry still update. A hidden composer yields its media subscription to the visible editor. Removing a selected replacement contribution resumes the previous producer without another focus or layout action. Physical behavior varies by terminal; negotiated support alone is not a native rendering certificate.

The old Ghost source has six frames at 180 ms each (about 5.6 fps). A new renderer cannot manufacture missing motion. Supply a higher-frame-rate APNG/GIF for a smoother original animation; no poses are invented or cross-faded by default.

Packs and replacement plugins​

Settings use shared plugin configuration scopes (mascot.json); commands save project overrides. Custom pack files are confined to the project, including symlinks. Map any state to a GIF, APNG or PNG; decoding follows the file header:

{
"enabled": true,
"paused": false,
"pack": "assets/my-ghost",
"animations": {"idle": "idle.apng", "thinking": "thinking.gif"},
"tools": {"my_search": "searching"},
"reactions": {"my-plugin:compiling": "editing"}
}

Unmapped names use <state>.apng. States are idle, thinking, searching, editing, planning, waiting_subagents, question, approval, done and error. Missing/bad states fall back to idle and then status, with one warning per state. Bounds are 2 MiB input, 512×512, up to 600 frames and 8 million decoded pixels per animation. Nonzero frame delays are preserved; unspecified zero delays use 100 ms for GIF and 10 ms for APNG. GIF transparent patches, interlace (decoder), disposal 2/3, and loop metadata are supported. Static images and finite loops stop at their final frame.

The bundled APNGs have unintentionally transparent dark outlines retaining RGB. Their existing restoreInk repair is enabled automatically only for the exact historical file fingerprints; new/replaced artwork is never repaired implicitly. Custom packs also default off. Use restoreInk:false for exact original transparency. GIFs never use this repair.

Replacement producers publish bounded frames with the portable API:

await $.ui.view('companion', 'My companion', {
attachment: {site:'composer', width:18, rows:6, label:'Ready'},
raster: {width, height, rgba: rgbaBytes.toString('base64'), alt:'Ready'},
lines: [],
});
await $.ui.remove('companion');

The host owns drafts, focus, submission and dialogs. Native pixels occupy only the reserved art rectangle, never the editable text. Slots use stable producer registration order: the latest contribution is selected, older producers get ui.visibility with visible:false, and resume when selected again. Image motion only runs when the slot is expanded (Ink: >=60 columns, >=24 terminal rows; OpenTUI: >=60 columns and >=20 rows of pane content). Text-only attachments remain supported. Plugin ui.reaction events and tool/subagent activity still select states without influencing the observed action.

Verification​

npm run visual:mascot runs the real PTY app with the diagnostic Kitty emulator; it verifies source dimensions, frame changes, pause, pet, resize, drafts, disable and restart. The emulator is not certification of a physical terminal. VISUAL_MASCOT_GIF=/absolute/path/to/test.gif selects a GIF in its isolated test project. VISUAL_EXPECT_MIN_FPS sets an explicit test-only observation threshold; reports distinguish observed changes from source fps. node tools/visual/sixel-reference-check.mjs optionally checks encoded pixels against ImageMagick's independent SIXEL decoder. SIXEL negotiation/cleanup and GIF compositing/timeline behavior also have package tests. Native GNOME hardware rendering has not been verified by these tests.