TheSmokeDev/hermes-talk

Hermes Talk

Realtime duplex voice for your own agent — talk to it, it runs real work in the background, it reports back out loud.

Hermes Talk is a realtime voice plugin for Hermes Agent: you speak, it answers out loud, and it calls the agent's own tools without leaving the conversation. It runs in the terminal (hermes talk), in a Discord voice channel (/talk join), in the Hermes dashboard Talk tab, and in Hermes Desktop.

It rides your ChatGPT or SuperGrok subscription. No API key required.

The realtime lanes are OpenAI Realtime (gpt-realtime-2.1, on your ChatGPT subscription through codex login), xAI Grok Voice (on an X Premium or SuperGrok login, no key), and Gemini Live (which does need a GEMINI_API_KEY — free-tier AI Studio keys work); GPT-Live is selected separately by TALK_VOICE_MODE=live and runs gpt-live-1-codex on the Codex subscription or gpt-live-1 on explicitly chosen API billing.

Talk calls the host's tools, delegates background work while you keep talking, reports results, and handles current approval requests. Provider support differs by surface; see the tables below.

What it is: a plug-in that adds interruptible, duplex speech-to-speech voice to an existing Hermes Agent install — one bidirectional audio session in which the agent uses its real tools, hands work to background agents, and speaks the results when they land.

Who it is for: people already running Hermes Agent who would rather talk to it than type at it, and who want to keep talking while it works. It is not a standalone assistant, and it does not replace Hermes's built-in turn-based voice mode — that is a different shape, and a good one.

🔊 Watch the 2:27 cut with sound — delegate, keep talking, hear the result land.

Real Realtime session, 8× speed in the GIF, recorded at v0.3.0. It does not demonstrate the new GPT-Live or Codex-worker integration.

Install

hermes plugins install --ref 7ad833e4fcedb2c9a9947d84d7fd89bdbd15d256 TheSmokeDev/hermes-talk --enable
pip install "hermes-talk[audio]"   # mic + speaker support (sounddevice); skip if dashboard-only

Needs Python ≥ 3.11 and a Hermes host ≥ v0.17. GPT-Live, shared task attachment and Codex workers need additional host capabilities; the full list is in Prerequisites.

The commit above is the published v0.22.0 source. Hermes v0.22.5 requires a full 40-character commit SHA for --ref; it does not accept a tag there. This also avoids an older community-catalog pin. For an existing pinned install, preserve local plugin edits, then run:

hermes plugins install --force --ref 7ad833e4fcedb2c9a9947d84d7fd89bdbd15d256 --enable TheSmokeDev/hermes-talk

Restart the gateway/dashboard and Desktop afterward. An unpinned Git install can use hermes plugins update hermes-talk.

Quickstart

Terminal — start here:

hermes talk

You are live: speak, and it answers out loud in the same breath. Ctrl+C hangs up. → Use

Discord — the call happens inside a voice channel, not in chat:

/voice join     # put Hermes in the voice channel first
/talk join      # Talk borrows that connection — it never opens a second one

Talk answers in the room everyone can hear; /talk leave ends it. Mutating tools stay denied until you set TALK_DISCORD_OPERATOR_USER_IDS. → Discord voice

Dashboard — browser audio, no local mic drivers:

hermes dashboard    # then open the Talk tab and hit Start

Allow the microphone and talk. You see the live transcript plus a list of background runs. → Dashboard tab

New in 0.22.0: run your Desktop sessions by voice

With OpenAI Realtime in Desktop, you can now say:

• “Start a new session for this project and run its tests.”
• “Check what the other agent is working on.”
• “Tell that session to check the failing test next.”
• “Steer the running session toward this approach.”
• “Show my capabilities, scheduled jobs, and messaging status.”

Talk discovers the actual project directories and saved conversations on the current connection/profile. It creates a separate session with your opening task, reads live progress and recent replies, and queues or steers your instructions while the original call stays connected. Local and SSH backends use their own supported session contracts. A queued message is reported as accepted, not completed.

Also fixed: Stop → Connect in the same conversation, Hermes-managed ChatGPT login detection, the current Hermes capability catalog, and typed input during unbound realtime calls. Grok's expanded built-in voice list and a speech-tag reference are included too.

Tested on stock Hermes v0.22.5 over SSH, including creating a session in a chosen directory, a real pwd tool call, a follow-up to another session, and steering a running task. These Desktop controls stay within the selected connection/profile. They do not add arbitrary screen control or external message sending. GPT-Live and shared voice-history attachment still require the additional host support described in the Desktop guide.

Every version with its receipts: CHANGELOG.md.

Providers and billing

The provider knob is fail-closed and never inferred from which keys exist. Grok supports all 28 documented built-in voices through TALK_GROK_VOICE (for example, carina); see the voice list. Per-lane detail and the credential order: docs/PROVIDERS.md. Speaking in a voice of your own: docs/CASCADE.md. Experimental xAI speech effects and phrase wrappers, including how native Talk differs from verbatim TTS: docs/XAI-SPEECH-TAGS.md.

Surfaces

Is it working?

hermes plugins list        # → hermes-talk · enabled · current version
hermes talk doctor         # → read-only: auth lane, provider, model/voice, audio, host lanes
hermes talk check          # → doctor + one live provider turn + one bounded Hermes run
# then, in any session: say "status report" — talk_status answers with
# version, auth lane, agent lane, and audio state.

Doctor is read-only by design: it names which lane came up and what is missing, and never writes, probes, or refreshes a token. check is the other half and is deliberately not read-only — one short provider turn and one short agent run, exit 0 only if every step passed. A mock can never go green.

Filing an issue? hermes talk diagnostics --bundle writes one redacted, owner-only file — versions, the names of the variables you have set, device and host facts, and every doctor outcome; no values, logs, prompts, transcripts, audio, or paths. It is safe to paste into a public issue and it is what the bug template asks for.

Upgrade with hermes plugins update hermes-talk — not a second install — then restart the gateway: a running process keeps executing the old code until you do.

The full diagnostic walk, every receipt, and the upgrade runbook: docs/OPERATING.md.

Documentation

Everything Hermes Talk does, in depth:

• OPERATING.md — install, upgrade, use, every knob, the three agent lanes, the Discord lane, the dashboard tab, current boundaries, troubleshooting.
• PROVIDERS.md — per-lane provider detail and the fail-closed OpenAI credential order.
• BACKGROUND-WORK.md — delegation, admission control, steering a running agent, and the capability bridge.
• CASCADE.md — the cascade lane: your own ElevenLabs voice over a realtime provider.
• GPT-LIVE.md — GPT-Live billing and voice, task attachment, Codex workers, operator acceptance.
• DESKTOP.md — Talk in the Hermes desktop app: the stock lane, and what the Talk-enabled build adds.
• VOICE-COMMANDS.md — say this, hear this, and what each spoken receipt commits to.
• REALTIME-ORCHESTRATOR.md — architecture map of the tool-calling realtime lane.
• dashboard-task-continuity.md — joining, continuing and reconnecting to a canonical Hermes task.
• recipient-routing.md — addressing an existing application task from a voice task.
• codex-workers.md — selecting a Codex background worker from a bound task.
• task-event-projection.md — the worker-side library that restores task and work state.
• passive-attachment-client.md — the shared passive-history client used by all three surfaces.
• PROVIDER-RECEIPT.md — how to report a provider lane that worked, or broke, for you.
• CAPABILITY-KERNEL-PORT.md — the capability-plugin kernel adaptation guide.

Design rules

The three that shaped everything else:

• Nothing fails quietly. A degraded backend, a missing tool, a run whose watcher died — each is said out loud in the conversation. A voice surface that silently does less than you asked is worse than one that refuses.
• The credential never leaves the process. Key or OAuth token hits exactly one OpenAI endpoint (the mint) and the socket only ever sees the ephemeral secret it returns.
• Hermes owns the tools and the session. The Realtime layer is ears, mouth, and turn-taking. It never owns the agent loop.

Background

Hermes Talk began as a plugin and became a reference implementation for the speech-to-speech contract Hermes core now carries:

• RFC NousResearch/hermes-agent#77111 — filed from this repo: a RealtimeVoiceProvider ABC for Hermes core.
• PR NousResearch/hermes-agent#101808 — the core contract, orchestrator, and first built-in provider, ported from this plugin's orchestrator and OpenAI transport. hermes-talk already publishes its three lanes on that contract (details).
• PR NousResearch/hermes-agent#97325 — a pointer to this plugin on the official Voice Mode docs page.

Status

Under active development; the PyPI badge above is the released version. Every version with its receipts: CHANGELOG.md. Open epics and threads: #19 provider-neutral Realtime voice platform, #32 operator-grade orchestration UX, #43 Talk over the Bot Mode roster, #44 channel-neutral voice transport.

Contributing

pip install -e ".[dev]" -r requirements-lint.txt, pytest -q, ruff check . — offline, no keys, seconds. Priorities, the path for each kind of change (a provider, a surface, a tool, a fix), the merge bar, and the one test trap on a box that has Hermes installed: CONTRIBUTING.md. First response within 24 hours; good first issue fits in one sitting; a provider that works or broke for you is a contribution too (docs/PROVIDER-RECEIPT.md).

Contributors adapting The Homie's v1.7.0 capability-plugin lessons to Hermes should use the capability-kernel port plan. It maps the reusable safety and lifecycle contracts onto Hermes-owned APIs; it does not claim that hot lifecycle support already exists here.

Contributors

@danclaw93: room-scoped spoken approvals send the request_id the Hermes run API reads, so they stop failing with HTTP 400 (0.17.1).

@HearthCore: diagnosed why Desktop Talk could never connect on a stock Hermes host, and made it connect there; the composer popover also stops closing when the pointer crosses the chat (#169, #170, 0.21.1).

@kvnloo: PulseAudio WebRTC echo cancellation on Linux, and the fix that stopped quiet words being clipped during playback (#81); semantic turn-detection controls across the three lanes (#107, in review). @TheAngryPit — a renderer-owned Realtime transport for the Hermes desktop app that keeps core as the single chat authority (#80, in review). @webdevtodayjason — field-tested feedback from a second live consumer on the upstream RealtimeVoiceProvider contract these lanes register on (hermes-agent#81404).

License

MIT. Adapted-source licenses and contributor credits are in THIRD_PARTY_NOTICES.md.