InfiniteWhispers/HermesAgent-MultiModel

HermesAgent-MultiModel

A fully-local, multi-model AI agent framework with Mixture-of-Agents (MoA) orchestration. Run a high-performance agentic stack on consumer GPUs (RTX 4080/5080) using Hermes + Ollama, with zero cloud dependencies.

Video DriveBy

🎯 Quick Summary

HermesAgent-MultiModel implements a local-first approach to multi-model inference, routing tasks to specialized models (planning, coding, reasoning, vision) via an intelligent orchestration layer. It includes:

• 8-model roster: gpt-oss-20b (tools), gemma4-heretic (fast), qwen3-14b (planning), qwythos-9b (reasoning), qwen25-coder (scripts), ornith-9b (SWE-Bench 69.4%), embeddings, vision
• Mixture-of-Agents (MoA): Query multiple models in parallel, synthesize via aggregator
• Local Ollama backend: No API keys, full privacy, 64K context windows
• Hermes integration: Task routing, personalities, tool-calling, agentic loops
• Optimized for RTX 4080/5080 (16 GB VRAM, 48+ GB RAM)

📦 What's in This Repo

├── README.md                          # This file — start here
├── .hermes/
│   ├── config.yaml                   # Hermes providers + routing config
├── docs/
    └── optimization-guide.md         # 11-section field guide: hardware, models, tuning

🚀 Getting Started

1. Prerequisites

• GPU: RTX 4080 / 5080 or equivalent (16 GB VRAM minimum)
• CPU: 16+ cores (Intel/AMD)
• RAM: 48+ GB (64 GB ideal)
• Storage: 100 GB free NVMe SSD
• OS: Ubuntu 22.04+, WSL2 on Windows 11, or macOS 12+
• Python: 3.10+ (Optional but useful when customizing and building scripts)

2. Install Ollama

# Inspect before running (recommended):
curl -fsSL https://ollama.com/install.sh | less

# Install:
curl -fsSL https://ollama.com/install.sh | sh

3. Pull & Build Models

See the full Ollama Pull Reference for all commands. Quick start:

# Direct pulls (registry)

ollama pull qwen2.5-coder:7b
ollama pull nomic-embed-text:latest
ollama pull qwen3-vl:8b

# Custom builds (pull base, then create)
ollama pull igorls/gemma-4-12B-it-heretic-GGUF:Q4_K_M
ollama create gemma4-heretic-12b -f ~/ollama-modelfiles/gemma4-heretic-12b.Modelfile

ollama pull gpt-oss:20b
ollama create gpt-oss-20b -f ~/ollama-modelfiles/gpt-oss-20b.Modelfile

ollama pull richardyoung/qwythos-9b-abliterated
ollama create qwythos-9b -f ~/ollama-modelfiles/qwythos-9b.Modelfile

ollama pull hf.co/bartowski/deepreinforce-ai_Ornith-1.0-9B-GGUF:Q4_K_M
ollama create ornith-9b -f ~/ollama-modelfiles/ornith-9b.Modelfile

# qwen3-14b-think alias (required — Hermes uses alias, not raw qwen3:14b tag)
ollama pull qwen3:14b
ollama create qwen3-14b-think -f ~/ollama-modelfiles/qwen3-14b-think.Modelfile

# Verify
ollama list

4. Configure Hermes

Edit ~/.hermes/config.yaml with your model provider block. Full reference: Hermes Config — Providers Block.

5. Tune Ollama for Performance

sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

Then restart:

sudo systemctl daemon-reload
sudo systemctl restart ollama

See Ollama Installation & Systemd Tuning for details on why these matter.

📚 Model Roster at a Glance

Co-loading rule: Do NOT load gpt-oss-20b + qwen3-vl-8b simultaneously (~24.5 GB exceeds RTX 5080). One or the other at a time.

🔄 Mixture-of-Agents (MoA)

Query multiple models in parallel, then synthesize via an aggregator:

Default setup:

• References: qwen3-14b-think, ornith-9b
• Aggregator: gpt-oss-20b

🛠️ Advanced Configuration

Hardware Tuning

If running on WSL2, set aggressive resource limits in C:\Users\<YourName>\.wslconfig:

[wsl2]
memory=40GB
processors=24
swap=32GB
localhostForwarding=true

Modelfile Authoring

Three models need custom Modelfiles (HF-hosted, temperature-baked, or name mismatch fixes). See Modelfile Authoring for examples.

KV Cache Tuning

• q4_0: 50% VRAM savings but 92% throughput loss at 64K context → DO NOT USE
• q8_0: 25% VRAM savings, <5% throughput loss → RECOMMENDED

Context Window & VRAM Budget

Formula: model_weights_GB + (64K tokens × 2 bytes × layers × heads × head_dim) / 1e9

Examples at 64K context, q8_0 KV cache:

• gpt-oss-20b: 12 GB weights + 3.5 GB KV = 15.5 GB total
• ornith-9b: 5.6 GB weights + 3.0 GB KV = 8.6 GB total

See VRAM Budget at 64K Context for the full table.

📖 Complete Documentation

• optimization-guide.md — 11-section deep dive

  • Hardware requirements & WSL2 tuning
  • Ollama systemd configuration
  • Model roster notes & comparisons
  • Modelfile authoring & pull reference
  • Hermes provider & routing config
  • MoA presets & personalities
  • Key pitfalls & troubleshooting
  • Upgrade paths & scaling advice

• Hermes Config Skeleton

  • Copy/Paste to your config.yaml what you need
    • Don't forget to make a backup ;)

🧪 Troubleshooting

Ollama Hangs

• Enable FlashAttention: OLLAMA_FLASH_ATTENTION=1

Out-of-Memory (OOM)

• Check co-loading rules (gpt-oss-20b + qwen3-vl-8b = too big)
• Verify WSL2 memory allocation if on Windows
• Review VRAM budget table; may need smaller models or less context

Modelfile Build Fails

• Ensure Ollama base is pulled first: ollama pull <base-tag>
• For Hugging Face models, allow extra timeout (metadata is large)
• Verify num_predict -1 is set (prevents silent truncation in agent loops)

Model Not Appearing in ollama list

• Check create command succeeded: ollama list | grep <model-name>
• If missing, re-run: ollama create <alias> -f <Modelfile>
• Verify FROM line in Modelfile points to correct base

🎓 Key Concepts

Task Routing: Hermes automatically selects the right model based on task type (tool use → gpt-oss-20b, fast conversation → gemma4-heretic, coding → qwen25-coder or ornith-9b).

Agentic Loops: Models like ornith-9b and gpt-oss-20b support tool-calling and multi-turn planning, enabling autonomous task completion.

Reasoning Blocks: qwythos-9b and ornith-9b emit <think>...</think> blocks; Hermes parses these automatically for better plan transparency.

MoA Synthesis: Responses from reference models are anonymized to prevent aggregator bias. Aggregator sees only the anonymous drafts + original question.

📋 Project Status

• Created: 2026-06-29
• Language: Python 100%
• License: MIT
• Maintainer: InfiniteWhispers

🔗 Related Links

• Ollama
• Hermes AI Agent - NousResearch Repo
• Hermes Home Dashboard Plugin

Ready to start?

1. Ensure GPU + 48+ GB RAM
2. Install Ollama
3. Pull & build 8 models (takes ~60 min on good internet)
4. Configure Hermes provider block

For detailed tuning, see optimization-guide.md.