hermes-labs-ai/lintlang

lintlang

lintlang is a deterministic static linter for AI agent configs, tool descriptions, and system prompts. It combines 7 structural detectors (H1–H7) with 6 HERM v1.1 scoring dimensions and runs without model or network calls.

Release status: 0.3.1 is the latest public release, on PyPI and GitHub.

This release also includes a separate provider-neutral preflight surface for one present instruction plus explicit context. See Provider-neutral preflight.

Run bash evals/sample-detection-rate.sh to check the bundled regression fixtures: four deliberately broken samples must be flagged and one designated clean sample must pass. This is a repository regression check, not an accuracy estimate or validation corpus.

AI agent configs can be syntactically valid while their language remains vague, unbounded, or structurally inconsistent.

lintlang flags a bounded set of those static patterns before runtime review — without calling a model.

• "My agent picks the wrong tool because the tool descriptions all sound the same."
• "We only catch prompt and config drift after the agent starts looping."
• "I want a prompt linter or agent-config linter that runs in CI with no model calls."
• "Our YAML is valid, but the instructions inside it are still bad."

pip install lintlang

lintlang scan samples/bad_tool_descriptions.yaml

LINTLANG v0.3.1
samples/bad_tool_descriptions.yaml

FAIL — 1 CRITICAL, 2 HIGH, 6 MEDIUM, 3 LOW
H1: Tool Description Ambiguity
  [CRITICAL] tool:process_ticket
  Tool 'process_ticket' has no description.

How it differs from LLM-based config review

Model-based agent-config review depends on a model runtime and may add API cost, latency, or output variability. lintlang skips the model and applies local static rules.

Detection rules are static regex + structural heuristics. File discovery and findings are deterministically ordered.

When to use it

Use lintlang when you author or review AI agent tool descriptions, system prompts, or config files and want a static prompt/config quality gate in CI before runtime testing.

When NOT to use it

• Semantic correctness — lintlang is structural. It catches vague tool descriptions, not wrong ones. ("delete_user" with empty description fails; "delete_user" pointing at the wrong table is invisible to lintlang.)
• Open-ended creative writing — H1–H7 are calibrated for agent configs and system prompts, not prose.
• Silent auto-fix or sending — repository scans do not rewrite; preflight corrections require an explicit source-bound preview/apply action and never contact a provider.
• Behavioral safety proofs — a clean lintlang scan is not evidence that an agent is safe or correct. Use runtime evaluation and domain review for behavioral claims.
• Input boundaries — JSON, YAML, plain text, .prompt, and Markdown are parsed as language-bearing inputs. Python files use AST extraction only for embedded prompts and agent-pipeline thresholds; lintlang does not lint general Python syntax, style, types, or program correctness. Arbitrary nested templates may not parse.

Static linter for AI agent tool descriptions, system prompts, and configs.

Valid syntax does not guarantee clear instructions. Ambiguous tool descriptions, missing bounds, and schema-language mismatches can create review obligations that code linters do not express. lintlang flags its shipped static patterns at authoring time and in CI with zero LLM calls.

Install

pip install lintlang

Requires Python 3.10+. One dependency (pyyaml). No API keys, network calls, or LLM calls.

Quick Start

# Scan a single file
lintlang scan agent_config.yaml

# Scan a directory (finds .yaml, .json, .txt, .md, .prompt, and .py)
lintlang scan configs/

# JSON output for CI
lintlang scan config.yaml --format json

# Fail CI on CRITICAL/HIGH findings
lintlang scan config.yaml --fail-on fail

# Fail CI on any MEDIUM+ findings
lintlang scan config.yaml --fail-on review

Preflight one present instruction without sending it anywhere:

printf '%s' 'Is it true that remote work always reduces productivity?' \
  | lintlang preflight - --include-snippets

Example Output

  LINTLANG v0.3.1
  bad_tool_descriptions.yaml
  ──────────────────────────────────────────────────

  ❌ FAIL — 1 CRITICAL, 2 HIGH, 6 MEDIUM, 3 LOW

  H1: Tool Description Ambiguity

    !! [CRITICAL] tool:process_ticket
      Tool 'process_ticket' has no description.
      → Add a specific description explaining WHEN to use this tool.

    ! [HIGH] tool:get_user_info
      Tool 'get_user_info' has a very short description (13 chars)
      → Expand to include purpose, when to use, expected input/output.

    ~ [MEDIUM] tool:handle_request
      Tool 'handle_request' starts with vague verb 'handle'.
      → Replace with a specific action verb.

  H2: Missing Constraint Scaffolding

    ! [HIGH] system_prompt
      System prompt defines tools but has no termination conditions.
      → Add: 'Maximum 5 tool calls per task. Stop and report after 2 failures.'

  ──────────────────────────────────────────────────
  lintlang v0.3.1 | H1-H7 structural analysis | Zero LLM calls

How It Works

lintlang gives you a verdict, not a score:

Each finding includes the pattern (H1-H7, plus P1/P2 for Python pipeline checks), severity, location, and a concrete suggestion. Suggestions are review aids, not guaranteed meaning-preserving fixes.

Why These 7 Detectors?

The shipped rule set groups its structural checks into seven categories for agent configuration and prompt surfaces. Each detector maps to a condition that general syntax and schema linters do not evaluate.

Tools such as yamllint, semgrep, and ruff cover syntax or source-code patterns. lintlang focuses on a different layer: whether descriptions are ambiguous or whether an instruction surface omits explicit termination conditions. Those findings are heuristic structural warnings, not predictions of runtime behavior. Current checked-in evidence is limited to unit tests and the small regression fixture described below; detector accuracy on external projects is unmeasured.

Structural Detectors (H1-H7)

H5: Context-Aware Negatives

H5 distinguishes between safety constraints and style negatives. Security rules like "Never expose API keys" are correctly exempted. Style issues like "Don't be verbose" are flagged with positive rewrites.

The checked-in samples/ directory contains a small regression set for detector behavior. It does not establish performance on external projects or production workloads.

Why not just use GPT-4?

No model/API cost, model latency, or prompt transmission is required. lintlang runs offline and applies deterministic structural checks; it complements rather than replaces model-based or human review.

CI Integration

GitHub Actions

- name: Lint agent configs
  run: |
    pip install lintlang
    lintlang scan configs/ --fail-on fail

Verdict-Based Gating

Input integrity is a separate fatal channel: every explicitly requested input, and every supported file discovered in a requested directory, must be readable and parseable. A missing, unreadable, or malformed input exits 1 regardless of --fail-on. JSON output keeps the affected path in the result list with "verdict": "ERROR" and a non-null "input_error"; it is never labeled PASS or silently omitted because another input scanned successfully.

Filter by Severity

# Only show CRITICAL and HIGH
lintlang scan config.yaml --min-severity high

# Only check specific patterns
lintlang scan config.yaml --patterns H1 H3

Programmatic API

from lintlang import scan_file, compute_verdict

result = scan_file("config.yaml")
if result.input_error:
    raise ValueError(f"lintlang could not scan the input: {result.input_error}")
verdict = compute_verdict(result)  # ERROR, PASS, REVIEW, or FAIL
print(f"Verdict: {verdict}")  # ERROR, PASS, REVIEW, or FAIL

for finding in result.structural_findings:
    print(f"  [{finding.severity.value}] {finding.description}")
    print(f"  → {finding.suggestion}")

# Scan a directory
from lintlang import scan_directory, compute_verdict

results = scan_directory("configs/")
for path, result in results.items():
    if result.input_error:
        print(f"{path}: ERROR — {result.input_error}")
        continue
    verdict = compute_verdict(result)
    print(f"{path}: {verdict}")

Supported Formats

lintlang auto-detects file format:

• YAML (.yaml, .yml) — OpenAI function-calling format, tool definitions
• JSON (.json) — OpenAI and Anthropic tool schemas, message arrays
• Plain text (.txt, .md, .prompt) — System prompts, instruction docs
• Python source (.py) — AST extraction of embedded prompts and language-bearing agent-pipeline thresholds. This is not Python code linting: lintlang does not judge style, types, control flow, or general program correctness. A Python parse error is reported only because the requested language-bearing input could not be extracted safely.

Unknown extensions are tried as JSON → YAML → plain text.

How Is lintlang Different?

lintlang treats tool descriptions, system prompts, and agent configs as lintable artifacts — static analysis for prose, like eslint for JavaScript.

Development

git clone https://github.com/hermes-labs-ai/lintlang.git
cd lintlang
pip install -e ".[dev]"
pytest

License

Apache 2.0

About Hermes Labs

Hermes Labs is an independent AI-reliability lab building open-source tools that catch silent failure modes in production AI. More at hermes-labs.ai.