Skip to main content

What's new in Agenthood: evals, observability, and release hardening

Date: August 14, 2026
Author: Agenthood Team

This digest covers v3.15.0 through v3.27.0, a concentrated series of releases that brought evaluation from schema definition to regression gating, made traces queryable and navigable, and tightened the CLI and CI gates.

Evaluation and quality

Eval suites, scoring, and baselines

  • v3.19.0 — eval suite format validation: Eval suites now have a defined format validated with Ajv, so invalid suites are rejected up front. Closes #293.
  • v3.24.0 — llm-as-judge runner: A new eval runner scores agent behavior with LLM-as-judge scoring, adding qualitative checks to the evaluation pipeline. Closes #310.
  • v3.25.0 — baseline comparison: Eval reports can now be compared against a stored baseline, making it possible to see whether a change improved or regressed behavior. Closes #311.

Regression gating and drift detection

  • v3.26.0 — eval CLI command: agenthood eval brings the pipeline to the terminal, with baseline regression gating that can block a release when results fall below the stored baseline. Closes #298.
  • v3.27.0 — replay evaluator: A replay evaluator detects behavior drift by replaying past runs against the current agent build, connecting observed behavior to the evaluation pipeline. Closes #314.

Observability and tracing

Trace capture and persistence

  • v3.17.0 — trace envelope and ring-buffer tracer: The observability layer now emits a trace envelope and keeps a lightweight in-memory ring buffer for recent activity. Also bumped nanoid to 3.3.18 to clear the auditor gate. Closes #292 and #295.
  • v3.20.0 — queryable trace store: Traces are persisted to a queryable store instead of living only in process memory, so they can be inspected after the fact. Closes #299.

Correlation and navigation

  • v3.21.0 — workflow and session correlation IDs: Traces carry workflow and session correlation IDs, making it easier to follow a request across components and retries. Closes #301.
  • v3.22.0 — trace CLI command: npx agenthood trace brings trace inspection to the terminal. Closes #302.

Trace pipeline fixes

  • v3.23.0 — clarified trace pipeline behavior: Review findings on the trace pipeline were addressed, cleaning up edge cases before the feature is relied on in production.

Cost estimation and metrics

Token and cost accounting

  • v3.18.0 — TokenCounter and CostEstimator: Core utilities for counting tokens and estimating cost were implemented, giving agents a way to reason about usage. Closes #296 and #297.
  • v3.23.0 — OpenCode Go pricing: The cost estimator now includes pricing for OpenCode Go models, keeping estimates accurate for that provider.

Per-member summaries

  • v3.23.0 — cost and quality summaries per member: Metrics now include per-member cost and quality summaries, giving a per-agent view of spend and quality. Closes #300.

CLI, init, and release engineering

CLI and init hardening

  • v3.15.0 — sanitized CLI operations: Multiple fixes tightened argument parsing, lockfile validation, refname checks, and Git command invocation. --provider values are allowlisted, refname validation mirrors git check-ref-format, and pr-sync Git and GitHub calls now use execFileSync.
  • v3.15.0 — init dry-run and safe eject: agenthood init gained a dry-run flag, and ejecting a skills directory now removes only member subdirectories, never unrelated skills.
  • v3.15.0 — verify lane-overlap checks: agenthood verify --strict now performs real lane-overlap checks, not just placeholder logic.

LLM provider robustness

  • v3.15.0 — clearer provider errors: Ollama connection failures produce friendly guidance, Groq fails fast when a key is missing, and OpenCode Go requests strip sampling extras that caused upstream 400 responses.

CI and workflow gates

  • v3.16.0 — PR descriptions must link an issue: A doorman gate now requires pull request descriptions to link to an issue; the gate itself was hardened, and the audit check was extracted from other workflow steps.
  • v3.16.0 — workflow cleanup: CI workflow boilerplate was deduplicated, and the setup-env composite now runs after checkout to ensure a reliable environment.

What's Next

The combination of trace persistence, replay evaluation, and baseline gating points to a tighter feedback loop: recorded behavior can be replayed for drift, and eval results can block regressions at release time. We expect these pieces to continue converging in upcoming releases.