Agent Testing Suite

Agent Testing Suite is a comprehensive, local-first evaluation and testing framework for AI agents. It provides complete observability into agent execution with trace capture, interactive visualization, multi-model support, and a flexible evaluation system. This is a dedicated framework for evaluating LLM-powered applications—not a generic testing suite.

Overview

Agent Testing Suite combines modular components to create a unified evaluation environment:

• Core Runtime — Orchestration, execution tracing, evaluation logic, and metrics
• CLI — Command-line interface for running evaluations and managing tests
• Dashboard — Interactive Next.js UI for inspecting traces and metrics in real time
• Proxy — Network traffic interception for capturing model API interactions
• Scraper — Data analysis and extraction utilities

Key Features

✨ End-to-End Tracing — Capture complete execution traces with full context
📊 Interactive Dashboard — Visual inspection of agent runs, metrics, and evaluation results
🔌 Multi-Model Support — Works with Groq, Claude, Gemini, OpenAI, and other providers
💻 Local-First — Run entirely locally with optional Docker support for databases
🧪 Rich Evaluators — Built-in support for assertions, semantic similarity, LLM critique, and custom metrics
🪟 Windows-Friendly — Convenient launcher script for easy dashboard access
📦 Complete Demo — Sample agent included to explore the full workflow

Requirements

• Node.js 20+
• pnpm 10+
• Docker (optional, for containerized database features)

Quick Start

Install

pnpm install

Launch Dashboard

Windows users (easiest): Double-click Start Agent Testing Suite.cmd to build and open the dashboard.

Terminal:

corepack pnpm --filter @agent-testing-suite/dashboard build
corepack pnpm --filter @agent-testing-suite/dashboard start

Dashboard runs at http://localhost:3579

Run Sample Agent

1. Create .env with your LLM API key:

   GROQ_API_KEY=gsk_...

2. Run the agent:

   node test-agent.cjs

3. No API key? Use dummy mode:

   node test-agent.cjs --dummy

4. Via Agent Testing Suite:

   node packages/cli/dist/index.js run "node test-agent.cjs"

Keep the dashboard open while running to see live traces and metrics.

Build & Test

# Build all packages
pnpm build

# Run all tests
pnpm test

# Build individual packages
corepack pnpm --filter @agent-testing-suite/core build
corepack pnpm --filter @agent-testing-suite/cli build
corepack pnpm --filter @agent-testing-suite/proxy build
corepack pnpm --filter @agent-testing-suite/dashboard build

Repository Structure

packages/
  cli/        Command-line interface for running evaluations
  core/       Core runtime, schemas, evaluators, metrics, orchestration
  dashboard/  Next.js web UI for trace visualization
  proxy/      Network traffic interception and export
  scraper/    Analysis and data extraction utilities

docs/         Documentation and guides
adapters/     Integration guides for various LLM providers

Project Philosophy

Agent Testing Suite is built around three core principles:

1. Local-First — All evaluation happens on your machine by default
2. Observable — Every decision and interaction is traceable
3. Extensible — Easy to add custom evaluators, metrics, and integrations

License

MIT