ai-benchmarking

Benchmarking the gap between AI agent hype and architecture. Three agent archetypes, 73-point performance spread, stress testing, network resilience, and ensemble coordination analysis with statistical validation.

FinMTM: A Multi-Turn Multimodal Benchmark for Financial Reasoning and Agent Evaluation

Eval framework. Define correct, test against it, get results.

A terminal widget that allows you to keep track of Codex usage against your current quota, command sessions and more ...

LLM Benchmark

HiddenBench: Benchmark for evaluating collective reasoning in multi-agent LLM systems

Grades agent completion outputs across weighted multi-criteria rubrics, computes inter-rater agreement variance, and flags inconsistent outlier score distributions | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550

Grades agent completion outputs across weighted multi-criteria rubrics, computes inter-rater agreement variance, and flags inconsistent outlier score distributions | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550

AI model hub, benchmark arena, and multi-provider playground built with Next.js, live metadata providers, and a fast REST cache layer.

AI agent evaluation framework for multi-participant coordination tasks. Built with LangGraph, custom MCP tools, and LLM-as-a-Judge evaluation. MSc dissertation project (University of Edinburgh, 2025).

ORBIT - Operational Research and Benchmarking Interface for Transformers

Open-source, local-first evaluation infrastructure for applied AI systems, built for developer and agent workflows.

🛡️ Artemis: A high-fidelity OpenEnv benchmark for evaluating AI agents in realistic SOC triage and cybersecurity incident response scenarios.

An evidence-hostile, container-isolated benchmark and behavior analysis platform for long-horizon AI software agents.

An open public-interest project exploring compassionate, responsible AI stewardship, ethical AI literacy, human agency, and long-term continuity.

NVIQ by Noct-Tech is an open-source AI benchmarking and reliability evaluation framework. Built to test speech cognition, conversational accuracy, uncertainty, context preservation, and decision-making, with reproducible benchmarks, provider integrations, and detailed reliability reports.

Uncertainty & Confidence Management (UCM): A healthcare AI benchmark suite for uncertainty recognition, justification boundaries, confidence calibration, proportionate action, and reassessment.

High-performance AI model evaluation & benchmarking platform. Features side-by-side LLM output comparison, multi-criteria scoring (JSONB), and task workflows. Built with NestJS, Prisma, PostgreSQL, React 18, and Terraform (AWS).

A comprehensive benchmarking platform for CPT, ICD-10, and HCPCS coding questions. Identifies the most reliable models for healthcare applications. Evaluates multiple AI models on medical coding expertise through iterative consensus-building.

Same prompt, several models -- stopwatch or true race; the starting model judges (time is only one dimension)