Skip to content

ROADMAP.md

Latest commit

 

History

History
229 lines (173 loc) · 6.84 KB

ROADMAP.md

File metadata and controls

229 lines (173 loc) · 6.84 KB
Raw

HaystackBench — Project Roadmap

Last Updated: 2025-01-01

v0.1.0 — MVP (Current)

Theme: Get a single task type working end-to-end with the two most popular providers.

Target: Researchers can reproduce existing NIAH results and run S-RT evaluations on Anthropic and OpenAI models.

Features

Core Engine

  • Synthetic needle generation (abstract template-based)
  • Paul Graham essay haystack source
  • Depth-based needle injection with paragraph boundary enforcement
  • Exact and substring match scoring
  • Async test runner with concurrency control
  • SQLite persistence with checkpoint/resume
  • Response caching (SHA256 key)

Task Types

  • S-RT: Single-Needle Retrieval with 2D heatmap

Providers

  • Anthropic (claude-3-5-haiku, claude-3-5-sonnet, claude-3-opus)
  • OpenAI (gpt-4o, gpt-4o-mini, gpt-4-turbo)

Config

  • Pydantic config schema
  • YAML experiment config files
  • quick, standard, full presets

Storage

  • SQLite schema with all tables
  • OS keyring credential storage

Visualization

  • 2D heatmap (matplotlib)
  • HTML report (self-contained)
  • JSON + CSV export

CLI

  • haystackbench setup — credential wizard
  • haystackbench run s-rt — run single-needle retrieval
  • haystackbench results show/plot/export
  • haystackbench serve — start web UI

Web UI

  • Dashboard page
  • Model setup page with credential management
  • New experiment wizard (S-RT only)
  • Results page with interactive heatmap
  • Results library with export

Documentation

  • README with quickstart
  • METHODOLOGY.md
  • CONTRIBUTING.md
  • QUICKSTART.md

v0.2.0 — Full Task Suite

Theme: Implement all planned task types and complete the provider roster.

Target: Comprehensive NIAH evaluation equivalent to or surpassing NeedleBench.

Features

Task Types

  • M-RT: Multi-Needle Retrieval with per-needle scoring
  • M-RS: Multi-Needle Reasoning (numeric synthesis)
  • ATC: Ancestral Trace Challenge (chain, tree, DAG)
  • Counting Task
  • Key-Value Retrieval
  • Conflicting Needles (primacy/recency bias)
  • Distractor Needles

Providers

  • Google (gemini-1.5-pro, gemini-1.5-flash, gemini-2.0-flash)
  • Mistral (mistral-large, mistral-medium)
  • Cohere (command-r, command-r-plus)
  • Groq (llama-3.1-70b, llama-3.1-8b, mixtral-8x7b)
  • Ollama (local models, auto-discover)

Scoring

  • Semantic similarity scoring (sentence-transformers)
  • LLM Judge scoring
  • Numeric match scoring with tolerance

Needles

  • Template-based needles (Jinja2 templates)
  • Relational needle sets for ATC
  • Key-value needle format
  • Conflicting needle pairs

Haystacks

  • Wikipedia article fetcher (configurable topic/language)
  • arXiv abstract fetcher

Visualization

  • Multi-model side-by-side heatmaps
  • Difference heatmap (model A - model B)
  • Radar chart (multi-task performance)
  • Line chart (accuracy vs. context length)
  • Depth profile chart (accuracy vs. depth)

Presets

  • needlebench_32k — NeedleBench 32K replication
  • needlebench_128k — NeedleBench 128K replication

Web UI

  • Full wizard support for all task types
  • Multi-model comparison view
  • Per-cell drill-down modal (prompt/response/score)
  • Dark mode

CI/CD

  • GitHub Actions CI pipeline
  • PyPI release automation

v0.3.0 — Advanced Features

Theme: Power user features, extensibility, and production-grade tooling.

Features

Haystack Sources

  • PDF/DOCX file upload (user-provided haystacks)
  • LLM-generated domain haystacks (legal, medical, code)
  • Synthetic paragraph generator (local, no API)

Needle System

  • Needle rotation (prevent memorization across benchmark versions)
  • Multi-language needles
  • Domain-specific needle vocabularies

Advanced Scoring

  • Multi-judge consensus scoring (3 models vote)
  • Per-needle latency tracking
  • Token efficiency score (did the model use minimal output?)

Runner Improvements

  • Priority queue (run cheap cells first, estimate total time)
  • Adaptive concurrency (auto-adjust based on rate limit responses)
  • Dry-run mode (show all cells without executing)

Visualization

  • Animated heatmap (show evolution across runs)
  • Colorblind-friendly palettes (viridis, magma)
  • PNG/SVG chart export from web UI
  • Shareable result URLs (base64-encoded state in URL)

CLI

  • haystackbench compare — generate comparison report for 2+ experiments
  • haystackbench watch — live-reload results as experiment runs
  • Shell completions (bash, zsh, fish)

Web UI

  • Experiment scheduling (run at specific time)
  • Email/Slack notification on completion
  • Multi-user support (shared results database)

Integration

  • Weights & Biases logging integration
  • MLflow experiment tracking integration
  • REST API authentication (API key for multi-user)

v1.0.0 — Stable Release

Theme: Production stability, performance, and ecosystem maturity.

Features

Stability

  • Stable Python API with semantic versioning guarantee
  • Database migration system (for upgrading from older versions)
  • Configuration schema versioning

Performance

  • Streaming responses (reduce time-to-first-token for long contexts)
  • Parallel haystack generation
  • Result compression for large experiments

Documentation

  • Full API reference (auto-generated from docstrings)
  • Video tutorials
  • Example gallery with real model comparisons
  • "Cookbook" for common research use cases

Ecosystem

  • Docker image (docker pull haystackbench/haystackbench)
  • Homebrew formula
  • GitHub Actions action (uses: haystackbench/run@v1)
  • Plugin system for custom task types

Community

  • Public leaderboard submission format
  • Standardized result file format (.hsb files) for sharing
  • Integration with Hugging Face Hub for model testing

Future Considerations (Post v1.0)

  • Multi-modal NIAH: Test image/document retrieval in vision-language models
  • Streaming NIAH: Test retrieval when model generates output token-by-token
  • Adversarial haystacks: Test robustness to injection attacks via the haystack
  • Non-English evaluation: Full support for Chinese, Japanese, and other languages
  • Custom judge training: Fine-tune small local judge models on NIAH-specific scoring

Version History

Version Date Highlights
v0.1.0 2025 Q1 Initial release — S-RT, Anthropic+OpenAI, CLI+UI
v0.2.0 2025 Q2 Full task suite, all 7 providers, comparison charts
v0.3.0 2025 Q3 Advanced features, integrations, power user tools
v1.0.0 2025 Q4 Stable release, Docker, plugin system