Skip to content

CONTRIBUTING.md

Latest commit

 

History

History
133 lines (98 loc) · 3.88 KB

CONTRIBUTING.md

File metadata and controls

133 lines (98 loc) · 3.88 KB
Raw

Contributing to HaystackBench

Thank you for your interest in contributing! This guide covers everything you need to get started.

Development Setup

# Clone the repo
git clone https://github.com/marmutapp/needlehaystack.git
cd haystackbench

# Create a virtual environment
python -m venv .venv
source .venv/bin/activate  # or .venv\Scripts\activate on Windows

# Install in editable mode with dev dependencies
pip install -e ".[all,dev]"

# Verify installation
haystackbench --version

Running Tests

# Run all tests
pytest

# Run with coverage report
pytest --cov=haystackbench --cov-report=html

# Run a specific test file
pytest tests/unit/test_needle.py -v

# Run tests without slow integration tests
pytest tests/unit/ -v

Code Quality

We use:

  • ruff for linting: ruff check haystackbench/
  • black for formatting: black haystackbench/
  • mypy for type checking: mypy haystackbench/

All three must pass before a PR is merged.

# Run all checks at once
ruff check haystackbench/ && black --check haystackbench/ && mypy haystackbench/

# Auto-fix what's possible
ruff check --fix haystackbench/ && black haystackbench/

Adding a New Provider

  1. Create haystackbench/providers/myprovider.py
  2. Subclass BaseProvider and implement all 3 abstract methods
  3. Register it in haystackbench/providers/__init__.py
  4. Add model info to the provider's list_models() return
  5. Update the cost table in haystackbench/core/runner.py
  6. Write tests in tests/unit/test_providers.py
  7. Add the provider to the README table
  8. Add the optional dependency to pyproject.toml
# haystackbench/providers/myprovider.py
from haystackbench.providers.base import BaseProvider, ModelInfo, ProviderResponse

class MyProvider(BaseProvider):
    async def complete(self, prompt, model, max_tokens, temperature, system_prompt):
        # Call your provider's API
        ...
        return ProviderResponse(content=..., prompt_tokens=..., completion_tokens=..., model=..., provider="myprovider")

    def list_models(self):
        return [ModelInfo("my-model-v1", "My Model v1", 128000, True, 1.0, 3.0)]

    def validate_credentials(self):
        try:
            self._get_client()
            return True
        except Exception:
            return False

Adding a New Task Type

  1. Create haystackbench/tasks/mytask.py
  2. Subclass BaseTask and implement all abstract methods
  3. Register in haystackbench/tasks/__init__.py
  4. Add to TaskType enum in haystackbench/config/schema.py
  5. Write tests in tests/unit/test_tasks.py

Adding a New Haystack Source

  1. Create or update haystackbench/haystacks/standard/mysource.py
  2. Implement fetch(target_tokens, **kwargs) -> str
  3. Register in HaystackSource enum in schema.py
  4. Handle caching in HaystackBuilder._fetch_source()

Commit Convention

We use Conventional Commits:

feat(tasks): implement distractor needles task
fix(injector): handle edge case at depth=100%
docs(methodology): clarify ATC scoring formula
test(evaluator): add LLM judge scoring tests
chore(deps): bump anthropic to 0.42.0
refactor(runner): extract cost estimation to helper module

Pull Request Process

  1. Fork the repo and create a feature branch: git checkout -b feat/my-feature
  2. Write your code with type hints and docstrings
  3. Add tests (coverage must stay ≥ 80%)
  4. Run ruff, black, and mypy — all must pass
  5. Open a PR with a clear description of what and why
  6. A maintainer will review within a few days

Project Governance

  • Issues are the preferred way to discuss bugs and features
  • For large changes, open an issue first to discuss the approach
  • PRs that reduce test coverage will not be merged

Code of Conduct

This project follows the Contributor Covenant Code of Conduct. Be kind, be constructive, be patient.