Skip to content

Add structured simulation observability: per-step event logging, component-level telemetry, and replay support #275

Description

@aditya-pandey-dev

Problem

Concordia currently has no standardized observability layer for simulation runs.
Researchers and engineers face three concrete problems:

1. No reproducible audit trail
When a simulation completes, there is no structured record of which component
produced which context, what the LLM was prompted with at each step, and what
decision was made. Debugging unexpected agent behavior requires manually
instrumenting every component with print statements.

2. No component-level telemetry
ProfiledLanguageModel exists but operates in isolation — there is no way to
attribute token costs or latency to a specific component (e.g., "AssociativeMemory
consumed 3x more tokens than expected on step 12"). For research and production
usage, this makes cost management and bottleneck identification impossible.

3. No simulation replay / partial recovery
If a long multi-agent simulation fails mid-run (LLM timeout, ThreadPoolExecutor
crash, OOM), all intermediate state is lost. There is no checkpoint/resume
mechanism, and no way to "replay" a simulation from a saved event log.

Impact

These gaps directly affect Concordia's stated use cases:

  • AI safety research: Requires full auditability of agent reasoning chains
  • Social science: Requires reproducible simulation records for publication
  • Economic modeling: Requires per-step state snapshots for analysis
  • Synthetic data generation: Requires structured logs to validate data quality

Proposed Solution

Add a lightweight SimulationLogger abstraction with the following interface:

class SimulationEvent:
    step: int
    agent_name: str
    phase: str  
    component_name: str
    prompt: str | None
    output: str
    token_usage: dict | None
    timestamp: float

class SimulationLogger(abc.ABC):
    def log_event(self, event: SimulationEvent) -> None: ...
    def flush(self) -> None: ...

Built-in implementations:

  • JsonLinesLogger — writes newline-delimited JSON to a file (replay-friendly)
  • InMemoryLogger — for testing and notebooks
  • NullLogger — zero overhead default (backward compatible)

The engine's _parallel_call_ and EntityAgent lifecycle hooks would emit
events to an injected logger. This is purely additive — existing code needs
zero changes.

Why This Matters

This is a foundational observability primitive that every serious Concordia
user needs but currently has to build themselves. A standardized interface
means community tooling (dashboards, replay tools, cost analyzers) can be
built on a stable API.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions