Skip to content

Fix calibration dict/dict multi-population contract #290

Description

@uriahf

Problem

create_calibration_curve() currently has inconsistent internal expectations for probs/reals dictionaries.

For matching population keys such as:

probs = {
    "Train": train_probs,
    "Test": test_probs,
}
reals = {
    "Train": train_y,
    "Test": test_y,
}

_make_deciles_dat_binary() computes the sum of all population sizes and requires each probability vector to have that total length. This rejects the natural keyed-population representation, including the example currently shown in Getting Started.

At the same time, _calculate_smooth_curve() and the time-dependent dataframe builder already treat matching keys as population-by-population pairs. The two calibration paths therefore use incompatible contracts.

Desired contract

When probs and reals are dictionaries with matching keys, pair each population independently:

probs["Train"] <-> reals["Train"]
probs["Test"]  <-> reals["Test"]

Each pair must have matching lengths, but different populations may have different sample sizes.

Acceptance criteria

  • create_calibration_curve() works for equal-size matching-key populations.
  • create_calibration_curve() works for unequal-size matching-key populations.
  • mismatched keys raise a clear ValueError.
  • within-population probability/outcome length mismatches raise a clear ValueError naming the population.
  • both calibration_type="discrete" and calibration_type="smooth" work under the same public input contract.
  • create_calibration_curve_times() continues to work with the same matching-key population contract after filtering by horizon.
  • regression tests cover equal and unequal population sizes.

Scope

This should be fixed before treating unequal-population calibration as a documented limitation. It is a correctness/API-consistency issue, not just a docs gap.

Related docs PR: #289

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions