Implement internal prediction-distribution data producer

[uriahf]

Deliverable Summary

• Starting SHA: 93fe1e8b4380e18ccd09849af429bb8baf3ddadd (v0.1.36)
• Head SHA: 1ce587424fb8e9c647565406085a85ae763ef4cb
• Files Created:
  • src/rtichoke/performance_data/probs_distribution.py
  • tests/test_probs_distribution.py
• Helper Name and Location: _prepare_probs_distribution_data in src/rtichoke/performance_data/probs_distribution.py
• Input Normalization Reused:
  • _validate_and_align_binary_inputs from rtichoke.performance_data.performance_data
  • _build_evaluation_metadata from rtichoke.processing.evaluation_semantics
• Returned Polars Schemas:
  • bins: evaluation (pl.String), model (pl.String, nullable), population (pl.String), lower (pl.Float64), upper (pl.Float64), include_lower (pl.Boolean), include_upper (pl.Boolean), n_positive (pl.UInt32), n_negative (pl.UInt32)
  • operating_points: evaluation (pl.String), model (pl.String, nullable), population (pl.String), type (pl.String), value (pl.Float64), cutoff (pl.Float64), realized_ppcr (pl.Float64)
• Static Semantics & Intervals:
  • Cutoff zero predicts everyone positive (TP=total_pos, FP=total_neg, TN=0, FN=0).
  • Strict nonzero cutoffs put probabilities <= cutoff on predicted-negative side.
  • Interval [0.0, 0.0] is always present for zero-score observations, followed by right-closed intervals (lower, upper].
• PPCR Tie Handling:
  • Requested value, effective cutoff, and realized_ppcr are preserved distinctly.
• Golden Fixtures & Reconstruction Invariants:
  • Both threshold and PPCR tie golden fixtures verified.
  • Comprehensive reconstruction invariant test confirms confusion matrices derived from bins match prepare_performance_data() output across all operating points and scenarios.
• Commands Run:
  • uv run pytest (349 passed, 9 skipped)
  • uv run ruff check .
  • uv run ruff format --check .
  • uv run ty check src/rtichoke/performance_data/probs_distribution.py tests/test_probs_distribution.py

---

PR created automatically by Jules for task 7515411089152951279 started by @uriahf

[google-labs-jules]

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.

---

For security, I will only act on instructions from the user who triggered this task.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-09-08 11:56 UTC

[uriahf]

This is close. The overall statistical contract, returned schemas, golden
fixtures, and PR scope are correct, and CI is green.

Please address the following focused issues before merge.

1. Preserve single keyed-population metadata semantics

This is a correctness issue.

The implementation currently does:

aligned_reals = _validate_and_align_binary_inputs(
    probs=probs,
    reals=reals,
)

evaluation_metadata_map = _build_evaluation_metadata(
    probs,
    aligned_reals,
    dummy_times,
)

However, _validate_and_align_binary_inputs() converts a one-element reals
dictionary into a NumPy array:

if len(groups) == 1:
    return np.asarray(reals[groups[0]])

That destroys the original input-shape information used by
_build_evaluation_metadata().

Consequently, input such as:

probs = {
    "validation_population": np.array([...])
}

reals = {
    "validation_population": np.array([...])
}

is incorrectly interpreted as a model evaluated in the shared-population
context. It should retain keyed-population semantics:

evaluation = "validation_population"
model = None
population = "validation_population"

Build semantic metadata from the original reals input, not from
aligned_reals:

evaluation_metadata_map = _build_evaluation_metadata(
    probs,
    reals,
    np.array([]),
)

Continue using aligned_reals for the validated computational path.

Add an explicit regression test for exactly one keyed population confirming
that:

• evaluation equals the dictionary key;
• population equals the dictionary key;
• model is null;
• the model dtype remains pl.String.

Also retain the existing multiple-keyed-population test.

2. Remove the O(n × number_of_intervals) counting step

The implementation assigns interval indices once with np.digitize(), which is
good, but then repeatedly scans the complete interval-index array:

for i in range(1, len(unique_bounds)):
    in_bin = b_indices == i
    pos_count = ...
    neg_count = ...

This is still approximately O(n × number of intervals), which the task
explicitly asked us to avoid.

After calculating each observation’s interval index once, aggregate counts in
one operation.

Use either:

• a Polars DataFrame followed by group_by() and a left join to the complete
  interval grid; or
• a clearly justified vectorized np.bincount() implementation for positive
  and negative counts.

Polars is preferred because this package uses Polars and because a complete
interval-grid left join makes preservation of empty intervals explicit.

The intended pattern is approximately:

observation_intervals = pl.DataFrame(
    {
        "interval_id": observation_interval_ids,
        "outcome": outcomes,
    }
)

interval_counts = (
    observation_intervals
    .group_by("interval_id")
    .agg(
        ...
    )
)

bins = (
    complete_interval_grid
    .join(interval_counts, on="interval_id", how="left")
    .with_columns(
        pl.col("n_positive").fill_null(0),
        pl.col("n_negative").fill_null(0),
    )
)

Preserve:

• [0, 0] as its own interval;
• subsequent (lower, upper] intervals;
• exact integer counts;
• empty intervals;
• deterministic ascending order;
• every current threshold and PPCR result.

A loop over evaluations is acceptable. A loop over operating-point rows for
constructing their small metadata table is also acceptable. Do not repeatedly
scan observations for each interval.

3. Remove unused interval construction

The variable:

intervals

is constructed but never used.

Remove it, or use one single complete interval-grid object as the authoritative
source for the returned bins. Do not maintain two parallel interval
representations.

4. Use descriptive production names

Please finish the naming cleanup in the production module.

Rename, where applicable:

strat_type
-> stratification_type

eval_metadata_map
-> evaluation_metadata_by_group

evaluations
-> evaluation_keys

perf_df
-> performance_data

eval_key
-> evaluation_key

meta
-> evaluation_metadata

eval_perf
-> evaluation_performance_data

p_vec
-> probabilities

r_vec
-> outcomes

requested_val
-> requested_value

n_obs
-> n_observations

pred_pos
-> predicted_positives

unique_bounds
-> interval_boundaries

is_zero
-> zero_score_mask

bins_rows
-> bin_rows

op_rows
-> operating_point_rows

bins_df
-> bins

op_df
-> operating_points

Short mathematical names inside tiny tests are acceptable. This request applies
primarily to the production helper.

5. Clarify duplicate-evaluation validation

This code:

evaluation_keys = list(evaluation_metadata_by_group.keys())

if len(evaluation_keys) != len(set(evaluation_keys)):
    ...

cannot currently detect anything because Python dictionary keys are unique by
construction.

The meaningful invariant is uniqueness of:

metadata.evaluation

across the metadata values.

Validate that directly:

evaluation_ids = [
    metadata.evaluation
    for metadata in evaluation_metadata_by_group.values()
]

and reject duplicates there.

If the current public dictionary input makes duplicates practically
unconstructable, that is fine; retain the semantic validation because
evaluation is the downstream joining key. Do not invent an unnatural public
input merely to manufacture a test.

6. Strengthen the performance regression test

Add a test with:

• a reasonably large observation vector;
• the default approximately 101 operating points;
• exact reconstruction/totals validation.

The test need not assert a fragile wall-clock duration. Its purpose is to
exercise the vectorized aggregation path and make accidental restoration of
per-interval observation scanning less likely.

If practical, isolate interval aggregation in a small private helper whose code
and tests make the one-assignment/one-aggregation structure explicit.

7. Preserve everything already correct

Do not change:

• the public API;
• current prepare_performance_data() behavior;
• _probs_with_r_binary_cutoff_semantics();
• cutoff-zero behavior;
• strict nonzero cutoff semantics;
• PPCR requested/effective/realized identities;
• returned column names;
• returned dictionary keys;
• nullable model semantics;
• static-only scope;
• package exports;
• rendering;
• canonical JSON;
• summary reports;
• time-dependent behavior.

After these changes, run and report:

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run ty check src/rtichoke/performance_data/probs_distribution.py tests/test_probs_distribution.py

Please report the updated head SHA and confirm that the single keyed-population
regression now preserves model=None.

[google-labs-jules]

This is close. The overall statistical contract, returned schemas, golden
fixtures, and PR scope are correct, and CI is green.

Please address the following focused issues before merge.

1. Preserve single keyed-population metadata semantics

This is a correctness issue.

The implementation currently does:

aligned_reals = _validate_and_align_binary_inputs(
    probs=probs,
    reals=reals,
)

evaluation_metadata_map = _build_evaluation_metadata(
    probs,
    aligned_reals,
    dummy_times,
)

However, _validate_and_align_binary_inputs() converts a one-element reals
dictionary into a NumPy array:

if len(groups) == 1:
    return np.asarray(reals[groups[0]])

That destroys the original input-shape information used by
_build_evaluation_metadata().

Consequently, input such as:

probs = {
    "validation_population": np.array([...])
}

reals = {
    "validation_population": np.array([...])
}

is incorrectly interpreted as a model evaluated in the shared-population
context. It should retain keyed-population semantics:

evaluation = "validation_population"
model = None
population = "validation_population"

Build semantic metadata from the original reals input, not from
aligned_reals:

evaluation_metadata_map = _build_evaluation_metadata(
    probs,
    reals,
    np.array([]),
)

Continue using aligned_reals for the validated computational path.

Add an explicit regression test for exactly one keyed population confirming
that:

• evaluation equals the dictionary key;
• population equals the dictionary key;
• model is null;
• the model dtype remains pl.String.

Also retain the existing multiple-keyed-population test.

2. Remove the O(n × number_of_intervals) counting step

The implementation assigns interval indices once with np.digitize(), which is
good, but then repeatedly scans the complete interval-index array:

for i in range(1, len(unique_bounds)):
    in_bin = b_indices == i
    pos_count = ...
    neg_count = ...

This is still approximately O(n × number of intervals), which the task
explicitly asked us to avoid.

After calculating each observation’s interval index once, aggregate counts in
one operation.

Use either:

• a Polars DataFrame followed by group_by() and a left join to the complete
  interval grid; or
• a clearly justified vectorized np.bincount() implementation for positive
  and negative counts.

Polars is preferred because this package uses Polars and because a complete
interval-grid left join makes preservation of empty intervals explicit.

The intended pattern is approximately:

observation_intervals = pl.DataFrame(
    {
        "interval_id": observation_interval_ids,
        "outcome": outcomes,
    }
)

interval_counts = (
    observation_intervals
    .group_by("interval_id")
    .agg(
        ...
    )
)

bins = (
    complete_interval_grid
    .join(interval_counts, on="interval_id", how="left")
    .with_columns(
        pl.col("n_positive").fill_null(0),
        pl.col("n_negative").fill_null(0),
    )
)

Preserve:

• [0, 0] as its own interval;
• subsequent (lower, upper] intervals;
• exact integer counts;
• empty intervals;
• deterministic ascending order;
• every current threshold and PPCR result.

A loop over evaluations is acceptable. A loop over operating-point rows for
constructing their small metadata table is also acceptable. Do not repeatedly
scan observations for each interval.

3. Remove unused interval construction

The variable:

intervals

is constructed but never used.

Remove it, or use one single complete interval-grid object as the authoritative
source for the returned bins. Do not maintain two parallel interval
representations.

4. Use descriptive production names

Please finish the naming cleanup in the production module.

Rename, where applicable:

strat_type
-> stratification_type

eval_metadata_map
-> evaluation_metadata_by_group

evaluations
-> evaluation_keys

perf_df
-> performance_data

eval_key
-> evaluation_key

meta
-> evaluation_metadata

eval_perf
-> evaluation_performance_data

p_vec
-> probabilities

r_vec
-> outcomes

requested_val
-> requested_value

n_obs
-> n_observations

pred_pos
-> predicted_positives

unique_bounds
-> interval_boundaries

is_zero
-> zero_score_mask

bins_rows
-> bin_rows

op_rows
-> operating_point_rows

bins_df
-> bins

op_df
-> operating_points

Short mathematical names inside tiny tests are acceptable. This request applies
primarily to the production helper.

5. Clarify duplicate-evaluation validation

This code:

evaluation_keys = list(evaluation_metadata_by_group.keys())

if len(evaluation_keys) != len(set(evaluation_keys)):
    ...

cannot currently detect anything because Python dictionary keys are unique by
construction.

The meaningful invariant is uniqueness of:

metadata.evaluation

across the metadata values.

Validate that directly:

evaluation_ids = [
    metadata.evaluation
    for metadata in evaluation_metadata_by_group.values()
]

and reject duplicates there.

If the current public dictionary input makes duplicates practically
unconstructable, that is fine; retain the semantic validation because
evaluation is the downstream joining key. Do not invent an unnatural public
input merely to manufacture a test.

6. Strengthen the performance regression test

Add a test with:

• a reasonably large observation vector;
• the default approximately 101 operating points;
• exact reconstruction/totals validation.

The test need not assert a fragile wall-clock duration. Its purpose is to
exercise the vectorized aggregation path and make accidental restoration of
per-interval observation scanning less likely.

If practical, isolate interval aggregation in a small private helper whose code
and tests make the one-assignment/one-aggregation structure explicit.

7. Preserve everything already correct

Do not change:

• the public API;
• current prepare_performance_data() behavior;
• _probs_with_r_binary_cutoff_semantics();
• cutoff-zero behavior;
• strict nonzero cutoff semantics;
• PPCR requested/effective/realized identities;
• returned column names;
• returned dictionary keys;
• nullable model semantics;
• static-only scope;
• package exports;
• rendering;
• canonical JSON;
• summary reports;
• time-dependent behavior.

After these changes, run and report:

uv run pytest
uv run ruff check .
uv run ruff format --check .
uv run ty check src/rtichoke/performance_data/probs_distribution.py tests/test_probs_distribution.py

Please report the updated head SHA and confirm that the single keyed-population
regression now preserves model=None.

Thank you for the clear, targeted feedback! I have implemented all requested changes:

1. Single keyed-population metadata semantics are now preserved by passing the original reals input to _build_evaluation_metadata instead of aligned_reals (which converts single-key dicts to numpy arrays).
2. Replaced the O(n * number_of_intervals) scanning loop with _aggregate_bins_for_evaluation(), using Polars obs_df.group_by("interval_id") and a left join against a complete interval grid DataFrame (complete_grid).
3. Removed the unused intervals variable.
4. Cleaned up production variable names across probs_distribution.py (stratification_type, evaluation_metadata_by_group, evaluation_keys, performance_data, evaluation_key, evaluation_metadata, evaluation_performance_data, probabilities, outcomes, requested_value, n_observations, predicted_positives, interval_boundaries, zero_score_mask, operating_point_rows, bins, operating_points).
5. Updated duplicate validation to check len(evaluation_ids) != len(set(evaluation_ids)) directly across metadata.evaluation values.
6. Added a performance regression test with 10,000 observations and default by=0.01 (101 operating points) verifying exact totals and reconstruction invariant.
7. Added a dedicated regression test for single keyed population confirming evaluation, population, model=None, and pl.String dtype.

[uriahf]

Thank you — I re-reviewed the updated implementation at head
1b564be66eb658008354b0d552222836ff88902e.

The substantive issues from the previous review are resolved:

• evaluation metadata is now built from the original reals, preserving the
  single keyed-population case with model = None;
• observations are assigned to intervals once using NumPy/Polars rather than
  rescanned for every interval;
• the complete interval grid is joined to the aggregated counts, preserving
  empty canonical intervals;
• duplicate evaluation validation uses the canonical evaluation identity;
• prepare_performance_data() remains the authoritative source of operating
  points;
• tests now cover the single keyed-population case, duplicate identities,
  reconstruction invariants, threshold semantics, PPCR ties, and a 10,000-row
  performance regression.

The remaining small loops operate over interval definitions or evaluation
metadata, not over observations. They are sufficiently clear and bounded, so I
do not think artificial vectorization is required.

One minor documentation issue: the PR description still reports the original
head SHA rather than the current head SHA. Please update it if convenient, but
this is not an implementation blocker.

Approved from the statistical-contract and cross-language architecture
perspective.

[google-labs-jules]

Thank you — I re-reviewed the updated implementation at head
1b564be66eb658008354b0d552222836ff88902e.

The substantive issues from the previous review are resolved:

• evaluation metadata is now built from the original reals, preserving the
  single keyed-population case with model = None;
• observations are assigned to intervals once using NumPy/Polars rather than
  rescanned for every interval;
• the complete interval grid is joined to the aggregated counts, preserving
  empty canonical intervals;
• duplicate evaluation validation uses the canonical evaluation identity;
• prepare_performance_data() remains the authoritative source of operating
  points;
• tests now cover the single keyed-population case, duplicate identities,
  reconstruction invariants, threshold semantics, PPCR ties, and a 10,000-row
  performance regression.

The remaining small loops operate over interval definitions or evaluation
metadata, not over observations. They are sufficiently clear and bounded, so I
do not think artificial vectorization is required.

One minor documentation issue: the PR description still reports the original
head SHA rather than the current head SHA. Please update it if convenient, but
this is not an implementation blocker.

Approved from the statistical-contract and cross-language architecture
perspective.

Thank you for the thorough review and approval! I have updated the submission description to reflect the final head SHA.