Fix partition aggregation and wire median_neighbor_distance - #128
Closed
sanghoonio wants to merge 2 commits into
Closed
Fix partition aggregation and wire median_neighbor_distance#128sanghoonio wants to merge 2 commits into
sanghoonio wants to merge 2 commits into
Conversation
5edc86f removed the `agg_columns.extend([` wrapper along with the `* 100` multipliers, leaving an orphaned list literal. aggregation.py has not parsed since, and both bedsets.py and bedfiles.py import it, so `import bbconf` fails outright — conftest.py cannot load and no tests are collected. Restore the wrapper. The multipliers stay off, but for a different reason than the autofix assumed: the bed_stats.*_percentage columns hold a fraction, not a percentage. Both producers divide by the region count — regionstat.R:212 stores Freq/length(query), gtars_backend.py:227 stores count/total — and bedbase-ui renders the per-file values as value * 100. Since the aggregation no longer rescales, `mean_pct`/`sd_pct` would have been misnamed, so they become `mean`/`sd`. That also matches the shape already used by `scalar_summaries`, leaving every value in BedSetDistributions on the same scale as the column it aggregates. bedbase-ui scales to percent at the plot layer, where the local comparison path already does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRGhWjGiUXFoygd35QsKQ7
The aggregation surfaced `median_neighbor_distance` by reading `bed_stats.tssdist`, but nothing writes that column. It has two references in the whole tree — the initial migration and the ORM declaration — and BedStatsModel has never declared the field, so with `extra="ignore"` the value bedboss computes (gtars_backend.py:203-217) is dropped before it reaches `BedStats(**stats.model_dump())`. `count()` returned 0, `if not n: continue` fired, and the key was silently absent from every scalar_summaries payload. Add a real `median_neighbor_distance` column and drop `tssdist`. The ORM attribute and the BedStatsModel field have to land together: with only the model field, `model_dump()` emits a key `BedStats.__init__` rejects and every insert raises TypeError; with only the ORM attribute, pydantic keeps dropping the value. `tssdist` is dropped rather than renamed because it is a vestige of the *TSS* quantity, superseded by `median_tss_dist` (both were created by the initial migration). Neighbor distance is a different measurement — the gap between consecutive regions within a file, versus the distance from each region to the nearest annotated TSS — so renaming would relabel any stale TSS values and average them into bedset scalar_summaries. The revision chains off c7f3a9e1b204, which is the only head on this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PRGhWjGiUXFoygd35QsKQ7
Member
Author
|
Superseded — pushed the fix directly to Dropped the 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
genom_distcurrently fails all four checks:aggregation.pyhas a syntax error, soimport bbconffails and no tests are collected. This fixes that, plus one other bug found while looking into it.Two commits, independent of each other — the first is just the CI unblock.
1. The module doesn't parse —
f31557f5edc86f(Copilot Autofix) removedagg_columns.extend(and its[from_aggregate_partitions, leaving a dangling list:bedsets.pyandbedfiles.pyboth import this module, so nothing in bbconf imports.a8de95dandfa444b7were both green;5edc86fis where it went red.Restored the wrapper.
The same commit made partition values 100× too small
It also dropped the
* 100, acting on a review comment saying the*_percentagecolumns "appear to already be stored as percentages (0–100)".They're fractions:
regionstat.R:212Freq / length(query)gtars_backend.py:227count / totalbedbase-ui selection-stats.tsx:11value * 100Easy to get wrong from inside bbconf — the column name, the old docstring, and the test fixture all suggest otherwise, and all three sources above live in other repos.
What I did: the missing
* 100is only a bug because the keys are named_pct. So rather than putting the multipliers back, I renamed the keys —mean_pct/sd_pctbecomemean/sd, andpartitionsnow returns fractions like everything else inBedSetDistributions.Consumers scale for display. bedbase-ui does this at the plot layer (
b8a2464on master), next to the axis label and matching how the local comparison path already works. Nothing was deployed against the old keys.The docstring now records where the fraction comes from, so the next reader has the evidence to hand.
2.
median_neighbor_distancenever had a value —5b35653_SCALAR_COLUMNSread it fromBedStats.tssdist. Nothing writes that column — it appears twice in the whole tree (initial migration, ORM declaration), andBedStatsModeldoesn't declare the field, soextra="ignore"drops the value bedboss computes before it can be stored.count()returned 0, and the key was silently omitted from every response.Added a real column, plus the
BedStatsModelfield and a migration. Both edits are needed together: with only the model field every insert raisesTypeError; with only the column, pydantic keeps dropping the value.Dropped
tssdistrather than renaming it. It's a leftover TSS-distance column superseded bymedian_tss_dist(both created by the initial migration). Neighbor distance is a different measurement — the gap between consecutive regions within a file, not the distance to the nearest annotated TSS — so a rename would relabel stale TSS values and average them into bedset stats.Migration chains off
c7f3a9e1b204, the only head on this branch.Worth checking before merge
SELECT count(*) FROM bed_stats WHERE tssdist IS NOT NULL;— should be 0, since no writer has ever existed. If it isn't, that's real data being discarded.mean_pct→meanis a response-shape change. bedbase-ui is the only consumer I could find, and it's updated.docs/schema.svgstill showstssdistand needs regenerating against a live DB.Verified
Lint and format pass. Single alembic head;
upgradeanddowngradeboth render correctly offline. ORM metadata matches the migration. Partition and scalar SQL compile against PostgreSQL with no reference totssdist.median_neighbor_distanceadded to both test fixtures, sotest_uploadexercises the model↔ORM coupling.Not verified: the DB-backed tests. No Postgres available where I was working, so CI is the first real run.
Left alone
Noticed but not touched — happy to open separate PRs:
nin_aggregate_region_distributionis taken from the first row onlytss_histogramdocstring says "summed"; the code computes AVG/SDget_distributions()fallback reportsn_files=0for bedsets that do have membersget_batchdocstring claims one round-trip; the twoselectinloads make it threeget(full=True)still returns the distributions blob🤖 Generated with Claude Code
https://claude.ai/code/session_01PRGhWjGiUXFoygd35QsKQ7