Conversation
`_col_name` rendered a non-column aggregate argument to text and kept only what followed the last dot, so `SUM(orders.gross - orders.tax)` became `expr: tax` and `SUM(COALESCE(orders.tax, 0))` became `tax, 0)`. Strip the dataset qualifier from each column reference on the parsed tree instead, keeping the surrounding expression intact. Fixes apache#385
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OssieToMSIConverterturned any aggregate argument that is not a single column into MetricFlow'sexprby rendering it to text and slicing at the last.:SUM(orders.gross - orders.tax)becameexpr: tax(andSUM(orders.tax)after a round trip),SUM(COALESCE(orders.tax, 0))became the unparseabletax, 0), and even the unqualifiedSUM(amount * 0.5)became5. NoConverterIssuewas emitted._col_namenow strips the qualifier from each column reference on the sqlglot tree_extract_agg_infoalready holds (node.transform(_unqualify_column)) and renders the result, soSUM(orders.gross - orders.tax)yieldsexpr: gross - tax, which is what MetricFlow expects inside the semantic model's subquery. The plain-column path, the_extract_agg_inforeturn value, and_strip_qualifier(still used by_find_dataset_for_colfor field lookups) are unchanged, so output only changes for arguments that were previously corrupted. Rendering already went throughnode.sql(), so no new normalisation is introduced. No new dependency.Tests: one parametrized forward test covering arithmetic (
SUM,MAX), function calls, casts, a decimal literal,COUNT(DISTINCT a || b)andPERCENTILE_CONTover a compound argument, plus one Ossie → MSI → Ossie round trip. All fail onmainand pass with the change; the 100 existing tests and 5 snapshots are unchanged.Out of scope, noted in the issue: the fallback path that re-wraps an unrecognised aggregate into
SUM(SUM(...))on export;_qualify_colon export re-qualifying only bare identifiers (its own comment defers compound and quoted expressions), which is why the round-trip test expectsSUM(gross - tax)and a ratio with a compound numerator now exports as(SUM(gross - tax)) / (SUM(orders.gross))(numerically identical within one semantic model;mainexported the wrong(SUM(orders.tax)) / (SUM(orders.gross))); and the SUM_BOOLEAN branch, which still keeps the qualifier in its condition (separate issue).Related Issues
Fixes #385
Testing
cd converters/dbt uv run pytestResult: 108 passed (100 existing + 8 new; 5 snapshots unchanged). The 8 new tests fail on
main.Also checked locally: the full suite under the declared floor
sqlglot==20.0.0(108 passed); every Ossie document inexamples/andconverters/*/tests/converts byte-identically onmainand this branch; MetricFlow'sSemanticManifestValidatoraccepts manifests containing every newexprshape with no errors or warnings; and a differential run of the old and new_col_nameover 31,608 generated aggregate expressions differs only when the rendered argument contains a dot, with no change in which expressions are recognised.CI matrix reproduced locally with uv on Python 3.11, 3.12, 3.13 and 3.14 against the locked dependencies (metricflow 0.211.0, sqlglot 30.12.0): 108 passed on each, and again with sqlglot pinned to the declared floor 20.0.0 and to the newest release 30.18.0. Rendering the converted metrics through MetricFlow 0.211 on DuckDB returns the same numbers as the original Ossie expressions for every affected shape, where
mainreturns the numbers of the sliced argument.Checklist
Specification
core-spec/and follow the existing structureOntology
ontology/are consistent with spec changesConverters
converters/is updated to reflect spec or ontology changesValidation
validation/are updated if the spec changedDocumentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedExamples
examples/are added or updated for any new spec constructs or converter supportTests
pytest/ CI green)Compliance