[DBT] Map config.meta onto Ossie custom_extensions instead of dropping it - #378
georgea-tk wants to merge 2 commits into
Conversation
…g it MSIToOssieConverter never read `config.meta` off dimensions, entities, measures, metrics, or semantic models, so any presentation/vendor metadata placed there (e.g. LookML group_label, value_format) vanished silently at the MSI -> Ossie boundary. Both ends of the mapping already exist: MSI exposes config.meta as an arbitrary dict, and Ossie's Field/Metric/Dataset/ SemanticModel all define custom_extensions with an open vendor_name string. Adds _custom_extensions_from_meta, mapping a non-empty config.meta onto one custom_extensions entry (vendor_name="DBT", data=json.dumps(meta)), wired into _convert_dimension, _convert_entity, _convert_measure, _convert_semantic_model, and the metric-building loop in convert(). Empty or absent config.meta continues to produce no custom_extensions, matching existing Optional-field conventions elsewhere in the converter. vendor_name is set to "DBT" per the schema's own example; open to renaming if maintainers prefer something else. Round-tripping through ossie_to_msi, and the equivalent drop in converters/snowflake (ossie#111), are left as follow-ups rather than folded into this PR. Fixes apache#303 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
862aeba to
1849a6f
Compare
QMalcolm
left a comment
There was a problem hiding this comment.
Thank you for putting this together! What has been done thus far, looks good 🙂 The PR body mentions that
Round-tripping through ossie_to_msi — open question in the issue on whether that's wanted
I think round-tripping would be desirable. I'd appreciate that being added before approval ❤️ If you disagree, let me know, and we can discuss it more and perhaps proceed as is.
Per review on apache#378: the forward direction (MSI -> Ossie) maps a non-empty config.meta onto one custom_extensions entry, but the reverse converter (ossie_to_msi) dropped custom_extensions entirely, so a document that had been through both directions lost its config.meta silently. Adds _config_from_custom_extensions, the inverse of the forward mapping: finds the DBT-vendor custom_extensions entry (if any) and reconstructs config.meta from its JSON payload. Wired into the same five call sites the forward fix touched: entity/dimension classification in _classify_field, the semantic-model-level dataset conversion, and metric conversion. Only the DBT-vendor entry round-trips — an extension written by another vendor (e.g. SIGMA) never originated from config.meta and has no meaning as one, so it's left alone. RATIO's synthetic numerator/denominator sub-metrics don't inherit the parent metric's config, since they have no origin of their own to carry metadata from. Verified beyond the new unit tests: ran an actual MSI -> Ossie -> MSI round trip through both converters and confirmed config.meta survives intact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Added — round-tripping now works via Verified an actual MSI -> Ossie -> MSI round trip preserves |
Summary
config.metais silently dropped when converting an MSI manifest to Ossie — dimensions, entities, measures, metrics, and semantic models can all carry arbitrary presentation/vendor metadata (e.g. a LookMLgroup_labelorvalue_format), and none of it survives the conversion. Both ends of the mapping already exist: MSI exposesconfig.metaas a dict, and Ossie'sField/Metric/Dataset/SemanticModelall definecustom_extensionswith an openvendor_namestring.This maps a non-empty
config.metaonto onecustom_extensionsentry (vendor_name="DBT",data=json.dumps(meta)) wherever it's currently dropped. Empty/absentmetastill produces nocustom_extensions.Two things intentionally left out of scope, per the issue:
ossie_to_msi— open question in the issue on whether that's wanted.converters/snowflake(Follow up on OSI ai_context and custom_extensions mapping in snowflake yaml #111) — a separate PR, since fixing either alone doesn't get metadata all the way through.vendor_name="DBT"follows the schema's own example; happy to change it if maintainers preferCOMMONor something else.Related Issues
Fixes #303
Checklist
Specification
Ontology
Converters
converters/is updated to reflect spec or ontology changesValidation
Documentation
Examples
Tests
converters/dbtsuite locally, 106 passedCompliance