Skip to content

[DBT] Map config.meta onto Ossie custom_extensions instead of dropping it - #378

Open
georgea-tk wants to merge 2 commits into
apache:mainfrom
georgea-tk:fix/config-meta-custom-extensions
Open

georgea-tk wants to merge 2 commits into
apache:mainfrom
georgea-tk:fix/config-meta-custom-extensions

Conversation

@georgea-tk

Copy link
Copy Markdown

Summary

config.meta is silently dropped when converting an MSI manifest to Ossie — dimensions, entities, measures, metrics, and semantic models can all carry arbitrary presentation/vendor metadata (e.g. a LookML group_label or value_format), and none of it survives the conversion. Both ends of the mapping already exist: MSI exposes config.meta as a dict, and Ossie's Field/Metric/Dataset/SemanticModel all define custom_extensions with an open vendor_name string.

This maps a non-empty config.meta onto one custom_extensions entry (vendor_name="DBT", data=json.dumps(meta)) wherever it's currently dropped. Empty/absent meta still produces no custom_extensions.

Two things intentionally left out of scope, per the issue:

vendor_name="DBT" follows the schema's own example; happy to change it if maintainers prefer COMMON or something else.

Related Issues

Fixes #303

Checklist

Specification

  • N/A — no spec change

Ontology

  • N/A

Converters

  • Converter logic in converters/ is updated to reflect spec or ontology changes
  • New tests added under the converter's test directory

Validation

  • N/A

Documentation

  • N/A — internal mapping, no user-facing doc surface changed

Examples

  • N/A

Tests

  • All existing tests pass — ran full converters/dbt suite locally, 106 passed
  • New functionality is covered by tests — dimension/entity/measure/metric/semantic-model config.meta, plus empty and absent meta

Compliance

  • ASF license headers present — no new files, only edits to existing licensed files
  • No third-party dependencies added

…g it

MSIToOssieConverter never read `config.meta` off dimensions, entities,
measures, metrics, or semantic models, so any presentation/vendor metadata
placed there (e.g. LookML group_label, value_format) vanished silently at
the MSI -> Ossie boundary. Both ends of the mapping already exist: MSI
exposes config.meta as an arbitrary dict, and Ossie's Field/Metric/Dataset/
SemanticModel all define custom_extensions with an open vendor_name string.

Adds _custom_extensions_from_meta, mapping a non-empty config.meta onto one
custom_extensions entry (vendor_name="DBT", data=json.dumps(meta)), wired
into _convert_dimension, _convert_entity, _convert_measure,
_convert_semantic_model, and the metric-building loop in convert(). Empty
or absent config.meta continues to produce no custom_extensions, matching
existing Optional-field conventions elsewhere in the converter.

vendor_name is set to "DBT" per the schema's own example; open to renaming
if maintainers prefer something else. Round-tripping through ossie_to_msi,
and the equivalent drop in converters/snowflake (ossie#111), are left as
follow-ups rather than folded into this PR.

Fixes apache#303

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@georgea-tk
georgea-tk force-pushed the fix/config-meta-custom-extensions branch from 862aeba to 1849a6f Compare September 10, 2026 09:06
@georgea-tk
georgea-tk marked this pull request as ready for review September 10, 2026 09:08

@QMalcolm QMalcolm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for putting this together! What has been done thus far, looks good 🙂 The PR body mentions that

Round-tripping through ossie_to_msi — open question in the issue on whether that's wanted

I think round-tripping would be desirable. I'd appreciate that being added before approval ❤️ If you disagree, let me know, and we can discuss it more and perhaps proceed as is.

Per review on apache#378: the forward direction (MSI -> Ossie) maps a non-empty
config.meta onto one custom_extensions entry, but the reverse converter
(ossie_to_msi) dropped custom_extensions entirely, so a document that had
been through both directions lost its config.meta silently.

Adds _config_from_custom_extensions, the inverse of the forward mapping:
finds the DBT-vendor custom_extensions entry (if any) and reconstructs
config.meta from its JSON payload. Wired into the same five call sites the
forward fix touched: entity/dimension classification in _classify_field,
the semantic-model-level dataset conversion, and metric conversion.

Only the DBT-vendor entry round-trips — an extension written by another
vendor (e.g. SIGMA) never originated from config.meta and has no meaning
as one, so it's left alone. RATIO's synthetic numerator/denominator
sub-metrics don't inherit the parent metric's config, since they have no
origin of their own to carry metadata from.

Verified beyond the new unit tests: ran an actual MSI -> Ossie -> MSI round
trip through both converters and confirmed config.meta survives intact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@georgea-tk

Copy link
Copy Markdown
Author

Added — round-tripping now works via ossie_to_msi._config_from_custom_extensions, the inverse of the forward mapping: it finds the DBT-vendor custom_extensions entry (if present) and reconstructs config.meta from it. Wired into the same call sites the forward direction touched (entities, dimensions, the dataset itself, and metrics — RATIO's synthetic numerator/denominator sub-metrics intentionally don't inherit it, since they have no origin of their own).

Verified an actual MSI -> Ossie -> MSI round trip preserves config.meta intact, not just via the new unit tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

msi_to_osi silently discards config.meta instead of mapping it to custom_extensions

2 participants