Skip to content

Migrate converters and Python SDK to flat semantic model documents - #396

Open
flyrain wants to merge 7 commits into
apache:mainfrom
flyrain:yf/migrate-converters-flat-model
Open

flyrain wants to merge 7 commits into
apache:mainfrom
flyrain:yf/migrate-converters-flat-model

Conversation

@flyrain

@flyrain flyrain commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrate the Python SDK and all affected converters to the single-model document format merged in #383. Ossie input/output now puts name, datasets, relationships, and metrics directly at the root alongside version; the legacy semantic_model wrapper is rejected instead of selecting its first entry.

version: 0.2.0.dev0
name: sales_analytics
datasets:
  - name: orders
    source: sales.public.orders
relationships: []
metrics: []

Rebased onto current main after #383 merged. This PR now contains only the SDK/converter migration and its CI/documentation updates. It also incorporates #397: root-level dialects and vendors are removed; expression dialects and vendor custom extensions remain supported.

Behavior

  • Flatten the shared OssieDocument API and update dbt, Sigma, Wisdom, Databricks, Honeydew, Omni, OrionBelt, Snowflake, GoodData, NVIDIA GSF, Microsoft/Power BI, Salesforce, and Polaris readers/writers, fixtures, snapshots, and usage documentation.
  • Remove obsolete root dialects/vendors fields and first-model/drop-extra-model behavior. Preserve vendor-native collections such as dbt's semantic_models.
  • Preserve the ontology mapping's embedded semantic_model structure and exclude standalone document metadata from that embedded object.
  • Add Polaris import --output-dir DIR to emit one document per nonempty namespace with deterministic, collision-safe filenames and no overwrites. Single-file/stdout imports require one nonempty namespace.
  • Trigger converter CI, including Microsoft, on shared schema/example changes and run shared SDK tests in Validation CI.

This is a breaking SDK and converter format change for the mutable 0.2.0.dev0 specification. Legacy documents must be migrated first; see the spec's migration guidance.

Checklist

  • Converter logic, fixtures, examples, and documentation reflect Define one semantic model per document without a wrapper #383.
  • Breaking behavior and bulk export migration described.
  • Focused regression coverage added or updated.
  • Ontology embedding and vendor-specific model collections preserved.
  • No runtime dependencies added.

Validation

  • Passed 199 focused SDK/converter tests, including dbt relationship, composite-key, template-safety, CLI, and snapshot regressions.
  • Passed Microsoft CLI import/export round-trip and schema validation; all 10 checked-in standalone converter fixtures validate against the merged schema.
  • Microsoft Ruff checks, changed-Python compilation, and whitespace checks pass. Comparison against the rebased branch found no new Ruff diagnostics in the other changed Python files; existing style findings remain.
  • Full converter regression suites and optional Microsoft TOM/live-engine checks were not run locally; full regression validation is left to CI.

@kayemkim

Copy link
Copy Markdown
Contributor

#393 was settled by removing the root dialects/vendors arrays from the schema (#397, now on main), and #383 has been asked to drop them in its rebase. This branch still keeps them in a few places that will need the same treatment: converters/sigma/tests/test_roundtrip.py asserts serialized["dialects"] == ["ANSI_SQL", "SIGMA"] and serialized["vendors"] == ["SIGMA"], the orionbelt test_schema_rejects_root_dialects_and_vendors is flipped to test_schema_accepts_root_dialects_and_vendors, the nvidia reader allow-lists both keys at the root, the honeydew tests expect a root vendors, and the SDK tests and python/README.md list them alongside version. I merged the schema change with this branch earlier; the conflict is confined to those spots and resolves by dropping them.

@flyrain
flyrain force-pushed the yf/migrate-converters-flat-model branch from b2bfb27 to 9c2a506 Compare September 16, 2026 17:27
@flyrain
flyrain marked this pull request as ready for review September 16, 2026 17:33
Copilot AI lite review requested due to automatic review settings September 16, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants