[OSSIE][DATABRICKS] Add Java Converter and Reformat by Haoranli503 · Pull Request #333 · apache/ossie

Haoranli503 · 2026-08-19T17:18:53Z

Summary

Adds a Java implementation of the Databricks converter alongside the existing Python
reference, and restructures converters/databricks/ to host both languages. The Java library is
the implementation maintained going forward; the Python converter becomes a reference copy slated
for deprecation.

Conversion is unchanged in spirit: pure YAML text in, YAML text out, reading and writing the two
formats as parsed maps and lists. The Java and Python implementations target the same Databricks
Unity Catalog Metric View YAML v1.1 contract, and the checked-in fixtures pin the expected output
for both directions (export: Ossie -> Metric View, import: Metric View -> Ossie).

Changes:

Restructure — the existing flat Python converter (src/, tests/, pyproject.toml,
README.md, uv.lock) moves under converters/databricks/python/ with no content changes,
making room for the Java tree.
Add java/ — the bidirectional converter as a Maven module: library, an
OssieDatabricksConverter command-line tool, JUnit 5 tests, and YAML fixtures. Package
org.apache.ossie.converter.databricks, following the conventions of the existing polaris and
salesforce Java converters (parent org.apache:apache:39; deps SnakeYAML / Jackson / JUnit 5;
apache-rat-plugin license audit; shade plugin for the executable CLI jar).
Add a root README.md describing the two-language layout and pointing at each
implementation's own README.
CI (.github/workflows/converter-databricks-ci.yml) — point the Python job at the new
python/ working directory, and add a Java job (mvn -B verify, Temurin JDK 21) mirroring
converter-polaris-ci.yml.

Related Issues

Checklist

Specification

Spec changes are included in core-spec/ and follow the existing structure
Spec changes have been discussed on the mailing list or in a linked issue
Breaking changes to the spec are clearly called out in the summary

Ontology

Ontology changes in ontology/ are consistent with spec changes
New or modified terms are defined and documented

Converters

Converter logic in converters/ is updated to reflect spec or ontology changes
New converters include tests under the converter's test directory

Validation

Validation rules in validation/ are updated if the spec changed
New validation cases are covered by tests

Documentation

docs/ is updated to reflect any user-facing changes
New features or behaviors are documented with examples where appropriate
CONTRIBUTING.md is updated if the contribution process changed

Examples

examples/ are added or updated for any new spec constructs or converter support

Tests

All existing tests pass (pytest / CI green)
New functionality is covered by tests

Compliance

ASF license headers are present on all new source files
No third-party dependencies are added without PMC/IPMC approval

jbonofre · 2026-08-28T05:11:25Z

@Haoranli503 I pushed some fixes here. Can you please take a look?

Haoranli503 · 2026-08-28T18:49:54Z

Hi @jbonofre , fixes look good. Thanks! I also pushed one commit addressing some issues we found when we are building native sql support based on the converter. Hope you can review those as well. Also feel free to change.

Haoranli503 · 2026-08-31T22:22:07Z

Hi @jbonofre , let's pause this a bit. I want to add a few more commits to address #321 and #326 and some findings from our sql function bug bash. Will notify you when these are in. Thanks a lot!

GMorrisXBL · 2026-09-03T15:30:12Z

Thanks @Haoranli503!

Move the Python converter under converters/databricks/python/ (content unchanged) and add a Maven Java module -- library, CLI (OssieDatabricksConverter), JUnit tests, and fixtures under java/, package org.apache.ossie.converter.databricks -- as the maintained implementation. Add a root README describing the two-language layout, and a Java build job (mvn -B verify, JDK 21) in converter-databricks-ci.yml, mirroring the polaris converter. Signed-off-by: Haoran Li <haoran.li@databricks.com>

Replace the Python README's forward-looking 'Future effort' section with a deprecation note: the Java converter under java/ is the maintained implementation; the Python copy is kept for reference and no longer actively extended. Signed-off-by: Haoran Li <haoran.li@databricks.com>

Build: - maven-shade no longer writes dependency-reduced-pom.xml into the module root, where apache-rat failed `mvn verify` on it as an unapproved file - configure surefire to include **/*Suite.java: the default includes match none of the test classes, so the build ran zero tests and still passed - drop the **/*.md rat exclude and restore the ASF header on both READMEs - align snakeyaml with the 2.3 that jackson-dataformat-yaml declares Converter: - qualifyMeasure matches the whole qualifier run and resolves it from the leaf, so an expression that already carries a join path is no longer qualified a second time (SUM(customer.customer.region.population)) - de-alias measure qualifiers on import, the inverse of the export rewrite and what resolveColumn already did for dimensions - match dropped names outside string literals when cascading drops, so a name that only occurs in a literal no longer drops an unrelated column - quoteReplacement the stash unicode-escape pass, which halved an escaped backslash run instead of re-emitting it verbatim - notice the ai_context object members and the foreign-vendor extensions dropped from a field or a metric - validate a join source on import with the rule the export applies, so a view that imports cleanly is always exportable again CLI: - name the directions from the Apache Ossie model's point of view, matching the library Javadoc and the Python CLI: export = Ossie -> Metric View - give each command its own selector flag instead of sharing one field, and resolve the command before parsing arguments so --help prints usage - print to stdout without the extra newline, so stdout and -o agree Tests: - generate join-qualified and nested-path measures, and one_to_many branches, in the property round-trip suites - add a regression test per fix, plus a CLI suite (the CLI had none)

Signed-off-by: Haoran Li <haoran.li@databricks.com>

…und trip A dimension's display_name maps to the Ossie Field label, but the Ossie Metric schema has no label, so a metric view measure's display_name was dropped in the MV -> Ossie -> MV round trip. Preserve it in the DATABRICKS custom_extensions stash (the same mechanism as format/window) in MetricViewToOssie.convertMeasure, and restore it in OssieToMetricView.convertMetric. Ports apache#326 (landed in Databricks runtime as databricks-eng/runtime#251390). Co-authored-by: Isaac <no-reply@databricks.com>

… complex_joins stash A Metric View join whose `on` is not an equi-join of simple `alias.column` pairs (a non-equi operator, a SQL-function-wrapped key, or an extra filter predicate) has no Apache Ossie relationship form: the relationship schema requires from_columns/to_columns. The converter used to abort the whole MV -> Ossie conversion on such a join. Instead of aborting, preserve the join under the model's DATABRICKS custom_extensions (complex_joins) and warn, rather than emitting a schema-invalid stub relationship with no columns. The reverse converter merges the stashed joins back into the relationship graph and restores each raw `on` verbatim, so such a metric view round-trips (nesting and one_to_many included). Condition-less (cross) joins still have no Apache Ossie representation and are still rejected. Also preserve an equi-join's original `on` verbatim when rebuilding it from the from/to columns would not reproduce it (a fact side qualified by the source table name rather than `source`, or an `on` over equal columns that would rebuild as `using`); canonical joins stash nothing. Ports apache#321 (landed in Databricks runtime as databricks-eng/runtime#251398). Co-authored-by: Isaac <no-reply@databricks.com>

…g tables Upstream apache#356 clarified the mapping table (a measure's display_name has no `label` on the Apache Ossie metric shape, so it rides in the DATABRICKS stash). The converter restructure turned the top-level README into a short pointer, so carry that clarification into the mapping tables now in python/README.md and java/README.md. Both converters stash a measure's display_name (Python via apache#356, Java via the measure display_name round-trip commit), so the tables match the behavior. Co-authored-by: Isaac <no-reply@databricks.com>

Haoranli503 · 2026-09-04T17:50:56Z

Hi @jbonofre , all fixes are in. Can you help tal? Thanks!

jbonofre · 2026-09-10T15:11:54Z

+      if (!isFact) {
+        // A dataset reachable by more than one path (diamond) appears once per path; the first
+        // wins, matching the order dimensions are emitted in.
+        datasetAliasPath.putIfAbsent(node.dataset, qualifier);


A dataset reached via two distinct join paths (a diamond) is silently mapped to only its first-seen alias path via putIfAbsent. A measure expression qualified by this dataset's bare name (SUM(shared.amount)) will always resolve to one arbitrary branch with no warning (unlike the dimension-path handling of the same ambiguity, which explicitily warns and drops).

For instance, fact orders -> dim1 -> shared and orders -> dim2 -> shared, plus SUM(shared.amount), silently emits SUM(dim1.dim1_shared.amount) and drops the dim2 branch.

Can we warn-and-drop here the same way the dimension path does?

Good catch. Fixed: a measure that names a diamond dataset by its bare name is now dropped with a notice, the same way the dimension path drops a complex expression it cannot unambiguously qualify. A measure addresses a dataset by name, so it cannot be split per branch the way a dimension column is, which makes warn and drop the consistent choice. Added a test measureNamingADiamondDatasetIsDroppedWithNotice covering the orders to dim1/dim2 to shared example.

jbonofre · 2026-09-10T15:14:26Z

+    return model;
+  }
+
+  private static void walk(String parentName, String parentAlias, List<Object> joins,


The import direction (walk/hasOtm) never enforces MAX_JOIN_NODES, while export checks it before and during recursion. A Metric View with 250+ joins imports cleanly but then fails to re-export.

I believe it breaks the documented guarantee that anything that imports cleanly can be exported again.

Should we add the check on import too?

Good catch, added the bound. MetricViewToOssie (MV to Ossie) now rejects a Metric View whose fact plus joins exceed MAX_JOIN_NODES, the same limit OssieToMetricView already enforces, so the two directions stay symmetric and the round trip holds. One note on the framing: the failure is actually the reverse. A Metric View with 200 or more joins exports cleanly today, but the emitted model then fails to import again, since the MAX_JOIN_NODES checks live on the Ossie to MV side. The fix rejects it at export instead. Added tests exportRejectsMoreJoinsThanImportCanRebuild and exportAtTheJoinLimitStillSucceeds.

jbonofre · 2026-09-10T15:18:54Z

+    }
+  }
+
+  private static void markOtmVisit(String modelName, Node node, boolean branchIsOtm) {


markOtmVisit rejects a join whose cardinality differs from its top-level branch's cardinality in either direction.

The Python reference (_mark_otm) only rejects reverting from one-to-many back to many-to-one: it explicitly allows a one-to-many join nested under a many-to-one branch. As written, a normal star-schema-with-fanout model (many-to-one to customer, then one-to-many to customer_notes) that Python accepts gets rejected here with "is one_to_many but descends from a many_to_one join".

Is the stricter check intentional, or should this match the Python behavior?

This one is intentional. The Java matches the Databricks engine's actual rule. A Metric View validates each top level join branch with Join.validateSubJoinCardinalities, seeded from that branch's top level join cardinality, and fails any descendant whose cardinality differs in either direction (Join.scala, invoked per top level join by SingleSourceMetricView.validateJoins). So your example, many_to_one to customer then one_to_many to customer_notes, is actually rejected by the engine at CREATE: Expected 'customer_notes' to have 'many_to_one' cardinality, but found 'one_to_many'. The Python _mark_otm is the one out of step: it only rejects reverting from one_to_many back to many_to_one, so it would emit that view and hit the engine error at CREATE instead. Its leniency does not actually help, it just moves the same rejection from conversion time to CREATE time. The Python copy is deprecated so we left it as is.

jbonofre · 2026-09-10T15:21:33Z

+    Map<String, Object> stash = null;
+    for (Map<String, Object> ext : customExtensions(obj)) {
+      if (VENDOR.equals(str(get(ext, "vendor_name")))) {
+        if (stash != null) {


readStash throws when more than one DATABRICKS custom_extensions entry is present.

Python's read_stash just returns the first match and ignores later duplicates.

An Ossie YAML with two such entries that Python converts fine gets rejected here.

Worth aligning with the Python behavior (first-wins) rather than erroring?

Good point, changed it. readStash no longer throws on a duplicate DATABRICKS custom_extensions entry. It keeps the first, ignores the rest to match first wins, and emits a converter notice so the dropped entry is not lost silently rather than vanishing the way a plain first wins would. So a model with two entries converts instead of failing, and the user still sees that one was ignored. Added a test duplicateDatabricksStashKeepsFirstAndWarns.

jbonofre · 2026-09-10T15:23:40Z

+      // cannot represent it (from/to columns are required). Rather than abort the whole
+      // conversion, warn here; the columns-less entry built below is stashed under the model's
+      // custom_extensions (complex_joins) so it round-trips, instead of an invalid relationship.
+      notices.warn("join '" + child + "'", "non-equi or unsupported join condition ('on: " + rawOn


This accepts a non-equi/filtered join on clause and preserves it via the new complex_joins stash, but the Python reference still raises ConversionError on the same input.

Is this divergence intentional/tracked, or should Java match Python's current (stricter) behavior?

Intentional and tracked. A join condition Apache Ossie cannot represent as a relationship (a non equi comparison, a function wrapped key, or an extra filter predicate) has no schema valid relationship form, since an Ossie relationship requires from_columns and to_columns. Rather than abort the whole export, the Java preserves the raw clause in a model level complex_joins stash inside custom_extensions, and the reverse converter rebuilds it. That keeps the round trip lossless for a join that carries real business logic, and the emitted model stays schema valid with no misleading columns less stub relationship, so a non Databricks consumer simply does not see the stashed join rather than seeing a wrong one. The Python reference still raises because it is the deprecated copy.

…th a notice A measure addresses a dataset by name, so a bare reference to a dataset reached by more than one join path (a diamond) cannot be unambiguously qualified. Warn and drop it, mirroring how the dimension path handles a complex expression on a diamond, instead of silently binding to one arbitrary branch.

…round trip symmetric The reverse conversion rejects a model with more than MAX_JOIN_NODES datasets, but the forward conversion had no matching bound, so a Metric View with too many joins could export to a model that could never be imported again. Reject it at export instead, with the same limit.

…stead of failing A duplicate DATABRICKS custom_extensions entry is malformed input. Rather than reject the whole conversion, readStash now keeps the first entry, ignores the rest, and emits a notice so the dropped entry is not lost silently.

Haoranli503 · 2026-09-10T20:28:42Z

Hi @jbonofre

Thanks for the careful pass, these are good catches. Some context that ties several together: the Java converter is now the maintained, on track implementation. It tracks the current Metric View engine rules, and it carries the newer decisions (for example the complex_joins stash). The Python converter is the original reference and is on its way to deprecation, so we are intentionally letting it keep some stale behavior rather than chasing parity into a copy we are retiring.

So where Java and Python differ, Java is the intended source of truth:

3 (cardinality nesting): Java matches the engine, which rejects a mixed cardinality branch in either direction. Python is looser and would emit a view the engine refuses at CREATE, so we are not matching Python here.
5 (non equi joins): Java stashes them via complex_joins. Python still rejects. This divergence is intentional and tracked.

The other three are improvements we applied:

1: drop a measure that names a diamond dataset, with a notice.
2: bound the export join count by MAX_JOIN_NODES so the round trip stays symmetric.
4: keep the first DATABRICKS stash entry and warn instead of failing on a duplicate.

Please help re-verify, thanks!

…avior Document the join-count bound on both conversion directions, the diamond-measure and duplicate-stash notices, and the model-level complex_joins stash for a non-representable join 'on' (dropping the stale line that listed non-equi joins as rejected).

jbonofre self-requested a review August 27, 2026 11:51

jbonofre force-pushed the databricks-converter-v2 branch from 366cccf to 5572106 Compare August 28, 2026 05:09

christianeu-db mentioned this pull request Sep 3, 2026

fix(Databricks): round-trip display_name on measures (#326) #356

Merged

17 tasks

Haoranli503 and others added 6 commits September 4, 2026 17:36

Address internal bug bash feedbacks

cf21f15

Signed-off-by: Haoran Li <haoran.li@databricks.com>

Haoranli503 force-pushed the databricks-converter-v2 branch from 9b270b8 to 7d94fc6 Compare September 4, 2026 17:38

jbonofre requested changes Sep 10, 2026

View reviewed changes

Haoranli503 added 3 commits September 10, 2026 19:18

Haoranli503 force-pushed the databricks-converter-v2 branch from 5b2da58 to 2de9015 Compare September 10, 2026 21:02

Uh oh!

Conversation

Haoranli503 commented Aug 19, 2026

Summary

Related Issues

Checklist

Specification

Ontology

Converters

Validation

Documentation

Examples

Tests

Compliance

Uh oh!

jbonofre commented Aug 28, 2026

Uh oh!

Haoranli503 commented Aug 28, 2026

Uh oh!

Haoranli503 commented Aug 31, 2026

Uh oh!

GMorrisXBL commented Sep 3, 2026

Uh oh!

Haoranli503 commented Sep 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Haoranli503 Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Haoranli503 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Haoranli503 Sep 10, 2026 •

edited

Loading

Haoranli503 commented Sep 10, 2026 •

edited

Loading