[Bug]: VECTOR LENGTH not supported in Spanner to BigQuery Change Stream Processor

[payaaam]

Related Template(s)

spanner_change_streams_to_bigquery

Template Version

2026-05-26-00_RC00

What happened?

I have a spanner enterprise table with a column

CREATE TABLE IF NOT EXISTS user_face_embeddings_buffalo_l (
    id         VARCHAR(36) NOT NULL,
    embedding  float4[] VECTOR LENGTH 512 NOT NULL -- this column caused problems
)

and I have the changestream to bigquery job deployed. Right when I landed this table with no data, the dataflow job started printing the following errors

Error message from worker: generic::unknown: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: java.lang.RuntimeException: Caught exception when constructing SpannerColumn for each column tracked by change stream spanner_snapshot_stream, message: Optional[Unsupported Spanner PostgreSQL type: real[] vector length 512], cause: null
    org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39)
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn$DoFnInvoker$6e2a6220.invokeSetup(Unknown Source)
    org.apache.beam.sdk.transforms.reflect.DoFnInvokers.tryInvokeSetupFor(DoFnInvokers.java:53)
    org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:442)
    org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.addRunnerForPTransform(FnApiDoFnRunner.java:181)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:304)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:876)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:510)
    org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:988)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:506)
    org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
    org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
    java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:840)
    Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Caught exception when constructing SpannerColumn for each column tracked by change stream spanner_snapshot_stream, message: Optional[Unsupported Spanner PostgreSQL type: real[] vector length 512], cause: null
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn.setUp(FailsafeModJsonToTableRowTransformer.java:197)
    Caused by: java.lang.RuntimeException: Caught exception when constructing SpannerColumn for each column tracked by change stream spanner_snapshot_stream, message: Optional[Unsupported Spanner PostgreSQL type: real[] vector length 512], cause: null
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerColumnsByTableName(SpannerChangeStreamsUtils.java:234)
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerTableByName(SpannerChangeStreamsUtils.java:119)
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerTableByName(SpannerChangeStreamsUtils.java:105)
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn.setUp(FailsafeModJsonToTableRowTransformer.java:190)
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.FailsafeModJsonToTableRowTransformer$FailsafeModJsonToTableRow$FailsafeModJsonToTableRowFn$DoFnInvoker$6e2a6220.invokeSetup(Unknown Source)
    org.apache.beam.sdk.transforms.reflect.DoFnInvokers.tryInvokeSetupFor(DoFnInvokers.java:53)
    org.apache.beam.fn.harness.FnApiDoFnRunner.<init>(FnApiDoFnRunner.java:442)
    org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.addRunnerForPTransform(FnApiDoFnRunner.java:181)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:304)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.addRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:258)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:876)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:510)
    org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:988)
    org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:506)
    org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:150)
    org.apache.beam.fn.harness.control.BeamFnControlClient$InboundObserver.lambda$onNext$0(BeamFnControlClient.java:115)
    java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    org.apache.beam.sdk.util.UnboundedScheduledExecutorService$ScheduledFutureTask.run(UnboundedScheduledExecutorService.java:163)
    java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    java.base/java.lang.Thread.run(Thread.java:840)
    Caused by: java.lang.IllegalArgumentException: Unsupported Spanner PostgreSQL type: real[] vector length 512
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.TypesUtils.informationSchemaPostgreSQLTypeToSpannerType(TypesUtils.java:98)
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.informationSchemaTypeToSpannerType(SpannerChangeStreamsUtils.java:489)
    com.google.cloud.teleport.v2.templates.spannerchangestreamstobigquery.schemautils.SpannerChangeStreamsUtils.getSpannerColumnsByTableName(SpannerChangeStreamsUtils.java:222)
    ... 26 more
    
    passed through:
    ==>
        dist_proc/dax/workflow/worker/fnapi_service_impl.cc:1347

Relevant log output