Example Code Throws Null Pointer Exception with 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12)

[Miao Wang (wangmiao1981)]

Describe the issue

I try to run some example code in a Spark notebook with 7.3 LTS (includes Apache Spark 3.0.1, Scala 2.12).

To Reproduce

Create a notebook and copy & paste the following code:

`
import org.apache.spark.sql.SparkSession
import com.microsoft.hyperspace._
import com.microsoft.hyperspace.Hyperspace
import com.microsoft.hyperspace.index.IndexConfig

val spark = SparkSession
.builder()
.appName("Hyperspace example")
.config("spark.some.config.option", "some-value")
.getOrCreate()

val deptLocation = "dbfs:/departments/"
departments
.toDF("deptId", "deptName", "location")
.write
.mode("overwrite")
.parquet(deptLocation)

val empLocation = "dbfs:/employees/"
employees
  .toDF("empId", "empName", "deptId")
  .write
  .mode("overwrite")
  .parquet(empLocation)

val departments = Seq(
(10, "Accounting", "New York"),
(20, "Research", "Dallas"),
(30, "Sales", "Chicago"),
(40, "Operations", "Boston"))

// Sample employee records.
val employees = Seq(
  (7369, "SMITH", 20),
  (7499, "ALLEN", 30),
  (7521, "WARD", 30),
  (7566, "JONES", 20),
  (7698, "BLAKE", 30),
  (7782, "CLARK", 10),
  (7788, "SCOTT", 20),
  (7839, "KING", 10),
  (7844, "TURNER", 30),
  (7876, "ADAMS", 20),
  (7900, "JAMES", 30),
  (7934, "MILLER", 10),
  (7902, "FORD", 20),
  (7654, "MARTIN", 30))

// Save example data records as Parquet.
import spark.implicits._

val deptLocation = "dbfs:/departments/"
departments
  .toDF("deptId", "deptName", "location")
  .write
  .mode("overwrite")
  .parquet(deptLocation)

val empLocation = "dbfs:/employees/"
employees
.toDF("empId", "empName", "deptId")
.write
.mode("overwrite")
.parquet(empLocation)

val hyperspace = new Hyperspace(spark)

val deptDF = spark.read.parquet(deptLocation)
val empDF = spark.read.parquet(empLocation)
val deptIndexConfig = IndexConfig("deptIndex", Seq("deptId"), Seq("deptName"))
val empIndexConfig = IndexConfig("empIndex", Seq("empId"), Seq("empName"))
Exceptions:java.lang.NullPointerException
at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.run(FileBasedSourceProviderManager.scala:96)
at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.signature(FileBasedSourceProviderManager.scala:80)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.$anonfun$fingerprintVisitor$1(FileBasedSignatureProvider.scala:53)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.$anonfun$fingerprintVisitor$1$adapted(FileBasedSignatureProvider.scala:51)
at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:190)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.fingerprintVisitor(FileBasedSignatureProvider.scala:51)
at com.microsoft.hyperspace.index.FileBasedSignatureProvider.signature(FileBasedSignatureProvider.scala:40)
at com.microsoft.hyperspace.index.IndexSignatureProvider.signature(IndexSignatureProvider.scala:45)
at com.microsoft.hyperspace.actions.CreateActionBase.getIndexLogEntry(CreateActionBase.scala:64)
at com.microsoft.hyperspace.actions.CreateAction.logEntry(CreateAction.scala:38)
at com.microsoft.hyperspace.actions.Action.begin(Action.scala:50)
at com.microsoft.hyperspace.actions.Action.run(Action.scala:90)
at com.microsoft.hyperspace.actions.Action.run$(Action.scala:83)
at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:30)
at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:43)
at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77)
at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:43)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:1)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:70)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:72)
at line04b8f3115af7439a97e3ee6d9355b81c80.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.(command-1356655024485756:74)`

Expected behavior

Environment

[EJ Song (sezruby)]

Hyperspace currently supports only Spark 2.4 and support for Spark 3.0 is on the way: #85.

[Miao Wang (wangmiao1981)]

EJ Song (@sezruby) Thanks! I also suspect the runtime version issue. Will create a new cluster with 2.4.

[Miao Wang (wangmiao1981)]

One more comment: EJ Song (@sezruby) Spark 2.4 is scale-2.11. By default, Hyperspace seems compiled with Scala-2.12.

[EJ Song (sezruby)]

Yes it's 2.12 by default, so you need to change the version as 2.11 in build.sbt.
It's being delayed to support Spark 3.0 for now. Thanks!

[Terry Kim (imback82)]

Or if you are using sbt CLI to build, you can do the following:

> ++ 2.11.12
[info] Setting version to 2.11.12
> compile

, or do the following to build both:

> +compile

[Miao Wang (wangmiao1981)]

I switched to Databricks runtime 6.4 (includes Apache Spark 2.4.5, Scala 2.11) with ADLS Gen2 as my storage (also tried DBFS).

It throws new exception if I try to create index.

java.lang.ClassCastException: org.apache.spark.sql.execution.datasources.SerializableFileStatus cannot be cast to org.apache.hadoop.fs.FileStatus at com.microsoft.hyperspace.index.sources.default.DefaultFileBasedSource$$anonfun$3.apply(DefaultFileBasedSource.scala:134) at scala.math.Ordering$$anon$5.compare(Ordering.scala:122) at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) at java.util.TimSort.sort(TimSort.java:220) at java.util.Arrays.sort(Arrays.java:1438) at scala.collection.SeqLike$class.sorted(SeqLike.scala:648) at scala.collection.AbstractSeq.sorted(Seq.scala:41) at scala.collection.SeqLike$class.sortBy(SeqLike.scala:623) at scala.collection.AbstractSeq.sortBy(Seq.scala:41) at com.microsoft.hyperspace.index.sources.default.DefaultFileBasedSource.signature(DefaultFileBasedSource.scala:134) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$signature$1.apply(FileBasedSourceProviderManager.scala:80) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$signature$1.apply(FileBasedSourceProviderManager.scala:80) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$run$1.apply(FileBasedSourceProviderManager.scala:97) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager$$anonfun$run$1.apply(FileBasedSourceProviderManager.scala:96) at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57) at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66) at scala.collection.mutable.ArraySeq.foldLeft(ArraySeq.scala:46) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.run(FileBasedSourceProviderManager.scala:96) at com.microsoft.hyperspace.index.sources.FileBasedSourceProviderManager.signature(FileBasedSourceProviderManager.scala:80) at com.microsoft.hyperspace.index.FileBasedSignatureProvider$$anonfun$fingerprintVisitor$1.apply(FileBasedSignatureProvider.scala:53) at com.microsoft.hyperspace.index.FileBasedSignatureProvider$$anonfun$fingerprintVisitor$1.apply(FileBasedSignatureProvider.scala:51)

[Miao Wang (wangmiao1981)]

EJ Song (@sezruby) Terry Kim (@imback82) any thoughts on the above exception?

[Miao Wang (wangmiao1981)]

EJ Song (@sezruby) I will first make it work for Spark 2.4 and rebuild runtime for Spark 3.0 later.

[Terry Kim (imback82)]

I believe SerializableFileStatus is a known issue in the Databricks (they seem to have a different implementation from the OSS Spark). We are looking into a workaround. cc Rahul Potharaju (@rapoth)

[Miao Wang (wangmiao1981)]

Glad to know it is not my problem. I can spend some time to see if we can walk-around or get some insights from DB engineering. I am closely working with them on other projects.

[Rahul Potharaju (rapoth)]

Thank you for your interest Miao Wang (@wangmiao1981)!

We are sorry you are hitting into this issue. We tested Hyperspace against open-source Apache Spark so there maybe a few minor modifications to make it work with Databricks. If you manage to find any workarounds or the changes needed to make it work after your discussion with DB engineering, please feel free to open a PR! Thanks again!

[Miao Wang (wangmiao1981)]

Response from Databricks:
yea, we'll have to manually convert it
val statuses = serializableStatuses.map { f =>
new FileStatus(f.length, f.isDir, f.blockReplication, f.blockSize, f.modificationTime, new Path(f.path)), blockLocations)
}

I am following up with them how to import it in the source code, because serializableStatuses is a Private class afaik.

I may be able to submit a PR once I get more details from Databricks.

[Miao Wang (wangmiao1981)]

Rahul Potharaju (@rapoth) our team member Andrei made Hyperspace working with 5.5 LTS, Spark 2.4.3 last night. He will submit a PR soon.

[Rahul Potharaju (rapoth)]

This is great news! Thank you for the update and look forward to the PR!

[Andrei Ionescu (andrei-ionescu)]

This has been fixed with #303.

[Andrei Ionescu (andrei-ionescu)]

Fixed by #303.