CreateIndex failed #423

Sergei Iarymov (soyarym) · 2021-04-18T22:57:00Z

Sergei Iarymov (soyarym)
Apr 18, 2021

Hello!
I use pyspark 2.4.7 and on-premise Hadoop cluster, and also Hyperspace v 0.4.0 by adding param "spark.jars.packages": {"com.microsoft.hyperspace:hyperspace-core_2.11": "0.3.0"} to the spark-session conf.
When I try to reproduce the example from the quick start guide:

sample_data = [(1, "name1"), (2, "name2")]

spark.createDataFrame(sample_data, ['id', 'name'])\
    .write\
    .mode("overwrite")\
    .parquet("warehouse/something_sb.db/hyperspace_test_1")

df = spark.read.parquet("warehouse/something_sb.db/hyperspace_test_1")

from hyperspace import Hyperspace

hs = Hyperspace(spark)

from hyperspace import IndexConfig

idxConfig = IndexConfig("test_index", ["id"], ["name"])
hs.createIndex(df, idxConfig)

I get this error:

Py4JJavaError: An error occurred while calling o324.createIndex.
: com.microsoft.hyperspace.HyperspaceException: Could not acquire proper state
	at com.microsoft.hyperspace.actions.Action$class.saveEntry(Action.scala:79)
	at com.microsoft.hyperspace.actions.Action$class.begin(Action.scala:53)
	at com.microsoft.hyperspace.actions.Action$class.run(Action.scala:90)
	at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:30)
	at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:43)
	at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:77)
	at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:42)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.lang.Thread.run(Thread.java:748)

What did I do wrong?
Thank you!

Answered by sezruby

Apr 19, 2021

Sergei Iarymov (@soyarym) Yes it's spark.sql.warehouse.dir by default and you can change the dir by setting this config: spark.hyperspace.system.path

View full answer

EJ Song (sezruby) · 2021-04-19T04:03:47Z

EJ Song (sezruby)
Apr 19, 2021

Hi,

Could you try again with "0.4.0" version?
Could you check the spark driver log if there's logError(s"Failed to write log with id = '$id'", ex)?

2 replies

Sergei Iarymov (soyarym) Apr 19, 2021
Author

Yep, but I've got the same error.
Yeah, you're totally right:

21/04/19 14:53:21 ERROR IndexLogManagerImpl: Failed to write log with id = '0'
org.apache.hadoop.security.AccessControlException: Permission denied: user=xxxxxx, access=WRITE, inode="/warehouse/indexes/test_index/_hyperspace_log/temp4837c652-bb48-472c-84a3-6828d00439eb":hive:hdfs:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:427)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:277)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1653)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
        at com.microsoft.hyperspace.util.FileUtils$.createFile(FileUtils.scala:39)
        at com.microsoft.hyperspace.index.IndexLogManagerImpl.writeLog(IndexLogManager.scala:155)
        at com.microsoft.hyperspace.actions.Action$class.saveEntry(Action.scala:78)
        at com.microsoft.hyperspace.actions.Action$class.begin(Action.scala:53)
        at com.microsoft.hyperspace.actions.Action$class.run(Action.scala:90)
        at com.microsoft.hyperspace.actions.CreateAction.run(CreateAction.scala:30)
        at com.microsoft.hyperspace.index.IndexCollectionManager.create(IndexCollectionManager.scala:47)
        at com.microsoft.hyperspace.index.CachingIndexCollectionManager.create(CachingIndexCollectionManager.scala:78)
        at com.microsoft.hyperspace.Hyperspace.createIndex(Hyperspace.scala:43)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at py4j.Gateway.invoke(Gateway.java:282)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=xxxxxx, access=WRITE, inode="/warehouse/indexes/test_index/_hyperspace_log/temp4837c652-bb48-472c-84a3-6828d00439eb":hive:hdfs:drwxr-xr-x
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:427)
        at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:277)
        at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
        at org.apache.hadoop.ipc.Client.call(Client.java:1475)
        at org.apache.hadoop.ipc.Client.call(Client.java:1412)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy13.create(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
        at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy14.create(Unknown Source)
        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
        ... 31 more

What does it mean?

Sergei Iarymov (soyarym) Apr 19, 2021
Author

Oh, now I understand - it tries to write to the /warehouse/indexes/.. , to which I can't have a permission to write, but I don't understand, why it does that.
In my opinion it should write to the /warehouse/something_sb.db/hyperspace_test_1/indexes/.., isn't it?

Sergei Iarymov (soyarym) · 2021-04-19T22:46:01Z

Sergei Iarymov (soyarym)
Apr 19, 2021
Author

I think I find the source of my problem - it is here:
https://github.com/microsoft/hyperspace/blob/master/src/main/scala/com/microsoft/hyperspace/index/PathResolver.scala

Here, to be precise:

def systemPath: Path = {
    val defaultIndexesPath =
      new Path(conf.getConfString("spark.sql.warehouse.dir"), "indexes")
    new Path(conf.getConfString(IndexConstants.INDEX_SYSTEM_PATH, defaultIndexesPath.toString))
  }

I suppose we can fix it by getting additional param like "spark.sql.warehouse.index.dir" from spark.conf, but If it's empty let it be by default "spark.sql.warehouse.dir".
EJ Song (@sezruby) What do you think?

0 replies

EJ Song (sezruby) · 2021-04-19T23:17:35Z

EJ Song (sezruby)
Apr 19, 2021

Sergei Iarymov (@soyarym) Yes it's spark.sql.warehouse.dir by default and you can change the dir by setting this config: spark.hyperspace.system.path

1 reply

Sergei Iarymov (soyarym) Apr 19, 2021
Author

Thanks, man, it works! :)
You can't imagine how stupid I'm feeling myself now.

Navigation Menu

Uh oh!

CreateIndex failed #423

Uh oh!

{{title}}

Uh oh!

Replies: 3 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

CreateIndex failed #423

Uh oh!

Sergei Iarymov (soyarym) Apr 18, 2021

Replies: 3 comments · 3 replies

Uh oh!

EJ Song (sezruby) Apr 19, 2021

Uh oh!

Uh oh!

Sergei Iarymov (soyarym) Apr 19, 2021 Author

Uh oh!

Sergei Iarymov (soyarym) Apr 19, 2021 Author

Uh oh!

Uh oh!

Sergei Iarymov (soyarym) Apr 19, 2021 Author

Uh oh!

EJ Song (sezruby) Apr 19, 2021

Uh oh!

Sergei Iarymov (soyarym) Apr 19, 2021 Author

Sergei Iarymov (soyarym)
Apr 18, 2021

Replies: 3 comments 3 replies

EJ Song (sezruby)
Apr 19, 2021

Sergei Iarymov (soyarym) Apr 19, 2021
Author

Sergei Iarymov (soyarym) Apr 19, 2021
Author

Sergei Iarymov (soyarym)
Apr 19, 2021
Author

EJ Song (sezruby)
Apr 19, 2021

Sergei Iarymov (soyarym) Apr 19, 2021
Author