Hi all, I am trying to setup hdfs as deepstore fo...
# troubleshooting
s
Hi all, I am trying to setup hdfs as deepstore for pinot which is deployed in kubernetes Cluster. I am getting below error with kerberose not able to authenticate. I'm not using kerberos authentication the hdfs config using property namehadoop.security.authentication/name valuesimple/value /property Error -
Copy code
2025/04/20 19:16:36.464 ERROR [PinotFSFactory] [Start a Pinot [SERVER]] Could not instantiate file system for class org.apache.pinot.plugin.filesystem.HadoopPinotFS with scheme hdfs
java.lang.RuntimeException: Could not initialize HadoopPinotFS
	at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:71) ~[pinot-hdfs-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.server.starter.helix.BaseServerStarter.initSegmentFetcher(BaseServerStarter.java:925) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.server.starter.helix.BaseServerStarter.start(BaseServerStarter.java:604) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:167) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:97) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:279) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:305) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:279) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
	at jdk.security.auth/com.sun.security.auth.UnixPrincipal.init(UnixPrincipal.java:71)
	at jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:134)
	at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
	at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
	at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
	at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2065)
	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1975)
	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719)
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3746)
	at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3736)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3520)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
	at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:67)
	at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52)
	at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72)
	at org.apache.pinot.server.starter.helix.BaseServerStarter.initSegmentFetcher(BaseServerStarter.java:925)
	at org.apache.pinot.server.starter.helix.BaseServerStarter.start(BaseServerStarter.java:604)
	at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:167)
	at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:97)
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:279)
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:305)
	at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:279)

	at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1986) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3746) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3736) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3520) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288) ~[hadoop-common-3.3.4.jar:?]
	at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:67) ~[pinot-hdfs-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	... 9 more
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
Please help in resolving this. I'm clueless Note : Pinot version is 1.2.0 and hdfs version is 3.3.4 Pinot config looks like this
Copy code
pinot.set.instance.id.to.hostname=true
      pinot.server.instance.realtime.alloc.offheap=true
      pinot.server.instance.enable.split.commit=true
      pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
      pinot.server.storage.factory.hdfs.hadoop.conf.path=/etc/hadoop-conf/conf/
      pinot.server.segment.fetcher.protocols=file,http,hdfs
         pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      pinot.server.grpc.enable=true
      pinot.server.grpc.port=8090
x
can you try out this image tag:
apachepinot/pinot:release-1.3.0-hdfs-patch-17-ms-openjdk
s
@Xiang Fu still same issue
x
s
We are not using kerberos authentication. Same is specified in the hdfs config.
Copy code
property
    namehadoop.security.authentication/name
    valuesimple/value
  /property
the pinot config looks like this
Copy code
pinot.set.instance.id.to.hostname=true
      pinot.server.instance.realtime.alloc.offheap=true
      pinot.server.instance.enable.split.commit=true
      pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
      pinot.server.storage.factory.hdfs.hadoop.conf.path=/etc/hadoop-conf/conf/
      pinot.server.segment.fetcher.protocols=file,http,hdfs
         pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
      pinot.server.grpc.enable=true
      pinot.server.grpc.port=8090
x
then maybe pinot is missing some libs
you might need to run an initContainer to copy some libs to the pinot classpath
I suppose you mount the volume to
/etc/hadoop-conf/conf/
right?
s
1. config(core-site.xml, hdfs-site.xml) is at
/etc/hapdoop-conf/conf
2. libs are also mouthed at
/usr/lib/hadoop-home/hadoop-hdfs
and provided in pinot server as CLASSPATH_PREFIX with values below
Copy code
/usr/lib/hadoop-home/hadoop-hdfs/hadoop-hdfs-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-annotations-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-auth-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-common-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/guava-27.0-jre.jar:/usr/lib/hadoop-home/hadoop-hdfs/gson-2.8.9.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-hdfs-native-client.jar
libraries above are already copied to pinot and mentioned in classpath
I am confused as to why there is an error about Kerberos authentication in this config when Kerberos authentication isn't part of this config.
Am i missing something?
x
what’s your core-site.xml and hdfs-site.xml file ?
s
config files for the hdfs
x
so when you don’t configure the auth, does it work?
s
I have only tried with simple auth configuration
Copy code
<property>
    <name>hadoop.security.authentication</name>
    <value>simple</value>
  </property>
and it is still throwing kerberos issue
x
ok, can you check the code of hdfs pinot fs plugin
and try to directly run that code to init with your hdfs conf
see if that works
s
sure... let me try this
It didnt work out
x
ok can you try to fix it?
I can help you review the code and build a new docker image
s
Hey @Xiang Fu, successfully integrated HDFS as deep store in Pinot! 🎉 Fixed the auth issue by adding this line to the
init()
method in
HadoopPinotFS.java:
in pinot-hdfs plugin UserGroupInformation.setLoginUser(UserGroupInformation.createRemoteUser("user_name")); The problem: When running in K8s pods, Pinot couldn't map the numeric user ID (from securityContext) to a username for Hadoop auth. Setting an explicit username bypasses this and works without Kerberos enabled.