Samridhi
04/20/2025, 7:27 PM2025/04/20 19:16:36.464 ERROR [PinotFSFactory] [Start a Pinot [SERVER]] Could not instantiate file system for class org.apache.pinot.plugin.filesystem.HadoopPinotFS with scheme hdfs
java.lang.RuntimeException: Could not initialize HadoopPinotFS
at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:71) ~[pinot-hdfs-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.server.starter.helix.BaseServerStarter.initSegmentFetcher(BaseServerStarter.java:925) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.server.starter.helix.BaseServerStarter.start(BaseServerStarter.java:604) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:167) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:97) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:279) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:305) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:279) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
at jdk.security.auth/com.sun.security.auth.UnixPrincipal.init(UnixPrincipal.java:71)
at jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:134)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2065)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1975)
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579)
at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3746)
at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3736)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3520)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:67)
at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52)
at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72)
at org.apache.pinot.server.starter.helix.BaseServerStarter.initSegmentFetcher(BaseServerStarter.java:925)
at org.apache.pinot.server.starter.helix.BaseServerStarter.start(BaseServerStarter.java:604)
at org.apache.pinot.tools.service.PinotServiceManager.startServer(PinotServiceManager.java:167)
at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:97)
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.lambda$run$0(StartServiceManagerCommand.java:279)
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:305)
at org.apache.pinot.tools.admin.command.StartServiceManagerCommand$1.run(StartServiceManagerCommand.java:279)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1986) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:719) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:669) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:579) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3746) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache$Key.init(FileSystem.java:3736) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3520) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540) ~[hadoop-common-3.3.4.jar:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288) ~[hadoop-common-3.3.4.jar:?]
at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:67) ~[pinot-hdfs-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
... 9 more
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
Please help in resolving this. I'm clueless
Note : Pinot version is 1.2.0 and hdfs version is 3.3.4
Pinot config looks like this
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.server.storage.factory.hdfs.hadoop.conf.path=/etc/hadoop-conf/conf/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.grpc.enable=true
pinot.server.grpc.port=8090Xiang Fu
apachepinot/pinot:release-1.3.0-hdfs-patch-17-ms-openjdkSamridhi
04/21/2025, 7:54 AMXiang Fu
Samridhi
04/21/2025, 8:46 AMproperty
namehadoop.security.authentication/name
valuesimple/value
/property
the pinot config looks like this
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.server.storage.factory.hdfs.hadoop.conf.path=/etc/hadoop-conf/conf/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.grpc.enable=true
pinot.server.grpc.port=8090Xiang Fu
Xiang Fu
Xiang Fu
/etc/hadoop-conf/conf/ right?Samridhi
04/21/2025, 9:07 AM/etc/hapdoop-conf/conf
2. libs are also mouthed at /usr/lib/hadoop-home/hadoop-hdfs and provided in pinot server as
CLASSPATH_PREFIX with values below
/usr/lib/hadoop-home/hadoop-hdfs/hadoop-hdfs-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-annotations-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-auth-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-common-3.3.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/guava-27.0-jre.jar:/usr/lib/hadoop-home/hadoop-hdfs/gson-2.8.9.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-hdfs-native-client.jarSamridhi
04/21/2025, 9:08 AMSamridhi
04/21/2025, 9:24 AMSamridhi
04/21/2025, 9:24 AMXiang Fu
Samridhi
04/21/2025, 9:48 AMXiang Fu
Samridhi
04/21/2025, 9:51 AM<property>
<name>hadoop.security.authentication</name>
<value>simple</value>
</property>
and it is still throwing kerberos issueXiang Fu
Xiang Fu
Xiang Fu
Samridhi
04/21/2025, 10:26 AMSamridhi
04/22/2025, 4:24 AMXiang Fu
Xiang Fu
Samridhi
04/24/2025, 10:02 AMinit() method in HadoopPinotFS.java: in pinot-hdfs plugin
UserGroupInformation.setLoginUser(UserGroupInformation.createRemoteUser("user_name"));
The problem: When running in K8s pods, Pinot couldn't map the numeric user ID (from securityContext) to a username for Hadoop auth. Setting an explicit username bypasses this and works without Kerberos enabled.