Hello Team, We are facing below error while integr...
# troubleshooting
r
Hello Team, We are facing below error while integrating HDFS as deep store with pinot 1.3
Copy code
2025/08/12 05:11:57.783 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 3.746 since launch
java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder
    at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42) ~[hadoop-common-3.2.4.jar:?]
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3370) ~[hadoop-common-3.2.4.jar:?]
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:125) ~[hadoop-common-3.2.4.jar:?]
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3424) ~[hadoop-common-3.2.4.jar:?]
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3392) ~[hadoop-common-3.2.4.jar:?]
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:485) ~[hadoop-common-3.2.4.jar:?]
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:233) ~[hadoop-common-3.2.4.jar:?]
    at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:68) ~[pinot-hdfs-1.3.0.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.controller.BaseControllerStarter.initPinotFSFactory(BaseControllerStarter.java:759) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:450) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:397) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:240) ~[pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:293) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:239) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:183) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:180) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine.executeUserObject(CommandLine.java:2045) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine.access$1500(CommandLine.java:148) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2457) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2419) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine$RunLast.execute(CommandLine.java:2421) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at picocli.CommandLine.execute(CommandLine.java:2174) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:174) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
    at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:210) [pinot-all-1.3.0-jar-with-dependencies.jar:1.3.0-c0023da298126af6a01b802a04b66da34ba16134]
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ~[?:?]
    ... 32 more
And our CLASSPATH_PREFIX env var looks like below:
Copy code
"/usr/lib/hadoop-home/hadoop-hdfs/hadoop-hdfs-3.2.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-annotations-3.2.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-auth-3.2.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-common-3.2.4.jar:/usr/lib/hadoop-home/hadoop-hdfs/guava-27.0-jre.jar:/usr/lib/hadoop-home/hadoop-hdfs/gson-2.9.0.jar:/usr/lib/hadoop-home/hadoop-hdfs/hadoop-hdfs-native-client.jar"
Any suggestions here?
c
I had issues with hdfs too in 1.3.0. I used an image with hdfs patch and they were fixed. I think the patch was said to be added in 1.4.0
r
Yeah, for us the issue was even with hadoop version. Pinot 1.3 works properly with 3.3.X and greater version of hadoop. We were trying with older version.
v
Therea a known issue in pinot 1.3 where hdfs as deep storage doesn't work. Check this https://apache-pinot.slack.com/archives/C011C9JHN7R/p1745203533389429?thread_ts=1745177277.188329&cid=C011C9JHN7R
👍 1
a
hi @Rohini Choudhary
whats your config for deepstore in controller and server
r