hi i try to use hdfs with pinot follow document --> https://docs.pinot.apache.org/basics/getting-started/hdfs-as-deepstorage it not working ###my config## ####controller config## pinot.service.role=CONTROLLER pinot.cluster.name=pinot-uat controller.host=pinot-uat01 controller.data.dir=hdfs://path/in/hdfs/for/controller/segment controller.local.temp.dir=/tmp/pinot/data/controller controller.zk.str=172.19.131.1162181,172.19.131.1172181,172.19.131.118:2181 controller.enable.split.commit=true controller.access.protocols.http.port=9000 controller.helix.cluster.name=pinot-uat pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.controller.storage.factory.hdfs.hadoop.conf.path=/etc/hadoop/conf pinot.controller.segment.fetcher.protocols=file,http,hdfs pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle=hdptest@TRUE.CARE pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab=/data/apache-pinot/keytab/hdptest.keytab controller.vip.host=pinotuat.true.care controller.vip.port=9000 controller.port=9000 pinot.set.instance.id.to.hostname=true pinot.server.grpc.enable=true ########Executable export HADOOP_HOME=/usr/lib/hadoop export HADOOP_VERSION=2.6.0-cdh5.16.2 export HADOOP_GUAVA_VERSION=11.0.2 export HADOOP_GSON_VERSION=2.2.4 export GC_LOG_LOCATION=/data/apache-pinot/logs/ export PINOT_VERSION=0.8.0 export PINOT_DISTRIBUTION_DIR=/data/apache-pinot export SERVER_CONF_DIR=/data/apache-pinot/conf export ZOOKEEPER_ADDRESS=172.19.131.1162181,172.19.131.1172181,172.19.131.118:2181 export CLASSPATH_PREFIX="${HADOOP_HOME}/client/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/client/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/client/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/client/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/client/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/client/gson-${HADOOP_GSON_VERSION}.jar" export JAVA_OPTS="-Xms8G -Xmx12G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:${GC_LOG_LOCATION}/gc-pinot-controller.log" ${PINOT_DISTRIBUTION_DIR}/bin/start-controller.sh -configFileName ${SERVER_CONF_DIR}/pinot-controller.conf ###########error log#### 2022/03/10 120641.771 INFO [StartControllerCommand] [main] Executing command: StartController -configFileName /data/apache-pinot/conf/pinot-controller.conf 2022/03/10 120641.843 INFO [StartServiceManagerCommand] [main] Executing command: StartServiceManager -clusterName pinot-uat -zkAddress 172.19.131.1162181,172.19.131.1172181,172.19.131.118:2181 -port -1 -bootstrapServices [] 2022/03/10 120641.843 INFO [StartServiceManagerCommand] [main] Starting a Pinot [SERVICE_MANAGER] at 0.012s since launch 2022/03/10 120641.847 INFO [StartServiceManagerCommand] [main] Started Pinot [SERVICE_MANAGER] instance [ServiceManager_poc-pinot01_-1] at 0.016s since launch 2022/03/10 120641.848 INFO [StartServiceManagerCommand] [main] Starting a Pinot [CONTROLLER] at 0.016s since launch WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/lib/hadoop/hadoop-auth-2.6.0-cdh5.16.2.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder at org.apache.hadoop.fs.FsTracer.get(FsTracer.java:42) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2803) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:186) at org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:65) at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:52) at org.apache.pinot.spi.filesystem.PinotFSFactory.init(PinotFSFactory.java:72) at org.apache.pinot.controller.BaseControllerStarter.initPinotFSFactory(BaseControllerStarter.java:518) at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:358) at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:308) at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:123) at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:93) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:233) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:285) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:232) at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:182) at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:149) at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:166) at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:186) at org.apache.pinot.tools.admin.PinotController.main(PinotController.java:35) Caused by: java.lang.ClassNotFoundException: org.apache.htrace.core.Tracer$Builder at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)

What version of Hadoop and Java? Cc: @User

export HADOOP_VERSION=2.6.0-cdh5.16.2 java --> java-11-openjdk.x86_64 (because use pinot 0.8.0 , in need to use newest version , first time i test with 0.9.3 it's error too.

can you try to put Pinot-hdfs shaded jar into your class path?

you mean export CLASSPATH_PREFIX=/data/apache-pinot/plugins/pinot-file-system/pinot-hdfs/pinot-hdfs-0.8.0-shaded.jar ?

Hmm Interesting, seems like still missing the lib

on hadoop lib ?

hmm, still same error?

Hadoop 2.6.0-cdh5.16.2

sure, meanwhile you can try the standalone job for segment ingestion

normally how pinot keep segment data.

Pinot stores a copy of data on deep store (hdfs/s3/etc). But Pinot servers keep a local copy for serving.

deepp store no need to input in controoler config ?

Yes deepstore needs to be configured on controller and sever. But that is separate from segment generation job on hdfs

oh it mean we no need to use deep storage (hdfs) if we use standalone job. how about table realtime (kafka) ?

No, I am saying that they are unrelated to each other. Controller data is stored in deep store (for real-time also)

thank you.