I am trying to start pinot with hdfs as deep stora...
# generala
Akash
04/13/2021, 2:51 PMI am trying to start pinot with hdfs as deep storage but getting error while starting the server
Copy code
bin/start-server.sh -zkAddress pinot1.plan:2181,pinot2.plan:2181,pinot3.plan:2181 -configFileName conf/server.conf Copy code
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
hadoop.conf.path=/local/hadoop/etc/hadoop/
pinot.server.storage.factory.hdfs.hadoop.conf.path=/local/hadoop/etc/hadoop/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.instance.dataDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.6.0-bin/data/PinotServer/index
pinot.server.instance.segmentTarDir=/home/akashmishra/hpgraph/apache-pinot-incubating-0.6.0-bin/data/PinotServer/segmentTard
Daniel Lavoie
04/13/2021, 2:55 PMHi, can you move this conversation to #troubleshooting? Also, error logs would be helpful to understand what is wrong.
a
Akash
04/13/2021, 2:56 PMSorry did not knew there was such channel. Will move it there thanks
r
RK
05/08/2021, 5:06 PM@User kindly share all the details. What all changes we need to do for implementing "hdfs as deepatorage in Pinot". I couldn't find any relevant document for the same.
a
Akash
05/08/2021, 6:34 PMAkash
05/08/2021, 6:34 PMHere is a setup information.
Akash
05/08/2021, 6:37 PMTo add HDFS jar, you need to put following information in you startup script.
Copy code
export HADOOP_HOME=/local/hadoop/
export HADOOP_VERSION=2.7.1
export HADOOP_GUAVA_VERSION=11.0.2
export HADOOP_GSON_VERSION=2.2.4
export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar"Akash
05/08/2021, 6:43 PMPing me if you face any issues.
r
RK
05/09/2021, 2:10 AMOkay Thanks a lot @User. But this is for importing the data from hdfs. I am consuming data from kafka topic and loading in pinot table(already done). Now this pinot table data I want to store on HDFS.