Hi Everyone, I am trying to integrate Kerberos Hadoop with Pinot.and using below configurations. Executables: export HADOOP_HOME=/usr/hdp/2.6.3.0-235/hadoop export HADOOP_VERSION=2.7.3.2.6.3.0-235 export HADOOP_GUAVA_VERSION=11.0.2 export HADOOP_GSON_VERSION=2.2.4 export GC_LOG_LOCATION=/home/hdfs/Pinot/pinotGcLog export PINOT_VERSION=0.7.1 export PINOT_DISTRIBUTION_DIR=/home/hdfs/apache-pinot-incubating-0.7.1-bin export HADOOP_CLIENT_OPTS="-Dplugins.dir=${PINOT_DISTRIBUTION_DIR}/plugins -Dlog4j2.configurationFile=${PINOT_DISTRIBUTION_DIR}/conf/pinot-ingestion-job-log4j2.xml" export SERVER_CONF_DIR=/home/hdfs/apache-pinot-incubating-0.7.1-bin/bin export ZOOKEEPER_ADDRESS=<ZOOKEEPER_ADDRESS> export CLASSPATH_PREFIX="${HADOOP_HOME}/hadoop-hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/lib/gson-${HADOOP_GSON_VERSION}.jar" export JAVA_OPTS="-Xms4G -Xmx16G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:${GC_LOG_LOCATION}/gc-pinot-server.log" controller.conf controller.data.dir=<fs.defaultFS>/user/hdfs/controller_segment controller.local.temp.dir=/home/hdfs/Pinot/pinot_tmp/ controller.zk.str=<ZOOKEEPER_ADDRESS> controller.enable.split.commit=true controller.access.protocols.http.port=9000 controller.helix.cluster.name=PinotCluster pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS pinot.controller.storage.factory.hdfs.hadoop.conf.path=/usr/hdp/2.6.3.0-235/hadoop/conf pinot.controller.segment.fetcher.protocols=file,http,hdfs pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle='hdfs@HDFSSITHDP.COM' pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab' pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle='hdfs@HDFSSITHDP.COM' pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab' controller.vip.port=9000 controller.port=9000 pinot.set.instance.id.to.hostname=true pinot.server.grpc.enable=true Kerbeous Information: kinit -V -k -t /home/hdfs/hdfs.keytab hdfs@HDFSSITHDP.COM Using default cache: /tmp/krb5cc_57372 Using principal: hdfs@HDFSSITHDP.COM Using keytab: /home/hdfs/hdfs.keytab Authenticated to Kerberos v5 ERROR MESSAGE: END: Invoking TASK controller pipeline for event ResourceConfigChange::15fc3764_TASK for cluster PinotCluster, took 278 ms START AsyncProcess: TASK::TaskGarbageCollectionStage END AsyncProcess: TASK::TaskGarbageCollectionStage, took 0 ms Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Trying to authenticate user 'hdfs@HDFSSITHDP.COM' with keytab '/home/hdfs/hdfs.keytab'.. Could not instantiate file system for class org.apache.pinot.plugin.filesystem.HadoopPinotFS with scheme hdfs java.lang.RuntimeException: Failed to authenticate user principal ['hdfs@HDFSSITHDP.COM'] with keytab ['/home/hdfs/hdfs.keytab'] at org.apache.pinot.plugin.filesystem.HadoopPinotFS.authenticate(HadoopPinotFS.java:258) ~[pinot-hdfs-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6]     Caused by: java.io.IOException: Login failure for 'hdfs@HDFSSITHDP.COM' from keytab '/home/hdfs/hdfs.keytab': javax.security.auth.login.LoginException: Unable to obtain password from user. at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:962) ~[pinot-orc-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6] at org.apache.pinot.plugin.filesystem.HadoopPinotFS.authenticate(HadoopPinotFS.java:254) ~[pinot-hdfs-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6] ... 15 more     Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901) ~[?:1.8.0_241] at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764) ~[?:1.8.0_241] at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617) ~[?:1.8.0_241] at org.apache.pinot.plugin.filesystem.HadoopPinotFS.authenticate(HadoopPinotFS.java:254) ~[pinot-hdfs-0.7.1-shaded.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6] ... 15 more     Failed to start a Pinot [CONTROLLER] at 21.954 since launch     java.lang.RuntimeException: java.lang.RuntimeException: Failed to authenticate user principal ['hdfs@HDFSSITHDP.COM'] with keytab ['/home/hdfs/hdfs.keytab'] at org.apache.pinot.spi.filesystem.PinotFSFactory.register(PinotFSFactory.java:58) ~[pinot-all-0.7.1-jar-with-dependencies.jar:0.7.1-e22be7c3a39e840321d3658e7505f21768b228d6] P.s. I am executing this hdfs user and for keytab file also user is hdfs .I have also given 777 access to hdfs.keytab file. someone Kindly suggest What is the issue here.I have read multiple blocks and everywhere found that it is because of wrong prinicpal/keytab file combination/user don't have access/give 777 access to file/try with different user. tried all the the options but nothing worked as of now.

is this same thread as: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1624842788293400

@User Yes, issue is same.After all the changes we are facing this issue, and looks like different issue not related to configuration.So @User started the seprate thread.is it fine to keep this separate thread so shall we move this to existing thread.?

oh, you mean use

pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle=hdfs@HDFSSITHDP.COM
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab=/home/hdfs/hdfs.keytab
pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle=hdfs@HDFSSITHDP.COM
pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab=/home/hdfs/hdfs.keytab

not

pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.principle='hdfs@HDFSSITHDP.COM'
pinot.controller.segment.fetcher.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab'
pinot.controller.storage.factory.hdfs.hadoop.kerberos.principle='hdfs@HDFSSITHDP.COM'
pinot.controller.storage.factory.hdfs.hadoop.kerberos.keytab='/home/hdfs/hdfs.keytab'