Hi all, I have ran into an issue with pinot minio...
# troubleshooting
v
Hi all, I have ran into an issue with pinot minion with kerberized hdfs where pinot minion upsert tasks fails with below error:
Copy code
UI:
org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:65)
	at org.apache.pinot.common.utils.fetcher.BaseSegmentFetcher.fetchSegmentToLocal(BaseSegmentFetcher.java:74)
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:124)
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchSegmentToLocal(SegmentFetcherFactory.java:132)
	at org.apache.pinot.common.utils.fetcher.SegmentFetcherFactory.fetchAndDecryptSegmentToLocal(SegmentFetcherFactory.java:165)
	at org.apache.pinot.plugin.minion.tasks.BaseTaskExecutor.downloadSegmentToLocal(BaseTaskExecutor.java:121)
	at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConversionExecutor.java:105)
	at org.apache.pinot.plugin.minion.tasks.BaseSingleSegmentConversionExecutor.executeTask(BaseSingleSegmentConv

Minion log shows:
2025/07/02 13:23:21.309 WARN [PinotFSSegmentFetcher] [TaskStateModelFactory-task_thread-3] Caught exception while fetching segment from: hdfs://xxxxxxx/controller_data/xxxxxxx/xxxxxxx__4__648__20250528T1621Z to: /tmp/PinotMinion/data/UpsertCompactionTask/tmp-9727a6d3-cc2d-44d0-9666-34939abbc356/tarredSegment
org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
        at jdk.internal.reflect.GeneratedConstructorAccessor43.newInstance(Unknown Source) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[?:?]
        at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121) ~[pinot-parquet-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88) ~[pinot-parquet-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1741) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1829) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1826) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[pinot-parquet-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
        at org.apache.hadoop.hdfs.DistributedFil
Pinot Server, Broker, and Controller are all able to read/write segments from HDFS using the configured keytab. I have recently added 3 Pinot Minion on new nodes, configured with the same keytab, principal, and Hadoop config path. However, when the Minion runs tasks like UpsertCompaction, it fails with the above error: Minion runs under pinot user (systemd) kinit is successful, and Kerberos ticket is visible via klist
pinot.minion.segment.fetcher.hdfs.hadoop.kerberos.principal=xxxxxx@xxxxxx
pinot.minion.segment.fetcher.hdfs.hadoop.kerberos.keytab=/etc/security/keytabs/pinot.keytab
pinot.minion.storage.factory.hdfs.hadoop.conf.path=/usr/hdp/xxxxx/hadoop/conf
Is there anything else Pinot Minion needs to perform Kerberos login internally? Does it require JAAS config explicitly even with keytab/principal settings?