Hi everyone!, I'm trying to configure Pinot with D...
# troubleshooting
t
Hi everyone!, I'm trying to configure Pinot with Deep Store using HDFS within a Docker setup. I haven't been able to find specific documentation for this. Has anyone successfully done this before or have any pointers on the necessary configurations? Any help would be greatly appreciated!
@Xiang Fu
in docker env, you need to mount your own hdfs conf or kerberos stuffs and passing correct environment variables
also ensure hdfs is accessible through your dns setup for the docker network
t
I tried converting it to a Docker-based setup and mounting the Hadoop config into the container using the following command: sudo docker run --rm -ti \ --name pinot-ingestion-job \ --network=pinot-net \ --mount type=bind,source=/home/admin/pinot/Tables,target=/ingestion \ --mount type=bind,source=/home/admin/pinot/config/hdfs_config,target=/hadoop-conf \ apachepinot/pinot:1.2.0 \ LaunchDataIngestionJob -jobSpecFile /ingestion/MPS_ADMN_PYMT_MAST.yaml However, I’m hitting the following error during ingestion: java.lang.IllegalArgumentException: Wrong FS: hdfs://..., expected: file:/// From the logs, it looks like the Pinot ingestion job is trying to use the local file system (file:///) instead of HDFS (hdfs://...). It seems like the Hadoop configuration is either not being picked up properly, or the HadoopPinotFS plugin is not configured correctly within the Docker container. Could you please advise if there's a specific way to ensure the HDFS configuration is loaded correctly in the Docker environment? Or is there any additional setup or package needed inside the container to make Pinot recognize and use HDFS?
x
for the file
/ingestion/MPS_ADMN_PYMT_MAST.yaml
did you configure HDFS pinotFS and use hdfs:// in the input/output file path?
t
x
what’s the stacktraces
t
admin@kube-master-1:~$ sudo docker run --rm -ti --name pinot-ingestion-job --network=pinot-net --mount type=bind,source=/home/admin/pinot/Tables,target=/ingestion --mount type=bind,source=/home/admin/pinot/config/hdfs_config,target=/hadoop-conf apachepinot/pinot:1.2.0 LaunchDataIngestionJob -jobSpecFile /ingestion/MPS_ADMN_PYMT_MAST.yaml 2025/05/07 084500.945 ERROR [LaunchDataIngestionJobCommand] [main] Got exception to kick off standalone data ingestion job - java.lang.IllegalArgumentException: Wrong FS: hdfs//10.44.251.2479000/pinot/PinotCluster_test/segments/MPS_ADMN_PYMT_MAST, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:824) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:119) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:910) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1236) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:905) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1862) ~[pinot-orc-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.plugin.filesystem.HadoopPinotFS.exists(HadoopPinotFS.java:133) ~[pinot-hdfs-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.spi.filesystem.NoClosePinotFS.exists(NoClosePinotFS.java:74) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.init(SegmentGenerationJobRunner.java:127) ~[pinot-batch-ingestion-standalone-1.2.0-shaded.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:148) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:129) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:132) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine.executeUserObject(CommandLine.java:2045) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine.access$1500(CommandLine.java:148) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine$RunLast.handle(CommandLine.java:2457) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine$RunLast.handle(CommandLine.java:2419) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine$RunLast.execute(CommandLine.java:2421) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at picocli.CommandLine.execute(CommandLine.java:2174) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:173) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:204) [pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a] java.lang.IllegalArgumentException: Wrong FS: hdfs//10.44.251.2479000/pinot/PinotCluster_test/segments/MPS_ADMN_PYMT_MAST, expected: file:/// at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:824) at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:119) at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:910) at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:1236) at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:905) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:462) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1862) at org.apache.pinot.plugin.filesystem.HadoopPinotFS.exists(HadoopPinotFS.java:133) at org.apache.pinot.spi.filesystem.NoClosePinotFS.exists(NoClosePinotFS.java:74) at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.init(SegmentGenerationJobRunner.java:127) at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:148) at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:129) at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:132) at org.apache.pinot.tools.Command.call(Command.java:33) at org.apache.pinot.tools.Command.call(Command.java:29) at picocli.CommandLine.executeUserObject(CommandLine.java:2045) at picocli.CommandLine.access$1500(CommandLine.java:148) at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465) at picocli.CommandLine$RunLast.handle(CommandLine.java:2457) at picocli.CommandLine$RunLast.handle(CommandLine.java:2419) at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277) at picocli.CommandLine$RunLast.execute(CommandLine.java:2421) at picocli.CommandLine.execute(CommandLine.java:2174) at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:173) at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:204)
x
can u paste the yaml file?
t
executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' segmentMetadataPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentMetadataPushJobRunner' jobType: SegmentCreationAndMetadataPush inputDirURI: 'hdfs//10.44.251.2479000/pinot/raw-data/MPS_ADMN_PYMT_MAST/' includeFileNamePattern: 'glob:*/.csv' outputDirURI: 'hdfs//10.44.251.2479000/pinot/segments/MPS_ADMN_PYMT_MAST' overwriteOutput: true segmentCreationJobParallelism: 5 pinotFSSpecs: - scheme: hdfs className: org.apache.pinot.plugin.filesystem.HadoopPinotFS configs: hadoop.conf.path: '/hadoop-conf' recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' tableSpec: tableName: 'MPS_ADMN_PYMT_MAST' schemaURI: 'http//pinot controller9000/tables/MPS_ADMN_PYMT_MAST/schema' tableConfigURI: 'http//pinot controller9000/tables/MPS_ADMN_PYMT_MAST' pinotClusterSpecs: - controllerURI: 'http//pinot controller9000' pushJobSpec: pushAttempts: 2 pushRetryIntervalMillis: 1000
x
which hadoop version are you using
did you configure those env variables:
Copy code
You must provide proper Hadoop dependencies jars from your Hadoop installation to your Pinot startup scripts.

Copy
export HADOOP_HOME=/local/hadoop/
export HADOOP_VERSION=2.7.1
export HADOOP_GUAVA_VERSION=11.0.2
export HADOOP_GSON_VERSION=2.2.4
export CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar"