Hi all , I am trying to push hdfs data in hybrid table. I have added offline table in pinot and now trying to push the hdfs file. When I am executing the final Hadoop jar command. It's showing pinot-plugins.tar.gz doesn't exist. Someone kindly suggest. Error: File file:/home/rah/hybrid/staging/pinot-plugin.tar.gz doesn't exits. I am attaching my config file. Here /user/hdfs is my hdfs location and /home/rah is local location .P.s. for staging and outputdir if I am giving hdfs Location then it's giving error. "Wrong FS:" hdfs://location-of- inputdir/filename.txt, expected: file:/// @Ken Krugler @Elon @Alexander Pucher @Ting Chen @Neha Pawar @Xiang Fu @Mayank Kindly suggest.

Hi @RK are you using the gcs plugin? Or are you on s3?

I will take a look the plugin jar issue for hdfs

Hi @Elon I m using hdfs

A few issues with your job spec: 1. You need to use

<hdfs://xxx>

for your staging directory. 2. You need to use

<hdfs://xxx>

for your

outputDirURI

.

@Xiang Fu @Ken Krugler this is complete log details when I am using local path as staging and output dir.

so i guess you start the job from your local this means hadoop job tries to add this uri into dist cache

/home/rah/hybrid/staging/pinot-plugins.tar.gz

Ok @Xiang Fu hadoop jar \ ${PINOT_DISTRIBUTION_DIR}/lib/pinot-all-${PINOT_VERSION}-jar-with- dependencies.jar \ org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand \ -jobSpecFile ${PINOT_DISTRIBUTION_DIR}/examples/batch/airlineStats/hadoopIngestionJobSpechybrid.yaml

this staging dir should be on hdfs as well I think

Ok @Xiang Fu let me try

Your

hadoop.conf.path

is in the wrong section. You have it as part of the

file

specification, but it needs to be part of the

hdfs

specification.

Error logs

@Ken Krugler huge thanks! we should document this into FAQ

This is the final file @Xiang Fu .

got it, so staging dir is hdfs and adding

hadoop.conf.path

Yes staging/ output/ input all are hdfs

It’s expected behavior, please see this: https://docs.pinot.apache.org/basics/components/table#hybrid-table

Oh okay @Xiang Fu so is there any way to test this hybrid table? like individually I have tested and I am able to see both real time as well as batch data, but if I want to explain it to someone like how hybrid storing both realtime and batch data then how should I prepare my Kafka and batch data to store both kind of data in pinot hybrid table.kindly suggest.

then you can prepare multiple days data

Ok @Xiang Fu so I am using current_ts column as timestamp in pinot. for realtime data the current_ts value is today's date. When I am doing select * from hybrid_realtime I am able to see the same current_ts. And from hdfs I m loading old data where current_ts is last year's 2020 date . I am able to see the same in hybrid_offline. But when I am doing select * from hybrid it's only showing Kafka topics current_ts same as hybrid_realtime and not the last year's current_ts which is available in hybrid_offline

What’s the max ts from offline table

Yes in offline table for testing I have loaded only 1 day data i.e. 30 aug 2020 data

Ok

Got it so if I load data for 29 and 30 then I should be able to see 29th from batch and today's date data from realtime

Yes

Ok Thanks alot @Xiang Fu let me try will update you if it's worksa

Usually we expect data overlapping between offline and real-time table

Ohh ok yes in real time scenario we will have overlapping data but for now for testing purpose I am creating data manually

Hi @RK - please start a new thread per question, it helps everyone else find useful conversations, thanks!

Sure @Ken Krugler