Hi Team, I have some data in hive table, I want to push that data into pinot, what's the best way to do it? Offline tables are fine for the use-case but I don't see any hive table connector in pinot batch spark jar.

Hi, Spark has native integration with Hive. So use spark hive connector for reading from Hive and use Spark Pinot connector to write data to Pinot. Hive <- Spark Hive Connector(read) - SparkDataFrame - Spark Pinot Connector (write) -> Pinot https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html

@Ashish Kumar Pinot currently provides Spark/MapReduce based data ingestion connectors but we don't have a way to push data directly from Hive table to Pinot. As @Ganesh mentioned, you should read your hive table from Spark/MapReduce and use our Pinot connectors to push data to Pinot

@Ganesh @Seunghyun could you point me to the pinot connectors which can be used to push data to pinot?

but this writes from file system to Pinot… I was thinking of writing spark Dataframe directly into Pinot using same spark application … something like Read Hive table using spark hive connector into dataframe and than do spark.write the dataframe into Pinot

@Ashish Kumar Sorry for the late reply. I somehow missed this. Currently, the connector requires to submit a separate job and assumes that the input is given. Dataframe write() support will need the separate connector that implement Spark’s sink interface. I will create an isue for this.