Hello, I'm new to Apache Pinot. I'm trying to run ...
# troubleshooting
r
Hello, I'm new to Apache Pinot. I'm trying to run it locally, and ingest data from Kafka topic into Pinot. I tried setting up Schema and TableConfig. But when I run LaunchDataIngestionJob . It fails.
If someone can please assist 🙏
m
For Kafka, you don’t need to launch any job. If the connection is set up correctly it should auto consume
r
It's not streaming data, it's a historical load kind of thing
a
For streaming data you dont need any job as Mayank mentioned. https://docs.pinot.apache.org/basics/getting-started/create-and-update-table-config
r
Hi Anand, But what if it's a batch ingestion ? It's a historical load data.
m
You said it fails, can you provide more info
r
# job-spec.yml
jobType: SegmentCreationAndPush inputDirURI: "" outputDirURI: "/tmp/pinot-segments" overwriteOutput: true tableName: external_provider_schema_OFFLINE schemaFile: /path/to/schema.json tableConfigFile: /path/to/tableConfig.json recordReaderSpec: dataFormat: "json" className: "org.apache.pinot.plugin.inputformat.json.JSONRecordReader" ingestionConfig: batchIngestionConfig: batchType: "kafka" kafkaIngestionConfig: kafkaTopicName: "myKafkaTopic" kafkaBrokerList: "localhost:9093" kafkaConsumerProps: auto.offset.reset: "earliest" group.id: "pinot-batch-ingestion" security.protocol: "SSL" ssl.truststore.location: "/path/to/truststore.jks" ssl.truststore.password: "yourTruststorePassword" ssl.keystore.location: "/path/to/keystore.jks" ssl.keystore.password: "yourKeystorePassword" ssl.key.password: "yourKeyPassword" startOffset: "0" endOffset: "100000" # Set this according to your batch size pushJobSpec: pushAttempts: 3 pushRetryIntervalMillis: 10000 pushParallelism: 2 pushToHost: "http://localhost:9000"
This is similar to my job-spec.yml It throws error :
Got exception to create to generate ingestionjobspec
m
You are specifying Kafka for batch ingestion. Kafka is for streaming ingestion
r
batchType: "kafka" -- Are you referring to this ?
m
Yes, that is incorrect
👍 1
r
Okay, but if a scenario has historical or batch data on Kafka topic, then what should I write as batchType ? Do you have any sample job-spec.yml for this scenario ?
m
Please understand Kafka is not a batch system.
You want to batch ingestion data that is at rest on some system like S3, or some data lake
What I think you might mean is there is older data in Kafka, or you are pushing data from your batch system into Kafka? If so, as far as Pinot is concerned it is a stream (it doesn’t matter if data is old or new), you can set real-time ingestion for data in Kafka
k
@Rishika We had a similar scenario. To solve the problem, we set up a real-time table like this: https://docs.pinot.apache.org/manage-data/data-import/pinot-stream-ingestion/import-from-apache-kafka. If needed, you can later move the data into an offline table: https://docs.pinot.apache.org/operators/operating-pinot/pinot-managed-offline-flows We haven’t done that part ourselves, so I can’t confirm how well it works. However, loading data from Kafka into Pinot in the real-time case was very fast.
thanks 1
👍 1