https://pinot.apache.org/ logo
a

Arun Vasudevan

05/06/2021, 10:00 PM
Hi…When trying to
Upload the schema and Table Config
- https://docs.pinot.apache.org/basics/getting-started/pushing-your-streaming-data-to-pinot#uploading-your-schema-and-table-config I am getting the following error…
Copy code
Sending request: <http://pinot-quickstart:9000/schemas> to controller: ea8d7bfc16ea, version: Unknown
{"code":500,"error":"org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata"}
However I am able to login to my kafka docker and describe and that works fine…
Copy code
bash-4.4# bin/kafka-topics.sh --bootstrap-server kafka:9092 --topic transcript-topic --describe
Topic: transcript-topic	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: transcript-topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Any idea where i am missing?
d

Daniel Lavoie

05/06/2021, 10:06 PM
Tumeout expiered while fetching topic metadata
is typically a sign that pinot was able to connect to your kafka instance but failing to load the topic. Double check the topic name
potentially the kafka hostname might also be the problem
a

Arun Vasudevan

05/06/2021, 10:26 PM
hmm….When i try to describe the topic specifying the hostname it works…
Copy code
incubator-pinot (master) ✗ docker exec \
  -t kafka \
  /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka:9092 \
  --topic transcript-topic --describe
Topic: transcript-topic	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: transcript-topic	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
Doesn’t this mean its already able to recognize the host.
Pinot UI is also able to list out the topics….
Apologies for some newbie questions here….. In the Streaming Example - https://docs.pinot.apache.org/basics/getting-started/running-pinot-in-docker#streaming-example I see that it says it starts the Pinot deployment by starting
Zookeeper
, `Kafka`…. But, i don’t see any other docker containers started other than
pinot-quickstart
….
k

Kishore G

05/06/2021, 11:30 PM
quickstart starts everything in one process
x

Xiang Fu

05/06/2021, 11:46 PM
you can follow this doc for running all components in different docker container: https://docs.pinot.apache.org/basics/getting-started/advanced-pinot-setup
a

Arun Vasudevan

05/06/2021, 11:56 PM
Awesome this is what i was looking for. Thanks! ☝️
Where should the
schemaFile
and the
tableconfigFile
be placed? Tried it with placing it in local as well as in controller..didn’t work..
Copy code
docker run \
    --network=pinot-demo \
    --name pinot-streaming-table-creation \
    0e536a319df3 AddTable \
    -schemaFile /tmp/pinot-quick-start/transcript-schema.json \
    -tableConfigFile /tmp/pinot-quick-start/transcript-table-realtime.json \
    -controllerHost pinot-controller \
    -controllerPort 9000 \
    -exec
Steps until here ☝️: • As of now running all the components in different docker containers…. • Created a kafka topic • Facing
FileNotFoundException
while creating table and schema
Tried it with placing it in local as well as in controller..didn’t work..
x

Xiang Fu

05/07/2021, 1:27 AM
You need to use -v to mount your local disk to docker container
Then you can use those files in the docker container
a

Arun Vasudevan

05/07/2021, 12:51 PM
Thanks now its able to pickup the schemaFile, but I am back to the old issue of
Timeout expired while fetching topic metadata
when trying with the docker command. So, i tried to execute
pinot-admin.sh
within the Controller Container… But, I am getting an exception
Schema is incompatible with tableConfig
Copy code
2021/05/07 12:36:02.787 ERROR [PinotAdministrator] [main] Exception caught:
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 400 (Bad Request) with reason: "Cannot add invalid schema: transcript. Reason: Schema is incompatible with tableConfig with name: transcript_REALTIME and type: REALTIME" while sending request: <http://172.18.0.3:9000/schemas> to controller: 31edf0f41322, version: Unknown
I am using the
Schema
and
Table Config
from this page - https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion Kindly advice!
I do see the schema is created thru the UI but the table is not created…
Found out the issue…. By just running the command
Copy code
bin/pinot-admin.sh AddTable     -tableConfigFile /tmp/pinot-quick-start/transcript-table.json     -exec
as the schema was created earlier it pointed out the issue in the
timestamp
field. So Fixed it in the table creation to have the same name as schema
Copy code
"segmentsConfig": {
    "timeColumnName": "timestampInEpoch",
Pushed a documentation PR see if this makes sense - https://github.com/pinot-contrib/pinot-docs/pull/29
m

Mayank

05/07/2021, 5:37 PM
Thanks a lot @Arun Vasudevan for debugging the issue almost on your own and contributing back to the community. We thrive because of members like you, much appreciated.
🎉 1