Hi team, I'm just facing this issue when i do run ...
# troubleshooting
m
Hi team, I'm just facing this issue when i do run query.
d
Your pinot broker is probably not healthy. Can you check the running status of your pods?
m
I just tried deleting the broker and restarted
and pinot stopped consuming data from kafka as well
for all tables
d
If your cluster is not healthy nothing will work as expected.
m
any other way to sort out?
d
What is the running status of your pods?
m
it's in running state
d
Any error in the logs of each pods?
m
This is for broker-0
d
what about the others?
m
This is controller-0
server-0
kindly do favor.
d
Seems like you have a schema error regarding a dateformat. Can you access the Cluster Manager UI and check if broker is live? Finally, which k8s provider are you running on?
k
Looking at the exception in the log that the time stamp format in the Kafka message does not match what you have in table schema
d
Schema error is a different problem.
m
GKE
d
I would suggest fixing your schema, then trying the query again. The error you got might be caused by a broker being restarted
m
brokers are in alive status
ok thats fine. but some table were consuming data. But now it got stopped.
can you point me which schema has error?
m
The lastest image you pasted has an error that indicates a bad date Feb 29 2021 (this year only had 28 days in Feb)
e
If you tail the server logs do you see
java.lang.OutOfMemoryError: Direct buffer memory
? The servers may be running but the direct memory may be hitting the limit.