https://pinot.apache.org/ logo
Join Slack
Powered by
Guys, I merged the latest changes into our fork, b...
# troubleshooting
a
Guys, I merged the latest changes into our fork, built again the docker image, deployed and now when we try to create a table we see this error in the controller: 
ClassNotFoundException: org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory
Any suggestions?
d
Are you running a customized PluginDir value?
a
Nope, actually we're using almost everything as it was in the first place
pluginsDir: /opt/pinot/plugins
d
what about 
plugins.include
?
a
This is the portion in values.yaml used to deploy in a kubernetes environment using helm:
sample.yaml
sample.yaml
d
can you list the content 
/opt/pinot/plugins
 of your within your forked image?
Also, can you extract your controller logs using this command?
Copy code
kubectl exec <controller-pod-name> -- cat pinotController.log > controller.log
The startup logs all plugins being loaded
a
This is what I see in 
/opt/pinot/plugins
d
If you can share your controller logs we’ll have a better understand of what is going on
a
controller.log
controller.log
d
Somehow, you have provided 
pinot-gcs
as 
plugins.include
. the plugin is loaded by default, so you don’t need specify it. Overwriting plugins.include will disable all other plugins, including the kafka one.
a
d
Yes, the documentation may induce in error since it doesn’t mention that 1- other plugins will be disabled, 2- it’s already part of the docker image.
I don’t see anything in your 
values.yaml
, so I guess it is part of your docker image fork?
a
It's in the 
jvmOpts
section of 
values.yaml
d
Not for the controller
You only shared the controller values
a
Sorry, wrong file
This is the one
sample.yaml
sample.yaml
d
anyways, just remove 
-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-gcs
from all your jvmArgs
a
Okay, I'll try that
I did that and I just deployed our table in pinot. For that table I set this properties:
Copy code
"realtime.segment.flush.threshold.time":"24h"
"realtime.segment.flush.threshold.size":"0"
"realtime.segment.flush.desired.size":"500M"
The thing is, despite having set 24 hours and 500MB as our limits in time and size for segments, I see this behavior in gcs, not sure if that's a good sign or not:
It's been consuming by roughly ten minutes or so and there is already 9 segments, one of them is not showing here yet, but look at those sizes, is that okay?
d
I’m not sure but I don’t think threshold.size 0 is helping out
Copy code
realtime.segment.flush.threshold.segment.size
realtime.segment.flush.threshold.time
realtime.segment.flush.threshold.rows
So you want something like
Copy code
"realtime.segment.flush.threshold.time":"24h",
"realtime.segment.flush.threshold.size":"500M",
"realtime.segment.flush.threshold.rows": "500000000",
Default values for
But my guess is that 500m records will always be bigger than 500MB 😛 So I would leave rows to default value.
a
Nice! I'll try those out
Thanks
So, just to confirm, this should be the properties to go on the table config, right?
Copy code
"realtime.segment.flush.threshold.time":"24h"
"realtime.segment.flush.threshold.segment.size":"500M"
"realtime.segment.flush.threshold.rows": "5000000"
d
Yes next to your kafka config
The rows config is not required since 5 million is the default
a
Ah, okay, nice
Interesting, now it consumed 5 million messages from kafka and stopped consuming, is that okay?
d
What are you logs saying?
a
Created only one segment, every time I try to get the number of records in pinot I get different results, I presume this is because the query get to a server that doesn't hold yet the whole segment, we have 2 replicas configured
d
Check the server logs
a
server.log
server.log
d
Zookeeper is down.
Copy code
21/02/12 10:26:44.302 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(mls-zookeeper.production.svc.cluster.local:2181)] Session 0x100722c6f210005 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
Unless that is an old message. Can you share the controller logs too?
a
No, actually it's been quite stable for now
I think that might be because I redeployed everything to use gcs in an attempt to configure it as the deep store
Another thing is, It stopped consuming after reaching 5 million messages but still hasn't stored the segment in gcs as it was doing previously
d
Can you share the controller logs?
there’s a high chance it’s simply failing to store the segment
General tip, whenever something is not working as expected, always check all the logs first
a
But I don't see any errors regarding gcs here
controller (1).log
d
How many controllers do you have?
a
3
d
get the logs from all of them 🙂
a
Okay, hold on
I see more data in controller 2,
controller-1.logcontroller-2.logcontroller-0.log
It took it quite some time to upload the segment data to gcs, but now I see it there. It didn't reach the 500MB goal though
Screenshot 2021-02-12 at 16.20.14.png
I saw a log regarding the minion component, I only started 1 instance with the default config values because I wasn't sure how we were supposed to use it
Now I see pinot started consuming again
Didn't change anything, I was just getting the logs out of every controller instance
d
how many servers do you have?
a
We have 3 servers
d
pinot-server
a
Yep, 3 pinot server instances
d
ok, keep monitoring and tell me if you keep witnessing the same behavior. You could also increase the 
rows
value 10 000 000, your segments should double in size
a
Okay, and then regarding the minion instance, should I also provide the same resources to it? How many minion instances should we have with this architecture?
d
Minion is used for operational tasks such as scheduled batch ingestion.
If you only need realtime streaming, you don’t need minion instances
a
Okay, then if I understood correctly, if we were to have offline tables and ingest data from any other source then we should increase the minion resources and instances. Is that correct?
d
Do your ingestion need, yes
Typically, a bunch of workers for an initial load, then just the right amount for the periodic new files to ingest
a
Ah, okay, so it depends on the workload for the batch ingestion
Many thanks!
d
Yes, it scales horizontally pretty well, it’s usually CPU bottlenecked because it’s responsible for generating the segments.
a
It works perfectly!
Last question, can we use this https://docs.pinot.apache.org/basics/data-import/pinot-file-system/import-from-gcp#job-spec to import data exported from bigquery?
d
As long as your can export the data with a schema matching the one of pinot, yes you can
a
Thanks
Open in Slack
PreviousNext
Powered by