Alexander Vivas
02/11/2021, 3:04 PMClassNotFoundException: org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory
Any suggestions?Daniel Lavoie
02/11/2021, 3:05 PMAlexander Vivas
02/11/2021, 3:07 PMpluginsDir: /opt/pinot/plugins
Daniel Lavoie
02/11/2021, 3:08 PMplugins.include
?Alexander Vivas
02/11/2021, 3:08 PMDaniel Lavoie
02/11/2021, 3:12 PM/opt/pinot/plugins
of your within your forked image?kubectl exec <controller-pod-name> -- cat pinotController.log > controller.log
The startup logs all plugins being loadedAlexander Vivas
02/11/2021, 3:21 PM/opt/pinot/plugins
Daniel Lavoie
02/11/2021, 3:22 PMAlexander Vivas
02/11/2021, 3:40 PMDaniel Lavoie
02/11/2021, 3:42 PMpinot-gcs
as plugins.include
. the plugin is loaded by default, so you don’t need specify it. Overwriting plugins.include will disable all other plugins, including the kafka one.Alexander Vivas
02/11/2021, 3:42 PMDaniel Lavoie
02/11/2021, 3:43 PMvalues.yaml
, so I guess it is part of your docker image fork?Alexander Vivas
02/11/2021, 3:45 PMjvmOpts
section of values.yaml
Daniel Lavoie
02/11/2021, 3:45 PMAlexander Vivas
02/11/2021, 3:49 PMDaniel Lavoie
02/11/2021, 3:50 PM-Dplugins.dir=/opt/pinot/plugins -Dplugins.include=pinot-gcs
from all your jvmArgsAlexander Vivas
02/11/2021, 3:51 PM"realtime.segment.flush.threshold.time":"24h"
"realtime.segment.flush.threshold.size":"0"
"realtime.segment.flush.desired.size":"500M"
The thing is, despite having set 24 hours and 500MB as our limits in time and size for segments, I see this behavior in gcs, not sure if that's a good sign or not:Daniel Lavoie
02/12/2021, 1:17 PMrealtime.segment.flush.threshold.segment.size
realtime.segment.flush.threshold.time
realtime.segment.flush.threshold.rows
"realtime.segment.flush.threshold.time":"24h",
"realtime.segment.flush.threshold.size":"500M",
"realtime.segment.flush.threshold.rows": "500000000",
Default values forAlexander Vivas
02/12/2021, 1:30 PM"realtime.segment.flush.threshold.time":"24h"
"realtime.segment.flush.threshold.segment.size":"500M"
"realtime.segment.flush.threshold.rows": "5000000"
Daniel Lavoie
02/12/2021, 2:09 PMAlexander Vivas
02/12/2021, 2:10 PMDaniel Lavoie
02/12/2021, 2:34 PMAlexander Vivas
02/12/2021, 2:34 PMDaniel Lavoie
02/12/2021, 2:35 PMAlexander Vivas
02/12/2021, 2:38 PMDaniel Lavoie
02/12/2021, 2:40 PM21/02/12 10:26:44.302 WARN [ClientCnxn] [Start a Pinot [SERVER]-SendThread(mls-zookeeper.production.svc.cluster.local:2181)] Session 0x100722c6f210005 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
Alexander Vivas
02/12/2021, 2:41 PMDaniel Lavoie
02/12/2021, 2:46 PMAlexander Vivas
02/12/2021, 2:56 PMDaniel Lavoie
02/12/2021, 2:57 PMAlexander Vivas
02/12/2021, 2:57 PMDaniel Lavoie
02/12/2021, 2:57 PMAlexander Vivas
02/12/2021, 2:58 PMDaniel Lavoie
02/12/2021, 3:22 PMAlexander Vivas
02/12/2021, 3:22 PMDaniel Lavoie
02/12/2021, 3:22 PMAlexander Vivas
02/12/2021, 3:23 PMDaniel Lavoie
02/12/2021, 3:23 PMrows
value 10 000 000, your segments should double in sizeAlexander Vivas
02/12/2021, 3:24 PMDaniel Lavoie
02/12/2021, 3:25 PMAlexander Vivas
02/12/2021, 3:26 PMDaniel Lavoie
02/12/2021, 3:27 PMAlexander Vivas
02/12/2021, 3:28 PMDaniel Lavoie
02/12/2021, 3:29 PMAlexander Vivas
02/12/2021, 6:28 PMDaniel Lavoie
02/12/2021, 6:30 PMAlexander Vivas
02/12/2021, 6:30 PM