do you limit the container size?

Seems pinot broker got shutdown by k8s

if you limit the container cpu/memory

then this shouldn’t happen

also tries to make init and limit to be same

so container size won’t change

ok i'll try it... Thanks!

It worked! Thanks

Hi All, I have an use case where I get a hight volume of financial time series data (Bid and Ask prices from an exchange) and my queries are like if the last two entries for a trade is less than a threshold then alert or if there is no update for a trade in the last 6 hours then alert..the volume is around 150K/sec so 1.5TB if I have to look for any update in the last 6 hours..some times it can be more than 6 hours I am still debating between druid, pinot, influxdb any suggestions?

hi all , is there a retention analysis in pinot source code?

by retention analysis - you mean like customer retention analysis?

Hi All,

pinot-input-format

plugin is not added to

pinot-distribution/pinot-assembly.xml

file.. Any specific reason for this plugin is not part of the distribution?

@User All input format plugins are part of the distribution. They get pulled in during the package generation.

are you facing any issue?

Please check https://github.com/apache/incubator-pinot/blob/master/pinot-distribution/pinot-assembly.xml#L87.

@User sorry for the typo. I was referring to

pinot-input-format/pinot-confluent-avro

plugin.

ah, you are right, I think we missed that

do you mind submitting a patch

sure, will do.

👍

Another related question - we are planning to migrate from

SimpleAvroMessageDecoder

to

KafkaConfluentSchemaRegistryAvroMessageDecoder

, does pinot allow it for the existing table without dropping the data?

not sure if that will work on the Kafka side, producers will be writing the data in a different format

you can achieve what you want without losing data but will incur downtime

As part of this migration, we will be updating the kafka producers to use schema registry for serialization.

what is the alternative for step-3 if there is no API?

set the segment flush threshold to 1 hour or something lower

wait for an hour and segment will get flushed,

Thanks @User

@User Please review: https://github.com/apache/incubator-pinot/pull/5248

How do we check lag for the pinot consumer ? Are there any metrics available ? I see it is using LLC (Low level consumer) right now.