Hi everyone... I am using realtime ingestion. Kafk...
# general
r
Hi everyone... I am using realtime ingestion. Kafka source has a retention period of 3 days, but the lag in Pinot said 4-5 days and was increasing. When I restated the servers the lag dropped to 2-2.5 days and reduced quite drastically and overnight dissipated. Can you pls help me debug this?
x
Pinot handles retention in batches, here are 2 reasons you may still see the older data: 1. Pinot retention manager is on controller to handle data deletion, please check the run interval. 2. Pinot consumes data from Kafka then persist, based on the flush threshold, the flushed batch may contain certain time range of the data. Pinot retention manager delete the data batch (segment) based on the end time, which means the batch will be deleted once all the events inside has passed the retention.
r
But what could have caused the ingestion delay?
x
oh, i think i misunderstand the question for why table has data older than retention. is the data ingested with latest timestamp? can you compare the kafka ingestion throughtput and the kafka data ingestion throughput
r
Yes ingestion is from latest
Sorry I didn't understand the difference between the two
Do you mean the rate of msg production on Kafka and the rate in which data is ingested in Pinot...
x
yes
say e.g. message rate to kafka is 1000 msg/sec, and to pinot is 800 msg/sec, then pinot cannot catch up kafka side
r
there is another issue... Initially we see high kafka data ingestion throughput but with time we see it slow down... As out heap usage is very high... we have 55 GB heap but in a week or so we are continuously reaching 90%.... So this lag is understood but I can't understand the sudden change in the value after the restart... is there something in way the metrics is calculated which take into account the current rate of ingestion???
r
Are you suggesting that ingestion lag is caused by lack of heap space?
Yes, there are ingestion rate metrics
r
Yes that's my hypothesis... but can't explain this sudden metric change... I see the rate reduces over time and heap usage increases and we do see higher GCs...
r
Re: ingestion metrics, I recommend using the
pinot_server_realtimeIngestionDelayMs_Value
metric
r
Ok...
r
When you say “higher GCs” do you mean more frequent or longer or both?
r
both...
we have 55GB heap and see usage of 50-51GB
r
Besides GCs, frequent segment completion could cause ingestion delays
r
Ok...
@robert zych Any specific metrics you recommend which I should use to see the segment completion rate or something which will give me better understanding of the reason for the delay...
r
No, I don't have a recommendation for the segment completion rate specifically. But if segment completion rate was too high, it would be visible via other indicators. For example, higher query latencies, larger amount of segments, low
flush.threshold
in tableConfig, and spikes/increases in the metric I mentioned yesterday.
r
Yes we do see higher query latencies and larger amount of segments....
r
In that case, increase
flush.threshold.segment.size
in your tableConfig
r
We have it at 1GB...
realtime.segment.flush.threshold.time: 72h realtime.segment.flush.threshold.segment.size: 1024M
r
What's your data rate?
r
it varies across the entire day... but in a day overall we would be ingesting 300M+ rows into pinot... that is after filtering... kafka msg consumption would be higher...
r
How many segments does your table have? What's your retention period?
r
7TB data with 51840 segments.... We have started ingestion around 3 months back... We are looking for 1.5years retention period...
r
7TB reported by Pinot, correct?
52K segments in only three 3 months is too much. Especially given your retention period.
At about 100K segments per table, ZK issues begin
r
Yes 7TB reported by Pinot... we are working on minion to merge segments... Yes... we were seeing ZK issue... so we increased the jute buffer for now...
Our retention configs are higher that what's recommended(500M).... but still we are generating a lot of segments...
is minion approach the only way forward to reduce the number of segment... or am I missing any configuration???
r
At 7TB and nearly 52K segments, your average segment size is only 134MB
besides using the minion task and increasing the segment size, you're only other potential option would be to reduce your retention period
r
Yes... but I don't understand the reason for segments with lower size... when the threshold is higher...
We have partial commit enabled... can that be the reason???
r
Partial commit?
r
pinot.server.instance.enable.split.commit=true....
r
Unlikely
r
ok... what can be the reason???
realtime.segment.flush.threshold.time: 72h realtime.segment.flush.threshold.segment.size: 1024M these are the table configs....
r
Segment.size is a guideline. Actual segment sizes may vary. How many records per segment at peak ingestion rate vs at lower ingestion rate?
r
500000-600000 rows...
but this is from random sampling of segments...
r
Looks like you segments are consistently small.
These metrics may be relevant for you:
pinot_server_llcSimultaneousSegmentBuilds_Value
pinot_server_segmentCount_Value
@Xiang Fu Raunak's table size is 7TB and has set
threshold.segment.size: 1024M
but his table >50K segments and random sampling shows that his segment contain 500K to 600K records per segment. Do you agree that the segments are too small? If so, and ideas why the segments are small?
x
Yes, I think the segment size threshold could be doubled to reduce the total number of segments
r
Yes, I also recommended increasing
flush.threshold.segment.size
. But given that the table has been consuming for 3 months, do you agree that the desired segment size isn't being achieved?
If the desired segment size (1GB) was being achieved, I would expect the table to have 7K segments (not >50K)
r
realtime.segment.flush.threshold.time: 72h realtime.segment.flush.threshold.segment.size: 1024M I understand both these number are guideline but what can be the reason for low segment size... 1G to 134MB.... What could be the cause of this??
r
Here’s another way inspect segment sizes…
It shows how to get diskSizeInBytes which will also confirm if you have smaller segments
The record count you provided earlier could be misleading if your schema has many dimensions
If your segments are indeed only 134MB then you could file an issue and use the row count threshold as a workaround
r
Thanks a lot @robert zych will use this to see the segment size.... And if there is an issue I would proceed with your suggestion... The schema has 53 fields...
r
53 is normal
x
I remember there is also a config of flush row size, I guess we should make it to be very large so only flush size take effect
r
We have kept it at 0.... It says that if that is the value... The above 2 params are used to decide flush rate
x
i see, still suggest to make it a very large number like 10MM rows
to avoid any possible variation
r
Sure... will do that...