Hello what is the average consuming latency that y...
# troubleshooting
g
Hello what is the average consuming latency that you see in your use cases as no matter what i do i see around 3-4 seconds latency, which is not ideal. Can you tell me what data would you need in order to help me determine if we can bring it down to under 1 second latency
x
how do you measure? is it kafka?
have you also measured the messaging infra e2e time based on a native consumer?
g
i measure it with the pinot metrics that are given as example and exposed through Prometheus and into Grafana
pinot_server_freshnessLagMs_99thPercentile this is the metric i use
https://docs.pinot.apache.org/configuration-reference/monitoring-metrics used those docs and the quick setup guide as well
x
if you found that p50 is ok then possibly this is due to kafka producer side glitch
also is this high ingestion speed topic?
if not then possible the partition has no data then it may also report larger freshness number(current time - last timestamp from row metadata)
g
sorry for late response, yes p50 is in 1s range, it is slow/medium, but has a lot of big spikes and also has 100 partitions, which as you said might cause given partitions to have less data and make the overall metric look worse
x
yeah, the usual way to test the latency is to inject your own markable data. Say from the producer side you can publish data stream with a unique key and ts, then from the pinot side you keep query this key and publish the freshness number by using the (currentTs - queriedTs)