Hello folks, looking for recommendations around logging for Pinot production cluster. There seem to be multiple configs to control this. But we wanted to ensure we don't over-do with logging at the same time have flexibility to look at Pinot logs for exceptions/errors ?

we recently went through some changes here. there’s no controls on exception/error logging, so you can’t really miss those. The main take away we had is pinot has somewhat explosive logging from queries: 1 log on the broker followed by N logs (1 per server). So every query gets N + 1 logs. For a high QPS system, this can turn into a lot of data. We ended up setting the log rate limit on the servers down to 50 but have left the brokers untouched so far because we don’t want to lose top level visibility on broker logs. We also patched the broker logs to remove some redundant info to lower log volume. But general recommendation is to cap the log rate for queries so log volume isn’t scaling with query volume in high QPS

Got it. Thanks for the reco.

those can also be large, but not the same scale. they’re not configurable, so you’d have to modify the code if you wanted to expose a way to turn them down

how are you setting the log rate limit? are you using burst filters?

I am assuming

pinot.broker.query.log.maxRatePerSecond

from here https://docs.pinot.apache.org/configuration-reference/broker

Sure, you can file a GH issue. And feel free to contribute as well, if you feel like giving back to the community 🙂

For sure. Will do and contribute as well.