Hey all! Is there a way to mirror traffic (reques...
# general
p
Hey all! Is there a way to mirror traffic (requests and responses) in Pinot? We want to monitor and analyse the traffic and react quickly in case of errors. Broker logs are unstructured and hard to analyse. We don’t know which user made this query or what is the source ip address or what is the query latency or number of segments scanned for this particular query. From time to time, we have found that prometheus metrics are not accurate. Simply pushing all requests (with username and all other headers ) and responses (just the query stats and error traces in case of errors, no data) to Kafka would be immensely useful to keep an eye on how Pinot cluster is utilized. We can even connect this kafka topic to an internal realtime table in Pinot and monitor everything right within Pinot and analyse traffic using SQL. Can this be done via some kind of plugin? I am willing to work on this if someone can point me to the right direction. Thanks!