when ingesting large amounts of streaming data, is...
# troubleshootingj
Josh Highley
03/12/2021, 6:38 PMwhen ingesting large amounts of streaming data, is there any mechanism for capturing/notifying about data errors other than looking through the server log files? We're ingesting millions of records as fast as Pinot can ingest them, so looking through logs isn't really practical.
d
Daniel Lavoie
03/12/2021, 6:40 PMPlease share your use case to this issue: https://github.com/apache/incubator-pinot/issues/6524
j
Josh Highley
03/12/2021, 6:43 PMthat seems to be at a table level -- connectivity issues or configuration issues. Is row-level monitoring applicable to that issue?
d
Daniel Lavoie
03/12/2021, 6:44 PMIt’s worth sharing as this bring another perspective to the general “ingestion” observability topic.
Daniel Lavoie
03/12/2021, 6:45 PMIf one row breaks ingestion, it’s equaly important to bubble up the details than a table misconfiguration.
Daniel Lavoie
03/12/2021, 6:45 PMI think both issues fits in the “table ingestion status” story
k
Kishore G
03/12/2021, 7:23 PM
there is a metric that you can monitor for errors during ingestion
j
Josh Highley
03/12/2021, 9:27 PMI see an error count in ServerMeter but it's just a counter. Per instructions here: https://docs.pinot.apache.org/operators/operating-pinot/monitoring#customizing-metrics I exposed the metrics but the counter is still 0 despite errors in the logs