Hi team, recently we had experienced Pinot server out of memory issue in a 4 server Pinot cluster when issuing one query select distinct id on entire table with only 1 billion record. We had to manually restart the Pinot servers pods to recover. Is this normal? Is there some index or Pinot cofigurariom we can add to this id or Pinot cluster to prevent it bring down the entire cluster.

We had run into a similar issue on 0.6 when doing a

DISTINCTCOUNT

on a high cardinality field. Only the broker was jammed, so restarting that process fixed things. From the stack trace, it seemed like the problem was caused by very large responses from the server processes causing blockage at the broker network layer. I don’t think there was a way to prevent this from happening, at least with that version.