Hello Team I am currently looking for documentation related to segment file compression and storage formats in Apache Pinot. I find the existing documentation somewhat unclear. From my understanding, regardless of whether we use JSON, Proto, or Avro, segments are always stored using some form of file compression. This seems to be independent of the serialization format used when writing messages to the topic. Is my understanding correct? We are planning to implement a Pinot-based solution in the cloud, but without a clear understanding of segment compression, we are unable to determine whether Pinot will be cost-effective in terms of storage. Could someone please provide clarification on this topic? Your insights will greatly assist us in making an informed decision regarding our Pinot-based application.

Pinot has its own segment format for data storage, its columnar and you can tune the columnar compression. https://docs.pinot.apache.org/basics/concepts/components/table/segment