Apache Pinot

even confluent’s docs mix them on the same page

<@UMTM7HSNN> Are you planning to write your own pluggable stream? Or, do you plan to use Kafka?

I ask because you quoted from the doc for pluggable streams.

Unless you are developing modules for streams other than kafka, that particular section is of little value.

<@UDSRQR2GP> switching to low level consumer and reingesting 300M records dropped zookeper log folder from 8gbs to 200Mbs

excellent. let us know how the progress is going.

looking at segments metadata and noticing that:

```column.itemSalePrice.cardinality = 445
column.itemSalePrice.totalDocs = 506250
column.itemSalePrice.totalRawDocs = 506250
column.itemSalePrice.totalAggDocs = 0```

It’s non zero when you enable star tree

want to try it as well. <https://pinot.readthedocs.io/en/latest/star-tree/star-tree.html#startreeindexconfig> doesn’t say where to put the config :slightly_smiling_face:

``` "startreeIndexSpec": {
      "dimensionsSplitOrder": [
        "dim1",
        "dim2",
        "dim3"
      ],
      "skipStarNodeCreationForDimensions": [],
      "functionColumnPairs": [
        "SUM__subtotal",
        "MAX__subtotal",
        "AVG__subtotal",
        "DISTINCTCOUNTHLL__items"
      ]
    }```

under tableIndexConfig and it didn’t do anything. Table was created and ingested, but totalAggDocs are still 0s

startreeIndexSpec -&gt; starTreeIndexSpec works

another question: aggregateMetrics -&gt; documentation says that it will sum all the metrics. Is SUM the only operation supported?

No. It’s the default one. You can apply any function min max etc

I thought Jackie added support for other agg functions as well