Hey team, i have few questions, can someone help ? 1) Queries are not returning results most of the time. Upon checking broker logs found the following :

Failed to find servers hosting segment: mytable_0_8_20211029T2056Z for table: mytable_REALTIME (all ONLINE/CONSUMING instances: [] are disabled, but find enabled OFFLINE instance: Server_ip_8098 from OFFLINE instances: [Server_ip_8098], not counting the segment as unavailable)

is this query timeout case? 2) I have set, flush.threshold.size to 10mn. But segments are getting created with lesser rows ( Total docs: 3.4mn). Is this expected? 3) What type of index is recommended on Realtime table with upsert mode on ? 4) In upsert mode, any limitation on "comparison time column" , i.e timestamp format, granularity? my table date column is in yyyyMMddHH format. comparison time column will be in timestamp format yyyy-MM-dd HHmmss

{
  "upsertConfig": {
    "mode": "FULL",
    "comparisonColumn": "anotherTimeColumn"
  }
}

5) Queries are timing out at 10secs, even after changing the values at broker and server level. anyother configs needs to be changed ? pinot.broker.timeoutMs pinot.server.query.executor.timeout

1. Please check table debug api, and external view. 2. need to look at your table config, there’s also a time based flush threshold. 3. Index and upsert are orthogonal, except startree is not supported with upsert. 4. No limitation on time column afaik 5. Can you pass timeout value from the query console and see if that works?

Table config @Mayank

"streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "lowlevel",
        "stream.kafka.topic.name": "topic",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.broker.list": "brokers",
        "realtime.segment.flush.threshold.rows": "10000000",
        "stream.kafka.consumer.prop.auto.offset.reset": "largest"
      }

I think that num rows is across all partitions

can you explain a bit please ? didn't understand. I was expecting with this conf was , a segment will have 10mn rows.

So let’s say you have 10 Kafka partitions, and you set threshold 1M, then each segment will be 100k rows

so, i have set threshold as 10mn in config, so each segment should have 10mn rows right?

How many Kafka partitions you have? Let’s say you have 3, then each segment will be 10/3 M

okay thanks