Hi Guys , need guidance with one of issue we are facing . We are testing upsert use case on realtime table , where realtime table has 20 million records and we are publishing additional 50million records. We have query loop running which basically validates total record count while upsert is in progress , we expect this query to return total count of 20 million consistently. Real time table has replication factor 2. We noticed that if one of our server goes down (We have 5 servers), pinot dont return result (i.e our count(*)query returns numberofSegmentsmatched=0, totaldocs=0, segmentqueried=0 etc ) for about 30secs before giving expected result. i.e 20 million. Also noticed when server comes back up total count is inconsistent for some time before returning expected result. Table remains in BAD state but keeps serving query with expected result after sometime. Our Table config "REALTIME": {    "tableName": "sometable",    "tableType": "REALTIME",    "segmentsConfig": {      "schemaName": "someschema",      "timeColumnName": "AuditDateTimeUTC",      "allowNullTimeValue": false,      "replication": "1",      "replicasPerPartition": “2”    },    "tenants": {      "broker": "DefaultTenant",      "server": "DefaultTenant",      "tagOverrideConfig": {}    },    "tableIndexConfig": {      "invertedIndexColumns": [],      "rangeIndexColumns": [],      "rangeIndexVersion": 1,      "autoGeneratedInvertedIndex": false,      "createInvertedIndexDuringSegmentGeneration": false,      "sortedColumn": [],      "bloomFilterColumns": [],      "loadMode": "MMAP",      "noDictionaryColumns": [some columns],      "onHeapDictionaryColumns": [],      "varLengthDictionaryColumns": [],      "enableDefaultStarTree": false,      "enableDynamicStarTreeCreation": false,      "aggregateMetrics": false,      "nullHandlingEnabled": false,      "streamConfigs": {        "streamType": "kafka",        "stream.kafka.topic.name": "sometopic",        "stream.kafka.broker.list": "host:9092",        "stream.kafka.consumer.type": "lowlevel",        "stream.kafka.hlc.bootstrap.server": "host:9092",        "stream.kafka.consumer.prop.auto.offset.reset": "largest",        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",        "realtime.segment.flush.threshold.rows": "0",        "realtime.segment.flush.threshold.size": "0",        "realtime.segment.flush.autotune.initialRows": “3000000",        "realtime.segment.flush.threshold.time": "24h",        "realtime.segment.flush.threshold.segment.size": “500M”      }    },    "metadata": {},    "quota": {},    "routing": {      "instanceSelectorType": "strictReplicaGroup"    },    "query": {},    "upsertConfig": {      "mode": "FULL",      "comparisonColumn": "AuditDateTimeUTC",      "hashFunction": "NONE"    },    "ingestionConfig": {},    "isDimTable": false  } }

Do you know why the server went down? Was there an expensive query issued?

we stopped server as part of test case to check how replication behaves when upsert is in progress

Oh ok. When server is restarted, it can take a bit to load all segments (depending on how many there are), which will explain the inconsistent result.

replication 2 .... actually we are more worried abt scenarion where pinot stops responding for 30 sec or so after server goes down

That is not an expected.

we were expecting to continue to get results even if server goes down as replication is set to 2 ..could it be any config issue?

I don’t think Pinot is expected to stop responding if one of the servers is down. There might be some other issue going on there.

select count(*) ...

Hmm, that should definitely not time out. How do you stop the server?

Hi Mayank , In external view its doent show any segment offline ...all segments are in consuming state to start with before stopping server , each segment having 2 consuming servers (replication 2) and we see only one server is consuming state for one of the segments after one of server is stopped .... but noticed it does take time for external view to update ...could it be reason why we dont get any result for time external view is updating ...

@Prashant Korade there are two types of failure scenarios

Thanks Kishore ... for this test case ...we stopped server ... by kill -9 ...

yeah, you can control the session timeout and reduce it to 15 sec or 10 sec

Thanks @Mayank @Kishore G for your inputs ...

Thanks for confirming it works @Prashant Korade