Hi Team, Im getting this error continously even my servers are running properly and status of the tables are in good state. help me to resolve the issue kubectl get pod -n dp-1-346 NAME READY STATUS RESTARTS AGE pinot-broker-0 1/1 Running 0 13h pinot-controller-0 1/1 Running 0 6m26s pinot-minion-stateless-84fc6899f9-2shqp 1/1 Running 0 13h pinot-server-0 1/1 Running 0 6m32s pinot-server-1 1/1 Running 0 6m39s presto-coordinator-0 1/1 Running 0 25h presto-worker-0 1/1 Running 0 25h zookeeper-0 1/1 Running 0 13h

Hi Naveen, looks like you are hitting the 10 sec timeout, try increasing it, yet better, look into controller/server logs for clues

Im getting same response for all the tables. Even the tables which as KB's of data

Without proper error from the logs, it's not possible to tell what is wrong by just looking at this generic error Naveen

Hi, Im working with Naveen. Gave timout seconds as 100002. Still gettings the same. Qurey JSON { "numRowsResultSet": 0, "partialResult": true, "exceptions": [ { "errorCode": 427, "message": "1 servers [pinot-server-0_R] not responded" } ], "numGroupsLimitReached": false, "numGroupsWarningLimitReached": false, "timeUsedMs": 100002, "requestId": "1020573879000000384", "clientRequestId": null, "brokerId": "Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099", "numDocsScanned": 0, "totalDocs": 0, "numEntriesScannedInFilter": 0, "numEntriesScannedPostFilter": 0, "numServersQueried": 1, "numServersResponded": 0, "numSegmentsQueried": 0, "numSegmentsProcessed": 0, "numSegmentsMatched": 0, "numConsumingSegmentsQueried": 0, "numConsumingSegmentsProcessed": 0, "numConsumingSegmentsMatched": 0, "minConsumingFreshnessTimeMs": 0, "numSegmentsPrunedByBroker": 0, "numSegmentsPrunedByServer": 0, "numSegmentsPrunedInvalid": 0, "numSegmentsPrunedByLimit": 0, "numSegmentsPrunedByValue": 0, "brokerReduceTimeMs": 0, "offlineThreadCpuTimeNs": 0, "realtimeThreadCpuTimeNs": 0, "offlineSystemActivitiesCpuTimeNs": 0, "realtimeSystemActivitiesCpuTimeNs": 0, "offlineResponseSerializationCpuTimeNs": 0, "realtimeResponseSerializationCpuTimeNs": 0, "offlineTotalCpuTimeNs": 0, "realtimeTotalCpuTimeNs": 0, "explainPlanNumEmptyFilterSegments": 0, "explainPlanNumMatchAllFilterSegments": 0, "traceInfo": {}, "tablesQueried": [ "anti_virus_44" ], "offlineThreadMemAllocatedBytes": 0, "realtimeThreadMemAllocatedBytes": 0, "offlineResponseSerMemAllocatedBytes": 0, "realtimeResponseSerMemAllocatedBytes": 0, "offlineTotalMemAllocatedBytes": 0, "realtimeTotalMemAllocatedBytes": 0, "pools": [ -1 ], "rlsFiltersApplied": false, "groupsTrimmed": false } Pinot broker logs pinot broker logs Message: dc6955a9-6b45-49ec-a4bc-eb9f1a21c215 handling task scheduled │ │ Message dc6955a9-6b45-49ec-a4bc-eb9f1a21c215 completed. │ │ 99 END:INVOKE CallbackHandler 4, /pinot-nanoheal/INSTANCES/Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@2174fa3b type: CALLBAC │ │ Delete message dc6955a9-6b45-49ec-a4bc-eb9f1a21c215 from zk! │ │ message finished: dc6955a9-6b45-49ec-a4bc-eb9f1a21c215, took 4 │ │ Message: dc6955a9-6b45-49ec-a4bc-eb9f1a21c215 (parent: null) handling task for brokerResource:t1_d44_census_REALTIME completed at: 1762335926480, results: true. FrameworkTime: 3 ms; HandlerTime: 1 ms. │ │ 99 START: CallbackHandler 4, INVOKE /pinot-nanoheal/INSTANCES/Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@2174fa3b type: CALL │ │ CallbackHandler 4 subscribing changes listener to path: /pinot-nanoheal/INSTANCES/Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], l │ │ CallbackHandler4, Subscribing to path: /pinot-nanoheal/INSTANCES/Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099/MESSAGES took: 1 │ │ No Messages to process │ │ 99 END:INVOKE CallbackHandler 4, /pinot-nanoheal/INSTANCES/Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@2174fa3b type: CALLBAC │ │ requestId=1020573879000000383,table=anti_virus_44,timeMs=100001,docs=0/0,entries=0/0,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/unavailable):0/0/0/0/0/0/0,consumingFreshnessTimeMs=0,ser │ │ Query processing exceptions: {427=1 servers [pinot-server-1_R] not responded} │ │ SQL query for request 1020573879000000384: select * from anti_virus_44 limit 10 │ │ Caught exception while deserializing data table of size: 1209 from server: pinot-server-0_R │ │ java.lang.NumberFormatException: Cannot parse null string │ │ at java.base/java.lang.Long.parseLong(Long.java:674) │ │ at java.base/java.lang.Long.parseLong(Long.java:836) │ │ at org.apache.pinot.core.transport.QueryRouter.receiveDataTable(QueryRouter.java:192) │ │ at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:73) │ │ at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:39) │ │ at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) │ │ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) │ │ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) │ │ at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) │ │ at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) │ │ at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) │ │ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) │ │ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) │ │ at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) │ │ at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) │ │ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) │ │ at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) │ │ at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) │ │ at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) │ │ at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) │ │ at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) │ │ at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) │ │ at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) │ │ at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) │ │ at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) │ │ at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) │ │ at java.base/java.lang.Thread.run(Thread.java:840) │ │ requestId=1020573879000000384,table=anti_virus_44,timeMs=100002,docs=0/0,entries=0/0,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/unavailable):0/0/0/0/0/0/0,consumingFreshnessTimeMs=0,ser │ │ Query processing exceptions: {427=1 servers [pinot-server-0_R] not responded}

Thanks, Look at the server logs for the exact

requestId

In your broker log it’s

requestId=1020573879000000384

. On pinot-server-0 and pinot-server-1, search for that requestId (or the same timestamp) and see if: • The server threw an exception while executing the query, • Or it completed and wrote a response (but perhaps with error metadata).

Broker log SQL query for request 1020573879000000408: select * from anti_virus_44 limit 10 Caught exception while deserializing data table of size: 1209 from server: pinot-server-0_R java.lang.NumberFormatException: Cannot parse null string at java.base/java.lang.Long.parseLong(Long.java:674) at java.base/java.lang.Long.parseLong(Long.java:836) at org.apache.pinot.core.transport.QueryRouter.receiveDataTable(QueryRouter.java:192) at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:73) at org.apache.pinot.core.transport.DataTableHandler.channelRead0(DataTableHandler.java:39) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840) requestId=1020573879000000408,table=anti_virus_44,timeMs=10002,docs=0/0,entries=0/0,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/unavailable):0/0/0/0/0/0/0,consumingFreshnessTimeMs=0,servers=0/1,groupsTrimmed=false,groupLimitReached=false,groupWarningLimitReached=false,brokerReduceTimeMs=0,exceptions=1,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs,RequestSentDelayMs);pinot-server-0_R=0,-1,0,0,1,offlineThreadCpuTimeNs(total/thread/sysActivity/resSer):0/0/0/0,realtimeThreadCpuTimeNs(total/thread/sysActivity/resSer):0/0/0/0,clientIp=unknown,queryEngine=singleStage,offlineMemAllocatedBytes(total/thread/resSer):0/0/0,realtimeMemAllocatedBytes(total/thread/resSer):0/0/0,pools=[-1],rlsFiltersApplied=false,query=select * from anti_virus_44 limit 10 Query processing exceptions: {427=1 servers [pinot-server-0_R] not responded} Pinot server-0 log [Consumer clientId=t1_d45_dart_dtag_emea_config_v1_REALTIME-dtag_emea_config-0, groupId=null] Resetting offset for partition dtag_emea_config-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}}. Processed requestId=1020573879000000408,table=anti_virus_44_REALTIME,segments(queried/processed/matched/consumingQueried/consumingProcessed/consumingMatched/invalid/limit/value)=105/105/1/1/1/0/0/0/0,schedulerWaitMs=0,reqDeserMs=1,totalExecMs=7,resSerMs=1,totalTimeMs=9,minConsumingFreshnessMs=1762342193209,broker=Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099,numDocsScanned=10,scanInFilter=0,scanPostFilter=60,sched=FCFS,threadCpuTimeNs(total/thread/sysActivity/resSer)=0/0/0/0, threadMemAllocatedBytes(total/thread/resSer)=0/0/0 [Consumer clientId=t1_d44_services_REALTIME-service_name-0, groupId=null] Seeking to offset 0 for partition service_name-0 [Consumer clientId=t1_d44_url_REALTIME-URL_44-0, groupId=null] Seeking to offset 0 for partition URL_44-0 [Consumer clientId=t1_d44_asset_REALTIME-asset_44-0, groupId=null] Seeking to offset 0 for partition asset_44-0 [Consumer clientId=windows_defender_44_REALTIME-windows_defender_44-0, groupId=null] Seeking to offset 94708 for partition windows_defender_44-0 [Consumer clientId=t1_d359_automation_test_REALTIME-1_359_automation_test-0, groupId=null] Seeking to offset 3570 for partition 1_359_automation_test-0 Consumed 0 events from (rate:0.0/s), currentOffset=86419, numRowsConsumedSoFar=3509, numRowsIndexedSoFar=3489 [Consumer clientId=windows_defender_44_REALTIME-windows_defender_44-0, groupId=null] Fetch position FetchPosition{offset=94708, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}} is out of range for partition windows_defender_44-0, resetting offset [Consumer clientId=t1_d359_automation_test_REALTIME-1_359_automation_test-0, groupId=null] Fetch position FetchPosition{offset=3570, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}} is out of range for partition 1_359_automation_test-0, resetting offset [Consumer clientId=windows_defender_44_REALTIME-windows_defender_44-0, groupId=null] Resetting offset for partition windows_defender_44-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}}. [Consumer clientId=t1_d359_automation_test_REALTIME-1_359_automation_test-0, groupId=null] Resetting offset for partition 1_359_automation_test-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}}. [Consumer clientId=t1_d45_scccm_REALTIME-sccm_status_45-0, groupId=null] Seeking to offset 2 for partition sccm_status_45-0 [Consumer clientId=t1_d45_dartautomation_v1_REALTIME-1_45_pinot_automation-0, groupId=null] Seeking to offset 7 for partition 1_45_pinot_automation-0 [Consumer clientId=t1_d44_batteryDetails_REALTIME-battery_details_44-0, groupId=null] Seeking to offset 0 for partition battery_details_44-0 [Consumer clientId=t1_d45_scccm_REALTIME-sccm_status_45-0, groupId=null] Fetch position FetchPosition{offset=2, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}} is out of range for partition sccm_status_45-0, resetting offset [Consumer clientId=t1_d45_scccm_REALTIME-sccm_status_45-0, groupId=null] Resetting offset for partition sccm_status_45-0 to position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}}. [Consumer clientId=t1_d45_dartautomation_v1_REALTIME-1_45_pinot_automation-0, groupId=null] Fetch position FetchPosition{offset=7, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[10.0.43.64:9092 (id: 0 rack: null)], epoch=0}} is out of range for partition 1_45_pinot_automation-0, resetting offset

Can you please get answers to the rest of the question?

curl -s http://<broker-host>:8099/version
curl -s http://<server-host>:8098/version
curl -s http://<controller-host>:9000/version

and

kubectl exec -it <broker-pod> -- curl -s http://<server-pod>:8098/health

your ports might need to be adjusted

I didnt get output from those commands, I have taken the version from the UI Broker Instance Config { "id": "Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099", "simpleFields": { "HELIX_HOST": "pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local", "HELIX_PORT": "8099", "pinotVersion": "1.4.0-SNAPSHOT", "queryMailboxPort": "35635" }, "mapFields": {}, "listFields": { "TAG_LIST": [ "DefaultTenant_BROKER" ] } } LiveInstance Config { "id": "Broker_pinot-broker-0.pinot-broker-headless.dp-1-346.svc.cluster.local_8099", "simpleFields": { "CURRENT_TASK_THREAD_POOL_SIZE": "40", "HELIX_VERSION": "1.3.2", "LIVE_INSTANCE": "1@pinot-broker-0", "SESSION_ID": "1021a4e11ad001b" }, "mapFields": {}, "listFields": {} } Controller version Instance Config { "id": "Controller_pinot-controller-0.pinot-controller-headless.dp-1-346.svc.cluster.local_9000", "simpleFields": { "HELIX_HOST": "pinot-controller-0.pinot-controller-headless.dp-1-346.svc.cluster.local", "HELIX_PORT": "9000", "pinotVersion": "1.5.0-SNAPSHOT" }, "mapFields": {}, "listFields": { "TAG_LIST": [ "controller" ] } LiveInstance Config { "id": "Controller_pinot-controller-0.pinot-controller-headless.dp-1-346.svc.cluster.local_9000", "simpleFields": { "CURRENT_TASK_THREAD_POOL_SIZE": "40", "HELIX_VERSION": "1.3.2", "LIVE_INSTANCE": "1@pinot-controller-0", "SESSION_ID": "1021a4e11ad007a" }, "mapFields": {}, "listFields": {} } Server-0 Instance config { "id": "Server_pinot-server-0.pinot-server-headless.dp-1-346.svc.cluster.local_8098", "simpleFields": { "HELIX_HOST": "pinot-server-0.pinot-server-headless.dp-1-346.svc.cluster.local", "HELIX_PORT": "8098", "adminPort": "8097", "grpcPort": "8090", "pinotVersion": "1.5.0-SNAPSHOT", "queryMailboxPort": "42867", "queryServerPort": "40427", "shutdownInProgress": "false" }, LiveInstance Config { "id": "Server_pinot-server-0.pinot-server-headless.dp-1-346.svc.cluster.local_8098", "simpleFields": { "CURRENT_TASK_THREAD_POOL_SIZE": "40", "HELIX_VERSION": "1.3.2", "LIVE_INSTANCE": "1@pinot-server-0", "SESSION_ID": "1021a4e11ad0083" }, "mapFields": {}, "listFields": {} }

This sounds like what's causing issues on your end

Broker: pinotVersion = 1.4.0-SNAPSHOT

Controller: 1.5.0-SNAPSHOT

Server-0: 1.5.0-SNAPSHOT

We did not change anything manually, is there any other possibility that it will change automatically while restarting. We are using the same template in other region, but there all the versions are matching. How to change the broker version to 1.5.0-SNAPSHOT?

I do not know how your pinot is deployed, so its hard for me to recommend something.

curl -X 'GET' '<http://domain|http://<domain>>/version' -H 'accept: application/json' {"pinot-kafka-2.0":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-avro":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-yammer":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-batch-ingestion-standalone":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-thrift":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-azure":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-gcs":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-hdfs":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-kinesis":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-protobuf":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-confluent-protobuf":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-distribution":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-csv":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-s3":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-segment-uploader-default":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-confluent-avro":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-clp-log":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-pulsar":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-confluent-json":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-dropwizard":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-timeseries-m3ql":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-adls":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-json":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-minion-builtin-tasks":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-parquet":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57","pinot-segment-writer-file-based":"1.5.0-SNAPSHOT-eaaf1ac519a2e6e3c19bec523d5ed580af637e57"} i tried using the above curl command and i got this output. does it make any difference with above versions he as shared?

We saw in previous output clear evidence of Brokers & servers being on different versions

now we edited the version from where we got the earlier versions

Hi, I'm getting a similar issue. When I look at the table segments, I see that it is still in Consuming state. And I do not see any logs in the server. Any leads will be helpful. Thanks.

This seems to be a problem

Caught exception while sending request … to server:
disco-rakesh-pinot-server-0.disco-rakesh-pinot-server-headless.disco-bff-rakesh.svc.cluster.local:
Name or service not known
java.net.UnknownHostException

The broker is trying to query a server at a fully qualified DNS name that doesn't exist.