Hi <@UDQU92KBK> <@UGRJA9TEH> the table query respo...
# troubleshootings
Sadim Nadeem
08/02/2021, 7:41 AMHi @Mayank @Xiang Fu the table query response becomes unstable on pinot sql query editor once we updated the schema of the table means some times it returns rows in response .. somtimes rows are empty .. the query does group by , select columns and filter cc: @Mohamed Sultan @Mohamed Kashifuddin @Manju Priyadharshini
x
Xiang Fu
08/02/2021, 8:05 AM
do you have query stats you can share ? for both resposnes
s
Sadim Nadeem
08/02/2021, 8:10 AMSometimes response empty sometimes not empty
Sadim Nadeem
08/02/2021, 8:11 AMIMG-20210802-WA0003.jpg,IMG-20210802-WA0002.jpg,IMG-20210802-WA0001.jpg,IMG-20210802-WA0000.jpg
Sadim Nadeem
08/02/2021, 8:17 AMquery is "select count(*) as countPerMin,timestampInEpoch,col1,col2,tenant_id,col3,col4,
col5,col6,col7,col8,substr(""time"",0,16) from myTable
WHERE col2 !=-1 AND
col3 = 'deathstar-c74d84667-2jdht'
AND col4 = 'tiefighter'
AND timestampInEpoch >= 1626345720000
AND timestampInEpoch <= 1626345720000
AND tenant_id = '728*&*285'
AND col1 = 'sadr'
AND col5 = 'default'
AND col6 = 'default'
AND col7 = '200'
AND col8 = 'nadr'
group by timestampInEpoch,col1,col2,tenant_id,col3,
col4,col5,col6,col7,col8,substr(""time"",0,16) limit 1000
"
Sadim Nadeem
08/02/2021, 8:17 AMcc: @Jackie @Ken Krugler
Sadim Nadeem
08/02/2021, 8:20 AMwe did load test for 100 threads using jmeter and response was always 200 ok
Sadim Nadeem
08/02/2021, 8:20 AMbut we didnt asserted the row size in repsonse json .. we are planning to assert that with load testing as well
m
Mayank
08/02/2021, 11:31 AM
Did you call reload api after schema change?
s
Sadim Nadeem
08/02/2021, 4:05 PMYep
Sadim Nadeem
08/02/2021, 4:05 PMReLoad all segments on controller uo
Sadim Nadeem
08/02/2021, 4:05 PMUi
j
Jackie
08/02/2021, 6:08 PM@Sadim Nadeem Seems the empty response queries actually timed out. Can you turn on the
Show JSON formats
Sadim Nadeem
08/03/2021, 4:59 AMhi @Jackie tried with Show JSON format enabled.. sometimes the rows array inside the resultTable have the data and sometimes empty for same query .. not consistent .. cc: @Mayank @Xiang Fu
j
Jackie
08/03/2021, 5:57 AMHow about the exception?
Jackie
08/03/2021, 5:57 AMAlso number of servers responded
s
Sadim Nadeem
08/03/2021, 6:18 AMno exception @Jackie
Sadim Nadeem
08/03/2021, 6:18 AMhow to check number of pinot servers responded .. we have 3 pods for pinot server
Sadim Nadeem
08/03/2021, 6:18 AMdo we need to check the pinot server logs
j
Jackie
08/03/2021, 6:47 AMIt is also part of the response metadata
Jackie
08/03/2021, 6:47 AMFrom the screenshot, one of the 3 servers didn’t respond
s
Sadim Nadeem
08/03/2021, 7:29 AMnone of the pinot server pods restarted in last 4 days . .still facing this issue
Sadim Nadeem
08/03/2021, 7:32 AMthis is the respoinse json when empty response came
Sadim Nadeem
08/03/2021, 7:32 AM{
"resultTable": {
"dataSchema": {
"columnNames": [
"timestampInEpoch",
"col1",
"countPerMin"
],
"columnDataTypes": [
"LONG",
"LONG",
"LONG"
]
},
"rows": []
},
"exceptions": [],
"numServersQueried": 3,
"numServersResponded": 2,
"numSegmentsQueried": 1333,
"numSegmentsProcessed": 1333,
"numSegmentsMatched": 86,
"numConsumingSegmentsQueried": 43,
"numDocsScanned": 1268,
"numEntriesScannedInFilter": 8641190,
"numEntriesScannedPostFilter": 2536,
"numGroupsLimitReached": false,
"totalDocs": 29377275,
"timeUsedMs": 10001,
"offlineThreadCpuTimeNs": 0,
"realtimeThreadCpuTimeNs": 1530254422,
"segmentStatistics": [],
"traceInfo": {},
"numRowsResultSet": 0,
"minConsumingFreshnessTimeMs": 1627975889277
}
Sadim Nadeem
08/03/2021, 7:33 AM-------------------------
rows array came empty
Sadim Nadeem
08/03/2021, 7:33 AMsometimes the rows array has data for same query and sometimes its empty
Sadim Nadeem
08/03/2021, 7:34 AMIt is also part of the response metadata-->no exception in this json as such
Sadim Nadeem
08/03/2021, 7:39 AMAlso when the data came in response in rows means query executed successfully and rows array was not empty .. then also no of servers responded is 2 as u can see in the screenshot here
b
Bruce Ritchie
08/03/2021, 2:42 PM"timeUsedMs": 10001
The query timed out.
m
Mayank
08/03/2021, 4:03 PM
Yes ^^
s
Sadim Nadeem
08/03/2021, 4:05 PMCan we increase the timeout somehow
Sadim Nadeem
08/03/2021, 4:06 PMWe have not manually enabled any timeout
x
Xiang Fu
08/03/2021, 4:31 PM
you can update timeout from query console
s
Sadim Nadeem
08/03/2021, 5:51 PMok thanks
Sadim Nadeem
08/05/2021, 4:11 AMhow to increase default timeout in pinot golang connector api
Sadim Nadeem
08/05/2021, 4:22 AM@Xiang Fu @Mayank @Jackie @Daniel Lavoie
Sadim Nadeem
08/05/2021, 4:22 AMSadim Nadeem
08/05/2021, 4:24 AMhere i see some timeout settings .. is it the query timeout
m
Mayank
08/05/2021, 4:34 AM
That seems like ZK session timeout setting (going by the name), which is not query timeout
s
Sadim Nadeem
08/05/2021, 5:04 AMin golang api ..we are also facing the default timeout issue . so need to figure out how to increase timeout when querying pinot using golang pinot conenctor
Sadim Nadeem
08/05/2021, 6:45 AMalso why there is a timeout when querying through API
Sadim Nadeem
08/05/2021, 7:22 AM@Kishore G @Neha Pawar @Kulbir Nijjer @Chinmay Soman
j
Jackie
08/05/2021, 4:45 PM@Sadim Nadeem The timeout can be embedded into the query body as a query option, e.g.
select ... option(timeoutMs=60000)Jackie
08/05/2021, 4:47 PMBasically we allow per-query timeout to override the default timeout. Default timeout is set via the instance config (broker config), which is not available as a client config
s
Sadim Nadeem
08/05/2021, 5:00 PMthanks @Jackie..it worked ..
👍 1