Ken Krugler
02/19/2021, 11:05 PMselect distinctcount(<super-high cardinality column>) from table
), and it did. The request timed out, even though I gave it a 100,000ms timeout, and now all queries (e.g. select * from crawldata limit 20) time out. I’ve looked at the controller/broker/sample of server logs, and don’t see any errors. In the broker log it looks like it’s getting no responses from servers:
2021/02/19 22:21:53.860 INFO [BaseBrokerRequestHandler] [jersey-server-managed-async-executor-59] requestId=41163,table=crawldata_OFFLINE,timeMs=10000,docs=0/0,entries=0/0,segments(queried/processed/matched/consuming/unavailable):0/0/0/0/0,consumingFreshnessTimeMs=0,servers=0/5,groupLimitReached=false,brokerReduceTimeMs=0,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);116.202.83.208_O=0,-1,0,0;168.119.147.123_O=0,-1,0,0;168.119.147.125_O=1,-1,0,0;168.119.147.124_O=1,-1,0,0;116.202.52.154_O=1,-1,0,0,query=select * from crawldata limit 20
But an example server log has this:
2021/02/19 22:21:43.864 INFO [QueryScheduler] [pqr-11] Processed requestId=41163,table=crawldata_OFFLINE,segments(queried/processed/matched/consuming)=213/1/1/-1,schedulerWaitMs=0,reqDeserMs=0,totalExecMs=2,resSerMs=1,totalTimeMs=3,minConsumingFreshnessMs=-1,broker=Broker_168.119.147.124_8099,numDocsScanned=20,scanInFilter=0,scanPostFilter=620,sched=fcfs
Trying to figure out which process or processes are borked because of the query, and why. Any ideas? Thanks!Xiang Fu
Ken Krugler
02/19/2021, 11:25 PMnumDocsScanned=20
, etc)Xiang Fu
Jackie
02/19/2021, 11:29 PMKen Krugler
02/19/2021, 11:30 PMJackie
02/19/2021, 11:31 PMKen Krugler
02/19/2021, 11:31 PMdistinctcount
, I guess each server has to send back to the broker all unique values for the column, for every segment that it’s processing.Jackie
02/19/2021, 11:31 PMKen Krugler
02/19/2021, 11:32 PMJackie
02/19/2021, 11:33 PMKen Krugler
02/19/2021, 11:37 PMJackie
02/20/2021, 12:02 AMKen Krugler
02/20/2021, 12:19 AMJackie
02/20/2021, 12:40 AMKen Krugler
02/20/2021, 12:59 AMJackie
02/20/2021, 1:01 AM