Hi, we are getting a 500 error (with no logs) when...
# troubleshootinge
Elon
06/30/2021, 3:47 PMHi, we are getting a 500 error (with no logs) when doing queries with Integer.MAX_VALUE as the limit on pinot 0.7.1 - I see from the code that should be the max limit. Is there something else we are running into? (there are no table quotas). Here is the full error, redacted hosts and table names to maintain super secrecy:) :
Copy code
Unexpected response status: 500 for request {"sql" : "select AAA, BBB from CCC limit 2147483647" } to url <http://XXXX>:XXXX/query/sql, with headers {Accept=[application/json], Content-Type=[application/json]}, full response nullElon
06/30/2021, 3:48 PMShould I create a github issue? I can also help and work on this, lmk.
Elon
06/30/2021, 3:49 PMThe controller has a stack trace but not the broker, and there are no logs other than that the request was run (in pinotBroker.log)
Elon
06/30/2021, 3:50 PMStack trace is basically that the post failed (from controller). Also server does not have any logs (maybe request never makes it that far)
m
Mayank
06/30/2021, 4:00 PM
Is there an OOM?
e
Elon
06/30/2021, 4:03 PMI don’t see an oom or anything in the logs. The main consumer of memory (ran jmap ) was the async logger. Also the table only has 4K rows. When I lower the limit the query runs. Then I raise it back and the query fails. No crashes or ooms.
Elon
06/30/2021, 4:05 PMHave you ever seen this? I am retrying in local docker with debug logging (on a test table though) - maybe I can find debug messages…
Elon
06/30/2021, 4:18 PMOk, more debug info, I see that it made it to the server:
Elon
06/30/2021, 4:18 PMUntitled
Elon
06/30/2021, 4:19 PMAnd broker parsed the query just before that (about 10ms before):
Elon
06/30/2021, 4:19 PMUntitled
Elon
06/30/2021, 4:20 PMSo it looks like something that happened after broker received the response from the server...
Elon
06/30/2021, 4:20 PMServer completed the response:
Copy code
Processed requestId=2,table=alltypes_REALTIME,segments(queried/processed/matched/consuming)=1/1/1/1,schedulerWaitMs=5,reqDeserMs=0,totalExecMs=4,resSerMs=0,totalTimeMs=12,minConsumingFreshnessMs=1625069569193,broker=Broker_f068e4137900_8099,numDocsScanned=11,scanInFilter=0,scanPostFilter=176,sched=fcfsj
Jackie
06/30/2021, 5:36 PMI can reproduce the issue, let me look further into it
❤️ 1
e
Elon
06/30/2021, 5:36 PMThanks!
Elon
06/30/2021, 5:39 PM@Jackie looks like it occurs after the
reduceOnDataTableElon
06/30/2021, 5:43 PMShould I create a github issue for it?
j
Jackie
06/30/2021, 5:57 PM@Elon Yes, please create an issue and we can post the findings there
➕ 1
e
Elon
06/30/2021, 5:57 PMwill do