All - We are facing a strange issue where the MSQ query controller just hangs when there is no data in the source Druid table (indexing from one Druid table to another or reindexing into same Druid table which is loaded from Kafka) for the specified time interval and criteria. The query controller doesn't launch any workers but doesn't end either. It sits in "RUNNING" state perpetually (we left it for almost 4 hours 🙂). I tried this from the web console as well with the same result (hanging query controller). The same issue doesn't arise when we ingest from files on S3 using EXTERN. If there is no data for the specified time interval and criteria, the query controller fails with "InsertCannotBeEmpty: No rows to insert for dataSource [..]" error and cancels all the spawned workers. Has anyone faced this issue before? Druid Version: 24.0.2 Thanks, AR.

If possible, try upgrading to the latest versions. I believe a few of these bugs were fixed post Druid 24

Thanks Laksh. We are trying to upgrade to a newer version. InsertCannotBeEmpty --> This is actually fine and gives a clear indication of why the query controller failed. The issue is when the Query controller just sits in "RUNNING" state and doesn't end (with an error or otherwise). Thanks, AR.

can you take a jstack of the controller task and share it here please

Is it waiting for locks ?

@Karan Kumar There was no lock on the time interval. I confirmed that. If there was a lock, then it would go into "WAITING" state for the lock to be released, correct? Just to be sure, I tried running the same REPLACE INTO OVERWRITE query from the console into a new Druid table. Same result, the query controller goes into RUNNING state and just waits. It doesn't launch any workers or tasks. @Laksh Singla I checked the log file of the controller. There were no exceptions in it. It didn't seem to be any resource related issue since it continued in "RUNNING" state for several hours and shut down promptly when I "killed" it manually from the console - no errors in the log file. There was no resource issue on the host as other tasks were running fine on it. I will see if I can get the jstack of the query controller. Thanks, AR.

yeah a jstack would be helpful. I vaguely remember seeing something similar a while back in older versions of MSQ. I am unable to find the github link for the same, but I can’t confirm or deny that it’s the same issue. Upgrading to the newest version would be helpful

Please take 4-5 jstacks. Each jstack should be taken after every 10-15 seconds

We are trying to upgrade to v27 which is the latest available in our internal repository. Will try to get the jstacks and share. Cumbersome as Slack is not accessible from within our org. 😞 Thanks, AR.