Apache Druid

http 504 is a timeout; it's certainly possible there's so many tasks that it takes too long for the call to come back

i'm not sure what version you're using but this was improved somewhat in recent versions

that being said if you have tons of tasks in the relevant time window, then reducing `druid.indexer.storage.recentlyFinishedThreshold` would certainly help with the calls the console makes

i typically set it to `PT24H` or `PT48H` in very busy clusters to minimize the times those API calls take

note that you can retrieve info for a specific older task by API if you know the ID

Hi Gian, thank you very much! That is really helpful! We will reduce the `druid.indexer.storage.recentlyFinishedThreshold` to improve the performance, and use API call to retrieve old task logs.

Quick question: do you know which version contains the improvement?

<@U0344FW86DD> Hi Gian, another quick question: does setting `druid.indexer.storage.recentlyFinishedThreshold=P1W` mean that the sys.tasks table would only contain 1 week of ingestion logs, or would the table still has all the ingestion logs in the past, but just that when querying ingestion logs the time would be restricted to past 1 week? Thanks in advance!

the task metadata would still be there in your metadata store (mysql/postgresql) but not visible in `sys.tasks`

(this is the main thing that speeds up the API calls)

Oh I see. That makes sense now. Thank you Gian!