This message was deleted.

Those logs are very valuable for my customers to self-help themselves.

☝️ this is definitely true! there's technical reasons why it doesn't happen: we don't have an easy way to redirect logs for different tasks to different files when they run in the same JVM with MM, we do it by capturing stdout of the forked

java

process

we don't have an easy way to redirect logs for different tasks to different files when they run in the same JVM

could we try sifting log appenders?

maybe? i am not a log4j2 expert so i don't 100% know what is possible

Sorry for this dumb question, but if you have the taskID already, don't you have enough information to spawn a new file logger into the same folder location?

I was going through the log4j documentation and it seems like https://logging.apache.org/log4j/2.x/manual/appenders.html#RoutingAppender can help with dynamically changing the file to log to. I will try to go over it in detail in a while.

implementation note: an issue is shared thread pools: if you look through the logs for a task that ran on MM you will notice that there are lots of different threads in play, and in the Indexer some of these are shared across tasks. for example, the processing thread pool (for queries on realtime tasks) and the thread pool used for merging and pushing segments (in UnifiedIndexerAppenderatorsManager) are both shared in the Indexer model. this is one of the reasons the Indexer is cool (better resource sharing). but i believe it also means some attention is needed to capture all these logs (or, deciding we're ok with not capturing them)

I created a Github issue to track it: https://github.com/apache/druid/issues/13082

To add to @Didip Kerabat’s idea, @Gian Merlino - it should be "easyish" to track the log file per indexer task in a data structure accessible by the shared thread pools, no?

the thing that seems tricky to me is the logger running in the thread won't necessarily know what task it's associated with

I see. Yeah, at least in my experience, I mostly have been looking at the ingestion related logs. For queries, I mostly try to look at metrics and exceptions.

we'd certainly be able to capture stuff logged by the main task thread itself

This seems to be the highest value information for the use case we are discussing. Primarily I use these logs to diagnose failing ingestion.

I just did a few more investigation on this logging issue in the comments. https://github.com/apache/druid/issues/13082 What do y’all think?