This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

10/12/2022, 7:57 PM

This message was deleted.

Adam Peck

10/12/2022, 8:00 PM

Task logs by default just come off the local disk of the indexer. If you want them to be stored longer term you can set the logs to go to S3 or some other deep storage. See: https://druid.apache.org/docs/latest/configuration/index.html#task-logging

JAY PATEL

10/12/2022, 8:10 PM

that’s what I have, set to s3.

Copy code

# Task launch parameters
        druid.indexer.task.baseTaskDir=var/druid/task
        druid.indexer.logs.type=s3
        druid.indexer.logs.s3Bucket=druid-deep-storage-test
        druid.indexer.logs.s3Prefix=druid/indexer-log
        druid.indexer.logs.disableAcl=true

Sergio Ferragut

10/12/2022, 8:31 PM

You can control the task log retention policy https://druid.apache.org/docs/latest/configuration/index.html#log-retention-policy

Sergio Ferragut

10/12/2022, 9:20 PM

When you set up the s3 logs, task logs are almost immediately moved from local storage to deep storage. The retention of those logs is independent of where they are stored.

druid.indexer.logs.kill.initialDelay

- how long after overlord start should the first log removal cycle occur

druid.indexer.logs.kill.delay

- how long between log removal cycles

druid.indexer.logs.kill.durationToRetain

- duration of log retention

JAY PATEL

10/13/2022, 2:07 PM

Thanks for that insight, I’m seeing logs in s3 but not in task console it self 🤔 which is weird and I thought it might be related to log4j?

Copy code

jvm.options: |-
    -server
    -XX:MaxDirectMemorySize=10240g
    -Djava.io.tmpdir=/druid/data
    -Dlog4j.debug
    -Dlog4j.shutdownHookEnabled=true
    -Duser.timezone=UTC
    -Dfile.encoding=UTF-8
    -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
    -Daws.region=us-east-1
  log4j.config: |-
    <?xml version="1.0" encoding="UTF-8" ?>
    <Configuration status="WARN">
        <Appenders>
            <Console name="Console" target="SYSTEM_OUT">
                <PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/>
            </Console>
        </Appenders>
        <Loggers>
            <Root level="info">
                <AppenderRef ref="Console"/>
            </Root>
        </Loggers>
    </Configuration>

Mark Herrera

10/13/2022, 5:54 PM

Jay, which version of Druid are you using? Someone on the Forum is reporting something similar after an upgrade to 24.

JAY PATEL

10/14/2022, 5:19 PM

yes 24.

Mark Herrera

10/14/2022, 5:48 PM

The other community member might create an issue on GitHub. Here's the thread. I'll link the other community member to this discussion as well.

👍 1

Gian Merlino

10/14/2022, 10:41 PM

I think what you're seeing is the

druid.indexer.storage.recentlyFinishedThreshold

Gian Merlino

10/14/2022, 10:43 PM

It governs how long complete tasks show up in the

sys.tasks

table, and therefore, which show up in the web console. By default, it's 24 hours, meaning complete tasks stop showing up after 24 hours. It's there to keep the size of the sys table manageable. However, the logs and metadata are still there in deep storage and metadata store, until and unless they are "killed". You can retrieve them using all the regular task APIs, if you know the task ID.

Gian Merlino

10/14/2022, 10:43 PM

You can also extend

druid.indexer.storage.recentlyFinishedThreshold

and they will "magically" show back up 🙂 This is because

sys.tasks

is really just a view on the

druid_tasks

table in metadata store, filtered by the recently-finished threshold.

👍 1

Open in Slack

Previous Next