This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

09/21/2022, 2:44 PM

This message was deleted.

Gian Merlino

09/21/2022, 2:55 PM

We did change the way logging is done in the default startup scripts, mainly in: https://github.com/apache/druid/pull/10147, https://github.com/apache/druid/pull/12570

Gian Merlino

09/21/2022, 2:55 PM

Idea was to move the logs to rotated files so people's disks don't fill up

Gian Merlino

09/21/2022, 2:55 PM

If you have your own logging setup you may want to provide your own log4j2.xml

Julian Reyes

09/21/2022, 3:08 PM

not really own setup, just working from whats offered out of the box

Gian Merlino

09/21/2022, 3:26 PM

ah, which box are you talking about?

Gian Merlino

09/21/2022, 3:27 PM

I mean did you follow some particular documentation? Maybe we need to fix something

Julian Reyes

09/21/2022, 3:27 PM

docker image 24.0.0 being used in helm chart

Gian Merlino

09/21/2022, 3:27 PM

which helm chart?

Gian Merlino

09/21/2022, 3:28 PM

I'm wondering if we can repro what you're seeing somehow and make some fixes accordingly

Julian Reyes

09/21/2022, 3:28 PM

https://github.com/apache/druid/tree/master/helm/druid • updated https://github.com/apache/druid/blob/master/helm/druid/Chart.yaml#L17 to 24.0.0 • updated https://github.com/apache/druid/blob/master/helm/druid/values.yaml#L20 to 24.0.0

Julian Reyes

09/21/2022, 5:05 PM

added in configVars the following

Copy code

DRUID_LOG4J: '<?xml version="1.0" encoding="UTF-8" ?><Configuration status="WARN"><Appenders><Console name="Console" target="SYSTEM_OUT"><PatternLayout pattern="%d{ISO8601} %p [%t] %c - %m%n"/></Console></Appenders><Loggers><Root level="info"><AppenderRef ref="Console"/></Root><Logger name="org.apache.druid.jetty.RequestLog" additivity="false" level="DEBUG"><AppenderRef ref="Console"/></Logger></Loggers></Configuration>'

and getting logs in k9s

Sergio Ferragut

09/21/2022, 5:10 PM

Does that mean that it is writing to <service>.stdout.log instead of ${druid.node.type}.log ? I've been digging through the

dockerfile

and it seems to me that with default values.yaml it is not defining the setting

druid.node.type

by default (which should probably change) but given this line in the dockerfile's entrypoint script

druid.sh

, I think you can update your values.yaml for the `helm install`: As a workaround, I think if you add the config item to each service section, you should get the correct naming for each log as in:

Copy code

historical:
  config:
    druid_node_type: historical

Julian Reyes

09/21/2022, 5:12 PM

I think thats what I was missing

druid_node_type

Julian Reyes

09/21/2022, 5:17 PM

however, if do that and remove

DRUID_LOG4J

, would k9s pick up logs and will also datadog collect logs?

Sergio Ferragut

09/21/2022, 5:40 PM

I'm not very familiar with k9s or datadog, but that would return the default log file name in each service to: historical.log, broker.log, etc. which is perhaps what the k9s/datadog setup is expecting?

Julian Reyes

09/21/2022, 5:40 PM

with the current

DRUID_LOG4J

I see logs in k9s, but I can try adding the node type and see

Sergio Ferragut

09/21/2022, 5:41 PM

If your workaround is working, I wouldn't mess with it. This is just a different workaround. I think the dockerfile and or helm chart need to consider this.

Julian Reyes

09/21/2022, 5:43 PM

this is a testing env/cluster so no issues with playing and check

Julian Reyes

09/21/2022, 5:54 PM

changed the values in coordinator config :

druid_node_type: 'coordinator',

but logs in the dir keeps showing up as

${sys:druid.node.type}.log

Sergio Ferragut

09/21/2022, 5:56 PM

I think there is one advantage to reverting back to default log4j2.xml, in that it uses the rotating logs approach. Can you. share the whole coordinator section of your values.yaml?

Julian Reyes

09/21/2022, 5:56 PM

I am using pulumi to configure the helm values:

Copy code

coordinator: {
                    replicaCount: 3,
                    config: {
                        druid_node_type: 'coordinator',
                        DRUID_XMX: '8200m',
                        DRUID_XMS: '8200m',
                        druid_coordinator_period: 'PT60S',
                        druid_coordinator_startDelay: 'PT90S',
                        druid_coordinator_loadqueuepeon_type: 'http',
                        druid_coordinator_balancer_strategy: 'cachingCost',
                        druid_coordinator_kill_pendingSegments_on: 'true',
                        druid_coordinator_loadqueuepeon_http_batchSize: '8'
                    }
                },

Julian Reyes

09/21/2022, 5:58 PM

in runtime.properties I have

druid.node.type=coordinator

Julian Reyes

09/21/2022, 5:58 PM

so values are applied

Sergio Ferragut

09/21/2022, 6:00 PM

that is weird...

Sergio Ferragut

09/21/2022, 6:23 PM

So, the difference in execution single server setup and dockerfile paths is that in the latter, we are relying on runtime.properties to define this property and

druid.sh

is what writes it out to that file and in the former it is defined as part of the

java

startup command as a

-D

parameter. I'm wondering whether the runtime.properties files is begin used... can you share the section of the coordinator log where it outputs the system properties, in a standalone example I see this:

Copy code

2022-09-19T16:17:22,191 INFO [main] org.apache.druid.cli.CliCoordinator - * druid.monitoring.monitors: ["org.apache.druid.java.util.metrics.JvmMonitor"]
2022-09-19T16:17:22,191 INFO [main] org.apache.druid.cli.CliCoordinator - * druid.node.type: coordinator-overlord
2022-09-19T16:17:22,191 INFO [main] org.apache.druid.cli.CliCoordinator - * druid.plaintextPort: 8081

Do you see

druid.node.type

among them?

Julian Reyes

09/21/2022, 7:02 PM

Need to check that again but given that I got the other issue fixed I need to test other things first

5 Views

Open in Slack

Previous Next