This message was deleted Apache Druid #troubleshooting

Join Slack

This message was deleted.

# troubleshooting

Slackbot

02/28/2023, 8:13 PM

This message was deleted.

Gian Merlino

03/01/2023, 2:23 AM

I think that would happen if you have a line starting with

["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query"]

Rather than

druid_extensions_loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query"]

Gian Merlino

03/01/2023, 2:23 AM

try double-checking for that

Josh Walker

03/02/2023, 9:00 PM

Hey thanks for the help and sorry for the slow reply. I've been supplying an env var to the container:

Copy code

druid_indexer_runner_javaOptsArray=["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]

and

Copy code

druid_extensions_loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query", "druid-kafka-indexing-service"]

I'm fairly sure they are being provided to the containers correctly. It's potentially interesting to note that for the

..._javaOptsArray

variable if you remove the

-Duser.

and the

-DFile.

options it doesn't throw the parse error. It seems to throw the error always when it comes to the

loadList

variable tough.

Josh Walker

03/02/2023, 9:02 PM

Maybe I'm not doing things correctly but locally this setup is fine when you run it under docker-compose following the appropriate docs. Puzzling me at the moment

Gian Merlino

03/07/2023, 1:36 PM

hmm. i wonder if something, somewhere is splitting these lines?

Josh Walker

03/07/2023, 2:24 PM

Yeah I'm pretty stumped to be honest. These can be set instead in a config file and loaded that way though? I think... I'll need to go back and reorganise how I'm doing that config then

Josh Walker

03/13/2023, 5:02 PM

Okay so I'm not sure how to even debug more into this unfortunately. I've looked in the container and the env var seems to be set okay:

Copy code

bash-5.1$ echo $druid_indexer_runner_javaOptsArray
["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]
bash-5.1$ echo $druid_extensions_loadList
["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-multi-stage-query", "druid-kafka-indexing-service"]

I only see the following log output:

Copy code

2023-03-12T23:21:47+00:00 startup service
Setting 25.0.1.126= in /tmp/conf/druid/cluster/misc/druid.host/runtime.properties
sed: bad regex '["-server", "-Xmx1g", "-Xms1g", "-XX:MaxDirectMemorySize=3g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]=': Invalid range end

I reread over the docs and it seems that using env vars to set the config state is the preferred way of doing things - just that then it comes to production you may wish to split up your environment files into service specific ones. It works fine locally with docker so I don't know what's causing these weird issues. I guess the only option is to try the more production approach of having the

DRUID_COMMON_CONFIG

set to a custom config file, although that's going to have a bit more friction when testing just now.

Gian Merlino

03/13/2023, 9:01 PM

Setting 25.0.1.126=

looks like values are getting confused for keys

Gian Merlino

03/13/2023, 9:02 PM

I suspect the same thing is happening with the loadList; since the

sed

error you get is from a line of code in a script that is trying to deal with a key

Gian Merlino

03/13/2023, 9:02 PM

I am not sure why this is happening, but perhaps that hint will be helpful…

Josh Walker

03/13/2023, 9:06 PM

Thanks for your help on this, it's much appreciated. I think I'll maybe do a little more debugging - might try switching the druid version and such. After that I'm just going to have to try the

DRUID_COMMON_CONFIG

approach 🤷‍♂️

Gian Merlino

03/13/2023, 9:10 PM

wish I could help more, but I am not super familiar with this way of running druid!

Josh Walker

03/13/2023, 9:13 PM

I mean perhaps I'm going about things in a crazy way? I've just been attempting to transition the local docker based setup which works nicely into a cloud based container deploy. Didn't seem too crazy in my head but I'm new to druid so perhaps there's an easier way to go about this

Gian Merlino

03/13/2023, 9:21 PM

if you're using k8s try checking out #C04Q0047B4M, there's a bunch of people there using docker via k8s in production

👍 1

Josh Walker

03/13/2023, 11:15 PM

Okay so for the record I'm a fool 😂 Turns out it wasn't executing the

druid.sh

with any service name argument. Hence why everything was being put in

misc

as seen in

/tmp/conf/druid/cluster/misc/

. The lesson here is don't try to wrangle new software and a new orchestration tool at the same time.

Gian Merlino

03/13/2023, 11:17 PM

oof, i guess that's because

SERVICE="$1"

at the top of

druid.sh

Gian Merlino

03/13/2023, 11:18 PM

i wonder if the script can be improved to make that kind of error more obvious!

Josh Walker

03/13/2023, 11:21 PM

Looking at the docs inside that file:

Copy code

# It takes one required argument (the name of the service,
# e.g. 'broker', 'historical' etc). Any additional arguments
# are passed to that service.

I guess if an argument was strictly required then I'd expect it to error out if one wasn't provided. Maybe people have found a use for running it without one though? Perhaps even a warning message when the first argument is not a valid service name would be a possible improvement.

Open in Slack

Previous Next