https://linen.dev logo
Join Slack
Powered by
Hi Druids! I am having trouble setting up tls wit...
# general
b
Hi Druids! I am having trouble setting up tls with druid. Before adding TLS, everything was running smoothly, and I could create users and authenticators But now, after enabling TLS, The UI on port 9088 is throwing an error that says: 
Caused by:</h3><pre>org.apache.druid.java.util.common.IAE: No userMap is available for authenticator with prefix: [MyBasicMetadataAuthenticator]
Also, port 8281 isn't opening up,, The corrdinator overload doesn't have any errors. but the broker shows the below error
Copy code
WARN [main] org.apache.druid.discovery.BaseNodeRoleWatcher - Watcher for node role [coordinator] returned an empty collection.
2024-05-27T09:32:33,021 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (2 of 9) in 2,675ms.
org.apache.druid.java.util.common.IOE: A leader node could not be found for [COORDINATOR] service. Check logs of service [COORDINATOR] to confirm it is healthy.
'm a bit stuck on this and would really appreciate any help or advice you can offer. Thank you
s
you need to paste here your tls configuration
b
Sure here it is
druid conf.rtf
This is my common.runtime.properties
I followed the instruction here to create tls certificates https://druid.struct.ai/2K79c5
s
• the 1st error thrown does not have to do with TLS, but with basic Auth. Your basic auth though appears to be correctly configured as far as i can see. • druid.server.https.validateHostnames -> Are you sure this is valid config? There is nowhere in the documentation • Do you have the proper permissions in your keystore and trustore? • druid.client.https.trustStoreAlgorithm is not required, and I’m not sure where you took the PKIX value from. I wouldn’t set it if I were you, or I would try first having it commented out until i have a working solution
it is possible that your leader node can’t advertise through zookeeper, if it’s even running (as you said in the logs). Double check firewalls and that every node can reach each other in the new TLS ports, but also in the ports that zookeeper uses
i suppose your zookeeper runs on 2181? Can you try what is the result of 
echo conf | nc IP 2181
where IP is the coordinator instance hostname/IP?
also it would be helpful if you pasted what’s in zoo.conf
In my understanding your zoo.conf file should be something like:
Copy code
tickTime=2000
dataDir=var/zk
clientPort=2181
initLimit=5
syncLimit=2

autopurge.snapRetainCount=5
autopurge.purgeInterval=1

4lw.commands.whitelist=stat, ruok, conf, isro

server.1=host1:2888:3888
server.2=host2:2888:3888
.....
server.N=hostN:2888:3888
if you run zookeeper in N instances
check if instances can communicate in the necessary ports
remember your TLS ports are different than your plaintext ports (or not?)
b
Thank you, will check and get back to you.
For a single node quickstart, do i need to configure zookeeper?
s
Yes, I think so. You will just configure it with one node. Zookeeper will advertise your coordinator
b
I added the host in zoo.cfg I also wanted to add the below error in coordinator logs, 
Sync failed for server[<https://IP:8283/>] while [Handling response with code[500], description[Server Error]]. Failed [18] times in the last [732] seconds.. Try restarting the Druid process on server[<https://IP:8283/>].: {exceptionType=org.apache.druid.java.util.common.ISE, exceptionMessage=Received sync response [500], class=org.apache.druid.server.coordination.ChangeRequestHttpSyncer} (org.apache.druid.java.util.common.ISE: Received sync response [500])
Copy code
sudo ss -tuln
Netid                 State                   Recv-Q                  Send-Q                                   Local Address:Port                                   Peer Address:Port                 Process                 
udp                   UNCONN                  0                       0                                              0.0.0.0:68                                          0.0.0.0:*                                            
udp                   UNCONN                  0                       0                                              0.0.0.0:111                                         0.0.0.0:*                                            
udp                   UNCONN                  0                       0                                            127.0.0.1:323                                         0.0.0.0:*                                            
udp                   UNCONN                  0                       0                                                 [::]:111                                            [::]:*                                            
udp                   UNCONN                  0                       0                                                [::1]:323                                            [::]:*                                            
tcp                   LISTEN                  0                       4096                                           0.0.0.0:111                                         0.0.0.0:*                                            
tcp                   LISTEN                  0                       128                                            0.0.0.0:22                                          0.0.0.0:*                                            
tcp                   LISTEN                  0                       4096                                                 *:9088                                              *:*                                            
tcp                   LISTEN                  0                       4096                                                 *:8282                                              *:*                                            
tcp                   LISTEN                  0                       4096                                                 *:8283                                              *:*                                            
tcp                   LISTEN                  0                       4096                                                 *:8291                                              *:*                                            
tcp                   LISTEN                  0                       4096                                              [::]:111                                            [::]:*                                            
tcp                   LISTEN                  0                       128                                               [::]:22                                             [::]:*                                            
tcp                   LISTEN                  0                       50                                                   *:8080                                              *:*                                            
tcp                   LISTEN                  0                       50                                                   *:2181                                              *:*
zoo cfg file
Copy code
#
# Server
#

tickTime=2000
dataDir=var/zk
clientPort=2181
initLimit=5
syncLimit=2

#
# Autopurge
#

autopurge.snapRetainCount=5
autopurge.purgeInterval=1
server.1=IP:2888:3888
s
is this 
IP
set?
b
yes
s
did you restart the process?
b
Do you have any other document I could use. followed the instruction here to create tls certificates https://druid.struct.ai/2K79c5
I have now removed all the tls configuration. it is working fine as non tls as of now
Is there any doc for tls setting in one node environment.
s
Not sure to tell you the truth. Strictly talking trustore is not necessary in one node env because afaik it’s used for inter-node communication. In any case, if your cluster works fine without TLS setup, I would suggest to try to setup the same ports as the plaintext ports. Don’t forget that you will need to set the ports for each service. For example in historical/runtime.properties you should have a line: 
druid.tlsPort=8083
(or the port that the service should be configured. Same with all the rest. For example for coordinator runtime.properties: 
druid.tlsPort=8081
But your problem doesn’t seem to be cert oriented.
My settings:
Copy code
druid.client.https.trustStorePath=/opt/apache-druid/conf/druid/${startup_conf}/_common/truststore.jks
druid.client.https.trustStorePassword=${ksp}
druid.client.https.trustStoreType=jks
druid.client.https.protocol=TLSv1.2
druid.server.https.keyStorePath=/opt/apache-druid/conf/druid/${startup_conf}/_common/keystore.jks
druid.server.https.keyStorePassword=${ksp}
druid.server.https.keyStoreType=jks
druid.enablePlaintextPort=false
druid.enableTlsPort=true
and the process to create keystore and trustore:
Copy code
### Setup keystore for druid using self-signed cert for SSL/TLS setup ###
keytool -genkeypair -keyalg RSA -keysize 2048 -dname "CN=*.${dns_internal_zone}, OU=infra, O=example, C=ES" -ext SAN=dns:localhost,ip:$LOCALIP -alias example -keystore /opt/apache-druid/conf/druid/${startup_conf}/_common/keystore.jks -keypass ${ksp} -storepass ${ksp} -validity 365

keytool -certreq -keystore /opt/apache-druid/conf/druid/${startup_conf}/_common/keystore.jks -alias example -dname "CN=*.${dns_internal_zone}, OU=infra, O=example, C=ES" -ext SAN=dns:localhost,ip:$LOCALIP -file /tmp/example.csr -storepass ${ksp}

openssl x509 -req -CA /tmp/ca.pem -CAkey /tmp/ca.key -CAcreateserial -in /tmp/example.csr -out /tmp/example.pem -days 3650
keytool -importcert -keystore /opt/apache-druid/conf/druid/${startup_conf}/_common/keystore.jks -alias rootca -file /tmp/ca.pem -noprompt -keypass ${ksp} -storepass ${ksp}

keytool -importcert -keystore /opt/apache-druid/conf/druid/${startup_conf}/_common/keystore.jks -alias dino -file /tmp/example.pem -noprompt -keypass ${ksp} -storepass ${ksp}

### Setup trustore for druid for client connections ###
keytool -import -noprompt -trustcacerts -keystore /opt/apache-druid/conf/druid/${startup_conf}/_common/truststore.jks -storepass ${ksp} -file /tmp/ca.pem -alias examplerootca

keytool -import -noprompt -trustcacerts -keystore /opt/apache-druid/conf/druid/${startup_conf}/_common/truststore.jks -storepass ${ksp} -file /tmp/example.pem -alias examplecert

### Setup system-wide CA keystore for java ###
keytool -importcert -keystore /etc/ssl/certs/java/cacerts -alias examplerootCa -file /tmp/ca.pem -noprompt -storepass changeit

openssl x509 -in /tmp/ca.pem -out /tmp/ca.crt
cp /tmp/ca.crt /usr/local/share/ca-certificates/exampleca.crt && update-ca-certificates
### Cleanup
rm -f /tmp/example.pem /tmp/example.csr /tmp/ca.pem /tmp/ca.key /tmp/ca.crt
obviously some things in the process need to be changed
b
The below is my coordinator-overlord runtime properties do I need to add druid.tls port parameter. what should the value be
Copy code
cat runtime.properties
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   <http://www.apache.org/licenses/LICENSE-2.0>
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
#

druid.service=druid/coordinator
druid.plaintextPort=8081

druid.coordinator.startDelay=PT10S
druid.coordinator.period=PT5S
druid.manager.segments.pollDuration=PT5S

# Run the overlord service in the coordinator process
druid.coordinator.asOverlord.enabled=true
druid.coordinator.asOverlord.overlordService=druid/overlord

druid.indexer.queue.startDelay=PT5S

druid.indexer.storage.type=metadata
s
This is wrong. Since you don’t use plaintext port as you have put in your common.runtime.properties, you should remove all 
druid.plaintextPort
from all config files and replace these with 
druid.tlsPort
4 Views
Open in Slack
PreviousNext
Powered by