You may also find permissions errors in your logs that would indicate the user doesn’t have the right perms.

In fact, I think you should see log messages of interest right at the start of when coord / over processes start

One area I tripped up on before was the right permissions

GRANT

--> https://druid.apache.org/docs/latest/development/extensions-core/mysql.html#setting-up-mysql

actually i tried using the "admin" user but again meta data not updated

just wondering whether i need to create any table in the DB?

Do you see any other errors in the logs?

Or warnings?

Do i need to add the mysql conection config to all process and nodes?

You could

grep

your logs for

druid.metadata.storage.connector

– if it’s applied, you’ll see lines showing up with the

connectURI

, and other stuff

2023-01-27T140500,313 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-25.0.0.jar2023-01-27T140500,406 INFO [main] org.apache.druid.cli.CliCoordinator - Coordinator is configured to act as Overlord as well (druid.coordinator.asOverlord.enabled = true).2023-01-27T140500,603 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-s3-extensions], jars: commons-logging-1.1.1.jar, httpcore-4.4.11.jar, aws-java-sdk-core-1.12.317.jar, jackson-dataformat-cbor-2.10.5.jar, ion-java-1.0.2.jar, druid-s3-extensions-25.0.0.jar, aws-java-sdk-sts-1.12.317.jar, commons-codec-1.13.jar, jmespath-java-1.12.317.jar, httpclient-4.5.13.jar, joda-time-2.10.5.jar2023-01-27T140500,607 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-multi-stage-query], jars: druid-multi-stage-query-25.0.0.jar2023-01-27T140500,620 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-lookups-cached-global], jars: druid-lookups-cached-global-25.0.0.jar, mapdb-1.0.8.jar2023-01-27T140500,623 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-parquet-extensions], jars: hadoop-mapreduce-client-core-2.8.5.jar, slf4j-api-1.7.36.jar, snappy-java-1.1.8.4.jar, commons-configuration-1.6.jar, javax.annotation-api-1.3.2.jar, hadoop-common-2.8.5.jar, parquet-jackson-1.12.0.jar, htrace-core4-4.0.1-incubating.jar, zstd-jni-1.5.2-3.jar, hadoop-auth-2.8.5.jar, parquet-avro-1.12.0.jar, parquet-format-structures-1.12.0.jar, jackson-core-asl-1.9.13.jar, avro-1.9.2.jar, parquet-hadoop-1.12.0.jar, parquet-common-1.12.0.jar, parquet-column-1.12.0.jar, druid-parquet-extensions-25.0.0.jar, audience-annotations-0.12.0.jar, hadoop-annotations-2.8.5.jar, hadoop-hdfs-client-2.8.5.jar, parquet-encoding-1.12.0.jar, jackson-mapper-asl-1.9.13.jar2023-01-27T140500,625 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-avro-extensions], jars: velocity-engine-core-2.2.jar, slf4j-api-1.7.36.jar, snappy-java-1.1.8.4.jar, javax.annotation-api-1.3.2.jar, lz4-java-1.8.0.jar, schema-repo-api-0.1.3.jar, jersey-common-2.30.jar, schema-repo-client-0.1.3.jar, kafka-schema-registry-client-5.5.1.jar, schema-repo-common-0.1.3.jar, druid-avro-extensions-25.0.0.jar, common-config-5.5.1.jar, zstd-jni-1.5.2-3.jar, swagger-annotations-1.6.0.jar, avro-1.9.2.jar, avro-ipc-jetty-1.9.2.jar, common-utils-5.5.1.jar, osgi-resource-locator-1.0.3.jar, jakarta.inject-2.6.1.jar, avro-mapred-1.9.2.jar, jersey-client-1.19.4.jar, schema-repo-avro-0.1.3.jar, gson-2.3.1.jar, jakarta.annotation-api-1.3.5.jar, avro-ipc-1.9.2.jar, kafka-clients-5.5.1-ccs.jar2023-01-27T140500,627 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [druid-aws-rds-extensions], jars: druid-aws-rds-extensions-25.0.0.jar, commons-logging-1.1.1.jar, httpcore-4.4.11.jar, aws-java-sdk-core-1.12.317.jar, jackson-dataformat-cbor-2.10.5.jar, aws-java-sdk-rds-1.12.317.jar, ion-java-1.0.2.jar, commons-codec-1.13.jar, jmespath-java-1.12.317.jar, httpclient-4.5.13.jar, joda-time-2.10.5.jar2023-01-27T140500,629 INFO [main] org.apache.druid.guice.ExtensionsLoader - Loading extension [mysql-metadata-storage], jars: mysql-metadata-storage-25.0.0.jar2023-01-27T140501,575 INFO [main] org.apache.druid.guice.JsonConfigurator - Skipping druid.emitter.logging.logLevel property: one of it's prefixes is also used as a property key. Prefix: druid2023-01-27T140501,844 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Loading MySQL metadata connector driver com.mysql.jdbc.Driver2023-01-27T140501,869 INFO [main] org.apache.druid.metadata.storage.mysql.MySQLConnector - Loading MySQL metadata connector driver com.mysql.jdbc.Driver2023-01-27T140501,870 INFO [main] org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}] for metrics and alerts, with dimensions

[Fri Jan 27 130428 2023] Command[coordinator-overlord] failed, see its logfile for more details [Fri Jan 27 130431 2023] Running command[coordinator-overlord]: bin/run-druid coordinator-overlord conf/druid/cluster/master [Fri Jan 27 130434 2023] Command[coordinator-overlord] exited (pid = 6183, exited = 1)

additional log

2023-01-27T134432,342 WARN [QuorumConnectionThread-[myid=1]-3] org.apache.zookeeper.server.quorum.QuorumCnxManager - Cannot open channel to 3 at election address /xx.xxx.xxx.106:3888 java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?] at java.net.Socket.connect(Socket.java:609) ~[?:?] at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:373) ~[zookeeper-3.5.9.jar:3.5.9] at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:436) ~[zookeeper-3.5.9.jar:3.5.9] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[

Please note that above error happens when i add MySQL configuration and mysql-metadata-storage extension in master server

also please note ZooKeeper is configured in Master Server and there are 3 nodes running Master Server

Also please note, i am using MySQL RDS version MySQL 8.0.28

I am just thinking for some reason ( may be due to mysql connectivity issue). master server is not able to start and hence ZK is also not able to start giving the above zookeeper issue

Is there a connectivity test that i can run from Matser Server to MySQL to confirm that connectivity is fine? something like metadata-init ?

Also lastly, do i need to download and keep following jars? mysql-metadata-storage-<DruidVersion>.jar mysql-connector-java-<MYSQLVERSION>.jar

# # Server # tickTime=2000 dataDir=/root/apache-druid-25.0.0/var/zk #dataDir=var/zk clientPort=2181 initLimit=5 syncLimit=2 # # Autopurge # autopurge.snapRetainCount=5 autopurge.purgeInterval=1 server.1=<ip-1>28883888 server.2=<ip-2>28883888 server.3=<ip-3>28883888

pleasec heck that the myid files are correct in each server

without zookeeper coming up you cannot check anything else

below are the steps followed to check connectivity from Master Server

1. cd ${DRUID_ROOT}

2. Ran below script

java -classpath "lib/*" -Dlog4j.configurationFile=conf/druid/cluster/_common/log4j2.xml -Ddruid.extensions.directory="extensions" -Ddruid.extensions.loadList="[\"mysql-metadata-storage\"]" -Ddruid.metadata.storage.type=mysql -Ddruid.node.type=metadata-init org.apache.druid.cli.Main tools metadata-init --connectURI="jdbcmysql//<DNS_Name>:3306/<DB_Name>" --user <user_name> --password <pwd> --base druid

At first run, metadata-init.log created with error : Could not find com.mysql.jdbc.Driver on the classpath.

After placing below two jars in ClassPath, and re-running script, connectivity worked and also we were able to start the cluster

mysql-metadata-storage-<DruidVersion>.jar MySQL JDBC connectormysql-connector-java-<MYSQLVERSION>.jar