This message was deleted.
# troubleshootings
Slackbot
03/01/2023, 9:08 AMThis message was deleted.
b
Bibek Sahoo
03/01/2023, 9:14 AMIf no tasks are running, then we could check overlord log for any kind of error or warning. Check the middle manager where it is submitted to we can check that log as well.
404 generally suggest server is not reachable, so check if your zookeeper is running fine and check for any errors in zookeeper/overlord log.
m
MarekM
03/01/2023, 9:20 AMI looked in the logs and can't find anything
MarekM
03/01/2023, 9:32 AMzookeeper working:
MarekM
03/01/2023, 9:32 AM2023-03-01T095545,395 INFO [main] org.apache.zookeeper.server.ContainerManager - Using checkIntervalMs=60000 maxPerMinute=10000
2023-03-01T095556,342 INFO [SyncThread:0] org.apache.zookeeper.server.persistence.FileTxnLog - Creating new log file: log.80c3
2023-03-01T095615,722 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290000, timeout of 30000ms exceeded
2023-03-01T095615,723 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290003, timeout of 30000ms exceeded
2023-03-01T095615,723 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290001, timeout of 30000ms exceeded
2023-03-01T095615,723 INFO [SessionTracker] org.apache.zookeeper.server.ZooKeeperServer - Expiring session 0x10003d4e8290002, timeout of 30000ms exceeded
MarekM
03/01/2023, 9:33 AMin middle: 2023-03-01T103216,635 WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-03-01T103217,726 INFO [main] com.google.inject.Guice - An exception was caught and reported. Message: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
java.lang.RuntimeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:59) ~[?:?]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.Elements.getElements(Elements.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.util.Modules$OverrideModule.configure(Modules.java:198) ~[guice-4.1.0.jar:?]
at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) ~[guice-4.1.0.jar:?]
at com.google.inject.spi.Elements.getElements(Elements.java:110) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) ~[guice-4.1.0.jar:?]
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) ~[guice-4.1.0.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:99) ~[guice-4.1.0.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:73) ~[guice-4.1.0.jar:?]
at com.google.inject.Guice.createInjector(Guice.java:62) ~[guice-4.1.0.jar:?]
at org.apache.druid.initialization.ExtensionInjectorBuilder.build(ExtensionInjectorBuilder.java:49) ~[druid-server-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
at org.apache.druid.initialization.ServerInjectorBuilder.build(ServerInjectorBuilder.java:118) ~[druid-server-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
at org.apache.druid.initialization.ServerInjectorBuilder.makeServerInjector(ServerInjectorBuilder.java:73) ~[druid-server-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:85) ~[druid-services-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62) ~[druid-services-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
at org.apache.druid.cli.Main.main(Main.java:112) ~[druid-services-26.0.0-SNAPSHOT.jar:26.0.0-SNAPSHOT]
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332) ~[?:?]
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352) ~[?:?]
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) ~[?:?]
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) ~[?:?]
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) ~[?:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) ~[?:?]
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226) ~[?:?]
at org.apache.druid.security.ranger.authorizer.RangerSecurityDruidModule.configure(RangerSecurityDruidModule.java:56) ~[?:?]
... 17 more
b
Bibek Sahoo
03/01/2023, 9:40 AMError “No FileSystem for scheme “hdfs”" seems to be a hdfs issue, could you check at hdfs side if there is for error message
Bibek Sahoo
03/01/2023, 9:42 AMWhat are the hdfs version and druid are they compatible and also check if libraries are same in below two locations , if not copy them and retry
copy all of the hadoop* jars from dist/druid/extensions/druid-hdfs-storage to dist/druid/extensions/druid-parquet-extensions
m
MarekM
03/01/2023, 9:57 AMI used hadoop-client-3.1.1 an druid 26.0.0-SNAPSHOT
MarekM
03/01/2023, 9:58 AMit worked for 2 months and suddenly stopped working
MarekM
03/01/2023, 9:59 AMcopying the jar didn't help
MarekM
03/01/2023, 10:07 AMi tried to change the reading from hdfs to local but it didn't work
s
Sergio Ferragut
03/02/2023, 12:39 AMyeah, local is meant for single server deployments with local deep storage. What are your deep storage settings?
druid.storage.*6 Views