https://linen.dev logo
Join Slack
Powered by
Hello - We are having issues running "index_parall...
# general
k
Hello - We are having issues running "index_parallel" tasks on a Druid 29.0.1 Cluster on GCP, thats configured to a DataProc cluster. The Data is in parquet format, and is hosted on a Google Storage Bucket. The Cluster is configured with the following extensions and any community extension is downloaded using the pull-deps tool.
Copy code
druid.extensions.loadList=["druid-kafka-indexing-service","druid-datasketches","druid-multi-stage-query","druid-google-extensions","druid-avro-extensions","druid-parquet-extensions","mysql-metadata-storage","druid-histogram","druid-lookups-cached-global","prometheus-emitter"]
When we run a "index_parallel" job with a text file either from google storage, the job runs successfully. When we run a "index_parallel" job with a parquet file from google storage, we get the following error (SEE IN THREAD) We found this begin a issue for other users: https://github.com/apache/druid/issues/10212, however their's is a much older version of Druid. Does anyone have any troubleshooting tips or suggestions, to get to the bottom of the issue?
Copy code
2024-11-20T00:19:43,347 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.SecurityUtil - Setting hadoop.security.token.service.use_ip to false
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.shaded.org.xbill.DNS.ResolverConfig (file:/opt/apache-druid-29.0.1/extensions/druid-parquet-extensions/hadoop-client-runtime-3.3.6.jar) to method sun.net.dns.ResolverConfiguration.open()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.shaded.org.xbill.DNS.ResolverConfig
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-11-20T00:19:43,385 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.Groups -  Creating new Groups object
2024-11-20T00:19:43,386 DEBUG [task-runner-0-priority-0] org.apache.hadoop.util.NativeCodeLoader - Trying to load the custom-built native-hadoop library...
2024-11-20T00:19:43,386 DEBUG [task-runner-0-priority-0] org.apache.hadoop.util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
2024-11-20T00:19:43,386 DEBUG [task-runner-0-priority-0] org.apache.hadoop.util.NativeCodeLoader - java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2024-11-20T00:19:43,386 WARN [task-runner-0-priority-0] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-11-20T00:19:43,387 DEBUG [task-runner-0-priority-0] org.apache.hadoop.util.PerformanceAdvisory - Falling back to shell based
2024-11-20T00:19:43,387 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2024-11-20T00:19:43,401 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
2024-11-20T00:19:43,410 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.UserGroupInformation - Hadoop login
2024-11-20T00:19:43,410 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.UserGroupInformation - hadoop login commit
2024-11-20T00:19:43,411 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.UserGroupInformation - Using local user: UnixPrincipal: druid-app
2024-11-20T00:19:43,411 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.UserGroupInformation - Using user: "UnixPrincipal: druid-app" with name: druid-app
2024-11-20T00:19:43,411 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.UserGroupInformation - User entry: "druid-app"
2024-11-20T00:19:43,411 DEBUG [task-runner-0-priority-0] org.apache.hadoop.security.UserGroupInformation - UGI loginUser: druid-app (auth:SIMPLE)
2024-11-20T00:19:43,412 DEBUG [task-runner-0-priority-0] org.apache.hadoop.fs.FileSystem - Starting: Acquiring creator semaphore for <hdfs://XXXXXX:8020>
2024-11-20T00:19:43,412 DEBUG [task-runner-0-priority-0] org.apache.hadoop.fs.FileSystem - Acquiring creator semaphore for <hdfs://XXXXX:8020>: duration 0:00.000s
2024-11-20T00:19:43,413 DEBUG [task-runner-0-priority-0] org.apache.hadoop.fs.FileSystem - Starting: Creating FS <hdfs://XXXXX:8020>
2024-11-20T00:19:43,413 DEBUG [task-runner-0-priority-0] org.apache.hadoop.fs.FileSystem - Loading filesystems
2024-11-20T00:19:43,416 DEBUG [task-runner-0-priority-0] org.apache.hadoop.fs.FileSystem - Creating FS <hdfs://XXXXX:8020>: duration 0:00.003s
2024-11-20T00:19:43,419 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider - handler[index_parallel_agg_adi_ump_flight_audience_laanlalf_2024-11-20T00:19:38.225Z] not currently registered, ignoring.
2024-11-20T00:19:43,419 DEBUG [task-runner-0-priority-0] org.apache.druid.indexing.common.task.AbstractTask - Not pushing task logs and reports from task.
2024-11-20T00:19:43,419 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Uncaught Throwable while running task[AbstractTask{id='index_parallel_agg_laanlalf_2024-11-20T00:19:38.225Z', groupId='index_parallel_agg_laanlalf_2024-11-20T00:19:38.225Z', taskResource=TaskResource{availabilityGroup='index_parallel_agg_2024-11-20T00:19:38.225Z', requiredCapacity=1}, dataSource='agg, context={forceTimeChunkLock=true, useLineageBasedSegmentAllocation=true}}]
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
	at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
	at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) ~[?:?]
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
	at java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
	at java.lang.ClassLoader.defineClass(ClassLoader.java:1017) ~[?:?]
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621) ~[?:?]
	at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579) ~[?:?]
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:576) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
	at java.lang.Class.forName0(Native Method) ~[?:?]
	at java.lang.Class.forName(Class.java:398) ~[?:?]
	at java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1210) ~[?:?]
	at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221) ~[?:?]
	at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265) ~[?:?]
	at java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300) ~[?:?]
	at java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3518) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3566) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3608) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174) ~[?:?]
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712) ~[?:?]
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557) ~[?:?]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290) ~[?:?]
	at org.apache.druid.data.input.parquet.ParquetInputFormat.initialize(ParquetInputFormat.java:68) ~[?:?]
	at org.apache.druid.data.input.parquet.ParquetInputFormat.createReader(ParquetInputFormat.java:98) ~[?:?]
	at org.apache.druid.data.input.impl.InputEntityIteratingReader.lambda$read$0(InputEntityIteratingReader.java:74) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:82) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.hasNext(CloseableIterator.java:93) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$1.hasNext(CloseableIterator.java:42) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.FilteringCloseableInputRowIterator.hasNext(FilteringCloseableInputRowIterator.java:66) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.data.input.HandlingInputRowIterator.hasNext(HandlingInputRowIterator.java:63) ~[druid-processing-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:96) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:945) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:548) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1214) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:551) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450) ~[druid-indexing-service-29.0.1.jar:29.0.1]
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-32.0.1-jre.jar:?]
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) ~[guava-32.0.1-jre.jar:?]
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-32.0.1-jre.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
	at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
	at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
	... 57 more
Error!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
	at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:213)
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:380)
	at org.apache.druid.cli.Main.main(Main.java:112)
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
	at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:571)
	at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
	at org.apache.druid.indexing.worker.executor.ExecutorLifecycle.join(ExecutorLifecycle.java:210)
	... 2 more
Copy code
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FileSystem
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:576)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:398)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1210)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1221)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
	at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3518)
	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3566)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3608)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3712)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3663)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:290)
	at org.apache.druid.data.input.parquet.ParquetInputFormat.initialize(ParquetInputFormat.java:68)
	at org.apache.druid.data.input.parquet.ParquetInputFormat.createReader(ParquetInputFormat.java:98)
	at org.apache.druid.data.input.impl.InputEntityIteratingReader.lambda$read$0(InputEntityIteratingReader.java:74)
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:82)
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.hasNext(CloseableIterator.java:93)
	at org.apache.druid.java.util.common.parsers.CloseableIterator$1.hasNext(CloseableIterator.java:42)
	at org.apache.druid.indexing.common.task.FilteringCloseableInputRowIterator.hasNext(FilteringCloseableInputRowIterator.java:66)
	at org.apache.druid.data.input.HandlingInputRowIterator.hasNext(HandlingInputRowIterator.java:63)
	at org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:96)
	at org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:945)
	at org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:548)
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179)
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1214)
	at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:551)
	at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179)
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478)
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 57 more

2024-11-20T00:19:43,423 INFO [Thread-309] org.apache.druid.cli.CliPeon - Running shutdown hook
2024-11-20T00:19:43,424 INFO [Thread-309] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
We found this thread on github, thats related to our problem: https://github.com/apache/druid/pull/12827
SOLUTION: Deploy the Dataproc gcs-connector-hadoop3 jar to the GOOGLE EXTENSION Folder, instead of the LIB Folder.
3 Views
Open in Slack
PreviousNext
Powered by