https://flink.apache.org/ logo
Join Slack
Powered by
Hello all! Question about the Flink SQL Client: ...
# troubleshooting
n
Hello all! Question about the Flink SQL Client: I've set up a fluss cluster where flink is needed. To make quick queries on the database, I've tried using the flink sql client. But queries only works when I use "LIMIT 10". If I don't, I get this error:
Copy code
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.iceberg.data.IcebergGenericReader
        ... 15 more
And the full error in the jobmanager out file:
Copy code
java.lang.RuntimeException: One or more fetchers have encountered exception
  at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcherManager.checkErrors(SplitFetcherManager.java:333) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at org.apache.flink.connector.base.source.reader.SourceReaderBase.getNextFetch(SourceReaderBase.java:228) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:190) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:422) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) ~[flink-dist-1.20.0.jar:1.20.0]
  at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) ~[flink-dist-1.20.0.jar:1.20.0]
  at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.NoClassDefFoundError: org/apache/iceberg/data/IcebergGenericReader
  at org.apache.fluss.lake.iceberg.source.IcebergRecordReader.<init>(IcebergRecordReader.java:65) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.lake.iceberg.source.IcebergLakeSource.createRecordReader(IcebergLakeSource.java:100) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.lake.reader.LakeSnapshotScanner.pollBatch(LakeSnapshotScanner.java:54) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.BoundedSplitReader.pollBatch(BoundedSplitReader.java:128) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.BoundedSplitReader.poll(BoundedSplitReader.java:121) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.BoundedSplitReader.readBatch(BoundedSplitReader.java:76) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.FlinkSourceSplitReader.fetch(FlinkSourceSplitReader.java:154) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:117) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
  at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
  ... 1 more
Caused by: java.lang.ClassNotFoundException: org.apache.iceberg.data.IcebergGenericReader
  at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) ~[?:?]
  at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) ~[?:?]
  at java.lang.ClassLoader.loadClass(ClassLoader.java:526) ~[?:?]
  at org.apache.fluss.lake.iceberg.source.IcebergRecordReader.<init>(IcebergRecordReader.java:65) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.lake.iceberg.source.IcebergLakeSource.createRecordReader(IcebergLakeSource.java:100) ~[fluss-lake-iceberg-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.lake.reader.LakeSnapshotScanner.pollBatch(LakeSnapshotScanner.java:54) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.BoundedSplitReader.pollBatch(BoundedSplitReader.java:128) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.BoundedSplitReader.poll(BoundedSplitReader.java:121) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.BoundedSplitReader.readBatch(BoundedSplitReader.java:76) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.fluss.flink.source.reader.FlinkSourceSplitReader.fetch(FlinkSourceSplitReader.java:154) ~[fluss-flink-1.20-0.9.1-incubating.jar:0.9.1-incubating]
  at org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:165) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:117) ~[flink-connector-files-1.20.0.jar:1.20.0]
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
  at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
  ... 1 more
Any ideas on what the problem could be? I'm happy to provide any other information if necessary.
a
I'd guess it's because you're missing the required Iceberg dependencies in your flink deployment. I've never used Fluss, but I imagine that it only tries to read from the tiered iceberg layer when you don't provide a LIMIT
Hmm. Weird issue. I'm looking at the contents of the 
fluss-lake-iceberg-0.9.1-incubating.jar
file, and it should already include the 
org.apache.iceberg.data.IcebergGenericReader
class. Perhaps check with the Fluss community. Apache Fluss has a dedicated slack
Where did you put 
fluss-lake-iceberg-0.9.1-incubating.jar
? In $FLINK_HOME/lib/, or is it coming in with the SQL Client session (--jar / ADD JAR / pipeline.jars)? If it's session-scoped, try to move it to $FLINK_HOME/lib/ on JobManager and all TaskManagers and restart.
It definitely some funky class loading issue. 
fluss-lake-iceberg-0.9.1-incubating.jar
seems to be a self-contained fat jar that includs all required dependencies. It could be that other iceberg-*-runtime / iceberg-core jars har present on the classpath that are conflicting with the fluss jar. If there are any try to remove them
n
Thank you for the response! We've tried putting the fluss-lake-iceberg jar in multiple places in different combinations. Here's the ones we've tried: fluss/lib fluss/plugins/iceberg flink/lib
a
When you search do other iceberg jar show up? They could be conflicting
(heading out the office now, so I won't reply until tomorrow. Good luck troubleshooting 🙂 )
2 Views
Open in Slack
PreviousNext
Powered by