Stacktrace.txt

This warning from the logs might be related:

2023-04-05 23:15:22,125 WARN  org.apache.hadoop.util.NativeCodeLoader                      [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

After going in the wrong direction for a while I've found this mention in the unit tests (link to highlight). The dependency on local file blocks is not documented anywhere and it doesn't fall back if there isn't a local disk or the scheme for file:// wasn't provided (also, maybe it should use local:// instead).

Looks like setting

fs.azure.data.blocks.buffer: "bytebuffer"

alone won't fix the problem. I'm getting the following missing dependency errors. I think Flink itself should be providing these classes, so I'm going to switch back to the container I built from source instead of the one where I just replaced the azure-fs-plugin.

Did a clean install and still getting the error, so I don't think its a version incompatibility. Could it be due to missing hadoop-common jars?

After rereading some documentation I think it could be due to a dependency missing a

<scope>provided

tag.

I went back through all my dependencies to ensure I'm not missing any

<scope>provided</scope>

tags. I use custom KeyProvider implementations, and normally I just copy my fat jar into

/opt/flink/plugins/azure-fs-hadoop/

. That has worked fine for reading from Azure, but I was worried it could be causing duplicated classes resulting in the

NoClassDefFoundError : "org/apache/hadoop/conf/Configuration"

, so I made a separate jar with just the KeyProvider classes but then I get errors that my KeyProviders can't be found. I also tried putting my entire jar into the

$FLINK_HOME/lib

folder (see here), but that didn't help because my KeyProvider classes couldn't be found thereafter. I'll work on that, but my solution worked for reading, so it should work for writing too. Also, the error doesn't match the described classloading problems (docs). Now I'm thinking that maybe I need to include a Hadoop installation inside my docker image so that Flink can load the missing

Configuration

class as described here. In the documentation it says the ADLS connector is self-contained (shouldn't need Hadoop), so I don't think I should need to include Hadoop just to use the ADLS connector. Thus, I think perhaps the ADLS connector or my code must be missing the compile dependency that should contain

org/apache/hadoop/conf/Configuration

.

To resolve

NoClassDefFoundError : "org/apache/hadoop/conf/Configuration"

is there a

<scope>compile</scope>

dependency that should be added?

I think the problem is this exclusion (link-to-highlight).

flink-gs-fs-hadoop

and

flink-s3-fs-hadoop

seem to have

hadoop-common

as dependencies while

flink-azure-fs-hadoop

and

flink-oss-fs-hadoop

have it excluded but not restored.

Similar to a solution on StackOverflow I just needed to add some missing dependencies in compile scope: org.apache.hadoop:hadoop-common, flink-hadoop-compatibility_2.12, and hadoop-mapreduce-client-core did the trick. I thought that hadoop-common shouldn't be in user jars, but my new understanding is if you don't have Hadoop on your cluster there isn't another way to provide the classes. The source as in master works fine, so no need to change how Hadoop classes are provided or anything. Was able to write output!

Although I was able to successfully write some output with the ADLS2 connector, I'm facing problems with Hadoop using too much memory for staging writes. If I use

fs.azure.data.blocks.buffer: "bytebuffer"

it exceeds off-heap memory. If I use

fs.azure.data.blocks.buffer: "array"

it exceeds heap memory. I almost wonder if there is a leak. In the case it really does need this much memory I need to be able to use the

fs.azure.data.blocks.buffer: "disk"

option with

fs.azure.buffer.dir: local:///mnt/pvc-local/tmp_azure-fs_buffer

, but despite setting those values I get the

org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file"

error. I've tried supplying

flink.hadoop.fs.file.impl: org.apache.flink.core.fs.local.LocalFileSystem

and

flink.hadoop.fs.defaultFS: local://

to no effect. I'm wondering if somthing is preventing Flink from configuring Hadoop correctly. This is the PR that introduced the new pattern to Hadoop. I could try an older version of Hadoop.

Do I need to do all the stuff in this tutorial or just download and untar the Hadoop dependency?: https://phpfog.com/creating-hadoop-docker-image/