https://flink.apache.org/ logo
Join Slack
Powered by
I’m having trouble getting the filesystem plugins ...
# troubleshooting
h
I’m having trouble getting the filesystem plugins to be picked up by flink when I’m using Python. When I drop jars in the 
/opt/flink/lib
folder in my Dockerfile, they are picked up just fine. However, when I move plugin jars to the 
/opt/flink/plugins/<plugin>
folder(s), they are not picked up, resulting in errors like: 
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 's3'. The scheme is directly supported by Flink through the following plugin(s): flink-s3-fs-hadoop, flink-s3-fs-presto.
The weird thing - when I use the same docker image with the kubernetes operator, I can use s3/azure to store the snapshots and HA data, so it seems like there’s nothing wrong with the Dockerfile itself.
the plugins are really there (shown here: azure, same issue)
m
That’s the Azure plugin, not S3
Azure FS is for paths to wash:// or abfs://
h
Yeah, both have the same issue
Copy code
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'wasbs'. The scheme is directly supported by Flink through the following plugin(s): flink-fs-azure-hadoop.
I am not 100% sure I am calling my job correctly though, I am doing 
docker run --entrypoint python -t my-pyflink-image path/to/my/py.py
for testing - when I deploy I use the CRD. Haven’t tried if that fixes it, yet
Think I found the culprit (no fix yet). Following the docs, I have installed pyflink using pip inside the docker container. Even though 
FLINK_HOME
is set, pip still installs flink again, in a different dir (
/usr/local/lib/python3.10/dist-packages/pyflink
). Because the 
FLINK_HOME
var is set correctly, the path for 
lib
is picked up, but apparently 
plugins
is taken from the python installation, rather than the global one. So I guess I have two questions; 1. is there a better way to install pyflink, so it uses the same flink installation? 2. is there a better way to run python jobs (
python main.py / pytest
) inside docker?
Running with
Copy code
docker run <...> standalone-job --job-classname org.apache.flink.client.python.PythonDriver -py /opt/flink/app/main.py
has similar behaviour. When I add the azure jar as a plugin, I get the following error:
Copy code
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file"
Placing it in 
lib
works, and I can write to a csv file in blob). Writing parquet doesn’t. Placing the jar in 
plugins/azure-fs-hadoop
exits with the most unhelpful 
Python process exits with code: 1
Placing the jar in 
lib
complains about 
java.lang.NoClassDefFoundError: org/apache/hadoop/mapreduce/lib/output/FileOutputFormat
m
Parquet requires Hadoop dependencies
h
Arent 
flink-sql-*
jars supposed to be fat jars, bundling their dependencies? Is there any documentation on how to get this to work with python?
m
Connectors, yes. But Parquet is a format, not a connector.
h
Got it. So apart from this dependency I should install hadoop? In maven there are no runtime dependencies of that jar listed, how do I figure out what jars to add, if not all of them?
@Martijn Visser do you think it’s possible to use maven in tandem with pyflink? Build a jar like the python driver placeholder, but bundled with all dependencies? Doing manual dependency management is not fun, and everything I’ve tried has just made things worse.
v
@Huib are you able to fix this ... We are also running into same issue while writing the parquet files into AzureBlob ... files are showing in `inProgress`state
Copy code
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "file"
h
Yeah, adding the hadoop jars to the java path worked. We decided to ditch the direct flink -> blob route, and go through kafka instead, so I don’t have a working example ready to go
Open in Slack
PreviousNext
Powered by