Hi everyone,
Trying out a simple job in pyflink ...
# troubleshootingr
Ryner Menezes
02/10/2023, 4:56 AMHi everyone,
Trying out a simple job in pyflink to read data from a file in adls using the data stream api
• The required jar file ‘flink-azure-fs-hadoop-1.16.0.jar’ is added to the plugins folder of both task and job managers and being called to the execution env using add_jar.
Yet, keep running into this error ‘org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'abfs'.’
required authentication access keys from the storage account are already added into the flink config yaml file.
Is there something that I’m missing out on?
m
Martijn Visser
02/10/2023, 7:07 AMFile systems are loaded via plugins, not via ADD JAR. They need to be added to a sub folder in the plugins directory, don’t forget to restart your cluster when that’s done
r
Ryner Menezes
02/10/2023, 3:17 PM@Martijn Visser Thanks for clarifying. Yes, I have added the jar file to a subfolder under plugins directory and restarted the cluster - no luck.
m
Martijn Visser
02/10/2023, 3:20 PMPlease check the logs if the JAR is loaded when starting the cluster. Perhaps it's an ownership issue on the file that causes issues
r
Ryner Menezes
02/10/2023, 6:22 PM@Martijn Visser I just tried searching for the jar files in the logs, when the cluster starts. I do not see it being added. I changed the permissions from root to flink. The error I get now is invalid authority for the file I'm trying to access.
On the logs I still do not see the jar getting added to classpath, only the ones mentioned in /opt/flink/lib folder can be seen getting added.
Ryner Menezes
02/10/2023, 6:34 PM Copy code
Caused by: abfs://....
has invalid authority. Copy code
Classpath: /opt/flink/lib/flink-azure-fs-hadoop-1.16.0.jar:/opt/flink/lib/flink-cep-1.16.0.jar:/opt/flink/lib/flink-connector-files-1.16.0.jar:/opt/flink/lib/flink-csv-1.16.0.jar:/opt/flink/lib/flink-json-1.16.0.jar:/opt/flink/lib/flink-scala_2.12-1.16.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.5.9.jar:/opt/flink/lib/flink-table-api-java-uber-1.16.0.jar:/opt/flink/lib/flink-table-planner-loader-1.16.0.jar:/opt/flink/lib/flink-table-runtime-1.16.0.jar:/opt/flink/lib/log4j-1.2-api-2.17.1.jar:/opt/flink/lib/log4j-api-2.17.1.jar:/opt/flink/lib/log4j-core-2.17.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.17.1.jar:/opt/flink/lib/flink-dist-1.16.0.jar::::m
Martijn Visser
02/10/2023, 6:35 PMYou’re not using Docker or anything I hope?
r
Ryner Menezes
02/10/2023, 6:36 PMTesting on docker, running as standalone.
m
Martijn Visser
02/10/2023, 6:40 PMI guess https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/docker/#using-filesystem-plugins is then applicable for you
r
Ryner Menezes
02/10/2023, 10:41 PMThanks @Martijn Visser This definitely was a useful link, we tried adding these to the env variables, also through the command mentioned. But same error still.
Copy code
'environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: jobmanager
flink.execution.mode: streaming
flink.enable-built-in-plugins: "true"'3 Views