We're saving Parquet data to S3 using the flink-s3-fs-hadoop plugin and checkpoints using the flink-s3-fs-presto plugin. We had problems with

java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration

, and they get solved by adding the following to

lib

directory with Ansible, but is this how we're supposed to fix this? Edit: version 1.17.1, and the plugins were copied as instructed by the manual, we're using pyflink.

flink_dependencies:
  - group_id: org.apache.flink
    artifact_id: flink-sql-parquet
    version: "{{ flink_version }}"
  - group_id: org.apache.hadoop
    artifact_id: hadoop-common
    version: "{{ hadoop_version }}"
  - group_id: org.apache.hadoop
    artifact_id: hadoop-mapreduce-client-core
    version: "{{ hadoop_version }}"
  - group_id: com.fasterxml.woodstox
    artifact_id: woodstox-core
    version: 5.3.0
  - group_id: org.codehaus.woodstox
    artifact_id: stax2-api
    version: 4.2.1
  - group_id: com.google.guava
    artifact_id: guava
    version: 11.0.2
  - group_id: commons-logging
    artifact_id: commons-logging
    version: 1.1.3

I believe you haven’t set the

HADOOP_CLASSPATH

option as outlined in https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#hadoop-dependencies

We have a standalone cluster in session mode, we don't have hadoop installed.

Parquet requires Hadoop

Ok, sort of makes sense. It might be good if the docs mention that you need to install hadoop if you use parquet. (It may be in the docs, but I haven't seen it.)

The tricky thing is where you start and stop with documenting those type of dependencies: the assumption is that if you use Parquet, you also understand that you have a dependency on Hadoop at this moment

I'm ok with this, but this whole thing is quite difficult for newcomers to set up.

Fully agreed. I'm all open for improvements and merging those 🙂