While scrolling through the Apache Flink JIRA, I came across this issue that mentions that Hadoop 2.8.3 are no longer supported as of Flink 1.15. I’m in the process of updating the job which relies handling this within the Dockerfile through the use of the shaded uber-jars something like this:

ENV GCS_CONNECTOR_VERSION="hadoop2-2.1.1"
ENV FLINK_HADOOP_VERSION="2.8.3-10.0"
ENV GCS_CONNECTOR_NAME="gcs-connector-${GCS_CONNECTOR_VERSION}.jar"
ENV GCS_CONNECTOR_URI="$artifactStorage/${GCS_CONNECTOR_NAME}"
ENV FLINK_HADOOP_JAR_NAME="flink-shaded-hadoop-2-uber-${FLINK_HADOOP_VERSION}.jar"
ENV FLINK_HADOOP_JAR_URI="$artifactStorage/flink-shaded-hadoop-2-uber/${FLINK_HADOOP_VERSION}/${FLINK_HADOOP_JAR_NAME}"

I was hoping to upgrade this to a more recent version but I’m struggling to find the latest shaded plugin. I see these Flink Shaded Hadoop 3 Uber ones within Maven Central, but I’m not entirely sure what version of the connector or Hadoop that those would correspond to. Any advice? Just trying to upgrade this to the latest supported version as of 1.15 while I’m already touching the job and related things about the deployment.

Those Flink Shaded Hadoop 3 Uber ones aren’t from ASF Flink, but from vendors

Thanks Martjin, what’s the best way to go about upgrading Hadoop in that case give the current way it’s being handled?

If you need anything with Hadoop, this is the default manual I always refer to https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#hadoop-dependencies

Yup, I had come across that. Hadoop in this case is really only used for handing interactions with GCS (since we use that to store checkpoints/savepoints, etc.)

I’ve seen a similar request. I’m wondering if it’s not possible to already shade in the needed Hadoop dependencies for GCS. I believe that’s also done for S3 Presto and S3 Hadoop

So reading through these docs, it makes it seem that I wouldn’t need to add these Hadoop jars as part of the Dockerfile? Some of the docs make it seem as long as I have the

GOOGLE_APPLICATION_CREDENTIALS

defined at the job/task manager levels, that it should be able to write to GCS for checkpointing/savepointing purposes. Does that seem correct?

Good to hear @Rion Williams

I’m not entirely sure how to show any additional logging when looking at the bucket itself. The 40-50% client error rate seems to be consistent across multiple environments and both regional and multiregion buckets. I’d assume that because they are specifically client errors that those would be originating from Flink itself. This particular bucket handles quite a few different Flink jobs (~6-7 depending on the environment) all of which handle checkpointing at various intervals (from 10s to 10m). I tried to adjust the checkpointing intervals but it doesn’t look like that had much of an effect on this though.