What is the difference between the two docker hub repositories for Flink images? The reason I am asking is that the 1.15.2 docker images are not yet there in the "non-ASF" repo.

The new Docker images are not yet published because Docker required a new OS. There's a discussion thread on the Dev mailing list on it. Should be available tomorrow

Normally, we release to both. IMO its questionable whether we need to release to both,

library/flink

should suffice. There was a case in the past, where releasing to

library/flink

took a long time, because its requires a Docker maintainer to merge a PR.

apache/flink

is fully under our control. In my opinion, we should still get rid of it, because two identical repositories are work for us and confuse users.

Thanks for the clarification.

It's a discussion we had several times, but so far it always ends with a stalemate with me saying we should drop

apache/flink

, and xintong saying we should keep it. Note that redundancy is the least of my concerns;

library/flink

forces us through an actual docker image review cycle (detecting things like unsupported images), and the

library/flink

images are periodically rebuilt for security updates (which the

apache/flink

images are not).

How much do you feel we get delayed by the need to go through the docker hub official procedure, PR merge on their side etc.? Maybe we could try to kick off the process a day or two prior to publishing the maven archetype, so that by the time the automation (like dependabot) starts sending messages we already have the images in place?

In my experience, if there is no issue its fast, <12h. If there is an issue, it might take longer because it needs to be resolved. But the issues are valid.

Sometimes the PRs get merged immediately, sometimes it takes longer (1-2 days). This time it got delayed by over a week because the base image is deprecated. But you're on the right track that this is mostly a process issue. We typically reach out to official-images once the release is done, aka when it's already too late to fix anything without delaying the release.

Usually, we don't announce the release without having the images in place.

I see, I guess the real problem here is that dependabot does not read our blog 🙂 (which it should, it's great)

They actually reached out to us in July about the deprecated images, but apparently none of us (11 different people) don't read their github notifications.

@Chesnay Schepler, BTW, is it necessary to provide PyFlink official image? There is PySpark official image on DockerHub.

necessary? No. Could we think about it? Sure.

@Dian Fu, WDYT?

There wasn't a lot of activity in FLINK-16845 / FLINK-28443 about this topic, so I'm not sure if there really is any demand for it.

@Chesnay Schepler, IMO, the PyFlink official image is useful for the users which use the K8S operator to run PyFlink application. Otherwise, the users only follow the official document which tell users how to build the PyFlink image and build by themselves.

Currently we have documented the steps on how to build a docker image with PyFlink installed and suggest users to build it themselves. This is definitely not convenient for users, however seems still acceptable. My only concern is that it will add burden to the release managers. If this is not a big problem, then I’m +1 to publish PyFlink images officially.

The first step would be to write an e2e test that actually makes use of what we have in the documentation. It it appears the current python kubernetes test does something completely different.

Although there is indeed some difference between the documentation (pip install apache-flink ) and the python kubernetes test (pip install apache-flink-*.tar.gz), I think there is no fundamental difference between them and so don’t really think this is necessary.