Hi All, We are using Flink 1.20, and the related ...
# troubleshooting
m
Hi All, We are using Flink 1.20, and the related kafka-connector.
Copy code
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka</artifactId>
    <version>3.4.0-1.20</version>
</dependency>
This version of kafka client has 2 vulnerabilities. These vulnerabilities are only fixed in v3.9.1 of kafka clients. We are wondering how to handle this? Is it recommended to exclude the kafka-client from the connector and then override it with the new version of kafka-client. We are afraid that it might not be as easy as it sounds. πŸ™‚ Is running the test cases for Flink sufficient to verify that the new version is flink compatible. Or are there other things we'd keep in my while doing this upgrade. PS: In the longer run, we might upgrade to Flink 2.0, but right now that is not on our plans.
t
You can override it but the main flink repo tests won't exercise those code paths. You could pull the v3.4 tag of the flink-connector-kafka repo and bump the kafka version to 3.9.1 there. You can then run the tests for that repo which will test all the relevant functionality.
If you push that change to your own fork on GH the CI will run (look under the Actions menu)
Or you can run the CI command locally using:
Copy code
mvn clean deploy -U -B --no-transfer-progress -Dflink.version=1.20.0 \
-DaltDeploymentRepository=validation_repository::default::file:/tmp/flink-validation-deployment \
-Dscala-2.12 \
-Prun-end-to-end-tests -DdistDir=<path to folder containing flink dist> \
-Dflink.convergence.phase=install -Pcheck-convergence \
-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false -Dmaven.wagon.httpconnectionManager.ttlSeconds=120 \
-Dlog4j.configurationFile="file://$(pwd)/tools/ci/log4j.properties" \
| tee /tmp/mvn_build_output.out
Assuming all that passes you could submit a PR to the
v3.4
branch it the kafka connector repo. Maybe with a bump to flink 1.20.3 as well. This is a good argument for a 3.4.1 release of the connector to cover flink 1.20.x users which you could raise on the flink dev mailing list.
m
Thanks @Thomas Cooper for the inputs. I will try this out.
Update: Did a quick test for this, and realised that upgrading to 3.9.x doesn't work straight up. There are some Internal classes that are marked as
@Deprecated
, which are implementing the Producer interface. Two new methods have been added in this Interface, due to which the compilation fails. Will maybe later invest some more time to understand how to implement these methods properly. πŸ™‚