does anyone know what the flink git tags mean. how...
# troubleshootingf
Felix Angell
04/19/2023, 10:33 AMdoes anyone know what the flink git tags mean. how does
release-1.15.2-rc2release-1.15rc2release-1.15Felix Angell
04/19/2023, 10:34 AMim on AWS KDA and I have a feeling it's using rc2 as I cannot resolve this kinesis consumer when i submit my job.
i guess this means I need to manually build the python mappings for this?
anyone using aws kda 1.15. also have any experience with this problem?
m
Martijn Visser
04/19/2023, 11:09 AM@Hong Teoh @Danny Cranmer ?
d
Dian Fu
04/19/2023, 11:13 AM@Felix Angell
release-1.15.2-rc2release-1.15.2release-1.15release-1.15.4h
Hong Teoh
04/19/2023, 11:18 AMYes, Dian Fu seems correct. There are no python wrapper for Kinesis in Flink 1.15.2.
https://github.com/apache/flink/blob/release-1.15.2/flink-python/pyflink/datastream/connectors.py
If I’m not mistaken, a possible workaround is to use the Table API… 👀
f
Felix Angell
04/19/2023, 11:19 AMdang ok. i think we are too far gone for the table api sadly
Felix Angell
04/19/2023, 11:20 AMis there much risk in copying over the python wrappers from a later version?
d
Dian Fu
04/19/2023, 11:20 AM@Felix Angell I think it should work.
👍 2
f
Felix Angell
04/19/2023, 11:21 AMnoice! i can give that a go and see how we get on 😄
Felix Angell
04/19/2023, 11:22 AMah. can i assume that FlinkKinesisConsumer exists in the KDA runtime for PyFlink 1.15? or do i need to bundle that jar dependency in myself
d
Danny Cranmer
04/19/2023, 11:24 AMIt is not bundled; you will need to bundle that
f
Felix Angell
04/19/2023, 11:25 AMis it ok to use this or does this mean i need to get a jar from flink do your eckon: https://github.com/awslabs/amazon-kinesis-connector-flink
d
Danny Cranmer
04/19/2023, 11:27 AMThis fork is for 1.8/1.11 only. Please use the Apache version
👍 1
f
Felix Angell
04/19/2023, 11:29 AMcool, cheers for the help all!
Felix Angell
04/19/2023, 11:34 AManother KDA question relating to python backporting stuff that is related to this sorry!
the WatermarkStrategy PyFlink API introduces 'source alignment' in version 1.16 despite being available as a feature in 1.15.
is it reasonable/safe to backport this or would it be safer to roll our own alignment (somehow).
d
Danny Cranmer
04/19/2023, 12:57 PMI do not think this is a KDA question. I am unsure on the answer, suggest repost as a general question.
f
Felix Angell
04/19/2023, 1:15 PMah that's a fair point! thanks