anybody know about Pyflink 1.15~1.151 can't sink w...
# randomh
HAJI
07/15/2022, 2:28 AManybody know about Pyflink 1.15~1.151 can't sink with datastream?
https://issues.apache.org/jira/browse/FLINK-27910
HAJI
07/15/2022, 2:29 AMyes lknow that make hidden files not yet finished....
i will do batch and change name....
It so sad...
my flink can work delay for 1~2 hours...
HAJI
07/15/2022, 2:36 AMour system, not yet need realtime...
we will use
kafka -> pyflink -> hiddenfile -> batch and change file name -> logstash -> elasticsearch -> kibana.
ok! it is perfect system for me!
HAJI
07/15/2022, 2:36 AMevery data stack is funny!
d
David Anderson
07/15/2022, 7:08 PMFLINK-27910 was fixed in 1.15.1. If you are having problems with files not being finalized, that’s probably something else.
Do you have checkpointing enabled? That’s required for the FileSink when streaming.
h
HAJI
07/15/2022, 8:53 PMThank you!!
HAJI
07/15/2022, 8:53 PMI will check.
HAJI
07/15/2022, 8:53 PMI use pyflink.
It also fixed?
d
David Anderson
07/15/2022, 9:23 PMShould be, yes.
h
HAJI
07/15/2022, 9:50 PMThankyou💜
2 Views