Ajay Bokka
02/20/2025, 1:49 AMpublic void startApplication(StreamExecutionEnvironment env) throws Exception {
env.setMaxParallelism(maxParallelism);
// Operator name is used in Flink UI and metrics reporting, while UID is used to scope operator's state in savepoints.
// It's recommended to set UIDs for all operators: https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/production_ready.html#set-uuids-for-all-operators
DataStreamTimeSeriesDataRecord dataSource = env.addSource(flinkKinesisConsumer)
.uid(TIME_SERIES_DATA_STREAM).name(TIME_SERIES_DATA_STREAM).setParallelism(kinesisOpenShardCount);
dataSource.keyBy(TimeSeriesDataRecord::getAccountId)
.window(TumblingProcessingTimeWindowsWithDistributedStagger.of(Time.seconds(windowSizeInSeconds)))
.aggregate(perAccountTimeSeriesDataAggregateFunction).uid(AGGREGATE_TIME_SERIES_DATA).name(AGGREGATE_TIME_SERIES_DATA)
.addSink(dynamoDBSink).uid(EVENTUAL_TIME_SERIES_DATA_STORE).name(EVENTUAL_TIME_SERIES_DATA_STORE);
env.execute("MoonsailDataService TimeSeriesData Ingestion Flink Application");
}
The above is our current FlinkApplication code.
We would like to switch to EventTimeProcessing but we are currently doing ProcessingTimeWindow. I understand that I need to update to TumblingEventTimeWindow with any custom stagger as for my application requirement.
Here, It states that with Flink-1.20 ApproximateArrivalTimestamp by kinesis is supplied as eventTime. So, we don't need to extract the timestamp.
Will upgrading the tumblingWindow and to newerVersion enough to achieve EventTimeProcessing.
Do we need to add a watermark strategy or Flink can achieve it internally ?Pedro Mázala
02/20/2025, 10:32 AMapproximateArrivalTimestamp.
But a watermark strategy is important because you need to define what would be a out of order record to you there. I mean, Flink may:
1. read shard 1. After X records it will export the watermark value approximateArrivalTimestamp as 10
2. read shard 2 and the first record has approximateArrivalTimestamp as 1 (because time is linear per shard). How would the application behave?
Check WatermarkStrategy.forBoundedOutOfOrderness or allowedLateness for windows.
I'd say it's also important to set auto watermark/idle time if there is a chance of having scattered data or times when there is no data coming from your kinesis.
But, afterall, I'd say the best way to validate it is to test the upgrade and check if it behaves as you'd expect.Ajay Bokka
02/27/2025, 10:32 PMPedro Mázala
02/28/2025, 9:43 AMDoes Flink-1.18 provides the support to assign the approximateArrivalTimestamp as the event timestamp ?This is more a connector support than Flink per se. On Flink docs it does not show this possibility on the docs. You'd have to validate that.
I’m considering the forMonotonousTimestamp strategy would be good as we are processing fast and so, will approximateArrivalTimestamp is assigned as Timestamp as it suggested in the Flink-1.18 documentation ?I didn't fully quite get this one.
Ajay Bokka
03/14/2025, 5:16 AM