Hello Team, I'm little new to Flink and I would li...
# troubleshooting
a
Hello Team, I'm little new to Flink and I would like to get some confirmation from the team here.
Copy code
public void startApplication(StreamExecutionEnvironment env) throws Exception {
        env.setMaxParallelism(maxParallelism);

        // Operator name is used in Flink UI and metrics reporting, while UID is used to scope operator's state in savepoints.
        // It's recommended to set UIDs for all operators: https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/production_ready.html#set-uuids-for-all-operators
        DataStreamTimeSeriesDataRecord dataSource = env.addSource(flinkKinesisConsumer)
                .uid(TIME_SERIES_DATA_STREAM).name(TIME_SERIES_DATA_STREAM).setParallelism(kinesisOpenShardCount);

        dataSource.keyBy(TimeSeriesDataRecord::getAccountId)
                .window(TumblingProcessingTimeWindowsWithDistributedStagger.of(Time.seconds(windowSizeInSeconds)))
                .aggregate(perAccountTimeSeriesDataAggregateFunction).uid(AGGREGATE_TIME_SERIES_DATA).name(AGGREGATE_TIME_SERIES_DATA)
                .addSink(dynamoDBSink).uid(EVENTUAL_TIME_SERIES_DATA_STORE).name(EVENTUAL_TIME_SERIES_DATA_STORE);

        env.execute("MoonsailDataService TimeSeriesData Ingestion Flink Application");
    }
The above is our current FlinkApplication code. We would like to switch to EventTimeProcessing but we are currently doing ProcessingTimeWindow. I understand that I need to update to TumblingEventTimeWindow with any custom stagger as for my application requirement. Here, It states that with Flink-1.20
ApproximateArrivalTimestamp
by kinesis is supplied as eventTime. So, we don't need to extract the timestamp. Will upgrading the tumblingWindow and to newerVersion enough to achieve EventTimeProcessing. Do we need to add a watermark strategy or Flink can achieve it internally ?
p
By the documentation I'd you'd start getting your watermark as the same as
approximateArrivalTimestamp
. But a watermark strategy is important because you need to define what would be a out of order record to you there. I mean, Flink may: 1. read shard 1. After X records it will export the watermark value
approximateArrivalTimestamp
as 10 2. read shard 2 and the first record has
approximateArrivalTimestamp
as 1 (because time is linear per shard). How would the application behave? Check
WatermarkStrategy.forBoundedOutOfOrderness
or
allowedLateness
for windows. I'd say it's also important to set auto watermark/idle time if there is a chance of having scattered data or times when there is no data coming from your kinesis. But, afterall, I'd say the best way to validate it is to test the upgrade and check if it behaves as you'd expect.
a
Thanks Pedro I have another question. Does Flink-1.18 provides the support to assign the approximateArrivalTimestamp as the event timestamp ? In our case, very less chances of out of order state but we will consider the idleness state. I’m considering the forMonotonousTimestamp strategy would be good as we are processing fast and so, will approximateArrivalTimestamp is assigned as Timestamp as it suggested in the Flink-1.18 documentation ?
p
Does Flink-1.18 provides the support to assign the approximateArrivalTimestamp as the event timestamp ?
This is more a connector support than Flink per se. On Flink docs it does not show this possibility on the docs. You'd have to validate that.
I’m considering the forMonotonousTimestamp strategy would be good as we are processing fast and so, will approximateArrivalTimestamp is assigned as Timestamp as it suggested in the Flink-1.18 documentation ?
I didn't fully quite get this one.
a
Hey Pedro, Can you help me to understand these gaps for me ? 1.Is "Global Watermarks" only apply to multiple source application or any other applications (we have a single source for one application) ? 2. Each slot generates it's own watermarks depending upon what it see. but windows trigger based on the slowest watermark across all slots ?