Hello everyone. Since few days I have a problem wi...
# troubleshooting
b
Hello everyone. Since few days I have a problem with deduplication logic for simple Flink app. Logic of the app is: • read from kafka • do simple transformation of kafka event (parse json and add some data) • write to dynamo db I saw few articles about deduplication, at first I was using grouping with tumbling window. Then I moved to sth like: https://stackoverflow.com/questions/69883008/flink-deduplication-and-processwindowfunction which also caused dynamo to throw exceptions. I decided to simplify the app even more and I moved to table API. I am using both sink partitioning and deduplication sql . I am running out of ideas what could be wrong. Event has key which is used as partion key to create the table, but it also has a timestamp which I am using for deduplication. It's AWS managed Flink, Job is running in parallel. Also I wrote integration test with dynamo testcontainer and faked duplicate events and it's passing. I am attaching simplified code of the app which grasps most important parts. Any help or suggestion is welcome. Thanks in advance!
Copy code
def run(applicationProperties: ParameterTool): Unit = {
    val env = StreamExecutionEnvironment.getExecutionEnvironment
    val source: KafkaSource[String] = createSource(applicationProperties, OffsetsInitializer.latest())
    val sourceStream = env.fromSource(source, WatermarkStrategy.noWatermarks(), "topic")

    val sink = s"""
                       |CREATE TABLE `table` (
                       |  `key` STRING,
                       |  `occurred_at` STRING,
                       |  `received_at` STRING
                       |) PARTITIONED BY ( key, occurred_at )
                       |WITH (
                       |  'connector' = 'dynamodb',
                       |  'table-name' = 'table',
                       |  'aws.region' = 'region'
                       |);
                       |""".stripMargin

    val settings = EnvironmentSettings
      .newInstance()
      .inStreamingMode()
      .build()

    val tableEnv = StreamTableEnvironment.create(env, settings)

    val stream = sourceStream.map(string => decode[RowClass](string).toOption.orNull)

    val inputTable = tableEnv.fromDataStream(stream)
    tableEnv.createTemporaryView("input", inputTable)

    tableEnv.executeSql(sink)
    tableEnv.executeSql(
      s"""
           |INSERT INTO `table`
           |SELECT key, occurred_at, received_at FROM (
           |  SELECT *,
           |  ROW_NUMBER() OVER (PARTITION BY key, occurred_at ORDER BY occurred_at DESC) AS rownum
           |  FROM input)
           |WHERE rownum = 1
           |""".stripMargin
    )
  }
b
I'm running into a similar issue, and can reproduce locally. Essentially, it looks like the upserts only work for events the job has "seen" before. If I have my table over kafka set to
'scan.startup.mode' = 'earliest-offset'
- and I run 5 events (with some duplicate keys to ensure upserting is correct) then - I restart the job, so my DDB has records from previous runs and now the kafka source will emit the 5 events from the previous run. Boom-
Copy code
Caused by: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Provided list of item keys contains duplicates
I'm going to tweak state settings to see if dedupe can survive between jobs
b
In my case I was using wrong partition key ( i used composite and table was configured to single key). Double check it on your side as it can be as simple as that
b
Hmm, so you were able to get upserts working by adding a sort key to the table?
(upserts across job starts I mean)
b
So table had partition key set, but in connector I used wrong one(composite instead of single one)
👍 1