Bartosz Wikiera
06/20/2024, 7:46 PMdef run(applicationProperties: ParameterTool): Unit = {
val env = StreamExecutionEnvironment.getExecutionEnvironment
val source: KafkaSource[String] = createSource(applicationProperties, OffsetsInitializer.latest())
val sourceStream = env.fromSource(source, WatermarkStrategy.noWatermarks(), "topic")
val sink = s"""
|CREATE TABLE `table` (
| `key` STRING,
| `occurred_at` STRING,
| `received_at` STRING
|) PARTITIONED BY ( key, occurred_at )
|WITH (
| 'connector' = 'dynamodb',
| 'table-name' = 'table',
| 'aws.region' = 'region'
|);
|""".stripMargin
val settings = EnvironmentSettings
.newInstance()
.inStreamingMode()
.build()
val tableEnv = StreamTableEnvironment.create(env, settings)
val stream = sourceStream.map(string => decode[RowClass](string).toOption.orNull)
val inputTable = tableEnv.fromDataStream(stream)
tableEnv.createTemporaryView("input", inputTable)
tableEnv.executeSql(sink)
tableEnv.executeSql(
s"""
|INSERT INTO `table`
|SELECT key, occurred_at, received_at FROM (
| SELECT *,
| ROW_NUMBER() OVER (PARTITION BY key, occurred_at ORDER BY occurred_at DESC) AS rownum
| FROM input)
|WHERE rownum = 1
|""".stripMargin
)
}Brad Murry
07/24/2024, 5:55 PM'scan.startup.mode' = 'earliest-offset' - and I run 5 events (with some duplicate keys to ensure upserting is correct)
then - I restart the job, so my DDB has records from previous runs and now the kafka source will emit the 5 events from the previous run.
Boom-
Caused by: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: Provided list of item keys contains duplicatesBrad Murry
07/24/2024, 5:56 PMBartosz Wikiera
07/24/2024, 5:59 PMBrad Murry
07/24/2024, 6:00 PMBrad Murry
07/24/2024, 6:00 PMBartosz Wikiera
07/24/2024, 7:47 PM