Darren Miller
06/06/2024, 12:02 AMDarren Miller
06/06/2024, 12:04 AM2024-06-05 08:37:58,266 WARN org.apache.flink.connector.dynamodb.sink.DynamoDbSinkWriter [] - DynamoDB Sink failed to persist and will retry 3 entries.
java.util.concurrent.CompletionException: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: The security token included in the request is expired (Se
rvice: DynamoDb, Status Code: 400, Request ID: CP72SFUMJPVEQ34AIP8LR4USCVVV4KQNSO5AEMVJF66Q9ASUAAJG)
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65) ~[blob_p-97f9ed006171f58fa637301cecdbea115b
f0e390-f29a092b3a696dce70d5a16823d9c1c1:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionRepo
rtingStage.java:51) ~[blob_p-97f9ed006171f58fa637301cecdbea115bf0e390-f29a092b3a696dce70d5a16823d9c1c1:?]
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
Some code:
if (options.loadDynamoDB) {
Properties sinkProperties = new Properties();
sinkProperties.put(AWSConfigConstants.AWS_REGION, options.awsRegion);
String dynamoDBTable = typeName + "-" + options.dynamoDbTableSuffix;
ElementConverterT, DynamoDbWriteRequest jsonConverter = new PojoElementConverter<>();
DynamoDbSinkT dynamoDbSink = DynamoDbSink.Tbuilder()
.setDynamoDbProperties(sinkProperties) // Required
.setTableName(dynamoDBTable) // Required
.setElementConverter(jsonConverter) // Required
.setFailOnError(true)
.build();
dataStream.sinkTo(dynamoDbSink);
}Darren Miller
06/06/2024, 12:09 AMKirill Ternovsky
06/06/2024, 2:13 AMThe first log entry says it will retry 3 times (see thread).Unless I'm reading it wrong, it looks like it's retrying three entries an unspecified amount of times. I've encountered something similar, where my processing function fails due to some service unavailability and then retries about a billion times, spamming gigs into the log.
Darren Miller
06/06/2024, 5:24 AMAhmed Hamdy
06/06/2024, 12:35 PMsetFailOnError(true) explicitly tells sink not to retry
If you could verify then we can raise a bug ticketDarren Miller
06/06/2024, 1:07 PMAhmed Hamdy
06/06/2024, 1:10 PMDarren Miller
06/06/2024, 1:13 PM