Hi everyone! I am new to Flink, but have been doin...
# troubleshooting
d
Hi everyone! I am new to Flink, but have been doing large scale stream processing for a long time. I have a surprising failure mode. I'm loading a DynamoDB table from my local cluster, and when my session token expires (different problem, under control), my job goes into this crazy retry loop. The first log entry says it will retry 3 times (see thread). After only a minute or two, my log file is hitting 100MB and I have over 20k instances of the security token message. I've since discovered "setFailOnError()", but that seems like a big hammer in the long run. The job itself is pretty trivial, but see the thread for some code. Thoughts? Thanks...
A bit of the log:
2024-06-05 08:37:58,266 WARN  org.apache.flink.connector.dynamodb.sink.DynamoDbSinkWriter  [] - DynamoDB Sink failed to persist and will retry 3 entries.
java.util.concurrent.CompletionException: software.amazon.awssdk.services.dynamodb.model.DynamoDbException: The security token included in the request is expired (Se
rvice: DynamoDb, Status Code: 400, Request ID: CP72SFUMJPVEQ34AIP8LR4USCVVV4KQNSO5AEMVJF66Q9ASUAAJG)
at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65) ~[blob_p-97f9ed006171f58fa637301cecdbea115b
f0e390-f29a092b3a696dce70d5a16823d9c1c1:?]
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionRepo
rtingStage.java:51) ~[blob_p-97f9ed006171f58fa637301cecdbea115bf0e390-f29a092b3a696dce70d5a16823d9c1c1:?]
at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
Some code:
if (options.loadDynamoDB) {
Properties sinkProperties = new Properties();
sinkProperties.put(AWSConfigConstants.AWS_REGION, options.awsRegion);
String dynamoDBTable = typeName + "-" + options.dynamoDbTableSuffix;
ElementConverterT, DynamoDbWriteRequest jsonConverter = new PojoElementConverter<>();
DynamoDbSinkT dynamoDbSink = DynamoDbSink.Tbuilder()
.setDynamoDbProperties(sinkProperties)              // Required
.setTableName(dynamoDBTable)                        // Required
.setElementConverter(jsonConverter)                 // Required
.setFailOnError(true)
.build();
dataStream.sinkTo(dynamoDbSink);
}
FWIW, the input dataset has a whopping three records. The runtime environment is a local linux VM running Flink v1.18.1.
k
The first log entry says it will retry 3 times (see thread).
Unless I'm reading it wrong, it looks like it's retrying three entries an unspecified amount of times. I've encountered something similar, where my processing function fails due to some service unavailability and then retries about a billion times, spamming gigs into the log.
πŸ‘€ 1
d
That's an interesting point.
a
@Darren Miller are you sure you set FailOnError to true, this shouldn't happen in this case
setFailOnError(true)
explicitly tells sink not to retry If you could verify then we can raise a bug ticket
d
Sorry, @Ahmed Hamdy - no need for a ticket for setFailOnError, but maybe the behavior without it. This code was copied after I added that call. It was not set before, and does effectively prevent retries, so it has the intended effect.
a
For infinite retries, We are tracking this on this ticket: https://issues.apache.org/jira/browse/FLINK-35541, unfortunately this is in fact a known issue, DDB sink generally fails fast for some detected fatal exceptions but it retries indefinitely otherwise.
d
Well, that certainly sounds like it. Thanks for the pointer - I'll keep an eye on that issue.
πŸ™Œ 1