Hi All, I am currently developing custom DynamoDB ...
# troubleshooting
s
Hi All, I am currently developing custom DynamoDB sink (For upserts) and wondering if I can test the sink with testHarness? or is there is any other way I can test the sink operator? Any doc would be really helpful gratitude thank you
a
Hi Satyasheel, Could you elaborate why you are not using the DDB sink in https://github.com/apache/flink-connector-aws? It would be great to create a Jira for feature requests if needed for the existing connector. Regarding the testing, the documentation and e2e tests on the same repo has multiple examples for testing AWS connectors
s
Hi Ahmed, as of now AWS connector for DynamoDB is using AWS DynamoDB Batch API, and with this API, PUT method replaces the whole record - in our use case we would like to update the record with a non null value - it the record is present (not replace the whole record) otherwise insert the record. Please correct me If wrong here - otherwise as suggested I will create a jira request for the same. Thanks, will check the tests in the repo.
a
TestHarness is meant for Flink internal testing or if you implement your own operator.
Usually, you'd test by using a flink mini cluster and create a respective topology
but that would imply that you could spawn the external endpoint in docker container
If you test your stuff internally, you could also connect to a test instance.
m
Hi Ahmed, we also have custom implementation for Dynamo sink just because we are working with multitable. Each table is dedicated for specific tenant in job that is multitenant. In addition to simple batch sink logic we need to do more updates in Dynamo for counters and audits and do it in one Dynamo transaction. Just to add more complexity to my question, we are using 2 parallel sinks that one of them is sending change event to Kafka and another talks with Dynamo. How do you think in my case may I use KeyedProcessFunction instead of my Dynamo sink to make sure that I’m sending change event in sink only after I finished all work with Dynamo?