Hi everyone, I am trying to create a Pyflink app and deploy it on AWS KDA. My source is a Kinesis stream and the sink is a DynamoDB. I understand i have to create a uber jar as AWS KDA only supports configuring a single jar. The behavior is same in local as well as KDA with a uber jar. While I create the uber jar either I can have Kinesis connector to work or DynamoDB connector to work but not both at the same time depending on which jar i included as the first jar in my pom.xml to create the uber jar. As I searched further this appears to depend on the first entry in META-INF/services/org.apache.flink.table.factories.factory with 2 entries - One for KinesisDynamicTableFactory and another KinesisDynamicTableSinkFactory. Whichever is on the top works fine. Seperating by ", " or new line doesn't help. Screenshot attached

@Siva Family PyFlink supports specifying multiple jars, see https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/python/dependency_management/#jar-dependencies for more details.

Can you share your pom.xml for creating the uber jar? it should not matter which is specified first.

@Jeremy Ber Attached is the pom.xml and thanks for the reply

this looks good to me but you’d have to see how the jar appears once archived

Jar looks good when i decompiled.. the shade plugin seems to have merged it perfectly fine. However while starting the KDA app, I get an error - Could not find any factory for identifier 'dynamodb' that implements 'org.apache.flink.table.factories.DynamicTableFactory'. If i switch the order of dependencies above DyanmoDB works fine but i get the same error for Kinesis.

hi TSiva have you seen this page: https://github.com/aws-samples/pyflink-getting-started/tree/main/packaging

Hi Jeremy, yes i have used the samples and packaging you had mentioned and have an app working fine with source as Kinesis and sink as Kinesis in AWS KDA. The same app works fine with the uber jar i built using the above pom.xml. The problem happens only when I add the uber jar and use Kinesis as source and Dynamo as sink. Only either one of them works depending on the order of dependency in my pom.

If it's happening locally I have to think it's related to the composition of the jars maybe overriding or something

it looks like the FactoryUtil discoverfactory is not able to identify that KinesisDynamicTableSinkFactory has to be used for Dynamodb

Hi, you need something like the ServiceResourcesTransformer as shown here to ensure that the referenced Factories are merged properly in the fat jar

Thanks.. will try with the transformers