Hi everyone, I’ve encountered an issue while tryi...
# troubleshooting
a
Hi everyone, I’ve encountered an issue while trying to use Flink CDC for SQL Server to SQL Server data synchronization. I’ve posted the detailed problem description on Stack Overflow. If anyone could take a moment to look at it and provide some guidance, I would greatly appreciate it! Here’s the link to my question: https://stackoverflow.com/questions/78752423/ Thank you in advance for your help!
z
• I think you should use this to your maven not mssql-jdbc-12.6.3.jre11.jar
Copy code
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-sqlserver-cdc</artifactId>
    <version>3.2-SNAPSHOT</version>
</dependency>
a
Hi,the latest dependencies I found are as follows
Copy code
<dependency>
    <groupId>com.ververica</groupId>
    <artifactId>flink-connector-sqlserver-cdc</artifactId>
    <version>3.0.1</version>
</dependency>
But error still occur
Copy code
Exception in thread "main" java.lang.RuntimeException: Cannot find factory with                                      identifier "sqlserver-cdc" in the classpath.

Available factory classes are:


        at org.apache.flink.cdc.composer.utils.FactoryDiscoveryUtils.getFactoryB                                     yIdentifier(FactoryDiscoveryUtils.java:62)
        at org.apache.flink.cdc.composer.flink.translator.DataSourceTranslator.t                                     ranslate(DataSourceTranslator.java:47)
        at org.apache.flink.cdc.composer.flink.FlinkPipelineComposer.compose(Fli                                     nkPipelineComposer.java:101)
        at org.apache.flink.cdc.cli.CliExecutor.run(CliExecutor.java:71)
        at org.apache.flink.cdc.cli.CliFrontend.main(CliFrontend.java:71)
a
谢谢你!不过这个我已经测通过了,是在flink-sql-client执行的,但是它的缺点就是,一旦exit sql-client,所有之前定义的table都会消失,insert对应的任务都会停掉,官方文档并没有详细解释用flink-sql怎么像jar包一样提交任务,而且flink-sql-client虽然已经简化很多了,但是比yaml文件配置的方式还是要复杂许多
我甚至还看了微软的官方文档,https://learn.microsoft.com/zh-cn/azure/hdinsight-aks/flink/change-data-capture-connectors-for-apache-flink 但是它的sink对象是kafka,我想找到一个能用的案例 按我心目中实现更加简单的排序应该是 pipeline>table api>datasteam api
z
你可以把你的建表语句写在Java呀,之前我用sqlcdc的时候也遇到过这个情况,但是后面看了是因为我的maven里没有引用正确的cdc的包,我当时也是使用jdbc,把jar包引用了就ok了,其次你可以看一下jar包版本和mssql的适配情况,我不确定是否3.0.1这个包适配你的SQL server版本
a
多谢您!我一下子就有信心了
z
客气,你可以先试试,如果有问题可以一起探讨交流
a
我继续看文档的时候,发现了这样的描述 •
flink-connector-<NAME>
which is a thin JAR including only the connector code, but excluding eventual third-party dependencies •
flink-sql-connector-<NAME>
which is an uber JAR ready to use with all the connector third-party dependencies 也就是说flink-sql-connector-sqlserver-cdc-3.1.0.jar其实是一个fat包,我简单打开看了,里面有jdbc的相关类,我对JAVA不是很熟,这是否意味着,我应该把flink/lib下的flink-connector-sqlserver-cdc-3.0.1.jar和mssql-jdbc-12.6.3.jre11.jar都删除才对?
相同功能的包按文档描述会造成依赖冲突
发现了一个有趣的地方,看黑体部分,刚好我的报错就是找不到工厂类,不过即使我移除那些jar包仍然报错找不到工厂类,只是分享一下,相同功能的包不要放进来 Flink uses Java’s Service Provider Interfaces (SPI) to load the table connector/format factories by their identifiers. Since the SPI resource file named
org.apache.flink.table.factories.Factory
for every table connector/format is under the same directory
META-INF/services
, these resource files will override each other when build the uber-jar of the project which uses more than one table connector/format, which will cause Flink to fail to load table connector/format factories. 在IDEA使用中,找到了一个这样的依赖,在maven test的时候可以发现有没有依赖冲突
Copy code
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-test-utils</artifactId>
    <version>1.19.1</version>
    <scope>test</scope>
</dependency>
z
是的,相同功能的包会造成冲突,这个就是Java蛋疼的地方了哈哈哈
a
有人在sw回我了,明确表示sqlserver不支持flink pipeline,也就是配置文件的方式同步,要我等PR😂 我现在在研究如何使用table api方式提交,执行仍然还是报错找不到工厂类,请问你能否分享一下经验,比如怎么判断当程序报错后,通过报错信息得出自己缺了什么依赖?因为代码实在是非常简单,仿照这个去做的 https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/try-flink/table_api/
z
我也是初学,报错基本上靠搜索引擎,基本上能解决百分之80,主要是搜索的时候把所有的casedby都搜一下🤣
a
这个问题我解决了,但是又遇到了新的问题😂
解决了,flink官网是要求用JDK11,我降级为JDK8,就好了,如何评价
z
哈哈哈这么说我也经常遇到Java版本导致的问题,真的很无语