wave hello folks I m pretty new to the Flink community and Apache Flink #troubleshooting

:wave: hello folks. I'm pretty new to the Flink c...

Michael Richardson

07/19/2024, 5:56 AM

👋 hello folks. I'm pretty new to the Flink community and would greatly appreciate any tips on my issue below. I tried writing a simple pyflink application that creates a table using the JDBC connector to an existing impala table. I know impala isn't listed as a database in the Flink JDBC documentation, but was hoping i could make it work by supplying the impala JDBC jar. I added the

ImpalaJDBC42.jar

and

flink-connector-jdbc-3.2.0-1.19.jar

files to the

pipeline.jars

and

pipeline.classpaths

. When i run the application I get the following error

Copy code

Caused by : java.lang.IllegalStateEException: Could not find any jdbc dialect factory that can handle url 'jdbc:impala://<hostname>:<port>/<db_name>' that implements 'org.apache.flink.connector.jdbc.dialect.JdbcDialectFactory' in the classpath

Will the flink JDBC connector only work with the listed databases in the documentation as they have "dialectfactories" as part of the flink-connector-jdbc, or is it possible to get additional databases like impala to work with a JDBC connection?

D. Draco O'Brien

07/19/2024, 7:21 AM

Hi Michael! Yes it is possible to get impala to work with a JDBC connection but it will require some custom work probably.

D. Draco O'Brien

07/19/2024, 7:23 AM

Flink JDBC Connector is extensive. First you will need to define the new dialect

Copy code

public class ImpalaJdbcDialect extends JdbcDialect {
    @Override
    public boolean canHandle(String url) {
        return url.startsWith("jdbc:impala:");
    }

    // implement other methods as needed. 
}

D. Draco O'Brien

07/19/2024, 7:24 AM

2) you will need to register the custom dialect. I think it can be in jar file or register like

Copy code

JdbcDialects.registerDialect(new ImpalaJdbcDialect());

D. Draco O'Brien

07/19/2024, 7:25 AM

If you need dynamic discover then create the following class in META-INF/services:

Copy code

org.apache.flink.connector.jdbc.dialect.JdbcDialectFactory

and list your dialect class there.

D. Draco O'Brien

07/19/2024, 7:26 AM

You can view the dialects for other db integrations to get a better idea about how to implement for Impala

Michael Richardson

07/21/2024, 9:08 PM

thanks @D. Draco O'Brien - i'll have a go at that today. thanks for tips.

Open in Slack

Previous Next