Anyone managed to get clickhouse sink connector wo...
# troubleshooting
y
Anyone managed to get clickhouse sink connector working?
p
Still jar issues?
y
no:
Copy code
Caused by: java.lang.NullPointerException
	at org.apache.flink.connector.clickhouse.util.ClickHouseJdbcUtil.getDistributedEngineFull(ClickHouseJdbcUtil.java:70) ~[flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT]
	at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat$Builder.build(AbstractClickHouseOutputFormat.java:212) ~[flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT]
	... 33 more
I built
flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT
out of master, which is documented to be unstable, yet it doesn't say which branch to use. Also, they are moving to apache... So, no idea where to go from there
p
It looks like the problem goes down to here. https://github.com/itinycheng/flink-connector-clickhouse/blob/fe0a767d300febdbe9a5[…]lickhouse/internal/connection/ClickHouseConnectionProvider.java IIRC clickhouse returns a null connection if there is an issue while connecting. Do you have a log like this?
y
Copy code
14:22:57.874 [flink-rest-server-netty-worker-thread-2] ERROR org.apache.flink.table.gateway.service.SqlGatewayServiceImpl - Failed to fetchResults.
org.apache.flink.table.gateway.service.utils.SqlExecutionException: Failed to execute the operation 32772067-0e4a-43c9-ae50-6cf3655f13d8.
	at org.apache.flink.table.gateway.service.operation.OperationManager$Operation.processThrowable(OperationManager.java:414) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
	at org.apache.flink.table.gateway.service.operation.OperationManager$Operation.lambda$run$0(OperationManager.java:267) ~[flink-sql-gateway-1.20.1.jar:1.20.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.lang.RuntimeException: Build ClickHouse output format failed.
	at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat$Builder.build(AbstractClickHouseOutputFormat.java:223) ~[flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT]
	at org.apache.flink.connector.clickhouse.ClickHouseDynamicTableSink.getSinkRuntimeProvider(ClickHouseDynamicTableSink.java:101) ~[flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT]
	at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink.createSinkTransformation(CommonExecSink.java:151) ~[?:?]
	at org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.java:214) ~[?:?]
	at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:168) ~[?:?]
	at org.apache.flink.table.planner.delegation.StreamPlanner.$anonfun$translateToPlan$1(StreamPlanner.scala:85) ~[?:?]
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233) ~[flink-scala_2.12-1.20.1.jar:1.20.1]....Caused by: java.lang.NullPointerException
	at org.apache.flink.connector.clickhouse.util.ClickHouseJdbcUtil.getDistributedEngineFull(ClickHouseJdbcUtil.java:70) ~[flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT]
	at org.apache.flink.connector.clickhouse.internal.AbstractClickHouseOutputFormat$Builder.build(AbstractClickHouseOutputFormat.java:212) ~[flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT]
	at org.apache.flink.connector.clickhouse.ClickHouseDynamicTableSink.getSinkRuntimeProvider(ClickHouseDynamicTableSink.java:101) ~[flink-connector-clickhouse-1.0.0.jar:1.0.0-SNAPSHOT]
	at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink.createSinkTransformation(CommonExecSink.java:151) ~[?:?]
	at org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.java:214) ~[?:?]
	at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:168) ~[?:?]
	at org.apache.flink.table.planner.delegation.StreamPlanner.$anonfun$translateToPlan$1(StreamPlanner.scala:85) ~[?:?]
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233) ~[flink-scala_2.12-1.20.1.jar:1.20.1]
	at scala.collection.Iterator.foreach(Iterator.scala:937) ~[flink-scala_2.12-1.20.1.jar:1.20.1]
p
Can you share which configs do you have on your ddl?
y
Copy code
) WITH (
    'connector' = 'clickhouse',
    'url' = 'clickhouse://localhost:8443',
    'username' = 'kafka_connector',
    'password' = '****',
    'database-name' = 'raw',
    'table-name' = 'tab1',
    'sink.batch-size' = '500',
    'sink.flush-interval' = '1000',
    'sink.max-retries' = '3'
);
p
If you're running this on top of docker, the localhost will not work as expected. Have you tried
Copy code
) WITH (
    'connector' = 'clickhouse',
    'url' = 'clickhouse://host.docker.internal:8443',
    'username' = 'kafka_connector',
    'password' = '****',
    'database-name' = 'raw',
    'table-name' = 'tab1',
    'sink.batch-size' = '500',
    'sink.flush-interval' = '1000',
    'sink.max-retries' = '3'
);
if you're running it locally
y
Well... I'm using a public hostname
👍 1
Just didn't want to post it there
you whink it'd a access issue?
Let me see if I have that
p
The issue relies in this bit of code. Where the connection is being null. This is probably due to access. This would be my guess
This is coming from Java jdbc driver connection
I'd also try on the url
jdbc:ch:http://...:8433
or even
http://...:8433
as your URL to be sure
y
So, I validated that my port is open
I'll make sure thw uql is good
God damn oit, bad URL
Here is the format
jdbc:clickhouse://htttps://myformat:8443
p
Nice! I'm happy you got there.
y
Oh my god that was a pain...finally worked... The accpected URL format is
Copy code
'url' = 'jdbc:clickhouse://myhost:8443?ssl=true',
these are all the needed jars for flink-connector-clickhouse-1.0.0.jar:
Copy code
RUN curl -L -o /opt/flink/lib/flink-connector-kafka-1.20.jar https://dp-assets-public.s3.us-east-1.amazonaws.com/flink-connector-kafka-3.4.0.jar
RUN curl -L -o /opt/flink/lib/kafka-clients-3.4.0.jar https://repo.maven.apache.org/maven2/org/apache/kafka/kafka-clients/3.4.0/kafka-clients-3.4.0.jar
RUN curl -L -o /opt/flink/lib/clickhouse-client-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-client/0.6.4/clickhouse-client-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-data-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-data/0.6.4/clickhouse-data-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-http-client-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-http-client/0.6.4/clickhouse-http-client-0.6.4.jar
RUN curl -L -o /opt/flink/lib/clickhouse-jdbc-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc/0.6.4/clickhouse-jdbc-0.6.4.jar
RUN curl -L -o /opt/flink/lib/jdbc-v2-0.6.4.jar https://repo1.maven.org/maven2/com/clickhouse/jdbc-v2/0.6.4/jdbc-v2-0.6.4.jar
RUN curl -L -o /opt/flink/lib/httpcore5-5.2.5.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/core5/httpcore5/5.2.5/httpcore5-5.2.5.jar
RUN curl -L -o /opt/flink/lib/httpclient5-5.2.3.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/client5/httpclient5/5.2.3/httpclient5-5.2.3.jar
RUN curl -L -o /opt/flink/lib/httpcore5-h2-5.2.5.jar https://repo1.maven.org/maven2/org/apache/httpcomponents/core5/httpcore5-h2/5.2.5/httpcore5-h2-5.2.5.jar
🙌 2