https://pulsar.apache.org/ logo
Join Slack
Powered by
Hey there, I am facing issue in using Pulsar JDBC ...
# general
y
Hey there, I am facing issue in using Pulsar JDBC sink connector for clickhouse. Versions: pulsar-io-jdbc-clickhouse-4.1.2.nar(includes clickhouse-jdbc-0.4.6-all.jar) and I have ClickHouse 25.x Facing table not found issue, has anyone faced it ? The connector is failing while using JdbcUtils.getTableId() method. Please if someone can suggest anything regarding this issue, that would be a great help Thanks
d
can you share your configuration details minus senstive info such as passwords?
y
Sure, sink connector config
Copy code
tenant: "public"
namespace: "default"
name: "clickhouse-jdbc-sink"
inputs:
  - "<persistent://public/default/synthetic_logs>"

configs:
  userName: "**"
  password: "***"

  
  jdbcUrl: "jdbc:<clickhouse://clickhouse:8123/default>"

  tableName: "pulsarsink_test"

  useTransactions: "false"

  batchSize: 1000
  timeoutMs: 1000
  useJdbcBatch: true
Error trace:
Copy code
componentType: SINK

2025-12-29T00:02:54,113+0000 [public/default/clickhouse-jdbc-sink-0] INFO  org.apache.pulsar.io.jdbc.JdbcAbstractSink - Opened jdbc connection: jdbc:<clickhouse://clickhouse:8123/default>, autoCommit: true
2025-12-29T00:02:54,130+0000 [public/default/clickhouse-jdbc-sink-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - Sink open produced uncaught exception: 
java.lang.Exception: Not able to find table: pulsarsink_test
    at org.apache.pulsar.io.jdbc.JdbcUtils.getTableId(JdbcUtils.java:112) ~[pulsar-io-jdbc-core-4.1.2.jar:?]
    at org.apache.pulsar.io.jdbc.JdbcAbstractSink.open(JdbcAbstractSink.java:102) ~[pulsar-io-jdbc-core-4.1.2.jar:?]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupOutput(JavaInstanceRunnable.java:1070) ~[?:?]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setup(JavaInstanceRunnable.java:261) ~[?:?]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:305) ~[?:?]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
2025-12-29T00:02:54,133+0000 [public/default/clickhouse-jdbc-sink-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [public/default/clickhouse-jdbc-sink:0] Uncaught exception in Java Instance
java.lang.Exception: Not able to find table: pulsarsink_test
    at org.apache.pulsar.io.jdbc.JdbcUtils.getTableId(JdbcUtils.java:112) ~[?:?]
    at org.apache.pulsar.io.jdbc.JdbcAbstractSink.open(JdbcAbstractSink.java:102) ~[?:?]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setupOutput(JavaInstanceRunnable.java:1070) ~[org.apache.pulsar-pulsar-functions-instance-3.2.0.jar:3.2.0]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.setup(JavaInstanceRunnable.java:261) ~[org.apache.pulsar-pulsar-functions-instance-3.2.0.jar:3.2.0]
    at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:305) ~[org.apache.pulsar-pulsar-functions-instance-3.2.0.jar:3.2.0]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
2025-12-29T00:02:54,137+0000 [public/default/clickhouse-jdbc-sink-0] INFO  org.apache.pulsar.functions.instance.JavaInstanceRunnable - Closing instance
2025-12-29T00:02:54,138+0000 [public/default/clickhouse-jdbc-sink-0] INFO  org.apache.pulsar.io.jdbc.JdbcAbstractSink - Closed jdbc connection: jdbc:<clickhouse://clickhouse:8123/default>
2025-12-29T00:02:54,138+0000 [main] INFO  org.apache.pulsar.functions.runtime.JavaInstanceStarter - RuntimeSpawner quit, shutting down JavaInstance
2025-12-29T00:02:54,139+0000 [main] INFO  org.apache.pulsar.functions.runtime.thread.ThreadRuntime - Unloading JAR files for instanceId 0 functionId b6e01d53-f686-4836-acb9-da1d58de4734 namespace default
2025-12-29T00:02:54,144+0000 [main] INFO  org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: <pulsar://localhost:6650>
d
can you verify that the table exists?
-- List tables in the default database SHOW TABLES FROM default; -- Or check specifically SELECT name FROM system.tables WHERE database = ‘default’ AND name = ‘pulsarsink_test’;
y
Yes, even from inside the pulsar container, I tried doing curl to clickhouse and was able to fetch the tables
I checked the code for JdbcUtils.java inside Apache Pulsar repo, it is being failed while trying to get the metaData from clickhouse Also I updated the clickhouse-jdbc driver version too inside the pulsar-io-jdbc-clickhouse-4.1.2.nar, tried with clickhouse-jdbc-0.6.0-all.jar , still same issue
d
Can you try different JDBC url formats?
Copy code
# Option A: Explicit catalog
jdbcUrl: "jdbc:<clickhouse://clickhouse:8123/default?catalog=default>"

# Option B: Use the newer driver protocol
jdbcUrl: "jdbc:<ch://clickhouse:8123/default>"

# Option C: Add compatibility properties
jdbcUrl: "jdbc:<clickhouse://clickhouse:8123/default?use_server_time_zone=true&databaseTerm=schema>"
Copy code
jdbcUrl: "jdbc:<clickhouse://mydomainiscool.cloud:8443/default?ssl=true>"
y
Hey @David K, I found the issue, I tried checking the logs of the queries in clickhouse query_logs as my exception was there while calling for metadata/system table details In clickhouse logs, I was able to find out the query via clickhouse-jdbc-driver, that confirms my pulsar sink connector is able to talk to clickhouse jdbc driver and then clickhouse jdbc driver is able to post the query to clickhouse Then I tried testing queries from the clickhouse-jdbc-driver to clickhouse and find out the exception : Magic is not correct - expect [-126] but got [70] Then I got to know it's the compression technique mismatch between clickhouse server and clickhouse jdbc driver. This driver exception of Magic not correct was returning empty rows, that's why I was getting Table not found issue
👍 1
I will input more deep dive details for this
Open in Slack
PreviousNext
Powered by