https://linen.dev logo
Join Slack
Powered by
Hi! I have been having issues with airbyte ui, bot...
# replication-troubleshooting
n
Hi! I have been having issues with airbyte ui, both hosted and cloud. Basically when setting up any new destination/source, any toggle on the setup page does not work, meaning when i click on a toggle it does not turn left or right. For example, I’m trying to setup a new clickhouse destination, there is SSL toggle which is turned off by default. When I attempt to turn on it it does not respond.
r
Thanks for sharing this, we will test it and get back to you cc @Mikhail Shustov, @Geoff Genz
n
BTW, I checked out a branch with the 
v0.43.16-helm
tag name, the ui issue seems to be fixed.
Also, who does handle the connections issues? I’m having issues with the Clickhouse connection, is it gonna be airbyte or Clickhouse who would help?
g
I think it depends on what kind of issues you're having. We (ClickHouse) just maintain the connector code, so if it's related to a code issue we can do our best to help.
n
@Geoff Genz so it looks like its having hard time parseing timestamps with dates. Here is the full log
35ed6b4e_2432_4b5b_8262_0613b3100f67_logs_7_txt.txt
also this is another weird thing ( you will see it in the log file above) 
dditional Failure Information: Error executing HTTP request <http://https>://upj5s2f0ge.us-east-2.aws.clickhouse.cloud:8443
its formating the url incorrectly
g
It looks like the scheme is being added to the host in the dbt connection profile
Copy code
host: <https://upj5s2f0ge.us-east-2.aws.clickhouse.cloud>
that's not a host, I'm not sure if this a configuration problem or a problem in our connector
n
Here is my config
If i remove https it won’t work
i did lots of trial and error just to get it working. I had to use 
https
connection
g
removing https and enable SSL Connection should work, but it may be bugged
n
i did
g
but https there definitely won't work
n
well it worked and it synced and talked to ch db
created db
moved the data into raw json table
but it failed when it was trying to transform
but yeah
So i checked out 
v0.43.16-helm
and tried to do connection without https and enabled ssl (it worked on the ui) but still it can’t connect to clickhouse
@User Any other dev (alpha/beta) version branch i could try? maybe it would work
baz/source-airtable-to-beta
worked for me, I was able to connect without https with ssl toggle is on! Also no issues with syncing
g
Sorry, I had to step away. We'll file an internal issue and investigate the problem.
n
Np, thanks for getting back.
@User One last question, do you happen to know if airbyte handle datetime conversion? I did sync between s3 and clickhouse the resulted table has timestamp col but it is parsed as string, is there a way that it can be parsed as timestamp or datetime?
g
Unfortunately I don't, I'm not an Airbyte expert at all (I'm responsible for the underlying dbt transformation project and the clickhouse client it uses).
n
Thanks
m
If i remove https it won’t work
@Noor Thabit How did you run Airbyte? I tested the ClickHouse connector on Airbyte Cloud, which works fine. Could you report an issue with the reproduction steps on https://github.com/airbytehq/airbyte/issues/new?assignees=&amp;labels=type%2Fbug%2C+needs-triage&amp;template=bug-report.md&amp;title=? The bug report could help us to identify the problem.
n
@Mikhail Shustov If you have Clickhouse cloud, they give you two option to connect. Natively using port 
9440
or through https port 
8443
. If I use 
9940
it does not connect, but if I use 
8443
it does, which mean it is using https?
g
Ah, that makes sense. Airflow does not support the native ClickHouse interface on port 9440. It uses the JDBC driver underneath, which is only http/https
n
Ok make sense now. Thanks for the clarification.
@Mikhail Shustov @Geoff Genz I’m having issues with syncing now, which is related to the clickhouse connection i think. I created an issue on github https://github.com/airbytehq/airbyte/issues/21328
g
Has it worked for you before on larger datasets?
n
No never. I tried with all kind of sync modes
g
I know that 'append' incremental materializations are not supported by the dbt-clickhouse version that Airbyte is using (at least without a custom configuration). Since that's the case, I would expect you to get an error or that option to disabled. We'll dig into it.
n
What about full rewrite?
according to the logs it seems like connection timeout.
Also it did partial data sync, which means it works fine?
g
You're using ClickHouse Cloud, right?
n
yese
g
A connection timeout could mean either a problem in dbt-clickhouse keeping the connection open, or a problem with your service. Can you share the relevant part of the logs?
n
its 380mb+
Copy code
Sync Failed
Last attempt:176.06 MB625,120 emitted records600,000 committed records20m 50s
Failure Origin: destination, Message: Something went wrong in the connector. See the logs for more details.
9:06AM 01/12
3 attempts

2023-01-12 16:18:24 - Additional Failure Information: java.lang.RuntimeException: java.util.concurrent.CompletionException: com.clickhouse.client.ClickHouseException: SSL peer shut down incorrectly, server ClickHouseNode [uri=<https://upj5s2f0ge.us-east-2.aws.clickhouse.cloud:8443/default>, options={sslmode=NONE,socket_timeout=3000000}]
g
Okay, that looks a Java exception, not a DBT (Python) exception. It could indicate something unhappy in your ClickHouse Cloud service
n
yea, and again i can see some data successfully extracted, transformed and loaded to clickhouse
m
While we're investigating, maybe you can implement the data exporting similarly to https://clickhouse.com/docs/en/integrations/redshift/redshift-pivot-to-clickhouse/? Is this a one-time operation, or do you need to do periodic synchronization?
n
@Mikhail Shustov Thanks for this. But yeah unfortunately I need periodic sync.
Airbyte is the right solution, if it only works
the same data that i’m trying to move to clickhouse is also in S3 (and snowflake) i tired to sync between s3 -> clickhouse, it worked but it treats everything as numbers and strings, it does not parse date, which is a big deal.
But snowflake -> clickhouse transformation is perfect. Some of the sync attempts produce partial syncing in clickhouse, but never able to get it to complete a full successful sync.
m
i tired to sync between s3 -> clickhouse, it worked but it treats everything as numbers and strings, it does not parse date, which is a big deal.
Did you try to s3 -> clickhouse sync via Airbyte or the ClickHouse built-in mechanism?
n
airbyte
s3 -> clickhouse is one time thing, i need sync functionality.
s3 -> clickhouse works fine with transformation and everything
but again, no syncing
3 Views
Open in Slack
PreviousNext
Powered by