Andres Gutierrez
01/26/2023, 12:01 PMincremental and delete+insert for Clickhouse as destionation from Airbyte? By delete+insert I mean this
If I'm understand correctly this requires dbt-clickhouse@1.3.2 and I see in Airbyte's master branch =1.3.1>
Also. How I can check in my Airbyte installation what version of dbt-clickhouse is installed?
In what docker container should I check?user
01/26/2023, 4:41 PMAndres Gutierrez
01/26/2023, 5:11 PMincremental+dedup
:HTTPDriver for <https://zytq4zf64e.eu-central-1.aws.clickhouse.cloud:8443> returned response code 500
Code: 36. DB::Exception: ALTER UPDATE/ALTER DELETE statement with subquery may be nondeterministic, see allow_nondeterministic_mutations setting. (BAD_ARGUMENTS)
Also this document in Clickhouse docs explains that incremental+dedup is slow because it creates a full copy of the table before adding the delta of the changes
https://clickhouse.com/docs/en/integrations/dbt/dbt-incremental-model#internals
Down below in that same page explain that delete_insert is more efficient
https://clickhouse.com/docs/en/integrations/dbt/dbt-incremental-model#deleteinsert-mode-experimentalAndres Gutierrez
01/26/2023, 5:12 PMdelete_insert incremental sync modeAndres Gutierrez
01/26/2023, 5:13 PMuser
01/27/2023, 9:42 AMallow_nondeterministic_mutations flag. There isn't any ETA in current Airbyte roadmap for Clickhouse destination, my recommendation is: export the normaization module https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-sql and edit as your need to improve or achieve conditions to run in Clickhouse Cloud.Andres Gutierrez
01/27/2023, 9:44 AMAndres Gutierrez
01/27/2023, 11:02 AM:HTTPDriver for https://[OUR_ID].clickhouse.cloud:8443 returned response code 500)
Code: 36. DB::Exception: ALTER UPDATE/ALTER DELETE statement with subquery may be nondeterministic, see allow_nondeterministic_mutations setting. (BAD_ARGUMENTS)
They responded this:
Hi Andres,
This is a known issue, the Airbyte integration do queries that are not supported by ClickHouse. I have open an escalation to the integration team to see if we could workaround this issue.Maybe Airbyte documentation should put a note about connecting with Clickhouse.cloud and this limitation when you want to use
incremental+dedupMarcos Marx (Airbyte)
01/27/2023, 11:06 AMAndres Gutierrez
01/27/2023, 11:09 AMGeoff Genz
01/27/2023, 11:14 AMAndres Gutierrez
01/27/2023, 11:25 AMGeoff Genz
01/27/2023, 11:32 AMAndres Gutierrez
01/27/2023, 11:34 AMGeoff Genz
01/27/2023, 11:34 AM