Shawn Wang
05/27/2022, 1:41 PMAugustin Lafanechere (Airbyte)
06/02/2022, 1:59 PMKha Nguyen
06/07/2022, 1:58 PMMarcos Marx (Airbyte)
06/23/2022, 6:39 PMOlivier AGUDO PEREZ
06/28/2022, 9:53 AMArkadiusz Grzedzinski
06/30/2022, 1:08 PMAshley Baer
07/01/2022, 12:48 PMSefath Chowdhury
07/08/2022, 12:36 AM(ie: Microservice_1_Database | public schema -> Huge_Postgres_Instance | microservice_1 schema)
There are two problems with this.
1. AWS DMS is not resilient to DDL changes on the source DB
2. A huge postgres instance is still a postgres instance -> designed to be OLTP and not OLAP (we wanted to use redshift but many existing analytics queries break. This is something we are okay with when moving to cloud agnostic snowflake)
----------------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------------
Desired Situation
----------------------------------------------------------------------------------------------------------------------
Airbyte -> Snowflake using AWS RDS postgres (CDC enabled)
----------------------------------------------------------------------------------------------------------------------
1. Does anyone have a stack like this, that uses airbyte to replicate to snowflake real-time?
2. What Logical Decoding Output Plug-in are you using (hopefully you are using AWS RDS instances and that plugin is complaint) and why?
3. Did you deploy airbyte in a pod into a k8 cluster? and if so, how did you determine the specs it needed? I would assume on-going-replication is a heavy lift, and am unsure how to calculate the specs for this deploymentGerard Barrull
07/11/2022, 10:41 AMAri Bajo (Airbyte)
07/27/2022, 7:47 PMJose Luis Cases
08/09/2022, 9:12 PMJose Luis Cases
08/09/2022, 9:12 PMMarcos Marx (Airbyte)
08/11/2022, 6:37 PMJames Egan
08/16/2022, 10:48 AMJames Egan
08/16/2022, 10:48 AMVincent Koppen
08/17/2022, 10:01 AMAbba
08/17/2022, 2:27 PMJames Egan
08/17/2022, 2:50 PMHakeem Olu
08/18/2022, 2:30 PMDeployment: Using docker for deployment
Airbyte Version: 0.39.39-alpha
Source name/version: Redshift
Destination name/version: Snowflake
Step: The issue is happening during sync
Description: Data not showing in snowflake from redshift.
Versions:
From the airbyte
Redshift: 0.3.11
Snowflake: 0.4.34
AWS Redshift version: 1.0.40182
Hakeem Olu
08/18/2022, 2:30 PMSebastian Brickel
08/22/2022, 10:38 AMad_group_performance_report_hourly
and campaign_performance_report_hourly
. This gives: Failure Origin: source, Message: Checking source connection failed - please review this connection's configuration to prevent future syncs from failing
and Additional Failure Information: Server raised fault: 'Invalid client data. Check the SOAP fault details for more information.
Including only {ad,campaign}_performance_report_{daily,monthly ,weekly}
works fine.
Does anyone have an idea why that could be and how I could fix that?
Thank youShawn Wang (Airbyte)
08/22/2022, 8:30 PMThomas
08/26/2022, 12:05 PMBrendan McDonald
08/30/2022, 7:23 PMmarketing_emails
object, however it seems to be limited at 250 records (we have a total of 720 in hubspot). I am assuming this is because of some sort of rate limit through the API. Is there a way to backfill all data if the only way around this is through an incremental load setup?
For further context, I was able to get all records via the API using python directly using pagination. I am just not sure how to configure this via the airbyte UI.
Looking at the source code, it looks like there is a 250 record limit setup for each pull. This is definitely a nooby question, but how do you get around the pagination limit here?Dmytro Vorotyntsev
08/31/2022, 5:11 AMAn analysis of the cluster’s workload and database schema identified columns that will significantly benefit from using a different compression encoding.All suggested tables are those configured with Postgres CDC (Deduped History) And its suggestion
ALTER TABLE "public"."tatable_1_scd" ALTER COLUMN "_airbyte_unique_key_scd" ENCODE lzo;
ALTER TABLE "public"."table_2_scd" ALTER COLUMN "_airbyte_unique_key_scd" ENCODE lzo;
ALTER TABLE "public"."table_3_scd" ALTER COLUMN "_airbyte_unique_key_scd" ENCODE lzo;
ALTER TABLE "public"."tatable_1_scd" ALTER COLUMN "_airbyte_emitted_at" ENCODE az64;
ALTER TABLE "public"."table_3_scd" ALTER COLUMN "_airbyte_emitted_at" ENCODE az64;
ALTER TABLE "public"."table_4_scd" ALTER COLUMN "_airbyte_emitted_at" ENCODE az64;
Is it a relevant suggestion? Would it break the airbyte sync logic if encoding updated?
ThanksShivam Thakkar
08/31/2022, 3:17 PMLucas Wiley
09/15/2022, 11:08 PMCould not connect with provided configuration. net.snowflake.client.jdbc.SnowflakeSQLLoggedException: Private key provided is invalid or not supported: rsa_key.p8: Cannot invoke "net.snowflake.client.jdbc.internal.org.bouncycastle.util.io.pem.PemObject.getContent()" because the return value of "net.snowflake.client.jdbc.internal.org.bouncycastle.util.io.pem.PemReader.readPemObject()" is null
swyx (Airbyte)
Parham
09/21/2022, 10:19 AMAlexis Charrier
09/30/2022, 12:28 PM500 An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem: <https://cloud.google.com/bigquery/sla>. If the error continues to occur please contact support at <https://cloud.google.com/support>. Error: 5423415
Google status page is not reporting any issue regarding Bigquery service 🤔 any idea ?