Hi everyone, Please help to understand how the Postgres Source with CDC suppose to work I have done the following (according to connector readme): • Created identity for all tables I’m going to use with Incremental | Dedeped + History sync strategy • Created publication and included all tables I want to sync (those which for Incremental as well as some for Full Refresh) • Created publication slot • Set connector details in the Airbyte • On the other end I use Amazon Redshift with S3 Staging option Initial sync took 25h

85.24 GB | 271,975,505 emitted records | 271,975,505 committed records | 25h 2m 5s | Sync

For the subsequent sync operation I expected to take less, because it should use publication slot to get only new entries for tables where Incremental | Deduped + History is selected, and rewrite completely those entries where Full refresh | Overwrite is chosen, But I don’t understand (missed the doc?) why the subsequent sync is taking even longer (still in progress since yesterday) and re-scanning all data from the source. UPDATE: The issue seems to be in another dimensions We hit Airbyte performance limit, so we have increased to more powerful CPU and increased Network throughput Also the Redhisft initial single-node has 100% storage full, so we are switching to multi-node setup with automatic storage scaling

How is googleads library resolved within source-google-ads connector https://github.com/airbytehq/airbyte/search?q=googleads shows 7 references, but nothing is declared in requirements.txt. Shouldn't it be part of requirements.txt as well

How to upload private key input as part of Airbyte custom connector The section on multiline input https://docs.airbyte.com/connector-development/connector-specification-reference#multi-line-string-inputs talks about entering the information as a String, but would it be possible to upload a PEM file as input to the connector during connector configuration

Hi! I'm trying to build a connection between appstore and bigquery. However, when I press sync schemas I don't get anything back from airbyte, and thus can't choose which tables I want to sync. Anyone know how to fix this?

Hi we tried installing airbyte on an ec2 instance : (t3.micro) - basis the documentation. the installation goes smoothly anf for the first 5 mins we are able to add sources / destiations. however post that everything becomes super slow and freezes. we cant even ssh into the instance. and need to restart. which also allows repeats the same issue after 5 mins. any suggestion / advice where we are going wrong ? the same worked perfectly on docker desktop in my mac.

Hi.. Can someone help me on this issue?.. I got really stuck on this issue for more than a week? When I run the full sync, its perfectly working working POSTGRES - SNOWFLAKE sync. But when I changed to INCREMENTAL+ DEPUDED HISTORY sync its failing with Insufficnet privileges issue in Snowflake. BUt I have all required access.

Insufficient privileges to operate on database 'SANDBOX',externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@475585c[additionalProperties={attemptNumber=0, jobId=69, from_trace_message=true}],stacktrace=Encountered an error:
Database Error
** 003001 (42501): SQL access control error:**
** Insufficient privileges to operate on database 'SANDBOX',retryable=,timestamp=1660936001968], io.airbyte.config.FailureReason@6769c101[failureOrigin=normalization,failureType=system_error,internalMessage=Encountered an error:**
Database Error
** 003001 (42501): SQL access control error:**
** Insufficient privileges to operate on database 'SANDBOX',externalMessage=Normalization failed during the dbt run. This may indicate a problem with the data itself.,metadata=io.airbyte.config.Metadata@5a05d4b3[additionalProperties={attemptNumber=0, jobId=69, from_trace_message=true}],stacktrace=Encountered an error:**
Database Error
** 003001 (42501): SQL access control error:**
** Insufficient privileges to operate on database 'SANDBOX',retryable=,timestamp=1660936001968]]]**

is there any special PERMISSION required for Snowflake connections?. or is this bug ?

Hello Airbyte enthusiasts! I am working on setting up a connection that needs to decrypt (PGP) the data as it pulls it from PostGres. Does anyone have some documentation recommendations or git examples? I'm having trouble getting this one implemented. Any help would be much appreciated.

is the BambooHR connector only available in self deploy? And has anyone been able to successfully sync custom fields?

hi i’m actually having issues at the very first step of installing airbyte

docker-compose up

fails due to missing docker manifests. i’m running docker desktop on an m1 macbook pro

Hi… how can I extract noramlization files like explained in https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt in a Kubernetes deployment? Thank you!

Hi! Have anyone experienced that the sync between App Store to BigQuery fails for sales_report but succeeds for subscription_event_report and subscription_report? And any idea how to fix it? I'm running • OS Version / Instance: MacOS • Memory / Disk: you can use something like 16Gb • Deployment: docker • Airbyte Version: 3.7 and getting the errors attached. Also created an issue here.

The specification step failed with the following error. How to resolve this issue? (.venv) ubuntu@ip-172-31-44-147:~/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer$ python3 main.py spec Traceback (most recent call last): File "/home/ubuntu/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer/main.py", line 9, in <module> from source_appstore_singer import SourceAppstoreSinger File "/home/ubuntu/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer/source_appstore_singer/__init__.py", line 1, in <module> from .source import SourceAppstoreSinger File "/home/ubuntu/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer/source_appstore_singer/source.py", line 15, in <module> from appstoreconnect import Api File "/home/ubuntu/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer/.venv/lib/python3.10/site-packages/appstoreconnect/__init__.py", line 1, i n <module> from .api import Api, UserRole File "/home/ubuntu/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer/.venv/lib/python3.10/site-packages/appstoreconnect/api.py", line 2, in <mo dule> import jwt File "/home/ubuntu/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer/.venv/lib/python3.10/site-packages/jwt/__init__.py", line 19, in <module> from .api_jwt import ( File "/home/ubuntu/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer/.venv/lib/python3.10/site-packages/jwt/api_jwt.py", line 4, in <module> from collections import Iterable, Mapping ImportError: cannot import name 'Iterable' from 'collections' (/usr/lib/python3.10/collections/__init__.py) (.venv) ubuntu@ip-172-31-44-147:~/work/software/airbyte/airbyte-integrations/connectors/source-appstore-singer$

Hi there, where can I find release notes for the

airbyte-cdk

? 🙂

Hello, Can someone help me with an issue with Zendesk Chat connector? I want to understand if current Connector supports Zendesk Chat that is an app inside Zendesk support. I have been following the docs to setup the connector. But it takes me to https://account.zopim.com/ where my login credentials do not work. Appreciate any help I can get

Is there any simple way to re-enable a disabled workspace?

Hey all. I created a custom connector and I was able to make it work when adding it via airbyte UI (localhost). How can I add my custom connector into the default list when starting up airbyte server? I tried adding it in this PR (but I commented for now because it didn't work) into

airbyte-config/init/src/main/resources/seed/source_definitions.yaml

but it didn't seem to be displayed automatically in airbyte's connector default list

Hi Team, i am also new to airbyte, I have question regarding connections, whenever we create a connection, webui ask for source/destination connectors and then only we can create connection, is there way to reutilize same source/destination connections

it is practically impossible to create source/destination connections eveytime for each job

please advise if any body knew, please let know

Where can I find Airbyte limitations? I added 683 connections and looks like some of them waiting to sync... p.s. why it is using so much cpu resource and less ram?

hi all, not sure if this is the channel to post a question. I just get started today with airbyte and managed to get it setup from my local machine. However, one of the container

airbyte-webapp

has been having troubles to bring up and giving errors below. Please feel free to let me know how this can be resolved thanks

2022/08/24 04:56:11 [emerg] 1#1: unknown "is_demo" variable
nginx: [emerg] unknown "is_demo" variable
2022/08/24 04:56:12 [emerg] 1#1: unknown "is_demo" variable
nginx: [emerg] unknown "is_demo" variable
2022/08/24 04:56:12 [emerg] 1#1: unknown "is_demo" variable
nginx: [emerg] unknown "is_demo" variable

git reset --hard 3134619daf1c2f9f747f8c46155cc55d492f22e1 and after start it.

Most sync log showing:

2022-08-24 04:42:24 INFO i.a.w.t.TemporalAttemptExecution(get):131 - Stopping cancellation check scheduling...

What does it means?

Hi there.

I am using AIRBYTE to transfer data from Mongo DB to Snowflake

I created source connection with Mongo DB successfully. But as I go to use this source connection with destination connection (snowflake), it stops the service of my Mongo DB. Please help.

Hello) I was wondering where does Airbyte that’s deployed locally stores connector credentials (if I don’t want to use GSP)?

I have configured the Airbyte on my cloud server from where I am using it through VPN.