Is this your first time deploying Airbyte: No OS Version / Instance: Amazon Linux, EC2 t2.medium Memory / Disk: 30Gb  Deployment: Docker  Airbyte Version: 0.35.32-alpha  Source name/version: Jira 0.2.18 Destination name/version: Redshift 0.3.25 Step: Syncing data to Redshift    Description: I’m trying to sync for the first time and the process doesn’t finish. All the tables get created in redshift but there is no data. Tried this multiple times. No luck. Attached the logs. Please help

Is this your first time deploying Airbyte: No OS Version / Instance: Amazon Linux 2 AMI x64 Deployment: Docker version 20.10.7 Airbyte Version: 0.35.31-alpha Source name/version: Twilio Destination: Azure SQL db Description: We have 2 twilio accounts that have separate calls each day. both are set as incremental updates. One we have running every 5 minutes, the other every 30 minutes. I can move this out, but the issue I'm seeing is, they both work just fine, but Airbyte consumes so much DataIO it's causing long waits on other queries, one thing that processes on this DB is an AAS tabular model, and when Airbyte is running it extends the model processing time from 20mins non-airbyte to over an hour and sometimes long enough to have to kill it. I've checked for locking, and there isn't any exclusive locks, just alot of consumer waits, or buffer waits. In monitoring the SQL performance metrics, the DataIO is consistently pegged at 60% when airbyte is on. Is there anything you can think of to reduce the load airbye takes, or a way to maybe speed up airbyte? If not, is there a way to schedule Airbyte at specific time slots, vs intervals?

Is this your first time deploying Airbyte: Yes OS Version / Instance: Contabo Ubuntu 20 Memory / Disk: 16Gb / 500 GB Deployment: Docker Airbyte Version: 0.35.28-alpha Source name/version: SFTP - FILE - exccel (xls) Destination name/version: BigQuery Table Step: Create Connection Description: When i create the conection i got this error "Failed to fetch schema. Please try again". I tested the source and the destination and they are ok.

Hey all, quick question about the Security & Privacy part from the documentation, especially the “data retention” part (https://docs.airbyte.com/project-overview/security)

Hi there, When I connect Stripe to BigQuerry, it's really strange, I only have 20 tables in Stripe and in GCP, Airbyte creates more than 100 tables ... Hundreds of empty tables... Any idea why ? Someone had the same issue ? Thanks for your help

👋 I successfully set up a Stripe Connection lately and run the initial Sync at first try with a manual run set up. However, when I retried lately it seems to be broken even after 3 retries approximately at the same stage. I suspect the issue to be linked to Stripe API blocking requests after too many was sent.

source - 2022-02-22 08:22:41 INFO () DefaultAirbyteStreamFactory(internalLog):97 - Backing off _send(...) for 20.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)
source - 2022-02-22 08:23:06 INFO () DefaultAirbyteStreamFactory(internalLog):97 - Caught retryable error '' after 3 tries. Waiting 20 seconds then retrying...
source - 2022-02-22 08:23:06 INFO () DefaultAirbyteStreamFactory(internalLog):97 - Backing off _send(...) for 40.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)
source - 2022-02-22 08:23:51 INFO () DefaultAirbyteStreamFactory(internalLog):97 - Caught retryable error '' after 4 tries. Waiting 40 seconds then retrying...
source - 2022-02-22 08:23:51 INFO () DefaultAirbyteStreamFactory(internalLog):97 - Backing off _send(...) for 80.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)
source - 2022-02-22 08:25:16 INFO () DefaultAirbyteStreamFactory(internalLog):97 - Caught retryable error '' after 5 tries. Waiting 80 seconds then retrying...
source - 2022-02-22 08:25:16 ERROR () DefaultAirbyteStreamFactory(internalLog):95 - Giving up _send(...) after 6 tries (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)
source - 2022-02-22 08:25:16 ERROR () DefaultAirbyteStreamFactory(internalLog):95 - Encountered an exception while reading stream SourceStripe

Ask for help submission from @Boggdan Barrientos *Is this your first time deploying Airbyte * No *OS Version / Instance * Ubuntu 20.04 / t3.2xlarge *Memory / Disk * 32GiB / 100 *Deployment * Docker *Airbyte Version * 0.35.5-alpha *Source name/version * Oracle DB 0.3.13 *Destination name/version * Redshift 0.3.25 *Step * Sync - incremental job *Description * I have error when running incremental synchronization. It had been working fine but now it fails. I had the table with basic transformation and now only in raw. I have other tables from the same database in other streams and they are working fine. Attached are the logs.

Hi team, Where can I specify requests.cpu & requests.memory for airbyte.workers (Helm Chart deployment)? Running into this error when attempting to add a new connector: Caused by: io.airbyte.workers.WorkerException: Failure executing: POST at: https://000.000.000/api/... Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "***************" is forbidden: failed quota: request-quota: must specify requests.cpu,requests.memory.

Has anyone been able to deploy on GKE behind IAP and accessing through your google login vs. iap tunneling? I've seen various posts on this but no clear answers?

Hi guys, First time posting here so please let me know if this is not the place. I’m trying to use the Airbyte rest api to set up sources and connections through automation but running into an issue with the rest api for listing connections: It keeps returning error

"Some properties contained invalid input."

but am am using the api as documented and this did work a couple of months ago when I built our original poc. I opened a github issue , not sure if that is the preferred thing to do, Any help would be greatly appreciated

Is this your first time deploying Airbyte: No OS Version / Instance: GCP c2-standard-4 Memory / Disk: 16Gb / 50GB SSD Deployment: Docker Airbyte Version: 0.35.32-alpha Source name/version: Google Ads (0.1.27) Destination name/version: BigQuery (0.6.8) Description: I am trying to set up this Google Ads connection and it seems like ONLY the account table is being synced successfully. All the other tables are empty after sync and shows the following logs while syncing:

2022-02-22 17:46:04 [44msource[0m > Setting state of ad_group_ads stream to {}
2022-02-22 17:46:05 [44msource[0m > Setting state of ad_group_ads stream to {}
2022-02-22 17:46:07 [44msource[0m > Setting state of ad_group_ads stream to {}
2022-02-22 17:46:08 [44msource[0m > Setting state of ad_group_ads stream to {}
2022-02-22 17:46:09 [44msource[0m > Setting state of ad_group_ads stream to {}
2022-02-22 17:46:10 [44msource[0m > Setting state of ad_group_ads stream to {}
2022-02-22 17:46:11 [44msource[0m > Setting state of ad_group_ads stream to {}
2022-02-22 17:46:11 [44msource[0m > Read 0 records from ad_group_ads stream
2022-02-22 17:46:11 [44msource[0m > Finished syncing SourceGoogleAds

Anyone got an idea of what might be happening here?

Hi team, Getting this request quota must be specified for sync pods error. I am specifying worker.resources and jobs.resources in ~/charts/airbyte/values.yaml. Are there any additional mappings I should look out for/ any thoughts?

pods "...sync-...." is forbidden: failed quota: request-quota: must specify requests.cpu,requests.memory.', type='io.fabric8.kubernetes.client.KubernetesClientException', nonRetryable=...

Ask for help submission from @Brad Fockler *Is this your first time deploying Airbyte * No *OS Version / Instance * Container-Optimized OS *Memory / Disk * left blank *Deployment * Kubernetes *Airbyte Version * 0.35.10-alpha *Source name/version * Mixpanel/0.1.9 *Destination name/version * BigQuery/0.6.5 *Step * left blank *Description * Sync produces the following errors

2022-02-22 13:58:59 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.  errors: $: null found, object expected 2022-02-22 13:58:59 [1;31mERROR[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null 2022-02-22 13:59:05 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.  errors: $: null found, object expected 2022-02-22 13:59:05 [1;31mERROR[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null 2022-02-22 13:59:22 [32mINFO[m i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.  errors: $: null found, object expected 2022-02-22 13:59:22 [1;31mERROR[m i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null

Which is followed by:

2022-02-22 13:59:31 [1;31mERROR[m i.a.w.DefaultReplicationWorker(run):141 - Sync worker failed. java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Source process exited with non-zero exit code 137  at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]  at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]  at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:134) ~[io.airbyte-airbyte-workers-0.35.10-alpha.jar:?]  at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:52) ~[io.airbyte-airbyte-workers-0.35.10-alpha.jar:?]  at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:174) ~[io.airbyte-airbyte-workers-0.35.10-alpha.jar:?]  at java.lang.Thread.run(Thread.java:833) [?:?]  Suppressed: io.airbyte.workers.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.   at io.airbyte.workers.protocols.airbyte.DefaultAirbyteSource.close(DefaultAirbyteSource.java:136) ~[io.airbyte-airbyte-workers-0.35.10-alpha.jar:?]   at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:118) ~[io.airbyte-airbyte-workers-0.35.10-alpha.jar:?]   at io.airbyte.workers.DefaultReplicationWorker.run(DefaultReplicationWorker.java:52) ~[io.airbyte-airbyte-workers-0.35.10-alpha.jar:?]   at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:174) ~[io.airbyte-airbyte-workers-0.35.10-alpha.jar:?]   at java.lang.Thread.run(Thread.java:833) [?:?]

Is this your first time deploying Airbyte: Yes OS Version / Instance: Amazon Linux 2, AWS Memory / Disk: 4.01476 Gb / 50 Gb Deployment: Docker Airbyte Version: 0.35.14-alpha Source name/version: Twilio Destination name/version: Amazon S3 Description: I'm trying to sync using parquet output format but it give me this error:

Exception in thread "main" java.nio.file.AccessDeniedException: s3-bucket: org.apache.hadoop.fs.s3a.auth.NoAwsCredentialsException: SimpleAWSCredentialsProvider: No AWS credentials in the Hadoop configuration

Also when I try with JSON/CSV format it works, and to mention that I attach an IAM role on the instance that runs Airbyte docker to read/write on the S3 bucket.

Is this your first time deploying Airbyte: No OS Version / Instance: Google Compute Engine VM (e2-highmem-2 - Intel Broadwell) running debian-10-buster-v20210512 Memory / Disk: 16GB RAM / 30 Gib Deployment: Docker Airbyte Version: 0.35.27-alpha Source name/version: airbyte/source-monday 0.1.3 Destination name/version:  airbyte/destination-bigquery 0.6.8 Description: Sync fails and gives requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: ... error even though connection test passes and am able to access the API with the token otherwise

Hi, looks like AirByte is hanging on my main server, how can I restart it?

Is this your first time deploying Airbyte: No OS Version / Instance: GKE Deployment: Docker Airbyte Version: 0.35.27-alpha Source name/version: airbyte/source-monday 0.1.3 Description: I see messages such as these in the

airbyte minio deployment

pod

API: SYSTEM()
Time: 04:47:08 UTC 02/23/2022
DeploymentID: f45f4b84-d973-46b6-9ff6-ae5fbd4d5c50
Error: Storage reached its minimum free disk threshold. (cmd.StorageFull)
1: cmd/data-usage.go:51:cmd.storeDataUsageInBackend()

Hi Folks, Can data be migrated every time I make modification to the source database using airbyte, rather then just waiting for it to sync every 5mins? Example -                    Source - Postgres Destination - MySQL If I make a change in Postgres - can it be transferred to mysql immediately or do I have to wait for the sync time - which has been set during connections?

Is this your first time deploying Airbyte: No OS Version / Instance: EC2 Deployment: Docker Airbyte Version: 0.35.26-alpha Source name/version: airbyte/source-zuora:0.1.3 Description:I noticed something strange while syncing Zuora to BigQuery. I had to add a new table to schema so airbyte told me that this will reset all existing, which I guess if fair enough, so it retested all table even the that there we're some already up-to-date,however, this is not the issue. It attempted to do 3 full sync even though 2 already complicated succesfuly, so I stopped it when it tried to do the third attempt. I let it run during the second because I thought this might be related to the way scheduled interval time. Any idea why is thig happening?

Hello, Is this your first time deploying Airbyte: Yes OS Version / Instance: Digital Ocean Droplet - Ubuntu Memory / Disk: 8Gb / 80Gb Airbyte Version: 0.35.32 Source name/version: MongoDB Destination name/version: BigQuery Step: sync successful but destination empty Description: We ran a sync of 22 elements. It ended with success status (9.64 KB | 22 emitted records | 22 committed records | 11s | Sync) but the destination table is empty and there is a validation error in the log file:

2022-02-21 154040 INFO i.a.v.j.JsonSchemaValidator(test):56 - JSON schema validation failed.

errors: $: null found, object expected

2022-02-21 154040 ERROR i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$1):70 - Validation failed: null

I used the exact same JSON schema with Redshift as destination and it worked. So i'm not sure whats wrong ?

Hello, I'm having issues with bootloader missing the latest migration, what should I do? (Solved)

any ideas here ? I have confirmed that DB connetivity can be established … just something isnt right with the initialization seems to silently fail on

i.a.i.s.j.AbstractJdbcSource(lambda$getCheckOperations$1):87 - Attempting to get metadata from the database to see if we can connect.

You need to edit the source.yml file created by normalization, let me know if you find the correct place

Is this your first time deploying Airbyte: Yes OS Version / Instance: Win 10 Pro 64 bit Memory / Disk: 16 GB Deployment: Docker Airbyte Version: 0.35.30-alpha Source name/version: mssql 0.3.17 Destination name/version: redshift 0.3.27 and local csv Description: Source and destination start correctly, source start querying data on 1,000 rows partitions. When getting to 100k+ rows, the destionation closes and get a sync fail. The query results vary, sometimes the process stops at 200K rows, 400K, 350K, etc.

ReplicationWorker(lambda$getReplicationRunnable$5):300 - Records read: 259000

2022-02-28 20:49:19 [32mINFO[m i.a.w.DefaultReplicationWorker(cancel):383 - Cancelling destination...

Is this your first time deploying Airbyte: Yes OS Version / Instance: Amazon Linux Memory / Disk: 8 GB Deployment: Docker Airbyte Version: 0.35.38-alpha Source name/version: MSSQL 0.3.17 Destination name/version: Snowflake 0.4.17 Description: Snowflake destination connector is throwing an error "Something went wrong within the destination connector" that I'm not sure how to interpret. The full logs are attached below (errors start around line 1350), any insight would be appreciated!

Is this your first time deploying Airbyte: No OS Version / Instance: EC2 t2.large Memory / Disk: 32Gb Deployment: Docker Airbyte Version: 0.35.32-alpha Source name/version: Square 0.1.4 Destination name/version: Postgres 0.3.15 Step: Retrieving tables from Square Description: Hi team! I am noticing a good amount fields that are missing from different Square tables. Most of them are Foreign Keys that are needed to link to other tables. Any idea why this may be happening?

Hi Team I have loaded Zendesk data to s3 in parquet file format. Note: the data contains json columns. Now, while I am trying to copy this s3 parquet file to redshift, I am getting the below issue: SQL Error [XX000]: ERROR: Spectrum Scan Error Detail: ----------------------------------------------- error: Spectrum Scan Error code: 15007 context: Unsupported implicit cast: Column s3://airbyte-sync-tn/data_sync/test_zendesk_super/tickets/2022_02_28_1646039367500_0.parquet.via, FromType: struct<struct<struct<byte_array,byte_array,byte_array,byte_array,byte_array,byte_array,byte_array,byte_array,map<b

Ask for help submission from @Gary K *Is this your first time deploying Airbyte * No *OS Version / Instance * Debian GNU/Linux 11 (bullseye) *Memory / Disk * left blank *Deployment * Kubernetes *Airbyte Version * 0.35.42-alpha *Source name/version * MySQL, 0.5.6 *Destination name/version * Postgres, 0.3.15 *Step * Mid-sync *Description * In the middle of a sync it errors out. All tests are ok, and a lot of data is transferred, but at what looks like the same point it just stops with no reason (apart from exit code 1). Normalisation occurs without error. The row size is small, and I've checked the data in the table where it stops without noticing any bad data weirdness. Any ideas on where/what to look at here?

Hello, I am using GCP Compute Engine for Facebook Ads(FB Marketing)Connection to BQ as a POC. I have observed that every time i setup this job our engine is completely inaccessible. I am unsure but should we provision a larger Instance for this? Since this is a POC i have just selected a e2-medium Machine Type while the rest of the config are in default. Thanks in advance for the help and clarification!

Is this your first time deploying Airbyte: No Deployment Kubernetes Airbyte Version: 0.35.5-alpha Source name/version: Postgres version 0.4.4 ( had to downgrade from most recent one, because of this issue, but the same also happend in 0.4.8) Destination name/version: Postgres version 0.3.14 Step: On sync Description: Hello 👋 I am syncing data between two postgres databases, and the process gets stuck on the “Executing finalization of tables”, it has been like this for the whole night and after that I had to cancel the process. What Is strange is, that when I exclude the biggest table from the list of tables that are being synced ( it has around 20GB in total) everything works correctly… octavia thinking Is there any known issue that airbyte cannot transfer tables bigger than some threshold? Or maybe some way to see more debug messages, to better investigate what is going on? Thank you for any help! octavia thanks (See the attached full logs, as well as the message where the logs got stuck for many hours )

2022-02-28 15:58:47 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-02-28 15:58:47 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - Executing finalization of tables.
2022-03-01 07:50:13 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$3):210 - Running sync worker cancellation...