I'm getting the following error while trying to sync the data from source to destination State code: 08S01; Message: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. I'm trying to sync the MySQL database from my local machine to BigQuery in Google cloud. can you please assist me?

Has anyone successfully installed Airbyte on K8s on AWS?

Hi all I am looking for `Entity Relationship Diagram’ for okta connector Please if i get view of the workflow in the ERD that would be big help

Hello, folks! I've just completed creating a custom Python connector for Gmail and while it passes the check (both locally and running in the container), the connection fails to write to S3 for what appears to be a JSON validation error (see attached logs). The odd part is it appears it's trying to use OAuth despite the fact that I successfully configured it to use service account credentials (see screenshot). Any thoughts on what I'm doing wrong here?

I am observing:

io.airbyte.workers.exception.WorkerException: Could not find image: airbyte/source-google-analytics-data-api:0.0.3

How can I configure airbyte to work on-premise in a firewalled environment with a corporate docker registry/proxy? I.e. how to tell airbyte to pull from a local Artifactory instance similar to:

<http://local_docker_registry.mycorp.com/airbyte/source-google-analytics-data-api:0.0.3|local_docker_registry.mycorp.com/airbyte/source-google-analytics-data-api:0.0.3>

?

Hi, some of my connections are in delay in case of several task running simultaneously. You can see it in log timestamps here:

2022-10-06 07:52:17 INFO i.a.w.t.TemporalAttemptExecution(get):134 - Stopping cancellation check scheduling...
2022-10-06 08:48:01 INFO i.a.w.t.TemporalAttemptExecution(get):108 - Docker volume job log path: /tmp/workspace/376/0/logs.log
2022-10-06 08:48:01 INFO i.a.w.t.TemporalAttemptExecution(get):113 - Executing worker wrapper. Airbyte version: 0.39.7-alpha

It looks like some kind of balancing, several connections waits for resources and then continue, but there is no log message. Can anyone explain? Thanks a lot!

Hello, I'm using airbyte on a google cloud VM instance, and I'm wondering if it's safe to use an instance scheduler to start and stop the VM instance. What happens if a connection still running when the scheduler stops the VM?

Hello everybody, I'm new to the community and I would like to tell you that you are building an awesome solution that makes everyone's life easier, so thanks for that. My question about MSSQL connector, when it will be BETA? and if there is any issue holding it from being BETA can I help fixing it? It is so important for my current project and I want to push it if I can. Regards

I am looking to use the Postgres CDC to Redshift on a table with very large JSON blobs. Ideally, I'd like to map those types to a SUPER column both in terms of usability as well as due to max size as jsonb can be > than redshift varchar(max). Is this possible or are there gotchyas to watch out for?

I upgraded from version

0.39.39

to

0.40.11

and the GA connector is starting to fail with this message:

Additional Failure Information: message='Job ran during migration from Legacy State to Per Stream State. One of the streams that did not have state is: io.airbyte.protocol.models.StreamDescriptor@3dc7762a[name=primary,namespace=<null>,additionalProperties={}]. Job must be retried in order to properly store state.', type='java.lang.IllegalStateException', nonRetryable=false

I tried resetting the connector but that did not make a difference. The sync mode being used is

Incremental Deduped + History

. EDIT: I was able to resolve this issue by manually deleting the state record from the app db.

Hi team - I'm having issues with Google Analytics (Universal Analytics), when I try to leverage the Custom Report feature. I can generate a custom report with no issues, up until I start adding goal conversions. This custom report works just fine:

[{"name": "nodus_test", "dimensions": ["ga:date","ga:source","ga:medium","ga:keyword", "ga:campaign", "ga:adContent"], "metrics": ["ga:sessions","ga:transactionRevenue"]}]

But this one generates entirely NULL results:

[{"name": "nodus_ga_customconversions", "dimensions": ["ga:date","ga:source","ga:medium","ga:keyword","ga:campaign","ga:adContent"], "metrics":["ga:sessions","ga:users","ga:goal1Completions","ga:goal2Completions","ga:goal3Completions","ga:goal4Completions","ga:goal5Completions","ga:goal6Completions","ga:goal7Completions","ga:goal8Completions","ga:goal9Completions","ga:goal10Completions","ga:goal11Completions","ga:goal12Completions","ga:transactionRevenue"]}]

Has anyone ever faced this issue? I have been banging my head against this for days, with no luck.

I’m evaluating the Airbyte open source version on AWS to replace another ELT tool we use and I’m happy with it so far even though some of the connectors are in alpha. I wonder how you use it for the prod environment. Does anyone use it on an EC2 with Docker for prod? How has your experience been so far? What kind of deployment do you recommend?

Hi Team. I am using /v1/sources/update to update source parameter value for my requirement. This API requires sourceId to be sent in payload. The sourceId however changes for different environment(DEV,STG,PRD). Is there a way we can use source name (stays same in all env) instead of sourceId for updating the source?

Hi Guys. I'm trying to pass array for one of field in spec.yaml file. But its giving me this error

Exception: Config validation error: 'didis_livelihood_plan' is not of type 'array'

   id:
      type:  array
      title: ID
      description: Unique identifier for one of your forms
      order: 3

What can I do here?

Hello, I'm trying to use Snowflake as a source, and having issues with schema discovery, even though I'm using accountadmin credentials... Would you be able to help? Please find server logs attached (the source is 4790d5f2-16b6-4cc9-bf61-d8ddc1429965)

Hi there, curious how you do handle CDC with snowflake?. If we want to pull data from snowflake

Can custom dbt transformations be used to secure data within Airbyte's destination namespace? E.g. basically running transformations in transit before data hits the destination. I believe the answer is yes, I am just curious if there is existing precedent or best practices in place.

This message was deleted.

hello guys I have this error The connection tests failed. Internal Server Error: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long. I install with helm , helm install gke cluster public or privite same error kubernete local same error

Hello Airbyte Community folks, I’ve run into a problem where it looks as though an incremental: deduped + history sync may not be deduping. The connection is from Postgres to Snowflake and is syncing one table. For the source table, the cursor field is a timestamp with tz,

updated_at

, and the primary key,

id

is a unique bigint generated from a sequence. There are ~286M records in the table. After the destination tables are generated, the

_scd

table has ~759M rows and the final table has ~450M rows. If I query the final table for `id`s having a count > 1, I get ~1.8M results. A similar query on the source table returns 0 results, as you might expect. There don’t seem to be any big errors in the initial table population or subsequent hourly syncs. Any suggestions would be happily received. Thanks!

Hello Airbyte community!! I know lots of folks would have asked this b4 and i did search for it, but wanted to make sure whats the latest, does mysql schema changes gets propagated into the destination?

Hello! I’m encountering an issue with connector scheduling. None of the connections I have will sync unless manually triggered despite having a 24 hour sync frequency.

Hi all, I’m running into an issue setting up a connection using Mixpanel. While I can create a Source ok, it appears that fetching the schema causes an error, because some of the

default_cursor_field

values are unexpectedly `null`:

2022-10-06 21:38:31 INFO i.a.v.j.JsonSchemaValidator(test):71 - JSON schema validation failed.
errors: $.catalog.streams[4].default_cursor_field: null found, array expected, $.catalog.streams[5].default_cursor_field: null found, array expected
Log4j2Appender says: JSON schema validation failed.
errors: $.catalog.streams[4].default_cursor_field: null found, array expected, $.catalog.streams[5].default_cursor_field: null found, array expected

(more info in thread)

Hey we're not seeing shopping data in our Bing connector. Found this page saying that CampaignType only returns search by default. https://learn.microsoft.com/en-us/advertising/campaign-management-service/getcampaignsbyaccountid?view=bingads-13 could this be why we're only getting search campaign data?

Is it possible to only use a public repo for custom DBT transformations?

Hi @channel, I need help, I build a connector to use airbyte API, when develop in local (`http://localhost:3000/api/v1/) everything work fine, but when deploy the solution and try to connect to the airbyte instance url throw an error 401 and I not understand why o the way to get auth token to send by headers to the Airbyte API

Hello community, Which are the JDBC parameters for increasing the query timeout for Mysql as a source. Cannot find this documented. Thanks in advance.

Hello community, Wanted to understand how to achieve parallelism/horizontal scale when using Kafka as destination? We have a requirement wherein we will be pushing transformed data from our data lake to Kafka for downstream consumption. Thanks.

Hello, I connect Bing Ads and Airbyte. I import two tables : ad_performance_report_daily and campaign_performance_report_daily. This two tables have a field 'CampaignName' but it is empty. Does anybody have met this problem ?

Hi there, a quick question regarding the S3 destination: I am using an EC2 instance with 16 or 32 Gigs of RAM to pull in a data source which, in total, provides several 100 Gigs uncompressed. I see in the logs what the total amount of data read is and when a buffer is flushed. I am using the default S3 destination settings for Parquet SNAPPY compression. Yet, every single run, the sync fails with OoM errors. Why would that happen if the buffer flushes every 200MB or so (default setting is 128MB even)? Am I missing something obvious why memory is drowning besides flushes?