Hi Team, I am trying to replicate data from MySQL to MySQL but I get this error. Anyway to solve this?

hey quick (but beginner!) question: if I edit the python code for a given connector (e.g. modifying the

expand

part of the confluence connector here) should I expect it the fields I add to show up in my destination? or does this fall into the category of creating a net new custom connector documented here

hi! wanted to resurface an error i've been running into with Intercom. hoping to get a quick review of this: https://airbytehq.slack.com/archives/C021JANJ6TY/p1676337035689279

Hi There , I am working on set up a mysql destination, but here is the error. and mysql version is 8

2023-02-22 065051 INFO i.a.w.t.TemporalAttemptExecution(get):136 - Docker volume job log path: /tmp/workspace/21b8e27d-707d-4df8-8377-53ad6867ea59/0/logs.log 2023-02-22 065051 INFO i.a.w.t.TemporalAttemptExecution(get):141 - Executing worker wrapper. Airbyte version: 0.40.26 2023-02-22 065051 INFO i.a.a.c.AirbyteApiClient(retryWithJitter):172 - Attempt 0 to save workflow id for cancellation 2023-02-22 065051 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-02-22 065051 INFO i.a.c.i.LineGobbler(voidCall):114 - ----- START CHECK ----- 2023-02-22 065051 INFO i.a.c.i.LineGobbler(voidCall):114 - 2023-02-22 065051 INFO i.a.c.i.LineGobbler(voidCall):114 - Checking if airbyte/destination-mysql:0.1.20 exists... 2023-02-22 065051 INFO i.a.c.i.LineGobbler(voidCall):114 - airbyte/destination-mysql:0.1.20 was found locally. 2023-02-22 065051 INFO i.a.w.p.DockerProcessFactory(create):120 - Creating docker container = destination-mysql-check-21b8e27d-707d-4df8-8377-53ad6867ea59-0-gezie with resources io.airbyte.config.ResourceRequirements@6b42d60a[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] 2023-02-22 065051 INFO i.a.w.p.DockerProcessFactory(create):164 - Preparing command: docker run --rm --init -i -w /data/21b8e27d-707d-4df8-8377-53ad6867ea59/0 --log-driver none --name destination-mysql-check-21b8e27d-707d-4df8-8377-53ad6867ea59-0-gezie --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e WORKER_JOB_ATTEMPT=0 -e WORKER_CONNECTOR_IMAGE=airbyte/destination-mysql:0.1.20 -e AUTO_DETECT_SCHEMA=false -e AIRBYTE_VERSION=0.40.26 -e WORKER_JOB_ID=21b8e27d-707d-4df8-8377-53ad6867ea59 airbyte/destination-mysql:0.1.20 check --config source_config.json 2023-02-22 065051 INFO i.a.w.i.VersionedAirbyteStreamFactory(create):97 - Reading messages from protocol version 0.2.0 2023-02-22 065051 ERROR i.a.c.i.LineGobbler(voidCall):114 - SLF4J: Class path contains multiple SLF4J bindings. 2023-02-22 065051 ERROR i.a.c.i.LineGobbler(voidCall):114 - SLF4J: Found binding in [jarfile/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2023-02-22 065051 ERROR i.a.c.i.LineGobbler(voidCall):114 - SLF4J: Found binding in [jarfile/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2023-02-22 065051 ERROR i.a.c.i.LineGobbler(voidCall):114 - SLF4J: Found binding in [jarfile/airbyte/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] 2023-02-22 065051 ERROR i.a.c.i.LineGobbler(voidCall):114 - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 2023-02-22 065051 ERROR i.a.c.i.LineGobbler(voidCall):114 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2023-02-22 065052 INFO i.a.w.i.DefaultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 INFO i.a.i.d.m.MySQLDestination(main):136 - starting destination: class io.airbyte.integrations.destination.mysql.MySQLDestination 2023-02-22 065052 INFO i.a.w.i.DefaultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 INFO i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json} 2023-02-22 065052 INFO i.a.w.i.DefaultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 INFO i.a.i.b.IntegrationRunner(runInternal):123 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination 2023-02-22 065052 INFO i.a.w.i.DefaultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 INFO i.a.i.b.IntegrationRunner(runInternal):124 - Command: CHECK

eGobbler(voidCall):114 - ----- START CHECK ----- neGobbler(voidCall):114 - neGobbler(voidCall):114 - Checking if airbyte/destination-mysql:0.1.20 exists... neGobbler(voidCall):114 - airbyte/destination-mysql:0.1.20 was found locally. ckerProcessFactory(create):120 - Creating docker container = destination-mysql-check-21b8e27d-707d-4df8-8377-53ad6867ea59-0-gezie with resources io.airbyte.config.ResourceRequirements@6b42d60a[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] ckerProcessFactory(create):164 - Preparing command: docker run --rm --init -i -w /data/21b8e27d-707d-4df8-8377-53ad6867ea59/0 --log-driver none --name destination-mysql-check-21b8e27d-707d-4df8-8377-53ad6867ea59-0-gezie --network host -v airbyte rsionedAirbyteStreamFactory(create):97 - Reading messages from protocol version 0.2.0 .LineGobbler(voidCall):114 - SLF4J: Class path contains multiple SLF4J bindings. .LineGobbler(voidCall):114 - SLF4J: Found binding in [jarfile/airbyte/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] .LineGobbler(voidCall):114 - SLF4J: Found binding in [jarfile/airbyte/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] .LineGobbler(voidCall):114 - SLF4J: Found binding in [jarfile/airbyte/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] .LineGobbler(voidCall):114 - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. .LineGobbler(voidCall):114 - SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[32mINFO^[[m i.a.i.d.m.MySQLDestination(main):136 - starting destination: class io.airbyte.integrations.destination.mysql.MySQLDestination faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[32mINFO^[[m i.a.i.b.IntegrationCliParser(parseOptions):118 - integration args: {check=null, config=source_config.json} faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[32mINFO^[[m i.a.i.b.IntegrationRunner(runInternal):123 - Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[32mINFO^[[m i.a.i.b.IntegrationRunner(runInternal):124 - Command: CHECK faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[32mINFO^[[m i.a.i.b.IntegrationRunner(runInternal):125 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'} faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[33mWARN^[[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonVali faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[33mWARN^[[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonV faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[33mWARN^[[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065052 ^[[33mWARN^[[m c.n.s.JsonMetaSchema(newValidator):338 - Unknown keyword multiline - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a Non faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065053 ^[[32mINFO^[[m i.a.i.b.s.SshTunnel(getInstance):170 - Starting connection with method: NO_TUNNEL faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065053 ^[[32mINFO^[[m c.z.h.HikariDataSource(<init>):80 - HikariPool-1 - Starting... faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065053 ^[[32mINFO^[[m c.z.h.HikariDataSource(<init>):82 - HikariPool-1 - Start completed. faultAirbyteStreamFactory(parseJson):78 - 2023-02-22 065153 ^[[1;31mERROR^[[m i.a.i.d.m.MySQLDestination(check):83 - Exception while checking connection: faultAirbyteStreamFactory(parseJson):78 - java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60002ms.

Hello all! I am using Airbyte 0.40.28 and from time to time some of the connectors fail, no matter which ones, giving very similar messages to each other. After it crashes, the application stops responding and all the computer resources (CPU, Disk usage) are raised almost to the maximum. I leave the logs attached to the message to see if anyone can give me a hand, because I don't know why this happens sometimes. Thank you in advance!

👋 Hello Team, I'm seeing quite often that airbyte worker & server pods are holding up memory for quite a long, even if no syncs are running. It continuously grows with time. It releases after I restart the pods, but this is not the perfect solution. Can someone give some insights on • how airbyte worker/server pods manage memory & how can I make sure that they release it after syncs are completed. • Is there any K8S configurations that I can add in

.env

file? Airbyte version: 0.40.6

Hi 👋 After upgrading to airbyte-helm 0.40.28 when I trigger a job with the API I got this error

Caused by: java.net.UnknownHostException: minio

Even if I don’t use use minio I should miss an env variable but it was working perfectly with airbyte 0.40.27

hey 👋 I'm having some troubles setting the jdbc_url_params setting in clickhouse destination. Airbyte seems to be ignoring the configuration even though it gets saved correctly. I've given more details in this ticket.

Hey guys, I'm currently working on a schema, which has an array of objects, which also could be empty. When I run the Basic Normalization, it doesn't create new tables for the objects inside the array (which is the behaviour I expect), although I specified the objects. So does anyone of you has an idea of how a schema should look like? This is my schema:

{
  "$schema": "<http://json-schema.org/draft-04/schema#>",
  "type": "object",
  "properties": {
    "key": {
      "type": "integer"
    },
    "value": {
      "type": "array",
      "contains": {
        "type": "object",
        "properties": {
          "blockNumber": {
            "type": "integer"
          },
          "blockHash": {
            "type": "string"
          },
          "transactionIndex": {
            "type": "integer"
          },
          "removed": {
            "type": "boolean"
          },
          "address": {
            "type": "string"
          },
          "data": {
            "type": "string"
          },
          "topics": {
            "type": "array",
            "items": {
              "type": "string"
            }
          },
          "transactionHash": {
            "type": "string"
          },
          "logIndex": {
            "type": "integer"
          },
          "parsedEvent": {
            "type": "object"
          }
        },
        "required": [
          "blockNumber",
          "blockHash",
          "transactionIndex",
          "removed",
          "address",
          "data",
          "topics",
          "transactionHash",
          "logIndex",
          "parsedEvent"
        ]
      }
    }
  },
  "required": [
    "key",
    "value"
  ]
}

Hello team! Has anyone here tried to use octavia in CI/CD? I’m trying to come up with a solution where we might keep our yaml definitions in our github repo and have our CI/CD pipeline handle the application of the new connectors/sources/destinations automatically. Any ideas/help would be greatly appreciated!

Hello All, I am trying the Freshdesk Source, but some of the tables are not allowed to incremental refresh even it has id and updated_at columns. What is the exact reason of this behaviour?

hellop team, I am building airbyte on local machine, but hang a long time. How to fix it?

Hello Team, im trying to use Airbyte Api to get connections from an Airbyte open source instance, my question is how can bypass the authentication flow, i found out that i need some sort of key ( AIRBYTE_API_KEY ) that the instance's owner must provide to get successful authentication ?

Hey team 👋 I’m getting the error

ERROR i.a.b.Bootloader(assertNonBreakingMigration):139 -

when launching docker compose up on airbyte OS

Hi Airbyte Team, I use the sync mode to

incremental- Append

and i want to know where the last Connection STATE details are stored in the server. In the UI, it appears here. But i want to know from where its getting stored

Has any kind soul generated and published a Python client from the Airbyte OpenAPI spec here?

Is there a specific requirement for the internal Postgres version supported by Airbyte? I couldn’t find any. We’re using an external RDS on AWS and wish to upgrade it’s version. Where can I find that requirement set? Thanks!

Hi, I’m using airbyte with docker compose using this guide from docs Install and launch Airbyte

mkdir airbyte && cd airbyte
curl -sOO <https://raw.githubusercontent.com/airbytehq/airbyte-platform/main/{.env,flags.yml,docker-compose.yaml}>
docker compose up -d

How can I downgrade to v0.40.30 ?

Hello. I am using Airbyte Open Source using Google Compute Engine. I had set up TEMPORAL_HISTORY_RETENTION_IN_DAYS because it kept taking more and more of disk space. But since I had set it up it is taking more and more of Memory. Below is the picture of Memory Utilization (Percent of memory used by this VM, excluding disk caches). Does anyone know why? and What I should to make sure it doesn’t keep increasing?

Hi there, I am trying to set up a connection between Stripe and Google BigQuery and am getting an "Oops! Something went wrong... Unknown error occurred" a few seconds into fetching the schema. This only occurs with the Stripe source and on our Airbyte deployed on EC2. The same process works fine when run on my local machine

Hi! How do I stop airbyte from expanding nested arrays into new tables, while keeping basic normalization? I.e. for a stream

foo

with an array object

bar

I’d like to have the normalized table

foo

without creating a table

foo_bar

Hi, Just wondered if I'd get a bit more visibility on a thread I opened a while ago for setting up a connector for AirCall through AirByte Open Source on WSL. I am currently stuck on the following page (after having tested the connector is actually pulling information from my source) with the error as per screenshot in the thread.

Hey there, trying out Airbyte Cloud for the first time. I was wondering if there should be any difference in speed/performance between Airbyte Cloud and Airbyte OSS - specifically whether the OSS version would outperform Cloud?

I'm running into a regular error from using the Dagster integration where API requests to trigger a sync meeet with a 409 reponse. The error that I'm seeing in the logs from Airbyte shows

"io.airbyte.commons.server.errors.ValueConflictKnownException: A sync is already running for: <connection_id>"

. Is this something that folks have experience with or a solution for?

Hello, When will be the next Airbyte release?

Hello , First time airbyte user here. Im trying to run the airbyte open source version and connect it to a MongoDB source and the destination is a postgresql database. When I add a connection , it takes a long time to fetch the schema of the data source(MongoDB) and then errors out saying "Unable to fetch schema.non-json response received " . Any pointers here ?

Are the kubernetes related deployment files removed in most rcent master purposefully? I no longer see airbyte/kube/overlays/stable this is what we are relying on to deploy airbyte to kubernetes

Hellow! I am trying to migrate airbyte from 0.39.37 to 0.40.32 but the database migration is not working. Does someone faced the same issue? It seems like the new version has less tables and differente schemas.