This message was deleted.

Hi everyone, I'm currently setting up Airbyte for migrating data from MongoDB to BigQuery on an AWS EC2 instance. I followed the steps provided in the Airbyte documentation and was able to set it up successfully. However, when trying to configure the migration, I am running into issues. I have used the same configuration that worked fine on the Airbyte free trial, but I'm facing issues with the self-hosted version. Any help or suggestions would be greatly appreciated. Thank you!

Hi, I would be gratefule if someone with a general understanding of the code base help me with https://airbytehq.slack.com/archives/C021JANJ6TY/p1679821251281029 I would like to help fix a bug, but need a better understanding of what the

hashid

is used for.

This message was deleted.

Hello! I'm working with the latest version using the release version of helm charts. airbyte-db failed two times during a couple of months with the following error:

PostgreSQL Database directory appears to contain a database; Skipping initialization

2023-04-20 10:49:34.764 UTC [1] LOG: starting PostgreSQL 13.10 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit
2023-04-20 10:49:34.765 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2023-04-20 10:49:34.765 UTC [1] LOG: listening on IPv6 address "::", port 5432
2023-04-20 10:49:34.769 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-04-20 10:49:34.776 UTC [21] LOG: database system was interrupted while in recovery at 2023-04-20 10:49:32 UTC
2023-04-20 10:49:34.776 UTC [21] HINT: This probably means that some data is corrupted and you will have to use the last backup for recovery.
2023-04-20 10:49:34.990 UTC [21] LOG: database system was not properly shut down; automatic recovery in progress
2023-04-20 10:49:34.994 UTC [21] LOG: redo starts at 5/F1A6B8D0
2023-04-20 10:49:35.232 UTC [21] LOG: redo done at 5/F3FFFFA0
2023-04-20 10:49:35.249 UTC [21] PANIC: could not write to file "pg_wal/xlogtemp.21": No space left on device
2023-04-20 10:49:35.385 UTC [22] FATAL: the database system is starting up
2023-04-20 10:49:35.394 UTC [1] LOG: startup process (PID 21) was terminated by signal 6: Aborted
2023-04-20 10:49:35.394 UTC [1] LOG: aborting startup due to startup process failure
2023-04-20 10:49:35.400 UTC [1] LOG: database system is shut down

My questions: • I don't think this is supposed to happen, so how to prevent this? • Are there solutions to recover other than wiping the database?

I am trying to create a custom report with GA4 integration in airybyte, it is giving me error, Here is my custom report parameters ,

[{"name": "traffic_source_breakdown", "dimensions": ["sessionMedium","googleAdsAccountName","campaignId","date","adUnitName","sessionManualAdContent"], "metrics": ["conversions","totalPurchasers","totalUsers","totalRevenue","advertiserAdCost","advertiserAdCostPerConversion","totalRevenue","returnOnAdSpend"]}]

error i am getting is can

HTTPError('400 Client Error: Bad Request for url: <https://analyticsdata.googleapis.com/v1beta/properties/321362882:runReport>')

someone please help!!

The next office hour is going to start in 10 minutes the Zoom Meeting to enter and talk about Airbyte issues and features.

Hey all - we are moving our airbyte instance to a new EC2. We have a lot of

Incremental | Append

syncs set up in it, but I know that the "latest CreatedOn" is stored locally on the EC2 somewhere. I want to avoid running a full-refresh on all of our connections just to move to a new EC2. Is there a way I can copy that data for the "latest CreatedOn" for each sync I have set up? It would save me so much time and headache if this is possible. I asked ChatGPT, and it's telling me there are state files somewhere, but I'm having a hard time locating them on EC2 airbyte instance.

Hey Everyone, I am trying to use the Amazon Seller Partner Source Connection. It seems like other people have been able to get this alpha source to work but I can't get past the test. This is the error I get and the log: Configuration check failed HTTPError('400 Client Error: Bad Request for url: https://sellingpartnerapi-na.amazon.com/reports/2021-06-30/reports') 2023-04-20 135712 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):142 - Docker volume job log path: /tmp/workspace/0d14cfdf-47ba-42f5-a49a-304220505c43/0/logs.log 2023-04-20 135712 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):147 - Executing worker wrapper. Airbyte version: 0.43.1 2023-04-20 135712 [32mINFO[m i.a.a.c.AirbyteApiClient(retryWithJitter):179 - Attempt 0 to save workflow id for cancellation 2023-04-20 135712 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0' 2023-04-20 135712 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - 2023-04-20 135712 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - ----- START CHECK ----- 2023-04-20 135712 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - 2023-04-20 135712 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0' 2023-04-20 135712 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1' 2023-04-20 135712 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1' 2023-04-20 135712 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable LAUNCHDARKLY_KEY: '' 2023-04-20 135712 [32mINFO[m i.a.c.EnvConfigs(getEnvOrDefault):1222 - Using default value for environment variable FEATURE_FLAG_CLIENT: '' 2023-04-20 135712 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - Checking if airbyte/source-amazon-seller-partner:1.0.1 exists... 2023-04-20 135712 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - airbyte/source-amazon-seller-partner:1.0.1 was found locally. 2023-04-20 135712 [32mINFO[m i.a.w.p.DockerProcessFactory(create):133 - Creating docker container = source-amazon-seller-partner-check-0d14cfdf-47ba-42f5-a49a-304220505c43-0-ldmbc with resources io.airbyte.config.ResourceRequirements@5ee1334d[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=] and allowedHosts null 2023-04-20 135712 [32mINFO[m i.a.w.p.DockerProcessFactory(create):184 - Preparing command: docker run --rm --init -i -w /data/0d14cfdf-47ba-42f5-a49a-304220505c43/0 --log-driver none --name source-amazon-seller-partner-check-0d14cfdf-47ba-42f5-a49a-304220505c43-0-ldmbc --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-amazon-seller-partner:1.0.1 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e USE_STREAM_CAPABLE_STATE=true -e FIELD_SELECTION_WORKSPACES= -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e FEATURE_FLAG_CLIENT= -e AIRBYTE_VERSION=0.43.1 -e WORKER_JOB_ID=0d14cfdf-47ba-42f5-a49a-304220505c43 airbyte/source-amazon-seller-partner:1.0.1 check --config source_config.json 2023-04-20 135712 [32mINFO[m i.a.w.i.VersionedAirbyteStreamFactory(create):145 - Reading messages from protocol version 0.2.0 2023-04-20 135713 [1;31mERROR[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):140 - Check failed 2023-04-20 135713 [32mINFO[m i.a.w.g.DefaultCheckConnectionWorker(run):115 - Check connection job received output: io.airbyte.config.StandardCheckConnectionOutput@54a0360a[status=failed,message=HTTPError('400 Client Error: Bad Request for url: https://sellingpartnerapi-na.amazon.com/reports/2021-06-30/reports')] 2023-04-20 135713 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):169 - Stopping cancellation check scheduling... 2023-04-20 135713 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - 2023-04-20 135713 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 - ----- END CHECK ----- 2023-04-20 135713 [32mINFO[m i.a.c.i.LineGobbler(voidCall):149 -

Hey Everyone; I’m replicating data into Snowflake from Salesforce. But, there’s a problem with the column names that are

CamelCase

as those are not getting quoted when they are normalized into Snowflake, resulting in

SHIPPINGSTATE

instead of

ShippingState

Is this a bug to report? Is there a simple way to tweak the normalization?

when does a custom DBT apply transformations to incoming data? Is it after it loads into the _airbyte table or before loading at all into the destination?

This message was deleted.

i'm trying to get started with airbye, and have it running on a eck cluster, source is mysql via an ssh proxy, destination is postgres. Its been workingfor small tables, but i'm now trying to replicate a 5m row table, around 4GB with incremental + dedupe. Its appears to load all the data, buth then fails and restarts. In the logs i see

The last packet successfully received from the server was 37 milliseconds ago. The last packet sent successfully to the server was 566,688 milliseconds ago.,externalMessage=Something went wrong in the connector. See the logs for more details.,metadata=io.airbyte.config.Metadata@3db364ff[additionalProperties={attemptNumber=0, jobId=5, from_trace_message=true, connector_command=read}],stacktrace=java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

not sure if this is the root cause or not (edited)

question - I have several tables that are synced incrementally every hour, but also fully refreshed (overwrite) every week. The thing is the _scd tables seem to keep growing. Is there a safe way to trim the _scd tables while keep the same sync mode running? Or, can I delete the _scd tables right after overwrite?

Question - Attempting to follow the instructions on the github release, and it appears there is a lack of environmental variables that are missing from the .env file that are expected in the docker-compose.yml, Where would I be able to find examples of correct values for a general install of the platform?

Question, I inadvertently deleted the volumes when bringing down Docker Compose, lost the workspace/connections. One specific connection from Salesforce to Snowflake brings a lot of data. If we recreate the connection, point to the same destination DB, will Airbyte perform a full resynch or an incremental based on the last synch time?

my Intercom-to-Snowflake sync has ballooned from taking a handful of minutes to 8-9 hours per sync, despite no increase in the amount of data synced • i verified that all of these syncs are using

airbyte/source-intercom:0.2.0

and

airbyte/destination-snowflake:0.4.61

• this is an incremental | deduped sync of just one table (conversation_parts). i'm using the default cursor field and primary key any idea what might be causing this or how to bring it back down to normal?

Quick question about CDC configs, I set up the following connection: Source: MSSQL (AWS RDS) Destination: Snowflake I chose the CDC replication option, and I've set everything up right in the MSSQL source for that. All the tables I am replicating are set to "incremental | deduped history". The flow runs fine and the output in Snowflake is correct. My question is specifically about the "Existing and New" vs "New Changes Only". I configured the connection with "Existing and New" since I needed the initial snapshot, but I was wondering if I am then supposed to switch it to "New Changes Only" after the first run. Basically I am wondering whether the "Existing and New" setting makes it so that each subsequent run is still doing a SELECT * from each table, rather than just reading from the CDC logs.

I am trying to setup Google BigQuery as a destination. But get the following error message:

Configuration check failed

Access Denied: Project my-project-name-384201: User does not have bigquery.datasets.create permission in project made-with-ml-384201.

In BigQuery console, my-project-name-384201 is shown in Explorer as the BigQuery project. Any thoughts? Thanks! Tony

Greetings all, quick question. Does anyone know where the source is for the woocommerce plugin? I have a module (woocommerce software licensing addin) that I'd like to also use, however, there is no plugin for it. I wouldn't mind writing one and submitting it. Thanks in advance.

Hello, I was looking for the Zendesk Sell connector but it seems to be down in the platform. Its present in the source_definitions.yaml though. Is there any place I can look up to check connector health? Thanks in advance

Is normalisation for databricks in the roadmap?

Hi, I need to increase the

max_execution_time

in my MySQL source since the Pipeline is timing out. There seems to be a bug that does not allow me to configure it in the jdbc parameter. Could you help me work around this for now?

{
  "identifiers": [
    {
      "channels": {
        "email": {
          "status": "subscribed"
        }
      },
      "id": "<mailto:milindupasan@gmail.com|milindupasan@gmail.com>",
      "type": "email",
      "sendWelcomeMessage": false
    }
  ],
  "firstName": "test",
  "country": "Sri Lanka",
  "lastName": "Pasan",
  "countryCode": "Sr",
  "state": "Western",
  "city": "Kalutara",
  "address": "National College,Kalutara",
  "postalCode": "ABCD"
}

Hi I want to pass this array in request body of the airbyte YAML . Is it possible to do that? There is no option in UI (attached image) So i edited YAML File like this. But it does not work.

version: 0.28.0
type: DeclarativeSource
check:
  type: CheckStream
  stream_names:
    - Contact
streams:
  - type: DeclarativeStream
    name: Contact
    primary_key: []
    schema_loader:
      type: InlineSchemaLoader
      schema: {}
    retriever:
      type: SimpleRetriever
      requester:
        type: HttpRequester
        url_base: <https://api.omnisend.com/v3/>
        path: /contacts
        http_method: POST
        request_parameters: {}
        request_headers:
          X-API-KEY: '{{ config[''api_key''] }}'
        request_body_json:
          firstName: "test"
          country: "Sri Lanka"
          lastName: "Pasan"
          countryCode: "Sr"
          state: "Western"
          city: "Kalutara"
          address: " National College,Kalutara"
          postalCode: "ABCD"
          identifiers:
            - channels:
                email:
                  status: subscribed
              id: <mailto:milindupasan@gmail.com|milindupasan@gmail.com>
              type: email
              sendWelcomeMessage: false
        authenticator:
          type: ApiKeyAuthenticator
          api_token: '{{ config[''api_key''] }}'
          header: '{{ config[''input''] }}'
      record_selector:
        type: RecordSelector
        extractor:
          type: DpathExtractor
          field_path: []
      paginator:
        type: NoPagination
spec:
  connection_specification:
    $schema: <http://json-schema.org/draft-07/schema#>
    type: object
    required:
      - api_key
    properties:
      input:
        title: Input
        type: array
      api_key:
        type: string
        title: API Key
        airbyte_secret: true
    additionalProperties: true
  documentation_url: <https://example.org>
  type: Spec

I created a PR for the issue I mentioned above: https://github.com/airbytehq/airbyte/pull/25386 It should be a very straight forward fix. I would appreciate a review! My Java skills are a bit rusty ;)

Hello everyone ! We are on airbyte 0.42.0, source connector postgres : 2.0.10, destination connector Bigquery : 1.2.16. From one day to another (no change in connector or VM), we switched to a perfectly running job to a 'running' job pulling no record, any idea of what has happened? No explicit error on connection or anything else, we are able to connect "manually" to the source with the same user and we do see some data. Hint : it looks like the i.a.d.j.s.AdaptiveStreamingQueryConfig(accept) step does not appear anymore after i.a.d.j.s.AdaptiveStreamingQueryConfig(initialize) while it was the case and everything was runnning 'fine'. Addition following questions in the thread : full refresh overwrite, standard replication. Many thanks for your hints 🕵️‍♀️ 🕵️‍♂️

Hello everyone ! I am trying to create connection using google sheet my sheet column have some of special character so i need to remove all from every connection if special character are exist, so how can i do that i need to change any setting from Airbyte or need to update code which i have setup in my local machine ? Thank you in advance

Hi all! I'm using helm to deploy Airbyte in a private EKS cluster, and I intend to clone the required images to a private repository such as ECR. Is it possible to modify the image repository, which includes source and destination connector images, from https://hub.docker.com/ to ECR? I came across an article on Using custom connectors, but I am unsure if there is a simpler approach to achieving my goal. 🙏

Hey Team! Right now, I have some table names in

snowflake

that have hypens

-

, and when I am piping

dynamodb

, they are being converted to underscores

_

. I can't change the table names in

dynamodb

to have underscores rather than hypens, so right now I am unable to insert data into them because of this issue (instead, a new table is being created with underscores). Any thoughts on how to resolve this?

This message was deleted.