@kapa.ai With OSS deployment by Docker Composer on GCP VM, do I need to expose Airbyte IP to the public for using Terraform to configure Airbyte objects?

Hi! new to Airbyte, product looks very good! Is there any kubernetes operator that allows to manage connectors using kubernetes manifests?

Hello! I have an issue when creating my sources using Terraform. I got a similar issue in both

airbyte_source_postgres

and

airbyte_source_zendesk_support

with provider in v0.3.4. I have some attributes that are ignored and sent as

null

to the API, resulting in a fail during the terraform apply. Here are my two sources:

resource "airbyte_source_zendesk_support" "******_zendesk_support" {
  name         = "Zendesk Support"
  workspace_id = var.workspace_id

  configuration = {
    credentials = {
      source_zendesk_support_authentication_api_token = {
        email       = "******"
        api_token   = var.zendesk_api_token
        credentials = "api_token"
      }
    }
    ignore_pagination = false
    source_type       = "zendesk-support"
    start_date        = "2023-05-01T00:00:00Z"
    subdomain         = "******"
  }
}

resource "airbyte_source_postgres" "******_postgres" {
  workspace_id = var.workspace_id
  name         = "CloudSQL Production"

  configuration = {
    host        = var.database_hostname
    port        = 5432
    database    = "******"
    username    = var.database_username
    password    = var.database_password
    source_type = "postgres"

    replication_method = {
      source_postgres_update_method_read_changes_using_write_ahead_log_cdc = {
        method = "CDC"
        publication = "******"
        replication_slot = "******"
      }
    }

    schemas = [******]
  }
}

When running the terraform apply, the

airbyte_source_zendesk_support.configuration.credentials

and

airbyte_source_postgres.configuration.replication_method

are set as null. Do you have any idea why this happens to these specific blocks. They both seem to respect the schema. Thank you by advance! Louis

Hi, i am running Airbyte on Kubernetes and wanted to know where i can find api docs. I want to create a microservice that communicates with the Airbyte server to create

source|destination|connection

with code. The official docs are pointing to

<http://api.airbyte.com|api.airbyte.com>

: https://reference.airbyte.com/reference/createsource.

Hello, I am having problem with the slack notifications. I currently using Airbyte OSS v0.50.7. from UI when I test a Slack webhook It says

Webhook test failed. Please verify that the webhook URL is valid

When I hit the notification API using the following params:

curl --location --request POST '<http://localhost:80002/api/v1/notifications/try>' \
--header 'Authorization: Basic ' \
--header 'Content-Type: application/json' \
--data-raw '{
    "notificationType": "slack",
    "sendOnSuccess": true,
    "sendOnFailure": true,
    "slackConfiguration": {
        "webhook":    "<https://hooks.slack.com/services/><EXTRA_CHARS>"
    },
    "customerioConfiguration": {}
}'

Getting an empty response with 200 status? I see server logs and I see nothing there. Is there a way to fix this.

Hello I am using Airbyte OSS. I have installed it in my local using docker. I am able to execute list connection, list destination, list source apis. But when I try to create source using following details. http://localhost:8006/v1/sources

Authorization:Basic xxxxxxxxxxxxxxx
accept:application/json
content-type:application/json

{
  "configuration": {
    "sourceType": "mysql",
    "port": 3306,
    "ssl_mode": {
      "mode": "preferred"
    },
    "replication_method": {
      "method": "STANDARD"
    },
    "tunnel_method": {
      "tunnel_method": "NO_TUNNEL"
    },
    "host": "xx.xx.xx.xx",
    "database": "sourcedb",
    "username": "root",
    "password": "xxxxx"
  },
  "name": "servermysql",
  "workspaceId": "544cc0ed-7f05-4949-9e60-2a814f90c035"
}

I always get following response

{
  "type": "<https://reference.airbyte.com/reference/errors>",
  "title": "unexpected-problem",
  "status": 500,
  "detail": "An unexpected problem has occurred. If this is an error that needs to be addressed, please submit a pull request or github issue."
}

Can anyone suggest in this issue ?

Hello, I am using

AirbyteTriggerSyncOperator

to trigger sync of my airbyte jobs using airfow. I want to trigger these tasks inside a dag conditionally. I.e. if a condition is false then I want to skip this task which triggers a sync job and move on with the rest of the DAG How do I do it?

Hello I'm trying to move my Airbyte config to the terraform provider, but I can't manage to find where we define the normalisation process for the connection Did someone manage to make it work ?

Hey again, when deploying a big Salesforce <> Redshift connection, terraform fail with a timeout error. Is there a way to configure this ?

context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Hello, Has anyone tried to create custom connectors using Terraform without clicking manually on the UI?

Hello, I am trying to use the Airbyte API in a script to to create 4 different connections between 4 postgres sources and the same bigquery destination. Once I was able to successfully create 4 connections. But I usually get one of these errors:

data: 'stream timeout'

or

data: 'upstream request timeout'

I believe the create connection api call is failing. AirByte OSS version:

0.50.30

Postgres Source version:

3.1.8

BigQuery Destination version:

2.0.16

Any ideas?

Hello, how do you specify only the list of tables you need to sync in airbyte dagster integration using "airbyte_sync_op.configured"

Hello I am working on a product where i want the functionality like the registered user can connect to their business profiles(facebook, snapchat) to get their analytics and can be stored in to the database so i need to know does airbyte provide any api which we can use as a third party api in our website to make it work, please do reply

Hello, I am trying to figure out how to config the type of jobs trigger (Reset, normal sync ...) for AirbyteTriggerSyncOperator on Airflow, is there anyway that we could config that from Airflow side? Thank you!

Hello, I have just realised airbyte automatically creates column data type with maximum size on the destination table, is this something that can be configured to specify my own column size?

yes | octavia apply

can I use this command to delete the existing resource on Airbyte UI? if I delete the respective config file in airbyte-octavia directory under connections, I want this command to delete the existing resource on Airbyte UI

Hey all, does the terraform provider support column selection when creating connections? I don't see it in the docs, but is it perhaps undocumented and/or will it be supported in the future?

Hi, is there an open api description for the airbyte api?

Hello, team I have checked the API documentation, and it is having APIs where I can get the details of source, connection, jobs, etc.. However, does AirByte provide any way to explore or view the data? I want to use the source connectors to get the data from a source and I want to perform operations on that data before loading it on the another destination? Is that possible on the Airbyte?

I'm using AWS session manager to access a ec2 instance on which i have deployed airbyte... can anyone help me access the airbyte ui? I can't ssh into my ec2....

@kapa.ai Is there anyway that we could create a custom connector with Docker Image using Terraform? Assuming we already have necessary image available in the current docker environment

I'm trying to access the airbyte api docs but am getting a 404 error...is this the wrong link? https://api.airbyte.com/

@kapa.ai I'm using Octavia and I would like to know if there is a way to get allowed values for resources in the .yml file without running octavia generate <resource_type>. The purpose is to create a yml configuration checker that will map the fields filled in by a user and compare to the allowed values and this according to the current version. I want to validate yml configuration files before deploying in CD

I have Airbyte set up to pipe data from HubSpot to Redshift yesterday. I decided to leave it running a sync overnight since there are 500k contact objects in HubSpot...but Airbyte only loaded like 50k in, and it took the whole night. Trying to figure out where I went wrong? • Infra: Kubernetes (EKS) • HubSpot API Tier: 150 calls per 10 seconds -- 500,000 calls daily (which we haven't even come close to) • Normalization setting:

Raw data (JSON)

Has anyone else experienced this? When we were using Stitch, it would load all 500k contacts in about 2 hours.

This message was deleted.

Hi everyone, I've just configured Airbyte open source and I'm trying to understand how the dbt integration works. Airbyte Cloud seems to integrate with DBT Cloud, how does the open source version work?

https://airbytehq.slack.com/archives/C02RRUG9CP5/p1695156764115469 Hi Guys, I am stuck at this issue, any help would be appreciated.

Everyone from Airbyte Team was removed from this channel because @[DEPRECATED] Marcos Marx archived the channel.

Everyone from Setel was removed from this channel because @[DEPRECATED] Marcos Marx archived the channel.

archived the channel