Hi, i was wondering if anyone could help me with how to alter your nginx config? nginx is not something I have come across before but I have read that adding a line to it for the parameter

client_max_body_size

can fix the 'client tried to send too large body' error that I am getting. I am deploying airbyte locally on my Mac by cloning the github repo and running on docker desktop. I found some files in airbyte/airbyte-proxy/ called nginx-auth.conf.template and nginx-no-auth.conf.template and tried adding the change there (see picture), but it didn't seem to help. Am I missing something? Where else should I be going to make this change?

Hello everyone, I'm working with the Airbyte API and I need to access a source, destination or connection by name. So I'm using the search method. For example to get a source by its name I'm using

*POST* /v1/sources/search

with the following body

{"name": "mySourceName"}

. However It doesn't seems to work. Adding the workspaceId doesn't change anything. Is it an issue ? Or I'm missing something ?

Has anyone deployed Airbyte on kubernetes (better if through Helm) and put it behind traefik before? Would you mind sharing the config. I’m pretty new on deploying things so would very appreciate some help! 😊

Hey team -- question on your Shopify connector. Can someone confirm this can be used to pull historical data stored in Shopify as well? I would assume any data stored in the tables you specify could be moved via the connector?

Has anyone figured out how to set up datadog monitoring with the helm deploy? I'm trying my best to follow what I'm seeing here, but I'm getting way lost haha.

hello everyone! when instantiating an airbyte record message is it mandatory to specify the data as a dictionary as

{"key": "value"}

or is it possible to pass multiple records in a single message using a list of dictionaries

[{"key": "value1"}, {"key": "value2"}]

? It is taking a lot of time to iterate over every single record in the dataset when to yield it as a record message, if I could have a list of 100000 records in a single message it would take a lot less time since the data I am trying to collect has millions of records

Hi all, Bing Ads source: I receive the keywordid in the bingads_keyword_performance_report_daily. BUT I can´t find in any report the keyword itself. Also, We´ve set the msclkid, but I can´t see it in any report. Does someone has encountered the same problem?

We have a lot of custom python ingestion code (that runs in airflow python operators) and does things like “connect to this ftp server and download a subset of the files, using a list of desired filenames / file types / whatever criteria” - we’re interesting in switching to airbyte as much as possible and are wondering: has anyone figured out a way to customize data ingestion in airbyte. to fit this sort of finicky stuff?

Hello! I am trying to re-install airbyte (open source) on my mac and create source again. I keep on getting this error when I try to do pip install -r requirement.txt. I can see from my PyCharm interpretor that airbyte-cdk is not present. However I am unable to figure out, how did this installation worked before and now it's not. I have followed same step before to implement http-api-example source from the and it worked just fine. But for some reason I am unable to get it working again. Any help?

hi guys, need help please. one of the pipelines i am running, all of the sudden it stopped creating new sync jobs and showing an error "Uncaught (in promise) Error: A sync is already running for". i tried restarting the docker stack, and restarted the server, but still the same. Reset data also same behavior. Any tips please on how to cancel it? p.s: there is no job running in the frontend and in the API (/api/v1/jobs/list) all jobs are success. :/

Hi Team, If I am creating a custom connector similar as s3 destination with some changes in this new custom connector, can I access base-java-s3 in the new custom connector? is it achievable?

Hi everyone! I´ve encountered problems in my Google Ads connection. I get less records in the campaigns table and only 200 records in display_keyword_performance_report table (with segments.date) of 2021.. Any ideas?

Hi every one I have facing mysql connection lost issue during the sync below are the details: mysql version =5.7 airbyte installed using docker on EC2 EC2 config: 1. os = Linux 2. Ram = 16G 3. cpu = 4

Hello guys. Anyone had this problem with Recurly source? It keeps stuck at this point and takes too long (1h25min) to pass from this step to the normalization step. I'm trying to get invoices and transactions.

[Github connector] Hi team. Is there a reason why there is no support for incremental replication for the

pull_request_commits

stream? Is it a technical limitation with GH's api?

How would I go about logging during execution? I sometimes get stuck for a looong time on

preparing tables at destination completed

and have no way of knowing wat happened

Is there a documentation page that explains design difference in the connector versions for cloud and open-source? We’ve tried Google Ads on open-source, it worked. Trying to switch it to the cloud version, and the connector for cloud has significant differences.

Hello! Here is a question related to using a cloud managed DB (AWS RDS Postgres) to host the Airbyte Jobs & Config databases. I’m currently am using the Airbyte managed versions of those DBs, but would like to start using a cloud hosted Postgres DB. Is there a way to migrate the Airbyte hosted Jobs Database/ Config Database to the cloud managed database without losing any existing Logs/ Config data?

I have some errors after installing the newest version of Airbyte:

Caused by: io.temporal.failure.ApplicationFailure: message='SQL [select * from "public"."connection" where "public"."connection"."id" = cast(? as uuid)]; Error while reading field: "public"."connection"."unsupported_protocol_version", at JDBC index: 22', type='org.jooq.exception.DataAccessException', nonRetryable=false

and

Caused by: org.postgresql.util.PSQLException: The column index is out of range: 22, number of columns: 21.

what are the minimum services that airbyte would be functional with? Could I turnoff webapp and cron (I am running all jobs manually)?

hello guys, how do you implement connectors for large datasets? I am having a hard time iterating through all of the records of a dataset with more than a few million records

Hey friends, wondering if there's anyone out there I could pick their brain about whats worked for them when hosting Airbyte on Google Cloud Compute Engine. I've had to pivot my deployment strategy pretty quickly, and lack the months of experience with the deployment on compute engine that I've had on Kubernetes. What I'm interested in: • What instance sizes per volume do you see working well? • How do you deploy new versions of aribyte against the compute instance? • Have you manged to templatize an instance with a 'latest startup script? • Have you managed to keep a persistent disk separate from the instance successfully? How did it go?

Hey All! I’d like to know more about the replication using CDC, we are testing it here and it seems that I can only dump the files in s3 using the JSON format. Am I doing something wrong? This is the message I got when airbyte exports the data as parquet file:

Airbyte K8s/Helm deployer here - I bumped the Helm Chart to

0.41.0

and attempted to deploy, but was getting a

CrashLoopBackOff

on the Webapp, with the below error msg.

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/11/18 19:35:23 [emerg] 1#1: host not found in upstream "$CONNECTOR_BUILDER_API_HOST" in /etc/nginx/conf.d/default.conf:6
nginx: [emerg] host not found in upstream "$CONNECTOR_BUILDER_API_HOST" in /etc/nginx/conf.d/default.conf:6

Is this your first time deploying Airbyte?: No OS Version / Instance: linux Memory / Disk: 4GB / 30 GB (per node) - up to 5 nodes Deployment: Kubernetes (Stable with resource limits) Airbyte Version: 0.40.18 Source name/version: Stripe (Latest) (Connection Test Successful) Destination name/version: Big Query (Latest) (Connection Test Successful) Step: During First Sync up. I've tried this many times. all the time Source connection check completes with success status. Destination check does not complete and start showing errors like in the log file below. (this log file does not contains source and destination checks but the error is same). Please help me here.

Hi guys! I have been exploring Airbyte to extract some data from HubSpot. I was able to extract all the data and insert it into a Postgres database. I just need some fields from the

deals

stream like

properties.dealname

,

properties.job_description

, and

properties.role_description

. The problem is that Airbyte gets ALL the fields from the

deals

stream. Is there a way to only get the fields that I need? Some of the fields are confidential and it is a problem to send them to a Postgres database (I have been working with a Sandbox Hubspot account for now).

Hi guys! I am tryng to connect a local hosted psql to another also localhosted airbyte on windows machine. I am about to get started to build a connector we need at my company, so this is the first step. But I can't make them connect. I have already launched the container for airbyte and have accessed https://localhost:8000 and all is fine, but when building the first source (a psql 14.1 DB) using airbyte's UI it won't connect. Even though if I connect via cdm

psql -h localhost -U postgres

it works fine after asking the password. I am attaching the image bellow with how I am setting up the connection at airbyte's UI and also my localhost psql credentials, I am putting something wrong?

Hey! Thanks for your time in advance tbh. So I have built a 'python Http source' connector locally following all the 4 steps spec check discover read and finally [docker build] . Now I'm not much of an expert in Docker but I would like if its possible to have my connector in my local Docker setup, as a docker image, to be shown in Airbyte UI portal in my browser as a new connector.

hi! are there any examples/tutorials that I can see to understand how to use httpstream class for nested api calls? First call is POST and next two would be GET. The response from first POST call would be used to make the url for second call (GET) and the response from second GET call will be used to make URL for third GET call and that call generates the response data which will be the stream that I need to get from my api. i.e. the schema for the data that I need will be for only third call's response.

Hello! When could we expect working Airbyte cloud API? Thanks!