Hi team, I cloned the latest Airbyte version which is 0.29.21-alpha which is also correctly reflected in the .env file as well as in kustomization.yaml but in charts/airbyte/Chart.yaml app Version is set to 0.29.13-alpha as shown in the attached screenshot, is this correct? Also values.yaml are showing the same old version

Hi team, I’m trying to debug connectors. I’m following documentation and trying to stream data from a source to destination using docker. I’m stuck and can’t understand why the destination can’t consume messages from pipe. https://docs.airbyte.io/connector-development/tutorials/building-a-python-destination https://docs.airbyte.io/connector-development/tutorials/building-a-java-destination I run:

docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-file:0.2.6 read --config /secrets/config_csv.json --catalog /sample_files/configured_catalog_csv.json > messages.json

cat messages.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:0.2.8 write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json

or just

docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/source-file:0.2.6 read --config /secrets/config_csv.json --catalog /sample_files/configured_catalog_csv.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:0.2.8 write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json

But I don’t see that

destination-csv

is consuming AirbyteMessages

cat messages.json | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/sample_files:/sample_files airbyte/destination-csv:dev write --config /secrets/config_d_csv.json --catalog /sample_files/configured_catalog.json
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):96 - {} - Running integration: io.airbyte.integrations.destination.csv.CsvDestination
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationCliParser(parseOptions):135 - {} - integration args: {catalog=/sample_files/configured_catalog.json, write=null, config=/secrets/config_d_csv.json}
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):100 - {} - Command: WRITE
2021-09-21 13:29:01 INFO i.a.i.b.IntegrationRunner(run):101 - {} - Integration config: IntegrationConfig{command=WRITE, configPath='/secrets/config_d_csv.json', catalogPath='/sample_files/configured_catalog.json', statePath='null'}
2021-09-21 13:29:01 WARN c.n.s.JsonMetaSchema(newValidator):338 - {} - Unknown keyword examples - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(<init>):157 - {} - initializing consumer.
2021-09-21 13:29:02 INFO i.a.i.b.FailureTrackingAirbyteMessageConsumer(close):80 - {} - Airbyte message consumer: succeeded.
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):198 - {} - finalizing consumer.
2021-09-21 13:29:02 INFO i.a.i.d.c.CsvDestination$CsvConsumer(close):214 - {} - File output: /local/tmp/out/_airbyte_raw_dmesh.csv
2021-09-21 13:29:02 INFO i.a.i.b.IntegrationRunner(run):153 - {} - Completed integration: io.airbyte.integrations.destination.csv.CsvDestination

Could you please give me a hint of what am I doing wrong? Is there some problem with config.json?

Hi all, I am exploring the Airbyte codebase. I am using IntelliJ, but unable to find a few packages like

AirbyteMessage

(image for reference, line 34). It is not available at

io.airbyte.protocol.models

What am I missing here?

Hi Team, I have a question regarding schema generated by Airbyte connector. Source : Github Destination : AWS S3 When I pull data from Github to S3, Airbyte is generating a schema which I am not able to directly use in Cube.js Table ( for github commits ) Generated By AWS Crawler from S3 data : Fields : 1. _airbyte_ab_id 2. _airbyte_emitted_at 3. _airbyte_data Everything related to commits goes into _airbyte_data column. Can I just make simple table of columns coming from source?

Good morning. Just a quick question: where could we find the statements for Table create for postgres-destination if it's possible? We want to modify ad-hoc the implementation of postgres-destination in order to refer the workspace_id. Thank You

Hello peoples! Right now, I’m looking into building out an open-source project. With that, I wanted to know if there are community maintainers for Airbyte in addition to the Airbyte organization maintainers for the repo? We are just looking to understand how open-source projects work at various organizations like Airbyte and dbt.

I have an idea that might be a bit out there. We'd really love to leverage Airbyte cloud, but we have some specifications about where our data can go both in transit and at rest and are currently implementing with the helm chart on k8s in our VPC. Most of that movement of data seems to be done by some

socat

pods that get spun up in the k8s cluster. I wonder if there could be a strategy that is could hosted for metadata, logs, RBAC, SSO/SAML, etc but an agent is deployed locally that establish a TCP connection with an Airbyte endpoint. The agent would then spin up the pods for moving data the same as it does today and collect logs and other metadata and ship it to Airbyte.

Hi team, who can guide me to run this project in IntelliJ ??

Hey y’all, are there any plans to cut a release this week?

Does anybody remember how such spec looked like in UI? Or do we have something like that?

workTime: {
                      type: "array",
                      title: "Work time",
                      items: {
                        type: "string",
                        enum: ["day", "night"],
                      },
                    },

Edit: Ok, looks like we do not have such specs. Figured out how it should look like.

Working on adding the new worker pods to the helm chart after the split from the scheduler to address https://github.com/airbytehq/airbyte/issues/6416. Do they have a http healthcheck like the server does with

/api/v1/health

? I'm seeing that they are listening on port 9000 for heartbeating, but I'm not sure if that is valid to use for a liveness or readiness probe

Dear Team, I have a question on retry policy. I have tried to do the following in a docker-compose dev run mode on laptop: • create source - file (https), 68K lines • attached destination (local file) • started sync • waited untill ~40% is downloaded (besed on log output

Records read: 29000

• then just disabled WiFi • job remains in

Running state

- leaving it for couple of minutes to simulate network issue • then I've re-enabled WiFi • Nothing changed, job is still in the same

Running

state, distination file is not updated since then. Are there any retry policies for such network/data availability issues? For me it looks like there is couple of issues: • I am not notified on sync error • there is no job continuation - seems that I need manually reset connection?

do we need to keep:

airbyte-queue

?

Hi! I just started a new connector and ran the code generator for an API connector. I didnt get far before I got an error I cant get past alone.

equests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: <https://provetcloud.com/{clinic_ID}/api/0.1/consultation/>

TIL: pressing

.

in github repo will open this repo in web version of VS-code. Pressing it in PR - will open this PR in web version on VS-code

Hello I am working new tap for fresh-work CRM api, to integrate it with Airbyte what all step i have to follow.

Does anyone have a cheat sheet for Version Control best practices for Airbyte connector development? For developing multiple new connectors in parallel (with teammates), should I just fork

airbyte/airbyte

from git, and then make a new branch for each connector? There is a lot written in the docs about connector dev, but i’m not finding anything specific for managing git (and the section on Monorepo Python Development + Gradle is a bit intimidating for airbyte novices looking to experiment with the platform). If someone from the dev team can confirm or correct the below, I am happy to record my process and contribute to the docs I understand that new connectors get containerized. If I am new to Airbyte, looking to quickly get started with a custom connector, and don’t intend to contribute to main in the immediate future…am I right to think that - 1. my fork with custom connector(s) never needs to be deployed to production and should only exist locally 2. my production airbyte instance should be deployed directly from

airbyte/airbyte

3. locally-developed custom connectors should be containerized and added to prod via the UI a. can private container images be added via the UI? I have only been able to add public

Hi Is there any plan to handle schema level changes in incremental - append mode?

The current behavior of Incremental is not able to handle source schema changes yet, for example, when a column is added, renamed or deleted from an existing table etc. It is recommended to trigger a Full refresh - Overwrite to correctly replicate the data to the destination with the new schema changes.

Ref: https://docs.airbyte.io/understanding-airbyte/connections/incremental-append What is the workaround that you propose to identify changes in schemas ? Do we need to handle schema change detection logic out side of airbyte and then trigger full refresh?

hi, good morning. We are a small company using the airbyte integrations to provide our customers with some insights. We have recently generated our python code with the openapi spec, but the connection get endpoint is complaining about having the wrong data type. the namespace field is defined as a string and the server is returning None. I have been looking for an issue on this but haven't found any, do you want me to open an issue about this? Thanks

Hello. I am interested in running the generateApiClient task in airbyte-api/build.gradle so I can have a java client to use in our application. When I do so, it builds the entire project and it cannot succeed since I am not running with sudo (I would prefer not to) and it can't access my .docker/config.json file when applying the 'airbyte-docker' plugin. I am wondering why I need to build the 'airbyte-docker' plugin simply to get the generated api client. Is there another way to build what I need, or should I be running as root? Thanks in advance.

Hi! Im trying to create a docker image for my simple api connector following instructions from this page. I get the following error message when building the image:

9 38.65   running build_ext
#9 38.65   building 'multidict._multidict' extension
#9 38.65   creating build/temp.linux-x86_64-3.7
#9 38.65   creating build/temp.linux-x86_64-3.7/multidict
#9 38.65   gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/usr/local/include/python3.7m -c multidict/_multidict.c -o build/temp.linux-x86_64-3.7/multidict/_multidict.o -O2 -std=c99 -Wall -Wsign-compare -Wconversion -fno-strict-aliasing -pedantic
#9 38.65   error: command 'gcc' failed with exit status 1
#9 38.65   ----------------------------------------
#9 38.66   ERROR: Failed building wheel for multidict
#9 38.66 Successfully built source-provet pyrsistent PyYAML wrapt pendulum
#9 38.66 Failed to build yarl multidict
#9 38.66 ERROR: Could not build wheels for yarl, multidict which use PEP 517 and cannot be installed directly
------
executor failed running [/bin/sh -c pip install --prefix=/install .]: exit code: 1

I am running on a mac which points python by default to 2.7. If I run python I use python3 and pip3. Can that be the problem here?

Hello everyone, I am working on singer tap-freshworks crm, while i am runing docker build command

pip is looking at multiple versions

because of this i am unable complete the docker build, i tried docker build for existing singer tap

quickbook

for this also i am facing same issue.

Hello, when running in dev mode, what is the correct procedure if I want to recover (backup) my locally stored data using docker? Thanks!

Hi! I have made a simple source connector which seems to work and now I would like to test it in the UI. I have run

docker build . -t airbyte/source-my-connector:dev

but now what? I have tried adding my connector in the source_definitions.yaml and restarted all local airbyte docker containers, but it does not appear among the sources in the UI. Any tips?

Hi, I want to test my connector's change on airbyte instance, Ive build dev image and tried to change version to "dev" but it says "The docker image cannot be found despite" I have it on my images list. Is it a bug? I remember I was doing same in the past and it worked

Hey Guys, I was wondering if it would be possible to prevent logging of each record sent by a custom source I am developing? • The records might have sensitive data which should not be propagated into log files • There might be millions of records causing a memory load on the browser when using the UI

Hi team, I need to have multiple connections with the same source. ex: multiple accounts in Shopify but one database, when using like this there is no way to identify which source data in the database, I'm trying to add an additional parameter with MySQL insert query. I but haven't found a way to get source details in the destination application. any helps ??

Hey guys, getting the following error with the

Intercom

source: (Maybe an issue with pagination?)

Hey, is it known issue that UI doesn’t support json schemas with $ref? @Artem Astapenko

fyi we had a ~100X cost saving on bigquery by writing a custom DBT that only denormalizes and transfers newer data to bigquery. to achieve the best result I needed to cluster raw table based on _airbyte_emitted_at column (select * where _airbyte_emitted_at>last-time basically bills for entire table size when non-clustered). my code here. Unfortunately we cannot ship it as is because it is not backward compatible but we can probably add that as an option into current bigquery destination or as its own destination.