Hi all, relatively new to Airbyte here. I’m currently ingesting a google sheet to Snowflake. We added a new column to the sheet and I believe that the connector will not update the schema during a regular sync--at least mine didn’t. I tried a reset as well, but it still doesn’t update the schema in Snowflake. I explored the UI in the connection as well as the source connector, and there doesn’t seem to be any way to update the schema. The only thing that worked was deleting the source connector entirely and recreating it. I saw during the initial setup that it’s “querying the schema” and so I believe that’s the only place/time that the schema can be changed. The problem here is that I can’t delete and recreate the source connector every time a column is added on the google sheet, particularly because I need to capture it in historical/append mode. It seems that Airbyte wants to reset the stream anytime I do anything, like changing sync modes, so I’m pretty sure I can’t just recreate the source connector and point it to previously gathered data. That’s my assumption at least. If I do that, I’ll lose the history. Either way, I’m hoping to confirm the following: The google sheet connector in fact does not allow for schema changes during regular syncs, without resetting data. Or am I overlooking something here? Thanks!

we have a thing happening where airbyte seems to remove permissions on a database schema every time it runs (so we have to run the grant access / grant select queries every time airbyte runs, which seems like it should be unnecessary) - does airbyte recreate the database schema every time it runs?

Are there any thoughts on best practice on Notification Webhook URL if we want to have multiple notification destinations? Should this be handled in the UI or should this be handled by the user setup (aws lambda, azure automation/function)?

Anyone ever used this to do more generic webhook notifications? Something like if I wanted to orchestrate Airbyte syncs with AWS step functions to kick off a lambda to signal the success of an async task in the step function.

does anyone know where I can find the implementation of the following method:

io.airbyte.persistence.job.models.IntegrationLauncherConfig.getIsCustomConnector()

because it doesn’t seems to be in the open source version I need to debug why it’s returning

null

Hey team, I've found a small issue for Shopify -> PostgreSQL. I've started pulling data from the orders and customers table and I've noticed there are some records that are not coming through (on my PostgreSQL it's literally missing the row of id's). I am validating by going on shopify itself. Has anyone else come across this issue where random amounts of records do not come through? I thought possibly the it is not pulling records with 'closed' status or something like that but that does not seem to be the case. Is there any other settings on airbyte that may filter the data getting ingested? Thanks! Using shopify custom API with all read access; shopify connector is 0.3.0 on Airbyte 0.40.23

If I successfully use the Python CDK to make a custom API connector, can I then get that custom source uploaded to my cloud.airbyte.io account?? is the Connector Builder UI available in Airbyte Cloud Accounts?? I am trying to connect to WIHP Hotel Meta-Search reporting API as a source, they have very clean API SWAGGER docs that provide code snippets in almost all programming languages. What is the cleanest and fastest way to get this as a source in Airbyte Cloud?

is the expectation that a destination is able to parse a date like this succesfully? or should the destination skip incorrectly formattted dates?

"2020-03-\n31T00:00:00Z\r"

https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/bases/standa[…]rations/standardtest/destination/DestinationAcceptanceTest.java

Hello, team. I pulled from airbyte hq for the first time in about 6 months. I have Airbyte running on a GCP Ubuntu instance. I am able to still build my custom connectors successfully with an example: sudo docker build . -t nmwdi/hydrovu:0.0.2 Now, whenever I try to build a connector that I did not create myself I get the following error:

sudo docker build . -t nmwdi/salesforce:0.0.2
Sending build context to Docker daemon 79.36kB
Step 1/11 : FROM python:3.9-slim
 ---> 86def1a51ade
Step 2/11 : RUN apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*
 ---> Running in 86d6f1200ec0
Get:1 <http://deb.debian.org/debian> bullseye InRelease [116 kB]
Get:2 <http://deb.debian.org/debian-security> bullseye-security InRelease [48.4 kB]
Get:3 <http://deb.debian.org/debian> bullseye-updates InRelease [44.1 kB]
Err:1 <http://deb.debian.org/debian> bullseye InRelease
 At least one invalid signature was encountered.
Err:2 <http://deb.debian.org/debian-security> bullseye-security InRelease
 At least one invalid signature was encountered.
Err:3 <http://deb.debian.org/debian> bullseye-updates InRelease
 At least one invalid signature was encountered.
Reading package lists...
W: GPG error: <http://deb.debian.org/debian> bullseye InRelease: At least one invalid signature was encountered.
E: The repository '<http://deb.debian.org/debian> bullseye InRelease' is not signed.
W: GPG error: <http://deb.debian.org/debian-security> bullseye-security InRelease: At least one invalid signature was encountered.
E: The repository '<http://deb.debian.org/debian-security> bullseye-security InRelease' is not signed.
W: GPG error: <http://deb.debian.org/debian> bullseye-updates InRelease: At least one invalid signature was encountered.
E: The repository '<http://deb.debian.org/debian> bullseye-updates InRelease' is not signed.
The command '/bin/sh -c apt-get update && apt-get install -y bash && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

Is anyone familiar with this?

Hello! Does anybody know if I can configure the number of job attempts on a connection basis? I'm running a job that syncs a large amount of data and, even though the first attempt manages to save some state and later fails, the second attempt will start from the previous state (same with third attempt). I think this makes sense for other connections but would like to explicitly set up the number of attempts for particular connections.

Hi people, doing a sync with source Postgresql and target Snowflake I'm getting this error: "Sync worker failed" and "airbyte-source gobbler IOException: Connection timed out. Typically happens when cancelling a job", always with the same duration time (4hours) and the same 74GB of data, anyone has happened or something to adjust? Thank you for your recommendations

I can see it in the documentation when used as a destination, but Is there any way to use a key/pair with Snowflake as a source too in Airbyte Cloud*?*

Hello all, i am trying to deploy airbyte in a managed kubernetes cluster via your provided helm chart. I am using an external Postgres and instead of minio i am using s3 directly. After the bootloader finishes, only the metrics and the cron pod start successfully. The other pods run into a CrashLoopBackOff. I think the root cause of the other pods not starting is, that the temporal pod terminates with this line in the log

unable to create open /etc/temporal/config/docker.yaml: permission denied

. Does anyone can help me out here? I am not able to run the pods privilaged, since it is not allowed in my cluster. In a local minikube environment everything started properly with my supplied values.yaml. Thank you in advance!

Hello team I'm trying to follow this guide to develop locally(I'm new to the project). I ran

SUB_BUILD=PLATFORM ./gradlew build

but I'm getting the following build failure

Task :airbyte-workers:test

ConnectionManagerWorkflowTest > Test that sync workflow failures are recorded > Test that normalization failure is recorded FAILED
    Wanted but not invoked:
    jobCreationAndStatusUpdateActivity.attemptFailureWithAttemptNumber(
        <Has failure from origin>
    );
    -> at io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$SyncWorkflowReplicationFailuresRecorded.testNormalizationFailure(ConnectionManagerWo$
kflowTest.java:1301)

    However, there were exactly 3 interactions with this mock:
    jobCreationAndStatusUpdateActivity.createNewAttemptNumber(
        JobCreationAndStatusUpdateActivity.AttemptCreationInput(jobId=111)
    );
    -> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    jobCreationAndStatusUpdateActivity.reportJobStart(
        JobCreationAndStatusUpdateActivity.ReportJobStartInput(jobId=111, connectionId=077f8f2b-4521-4fb6-b8c9-57a20b4bdc19)
    );
    -> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    jobCreationAndStatusUpdateActivity.isLastJobOrAttemptFailure(
        JobCreationAndStatusUpdateActivity.JobCheckFailureInput(jobId=1, attemptId=1, connectionId=077f8f2b-4521-4fb6-b8c9-57a20b4bdc19)
    );
    -> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at app//io.airbyte.workers.temporal.scheduling.ConnectionManagerWorkflowTest$SyncWorkflowReplicationFailuresRecorded.testNormalizationFailure(ConnectionMan$
gerWorkflowTest.java:1301)

144 tests completed, 1 failed, 1 skipped

> Task :airbyte-workers:test FAILED

It seems like there is a test in airbyte-workers that is failing. I haven't modified the code at all after cloning it.

Does anyone try to connect Redshift as source? I got below error: Redshift connector version - 0.3.51 Helm chart version- 0.43.2 server & worker latest one

2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - integration args: {check=null, config=source_config.json}
Log4j2Appender says: Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Running integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
Log4j2Appender says: Command: CHECK
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Command: CHECK
Log4j2Appender says: Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Integration config: IntegrationConfig{command=CHECK, configPath='source_config.json', catalogPath='null', statePath='null'}
Log4j2Appender says: Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-12-14 09:48:51 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
Log4j2Appender says: Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2022-12-14 09:48:51 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
Log4j2Appender says: The "standard" upload mode is not performant, and is not recommended for production. Please use the Amazon S3 upload mode if you are syncing a large amount of data.
2022-12-14 09:48:51 [33mWARN[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):117 - The "standard" upload mode is not performant, and is not recommended for production. Please use the Amazon S3 upload mode if you are syncing a large amount of data.
Log4j2Appender says: Using destination type: STANDARD
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Using destination type: STANDARD
Log4j2Appender says: HikariPool-1 - Starting...
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Starting...
Log4j2Appender says: HikariPool-1 - Start completed.
2022-12-14 09:48:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Start completed.
Collecting content into /tmp/toBePublished15349657165706088974.tmp before uploading.
Publishing to S3 (bucket=hk01prod-data-airbyte; key=job-logging/workspace/536cef4f-f26d-44af-b866-f57c69737c09/0/logs.log/20221214094946_project-airbyte-main-worker-f5b986846-4zgqm_a256005acc5d4e89908d10c88acaa665):
Log4j2Appender says: HikariPool-1 - Shutdown initiated...
2022-12-14 09:49:51 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Shutdown initiated...
Log4j2Appender says: HikariPool-1 - Shutdown completed.
2022-12-14 09:49:53 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - HikariPool-1 - Shutdown completed.
Log4j2Appender says: Completed integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:49:53 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - Completed integration: io.airbyte.integrations.destination.redshift.RedshiftDestination
Log4j2Appender says: completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:49:53 [32mINFO[m i.a.w.i.DefaultAirbyteStreamFactory(internalLog):120 - completed destination: class io.airbyte.integrations.destination.redshift.RedshiftDestination
2022-12-14 09:49:54 [32mINFO[m i.a.w.p.ExitCodeWatcher(persistExitCode):113 - Received exit code 0 for pod ion-redshift-check-536cef4f-f26d-44af-b866-f57c69737c09-0-jljhp
Log4j2Appender says: (pod: project-airbyte / ion-redshift-check-536cef4f-f26d-44af-b866-f57c69737c09-0-jljhp) - Closed all resources for pod
2022-12-14 09:49:54 [32mINFO[m i.a.w.p.KubePodProcess(close):745 - (pod: project-airbyte / ion-redshift-check-536cef4f-f26d-44af-b866-f57c69737c09-0-jljhp) - Closed all resources for pod
Log4j2Appender says: Stopping cancellation check scheduling...
2022-12-14 09:49:54 [32mINFO[m i.a.w.t.TemporalAttemptExecution(get):163 - Stopping cancellation check scheduling...
Log4j2Appender says:
2022-12-14 09:49:54 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -
Log4j2Appender says: ----- END CHECK -----
2022-12-14 09:49:54 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 - ----- END CHECK -----
Log4j2Appender says:
2022-12-14 09:49:54 [32mINFO[m i.a.c.i.LineGobbler(voidCall):114 -

Hi team. What is the process to get a helm chart version bump PR merged? I believe this one is quite important given the fact that the missing

TEMPORAL_HOST

variable prevents Airbyte from functioning properly. Here's the PR: https://github.com/airbytehq/airbyte/pull/20320

Hi Everyone, im wondering if someone can help. Im trying to pull data from Jira into a SQLite DB but i havent been successful in what seems to be a simple ingestion

can anyone help on this please?

Hi, I am trying to modify the hubspot connector to fetch a new table. But I am not able to send the data to the destination connector, i am receiving null at destination.

Hi Everyone I'm starting to work with Airbyte but I have some questions about data sources. Is there any way to define custom REST api data source. For example I have couple of custom REST apis from where I need to fetch data. On Airbyte sources I only see already defined services that can be defined as sources?

Hello I am trying to add source and sync but when I am doing i am getting zero records in db and empty list logs. I am adding logs to this message as well Thanks in advance.

Hello everyone, I want to setup an external database (RDS aurora postgreSQL cluster) for a vanilla docker compose stack. I start commenting the db service/volume and start to see than I need temporal and temporal_visibility database after digging errors logs, and now airbyte_metadata relation is missing .... It seems than the documentation isn't explicit enough on it (but maybe i'm wrong), and I want to setup the least privileges. Do someone can shared me some insights about what's really needed to doing this properly ? like a SQL script will be perfect 😄 Thanks !

Hi Team, I want Kubernetes ServiceAccountName for Check & Sync jobs. How can I propagate from worker to job pod, I have checked the documentation I didn't find anything. Thanks for help in advance.

Hey all, I've run out of disk space on my Airbyte installation (30GB). I've tried a few things, but hitting a brick wall on all of them:

docker images prune --all
docker volume prune
Total reclaimed space: 0B

docker run -it --rm --volume airbyte_workspace:/data busybox
docker: open /var/lib/docker/tmp/GetImageBlob873569938: no space left on device.

docker run -it --rm --volume airbyte_workspace:/data busybox ls /data/9/2
docker: open /var/lib/docker/tmp/GetImageBlob031057609: no space left on device.

Hi everyone, Does anyone have experience trying to move servers while still keeping all the data and connections? We have to move not only the server where the Docker is running but also the MSSQL server that we use as a destination. It would be great to not lose all the data we have already fetched. Are there any tips or guides for doing this?

Hi everyone, the authentication to my API is through a POST call with a secret token and the token I get back is the one I have to use in the following calls. How can I do that with low code connector?

Hey friends! So we’re trying to transition from Fivetran to Airbyte and from the technical side we were investigating Airbyte on ECS, which is not available yet due to containers having to create other containers. In relation to that my question is: do y’all use Docker-in-Docker (dind) or some other way of creating containers with other containers?

Hi Folks, Just a quick question - I see Airbyte is using 1.7.0 Temporal. Are there any plans to upgrade to any newer version? And if so, do we know which release is targetted? 🙏

Hey y'all, I've deployed Airbyte locally and I'm trying to read a file as a source, but I'm not sure how to get my file into the tmp/airbyte_local mount so Airbyte can see it... any thoughts?

Hey did anyone faced with

non-json response

error while establishing connection with Shopify?