Hi, I'm looking at airbyte in order to load some data from a third party bigquery warehouse and transform it for our use since I am unable to do this inside of the bigquery instance. Is it possible to set up a connection so that it only extracts data that is later than a specified key, for example only extracting data from 2021-01-01, but ongoing as new data gets added?

Hello, Please tell me if there is a connector for transferring data from Google BigQuery to Clickhouse?

Hey All,

Hey All, is it possible to use Airbyte with Google Cloud Compose?

AIrbyte seems to be taking the db name and prepending it to the table names. It also seems to be making a duplicate table with that naming convention.. Its a connection between postgres and snowflake. Anyone encounter this before?

I tried the getting started guide with Docker, but Airbyte server doesn't want to start, as it gets stuck on waiting for a namespace in temporal

I get:

airbyte-server | 2022-03-20 17:08:22 WARN i.a.w.t.TemporalUtils(getTemporalClientWhenConnected):194 - Ignoring exception while trying to request Temporal namespaces:
airbyte-server | io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline exceeded after 4.998910674s.

Temporal doesn't show any errors

I tried it with

0.35.58-alpha

as well as with the lastest .60

What am I doing wrong?

Any reason why airbyte isn't on the terraform registry?

has renamed the channel from "getting-started" to "airbyte-for-beginners"

Hello Community, We are in the initial phases of evaluating Airbyte OSS for our ELT needs. User attribution and security is a primary need for us. We use Okta IDP and found following resources on security documentation page - https://developer.okta.com/blog/2018/08/28/nginx-auth-request However, we will be running airbyte in our private network and would like to focus more on user attribution to actions (new connector addition, deletion, executions etc.) Is there any way to propagate user information propagated from okta to airbyte workspaces? Alternatively, can we use multiple workspaces to keep a track of user actions?

Hello Everyone, Can we use log based replication in near real time(mysql to mysql) in Airbyte? As of now i can see the minimum batch time required is 5 minutes. I want to do something similar to Streamsets replication in real time

Hello, I am a beginner and unable to connect a file located in a Personal OneDrive ( shared and available to anyone with a link ) as my data source. I have added a source - One drive sheet that was successful, but fails to load the schema whenever I try create a connection. Your guidance will be greatly appreciated.

Hello! Please tell me if the Arbyte service can help in our task: it is necessary to transfer data from Google Bigquery to Clickhouse. If yes, how long (a few days or a few hours) can it take to migrate a 1.8 Tbyte table with 1000000000 (1 billion) rows?

Hello I’m working as a data engineer in Finance domain and new to Airbye, he to explore more on airbyte.

If I use

File

as destination on the local

docker-compose

airbyte environment for testing. Where will this file be written to? I suppose to one of the docker container runtimes? Or will it write to my local system?

Hello! I created incremental connection to S3. It creates new file each time it is runned. How can I get name of created file?

Hi All, how can we dbt with airbyte, dbt is transformation but with airbyte we are just extracting and loading the data. what is the benefit and for what purpose dbt is used with airbyte

Hi ! I'm new to Aribyte. I have a Postgres CDC -> S3 connection running (its a lerge database). On thing that is not clear to me is the difference between all Sync Modes. The docs (also this) says that

Incremental - Append Sync

and

Full Refresh Sync

are supported for S3, but when I set up the connection, I can only chose

Full refresh | APPEND

and

Full Refresh | Overwrite

or I get the

The form is invalid. Please make sure that all fields are correct.

error. check the screenshot ! What am I missing ? Thanks !

Hey guys, Question, one of my sources is a vendor hosted Oracle database that only provides us read access to views, not tables. Is there any way to handle incremental in this situation?

What causes some tables to have Incremental Append as an option, and others not? Is this expected ? The two prints are from the same connection. (Postgres CDC -> S3 Parquet connection) If this is expected, I will need to create two connections for the same database. On with and other without CDC. In the ones that do not support Incremental | Append with CDC, I'll chose a cursor field.

@here : Airbyte version :_0.35.61-alpha ... have question currently if airbyte has source connector which supports custom queries as source (this custom query is defined at airbyte level but not at source)._

Hi Team, I’m setting up Airbyte locally on my mac with k8 deployment. While setting up the source connection with MSSQL getting below error, is there someone can guide me on this. Log4j2Appender says: kubectl cp /tmp/cd621464-26b9-4286-9a96-221189d04df7/source_config.json default/source-mssql-sync-d1329155-996c-4ec8-87ce-d589b6a90cea-0-rqgdy:/config/source_config.json -c init 2022-03-30 045601 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):158 - Completing future exceptionally... io.airbyte.workers.WorkerException: Error while getting checking connection. at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:84) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at java.lang.Thread.run(Thread.java:833) [?:?] Caused by: io.airbyte.workers.WorkerException: java.io.IOException: Cannot run program “kubectl”: error=0, Failed to exec spawn helper: pid: 137, exit value: 1 at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:150) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:58) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] ... 3 more Caused by: java.lang.RuntimeException: java.io.IOException: Cannot run program “kubectl”: error=0, Failed to exec spawn helper: pid: 137, exit value: 1 at io.airbyte.workers.process.KubePodProcess.copyFilesToKubeConfigVolume(KubePodProcess.java:284) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:514) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:146) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:58) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.35.61-alpha.jar:?] ... 3 more

hi there ! I am trying to apply this tutorial and I noticed that with version 0.35.59-alpha, the command to extract normalization files uses dbt=1.0.0

docker run --rm -i -v airbyte_workspace:/data -w /data/$NORMALIZE_WORKSPACE/normalize --network host --entrypoint /usr/local/bin/dbt airbyte/normalization debug --profiles-dir=. --project-dir=. 
09:55:47  Running with dbt=1.0.0

But when running the transformation in Airbyte will fail with

2022-03-30 10:00:19 INFO i.a.c.i.LineGobbler(voidCall):82 - fishtownanalytics/dbt:0.19.1 was found locally.
2022-03-30 10:00:19 INFO i.a.w.p.DockerProcessFactory(create):106 - Creating docker job ID: 17
2022-03-30 10:00:19 INFO i.a.w.p.DockerProcessFactory(create):158 - Preparing command: docker run --rm --init -i -w /data/17/0/transform --log-driver none --network host -v airbyte_workspace:/data -v /tmp/airbyte_local:/local -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE= -e AIRBYTE_VERSION=0.35.59-alpha --entrypoint /bin/bash fishtownanalytics/dbt:0.19.1 entrypoint.sh run
2022-03-30 10:00:19 dbt > Running from /data/17/0/transform/git_repo
2022-03-30 10:00:19 dbt > detected no config file for ssh, assuming ssh is off.
2022-03-30 10:00:19 dbt > Running: dbt run --profiles-dir=/data/17/0/transform --project-dir=/data/17/0/transform/git_repo
2022-03-30 10:00:21 dbt > Running with dbt=0.19.1
2022-03-30 10:00:21 dbt > Encountered an error while reading the project:
2022-03-30 10:00:21 dbt >   ERROR: Runtime Error
2022-03-30 10:00:21 dbt >   at path []: Additional properties are not allowed ('dispatch', 'model-paths', 'packages-install-path', 'seed-paths' were unexpected)

Is there a way to set the dbt version in

airbyte/normalization

or is there a stable version to use ? Thanks !

Hi there! I'm using

<http://logging.info|logging.info>

in my airbyte code (request_params function) but my logs don't show up when running in the airbyte environment, what am I doing wrong?

Hi team, i had an issue with dbt in snowflake and with the create and replace statement it is not possible to restore all data with marvelous snowflake function time travel, any possibility to change create or replace by creat or replace if not exist??

are there any docs on parent/child streams? dont see any

Hello team. I am new to Airbyte. I am trying the generate source connector tutorial on Windows with Docker installed:

airbyte/airbyte-integrations/connector-templates/generator

I ran ./generate and received this error message:

While trying to generate a connector, an error occurred on line 38 of generate.sh and the process aborted early.  This is probably a bug.

That line in generate.sh is:

docker run --rm -it --name airbyte-connector-bootstrap --user "$_UID:$_GID" -e HOME=/tmp -v "$(pwd)/../../../.":/airbyte airbyte/connector-bootstrap

Does anyone know what could be the problem?