Help me pls.Thank u so much

Anyone experienced this error? How could I fix it?

I have data source .txt format, How do i process data and import to SQL Server

Hi, so, we really want to run airbyte on ECS (and if possible, fargate). we're willing to consider some makeshift hacking to make this work. is there any way to perhaps override the container launch commands used by the orchestrator to run a script that will start ECS tasks, or something like that?

Hi Airbyte team, in the DefaultSyncJobFactory.create() method there are references to connector-specific resource requirements, and in EnvConfigs.java there is a reference to an environment variable that theoretically can enable such a thing. However, I am unable to find anything in the connector development documentation that shows how to configure those. I would love to learn how to use this feature. Is there documentation anywhere that could help me?

Hi team, I encountered a normalization error when I tried to replica a MySQL table,

dbt found two resources with the name "tablename_ab1". Since these resources have the same name,
  dbt will be unable to find the correct resource when ref("tablename_ab1") is used. To fix this,
  change the name of one of these resources:
  - model.airbyte_utils.tablename_ab1 (models/generated/airbyte_ctes/database1/tablename_ab1.sql)
  - model.airbyte_utils.tablename_ab1 (models/generated/airbyte_ctes/database2/tablename_ab1.sql)

I know why this error happens, because I have 2 different tables, they are in 2 different source databases, but with the same tablename, so Airbyte use the same model name, which confuses dbt. so how can I solve this kind of error? please help me, thanks!

Hi. I building a python source and the auth method is oauth2 client credentials method. Oauth2Authenticator is what I need and would love to know where can I get docs about it and what it accepts and how it works. I tried looking in the source file i.e. airbyte_cdk\sources\streams\http\auth but it seems to say deprecated all over. I did find the new classes in requests_native_auth though still not sure which class to use for what I need Thank you.

Hello while trying to add source Im getting TIMEOUT ERROR

2022-09-08 08:57:50 INFO i.a.w.t.TemporalAttemptExecution(get):107 - Cloud storage job log path: /workspace/57c296ea-f665-4f5c-8c5d-e45b49252bb4/0/logs.log
2022-09-08 08:57:50 INFO i.a.w.t.TemporalAttemptExecution(get):110 - Executing worker wrapper. Airbyte version: 0.40.0-alpha
2022-09-08 08:57:51 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = rce-bigquery-check-57c296ea-f665-4f5c-8c5d-e45b49252bb4-0-qqxxq for airbyte/source-bigquery:0.2.0
2022-09-08 08:57:51 INFO i.a.w.p.KubeProcessFactory(create):103 - rce-bigquery-check-57c296ea-f665-4f5c-8c5d-e45b49252bb4-0-qqxxq stdoutLocalPort = 9024
2022-09-08 08:57:51 INFO i.a.w.p.KubeProcessFactory(create):106 - rce-bigquery-check-57c296ea-f665-4f5c-8c5d-e45b49252bb4-0-qqxxq stderrLocalPort = 9025
2022-09-08 08:57:51 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$9):587 - Creating stdout socket server...
2022-09-08 08:57:51 INFO i.a.w.p.KubePodProcess(lambda$setupStdOutAndStdErrListeners$10):605 - Creating stderr socket server...
2022-09-08 08:57:51 INFO i.a.w.p.KubePodProcess(<init>):518 - Creating pod rce-bigquery-check-57c296ea-f665-4f5c-8c5d-e45b49252bb4-0-qqxxq...
2022-09-08 08:57:52 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):308 - Waiting for init container to be ready before copying files...

│ Caused by: io.temporal.failure.TimeoutFailure: message='activity timeout', timeoutType=TIMEOUT_TYPE_SCHEDULE_TO_CLOSE                                                                                                                             │
│ 2022-09-08 09:34:40 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):100 - Unexpected error while checking connection:

On minikube with local cluster work properly but on gke Im getting TIMEOUT

helm install airbyte airbyte/airbyte

Hi everyone, does the Google Analytics v4 still require a View ID? I updated to airbyte version 0.40.3 which I thought had that fix but still seeing View ID as a required field.

Hi everyone, I am new to airbyte and dbt and i have created a connection moving data from mysql to bigquery (incremental + deduped), how can i use dbt to remove some of the columns on the tables and edit the data types. Any tutorial or explanation would be helpful\

Hello. I am currently trying to upgrade to the latest airbyte release. I have been using airbyte to sync from postgres to s3. Postgres source still works. But when I try to setup my bucket destination, it gives me 403 error, even though I know I am providing the correct creds. Airbyte logs mentions that the bucket does not exist, but it does. However, the bucket name has a digit in the beginning, in this case

7b-lakehouse-dev

. Is there a chance that in the latest s3 destination it does not like the

digit

as the first letter in the bucket name?….

Hello I want to connect airbyte with Google analytics but I get this error, is there a solution for this ,? Thankyou

The connection tests failed.

'Please check the permissions for the requested view_id: 271470929. Cannot retrieve data from that view ID.'

How to dockerize a custom connector for deployment into Airbyte

Hi Everyone, Can anyone share a script for deleting old logs from a workspace? The server storage disk is filled up

As a question; before a post a more through forum post; does the google sheets connector auto-refresh auth when run self-hosted?

Hi there folks!

TL;DR; Can I run a single-stream sync for any Postgres to Postgres connection?

Is there any way of syncing a SINGLE stream from a connection? I found no API calls for this here. Use case: • We have a pretty big hourly sync (postgres to postgres) • Sometimes we want to update only a subset of tables from this sync • Running the pretty big sync takes time and we want to do this faster. I thought of creating another connection with the same SRC <-> DST but I'm afraid this will mess incremental cursors.

Hi all, airbyte sync takes 38 seconds on my local dev machine and 2m 33 seconds in GCP, for the exact same connection params. What we want to know is which parameters to optimize. How do we find out if its memory bound, CPU bound or Postgres write limited. Any idea on how to improve performance in GCP?

I have published my custom connector's private docker image as part of ghcr.io (Github Container Registry). Airbyte is unable to fetch the docker image, do I need to provide the personal token for my user for Airbyte to authenticate Configured the following in source_definitions.yaml file. - name: MyConnector sourceDefinitionId: fd169cfe-16e4-42d5-887e-f440c4a43d79 dockerRepository: ghcr.io/mycompany/myconnector dockerImageTag: 0.0.1 Exception: Error while getting spec from image ghcr.io/mycompany/myconnector:0.0.1', type='java.util.concurrent.ExecutionException', nonRetryable=false

Hello. I am currently trying to upgrade to the latest airbyte release. I have been using airbyte to sync from postgres to s3. Postgres source still works. But when I try to setup my bucket destination, it gives me 403 error, even though I know I am providing the correct creds. Airbyte logs mentions that the bucket does not exist, but it does. However, the bucket name has a digit in the beginning, in this case

7b-lakehouse-dev

. Is there a chance that in the latest s3 destination it does not like the

digit

as the first letter in the bucket name?….

Why is it necessary to have

CREATE DATABASE

permissions for the airbyte <-> clickhouse user? It will not be creating any databases

Just created an Airbyte Cloud account and tried to setup JIRA as a source and it’s not showing on the list of available sources. What gives?

docker build of custom connector fails while trying to include opencv-python as part of Dockerfile. Any idea what could be going wrong with the docker build

Collecting numpy>=1.17.3

Downloading numpy-1.23.3.tar.gz (10.7 MB)

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 51.6 MB/s eta 0:00:00

Installing build dependencies: started

Installing build dependencies: finished with status 'done'

Getting requirements to build wheel: started

Getting requirements to build wheel: finished with status 'done'

Preparing metadata (pyproject.toml): started

Preparing metadata (pyproject.toml): finished with status 'done'

Building wheels for collected packages: opencv-contrib-python, numpy

Building wheel for opencv-contrib-python (pyproject.toml): started

Building wheel for opencv-contrib-python (pyproject.toml): still running...

Building wheel for opencv-contrib-python (pyproject.toml): still running...

Building wheel for opencv-contrib-python (pyproject.toml): still running...

Building wheel for opencv-contrib-python (pyproject.toml): still running...

Building wheel for opencv-contrib-python (pyproject.toml): still running...

Building wheel for opencv-contrib-python (pyproject.toml): finished with status 'error'

error: subprocess-exited-with-error

× Building wheel for opencv-contrib-python (pyproject.toml) did not run successfully.

│ exit code: 1

╰─> [917 lines of output]

Not searching for unused variables given on the command line.

CMake Error: CMake was unable to find a build program corresponding to "Ninja". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.

-- Configuring incomplete, errors occurred!

See also "/tmp/pip-install-y_ady_xo/opencv-contrib-python_aacd61cbb47847d481cda42de305140b/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log".

Not searching for unused variables given on the command line.

-- The C compiler identification is GNU 10.3.1

I just created a new connection (Google Analytics Data API <> Redshift and I keep getting this error and not really sure why. I was using the same GA account settings for my old connection and it worked fine. Anyone have an idea what might be causing this?

"error": {
    "code": 429,
    "message": "Exhausted property tokens for a project per hour. These quota tokens will return in under an hour. To learn more, see <https://developers.google.com/analytics/devguides/reporting/data/v1/quotas>",
    "status": "RESOURCE_EXHAUSTED"

Would someone be so kind to point me in the direction of how I could override the dockerRepository settings within source_definitions.yaml within a running Airbyte install? I cannot for the life of me figure out where this file ends up in a running instance. I even looked inside the postgres tables but could not find where these settings are stored. BTW, context is that I have to use a private registry for these with different paths.

is there any way to get google calendar data from airbyte?

Hello, I am getting the following error once I pull the repository and execute docker-compose, any advice?

airbyte-server      | 2022-09-13 13:36:32 ERROR i.a.s.ServerApp(main):320 - Server failed
airbyte-server      | io.airbyte.db.check.DatabaseCheckException: Timeout while waiting for database to fulfill minimum flyway migration version..
airbyte-server      | 	at io.airbyte.db.check.DatabaseMigrationCheck.check(DatabaseMigrationCheck.java:51) ~[io.airbyte.airbyte-db-db-lib-0.40.5.jar:?]
airbyte-server      | 	at io.airbyte.server.ServerApp.assertDatabasesReady(ServerApp.java:159) ~[io.airbyte-airbyte-server-0.40.5.jar:?]
airbyte-server      | 	at io.airbyte.server.ServerApp.getServer(ServerApp.java:180) ~[io.airbyte-airbyte-server-0.40.5.jar:?]
airbyte-server      | 	at io.airbyte.server.ServerApp.main(ServerApp.java:317) [io.airbyte-airbyte-server-0.40.5.jar:?]

Hello, I set up trial of Airbyte to test databricks destination but it nowhere to be found 😕 Is it not available on trial?

Hey Guys, I was able to run airbyte. but how to keep it running on Linux (ec2 instance). whenever I close the window it stops.

Hey all. I'm having a strange scenario where a postgres source is not showing views in the DB when I'm using CDC. When I switch the source to use replication method of standard, the views show up.

Suggestions on best approaches to troubleshoot hanging airbyte instances? Have one instance running alone on a google VM with 3 jobs (1 postgres db, 2 API-based syncs). While the jobs can complete sync via one-off manual execution, the entire instance seems to get stuck every few days and requires bouncing docker to get it going again.