are airbyte logs deleted after a certain point automatically? Or does it keep taking the disc space unless manually deleted?

How do I update an installation of Airbyte to include new updates? Specifically, I need this in my deployment (once it's fixed and pushed to the main branch). https://github.com/airbytehq/airbyte/pull/27390

I am getting the following error when I try to deploy Airbyte locally

./run-ab-platform.sh: line 66: curl: command not found

Could anyone help debug this issue.?? I am running deploying Airbyte on Ubuntu 22 machine

I am getting the following (new) error when I deploy Airbyte locally on ubuntu 22 machine.

Top-level object must be a mapping

Can someone help me debug it?

Is it possible to specify SSL communication in the DB required by Airbyte? If so, is it necessary to prepare server-side certificates?

Is there something wrong with Oauth in Airbyte? I am trying to create a low cdk connector for Zoho Books. When I use OAUTH as the authentication method, I get server error. But when I choose bearer as the authentication method, I get the data.

While I try to authenticate Zoho Books through Airbyte Oauth, I am getting the below error

access_token - Traceback (most recent call last):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 182, in _read_stream
    yield from AirbyteEntrypoint(source).read(source.spec(self.logger), config, configured_catalog, {})
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py", line 141, in read
    yield from self.source.read(self.logger, config, catalog, state)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 136, in read
    yield from super().read(logger, config, catalog, state)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 141, in read
    raise e
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 121, in read
    yield from self._read_stream(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 196, in _read_stream
    for record in record_iterator:
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/abstract_source.py", line 333, in _read_full_refresh
    for record_data_or_message in record_data_or_messages:
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 100, in read_records
    for record in self.retriever.read_records(sync_mode, cursor_field, stream_slice, stream_state):
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 398, in read_records
    for record in records_generator:
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 439, in _read_pages
    request, response = self._fetch_next_page(stream_slice, stream_state, next_page_token)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 453, in _fetch_next_page
    request = self._create_prepared_request(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py", line 280, in _create_prepared_request
    return self._session.prepare_request(requests.Request(**args))
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/requests/sessions.py", line 486, in prepare_request
    p.prepare(
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/requests/models.py", line 372, in prepare
    self.prepare_auth(auth, url)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/requests/models.py", line 603, in prepare_auth
    r = auth(self)
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 28, in __call__
    request.headers.update(self.get_auth_header())
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 33, in get_auth_header
    return {"Authorization": f"Bearer {self.get_access_token()}"}
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/requests_native_auth/abstract_oauth.py", line 38, in get_access_token
    token, expires_in = self.refresh_access_token()
  File "/root/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/sources/declarative/auth/oauth.py", line 116, in refresh_access_token
    return response_json[self.get_access_token_name()], response_json[self.get_expires_in_name()]
KeyError: 'access_token'

however, when I run the python code with the exact same token refresh endpoint, I get the necessary values. Why is it happening?

🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

Is there a way to specify what pods are attached to node groups in a kubernetes cluster? It looks like you have specify each pod, but I wasn’t sure if there was a global way to do that in the helm chart

Is there a way to use aws-efs for storage class instead of aws-ebs. I went through airbyte helm chart and could not figure this out. Can someone please help me with this? Many thanks for considering my request. 🙂

Is it possible to use dedicated database for meta data of airbyte? Or any way to free some memory, as my connector is failing

how can free buffer memory of airbyte?

I have Airbyte 0.50.3 deployed on Kubernetes. I did a big jump from 0.40.22 to the latest version. I had a few connections set up - some had normalization used, some didn’t. The currently deployed platform however has an empty Transformation tab as per screen below and all destinations that had normalization toggled are failing. Attaching example log - Airtable. dc7dd622_3d4a_4164_bbf2_eb4e35990748_logs_13858_txt (2).txt (942.7 KB) Please advise I’m starting to lose hope…

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!

When using the Octavia CLI to update the Airbyte instance with

octavia apply -f <file>

I notice that no authentication is required. Does this mean the Web UI username / password is only on the Web App side and that there's no API authentication? That would allow anybody to modify the connections on the Airbyte instance.

hey all, when upgrading airbyte, is there a way to maintain configuration data? Ie keep logs, connection configs etc. going from a version

0.40.0-alpha

using kustomize to using helm. For a different environment, we did a full delete and redeployed.

This message was deleted.

I'm seeing a lof of this in our Kubernetes-hosted Airbyte (from airbyte-worker):

2023-06-20 18:45:32 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$5):198 - Completing future exceptionally...
io.airbyte.workers.exception.WorkerException: Unexpected error while getting checking connection.
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:127) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:43) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$5(TemporalAttemptExecution.java:195) ~[io.airbyte-airbyte-workers-0.44.8.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.exception.WorkerException: Cannot invoke "io.fabric8.kubernetes.api.model.Pod.getMetadata()" because "this.podDefinition" is null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:113) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	... 3 more
Caused by: java.lang.NullPointerException: Cannot invoke "io.fabric8.kubernetes.api.model.Pod.getMetadata()" because "this.podDefinition" is null
	at io.airbyte.workers.process.KubePodProcess.close(KubePodProcess.java:783) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.process.KubePodProcess.cleanup(KubePodProcess.java:701) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.process.KubePodProcess.<init>(KubePodProcess.java:633) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:144) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:113) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:69) ~[io.airbyte-airbyte-commons-worker-0.44.8.jar:?]
	... 3 more

Any idea what's up?

getting error response from daemon: manifest for airbyte/connector-atelier-server:0.40.26 not found: manifest unknown: manifest unknown when trying to run ./run-ab-platform.sh

Hi Team, getting "consistent query buffer is full, cannot accept new consistent queries", could you please suggest how to free buffer?

I am trying to deploy airbyte using helm on EKS, getting this unknown error, can anyone help:

Error: UPGRADE FAILED: template: airbyte/charts/webapp/templates/secrets.yaml:14:60: executing "airbyte/charts/webapp/templates/secrets.yaml" at <.Values.global.secrets>: wrong type for value; expected map[string]interface {}; got interface {}

Hi, sorry if I’m missing something obvious as I am new to this: I am trying to launch airbyte locally on Mac. I installed docker as per instructions - docker compose is on docker desktop I believe - and then when launching get the following error.

docker compose v2 not found! please install docker compose!

What can I do to debug and solve?

we have airbyte running over openshift we need to basic authentication on top of it

:~/airbyte$ ./run-ab-platform.sh 

 █████╗ ██╗██████╗ ██████╗ ██╗   ██╗████████╗███████╗
██╔══██╗██║██╔══██╗██╔══██╗╚██╗ ██╔╝╚══██╔══╝██╔════╝
███████║██║██████╔╝██████╔╝ ╚████╔╝    ██║   █████╗  
██╔══██║██║██╔══██╗██╔══██╗  ╚██╔╝     ██║   ██╔══╝  
██║  ██║██║██║  ██║██████╔╝   ██║      ██║   ███████╗
╚═╝  ╚═╝╚═╝╚═╝  ╚═╝╚═════╝    ╚═╝      ╚═╝   ╚══════╝
                                            Move Data

docker compose v2 not found! please install docker compose!

:~/airbyte$ docker-compose -v
Docker Compose version v2.12.2

I am getting error while starting airbyte latest version

Hi there, Could someone please provide guidance on deploying Airbyte using the octavia-cli command, GKE, and CI/CD (GitHub Actions)? I have reviewed various threads but haven’t been able to find the answer I’m looking for. Currently, I’m attempting to achieve deployment using the following method, but I’m unsure how to launch the octavia-cli command. 1. Airbyte is deployed in a pod on GKE. 2. During deployment, I access the Airbyte pod on GKE from GitHub Actions. 3. --- Processing within the Airbyte pod on GKE --- 4. The Airbyte pod contains a git folder for the project I’ve been developing, so I execute a git pull to fetch the latest source code. 5. I execute the octavia apply command to update Airbyte. I’m unable to figure out how to place the octavia command in step 4. It seems that the provided install.sh file pulls the airbyte/octavia-cli:${VERSION} Docker image and creates an alias. Here is the GitHub repository where the install.sh file can be found: link

Airbyte allows us to use git repositories to setup dbt transformation. If the repo is private, then Airbyte expects us to paste the personal access token as part of the URL. However, GitHub recommends that we treat the token just like a password. Airbyte simply shows the token on the UI in plain text! I would like Airbyte to treat it like a password and hide the token on screen. Is there any alternative to avoid showing the password in plain text?

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves

Hi! We are running airbyte 0.41.0 in GKE and are having issues updating or adding sources. This is the error we get when trying to add a source:

ConnectorCard.module-f2c29f2c.js:20 Error: non-json response
    at qy (ConnectorCard.module-f2c29f2c.js:38:20944)

We have restarted the workloads already, but the error persists. Any pointers?

Hey Team, I am looking at deploying Airbyte using Rancher and connection to an external Postgres Database. Does anyone have experience with this? As I will be deploying it to run using Rancher v1.5.1 i think the best way is to dissect the docker-compose and deploy each image as separate services. I will also be looking at scaling up the workers as needed hence the reason for this setup. Does anyone have advice?

Hi, I need help with Requests and Limits for

destination

and

source

pods. I deploy Airbyte using Helm latest charts, and I see that my

destination

and

source

pods for some connections are requesting too little for their loads, even passing the limits. In the Helm charts, there are no specific config for

destination

and

source

. It seems that the equivalent to tweak these pods is the

jobs

resources. Yet, even when I change the request for

jobs

, I don't see these requests being reflected on destination and source pods.