Hi there we have airbyte running in k8s via the helm install route, and we been getting the available replica issue more frequently now. Off course it heals by itself just curious what others are doing to get around this issue

Hello, I'm evaluating the use of airbyte and in the comparison of versions, I understood that the SSH tunnels functionality only exists in the paid version. I would like to explore this point further, in the case of the free function does it mean that the connection between airbyte and my database will not be done through an encrypted tunnel?

🔥 Community Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 1pm PDT click here to join us on Zoom!

hi we are running airbyte oss, with multiple connectors ( >250) and the snowflake cost is getting pretty expensive to sync. we are syncing shopify to snowflake using Append Dedup. I see a lot of cloud services usage. We use s3 stage, XS-Small warehouse 1min TTL any recommended ways for us to reduce this cost? Does anyone knows what in particular causing the big cloud services consumption? + I ran into this roadmap plan: https://github.com/airbytehq/airbyte/issues/25194 and I believe it will have a big impact on costs for snowflake. does anyone know when is this expected to be live?

Hi ,

I am looking for help in migrating DB from docker to external PG RDS instance .

Hi Everyone, Is there any sort of alerts that we could setup on Airbyte in case one of our pipeline fail?

Hi everyone! I’ve got a small question regarding the documentation around running Airbyte Open Source on Kubernetes. As stated here the recommended way to configure Airbyte on K8s is going via Kustomize overlays to setup env variables. But at the same time the docs says that Kustomize is deprecated since version 0.40. All I really want to do is figure out how to enable basic authentication on a Helm deployment. Can I do this by setting the BASIC_AUTH_*-variables as noted here in my

values.yaml

?

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom!

Hey! I’m having trouble trying to setup Airbyte to use a CloudSQL as an external database, and am wondering about your thoughts about the security aspect of using the default database directly in Kubernetes. If the credentials for destinations and connections is saved to this database it would be by far the most sensitive database in my organization and should also ideally be the most secure. My fear is that a sneaky pod from a different namespace could fetch the secrets and env vars for the DB connection, connect to and get credentials stored in the database and worst case use those to fetch our data from third-parties where we can’t enforce much restrictions on API key usage and such. A tight network policy for the namespace might be in order. What measures are taken in the Airbyte OSS installation to make sure that only airbyte can access the database? What other best practices should I do if I decide to go for the default DB in Kubernetes?

Hi. I'm having trouble deploying Airbyte OSS to my computer. I've installed Docker, and now am trying to clone the Airbyte repo from Github in PowerShell. However, this download keeps getting stuck at 53%. I'd appreciate any advice anyone can give 🙂

Attempted to upgrade to 0.50.1 from 0.44.4 (through helm chart) but seems like the migration from the bootloader is giving issues - It’s complaining about missing columns. Anyone facing the same issue while upgrading?

Caused by: org.jooq.exception.DataAccessException: SQL [alter table "schema_management" drop auto_propagation_status]; ERROR: column "auto_propagation_status" of relation "schema_management" does not exist

How can I see created and deleted destinations in airbyte?

Hi, I want to send sync notifications to OpsGenie. Looks like currently only Slack webhooks are supported. Is there any open PR for OpsGenie integration?

hey all, we are having trouble configuring

<http://external-dns.alpha.kubernetes.io/hostname|external-dns.alpha.kubernetes.io/hostname>

in our ingress file for the new airbyte version. Does anyone know what ingress networking changes might be the cause from Kustomize to helm? We were previously using kustomize. We have successfully deployed it but cannot access it unless we forward the pod to our local machine. This has to do with the airbyte-webapp to my understanding. Any ideas? cc @Marco Hemken

Good morning everybody. I'm in a new type of occurrence with Airbyte, all connections from a given source being disabled by themselves. (this has happened 3x in the last week in different sources, before that it hasn't happened even once) Has anyone experienced this?

Hello! I have had airbyte deployed for a couple months with EKS, but this morning I tried to switch to use S3 for external logging. When deployed I am getting the following error from the airbyte-server pod

2023-06-14 13:52:01 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: null
java.lang.IllegalArgumentException: null
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:131) ~[guava-31.1-jre.jar:?]
	at io.airbyte.config.storage.DefaultS3ClientFactory.validate(DefaultS3ClientFactory.java:32) ~[io.airbyte.airbyte-config-config-models-0.50.1.jar:?]

What does the AUTO_DETECT_SCHEMA environment variable do?

I have Airbyte running on GCP, successfully. But it's just randomly started going very slowly - the logo spinning for a while and then "Oops something went wrong, Unknown error occured". Any idea on how I diagnose what's going on in the instance? There's no connections at the moment, I was just in the process of setting one up.

🔥 Office Hours starts in 15 minutes 🔥 Topic and schedule posted in #C045VK5AF54 octavia loves At 16:00 CEST / 10am EDT click here to join us on Zoom octavia loves

@Sunny Hashmi (Airbyte) I’m asking about node affinity for db and minio…

Hi guys - I have octavia to manage all my sources/dstinations and connections. On thing I'm struggling with is after running

octavia apply

for a connection, it updates, stores the new state. But if I immediately re-run apply for the same connection it says the state needs to be updated and re-runs. I have just updated the state, I can see the state locally being updated. So it should say "no change needed". Any ideas on how I deal with this one?

Hello everyone. I’m trying to upgrade Airbyte (deployed on GKE) from v0.40.17 version to 0.50.1 using Helm. The upgrade seems to work well (the webapp starts correctly) but if I open the “Connections” or “Sources” or “Destinations” pages I get a “Oops! Something went wrong… - Unknown error occurred” error. In the airbyte-webapp deployment logs there is a

"java.lang.NumberFormatException: For input string: "1000000000000000000000000000000000" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?] at java.lang.Long.parseLong(Long.java:708) ~[?:?] at java.lang.Long.valueOf(Long.java:1156) ~[?:?] at org.jooq.tools.json.Yylex.yylex(Yylex.java:659) ~[jooq-3.17.8.jar:?]"

error. It comes with the following ERROR log line :

"POST /api/v1/web_backend/connections/list HTTP/1.0" 500 1329

I tried to ugrade the connectors but it still does not work and I can’t figure out what’s going wrong exactly. If I rollback & redeploy the 0.40.17 version I still get an error (but another one) and have to rollback the postgres (external) database too to make it work again. Can someone help me with this issue (like any advice on how to fix / troubleshoot this) ? Thanks !

Hello everybody. I have a security related question in relation to credentials being injected into profiles.yml when using the dbt normalisation feature. These should really be populated via environment variables during runtime. Is there a fix in the pipeline for this? As a security conscious organisation we won't be able to enable the dbt feature until this is fixed. https://docs.getdbt.com/reference/dbt-jinja-functions/profiles-yml-context

I also have a question in relation to the Snowflake destination connector. Are there plans to add Azure blob storage as an external staging area? The only alternative is to use internal staging but this will require files to be stored locally before they can be uploaded to the internal stage and copied into a Snowflake table. This will be of course be less performant than using an external stage which can be read directly into Snowflake via COPY INTO. The Web UI is also stating that the internal stage option is recommended for performance and scalability. Clearly this is the less optimal approach compared to an external stage. Any feedback on this would be appreciated. https://docs.airbyte.com/integrations/destinations/snowflake/

Hey Airbyte team, what are the supported ways to route ingress traffic to Airbyte helm chart?

🔥 Community Office Hours starts in 15 minutes 🔥 At 1pm PDT click here to join us on Zoom!

Hey Airbyte Team, does Airbyte OSS support creating multiple workspaces ?

We've recently deployed airbyte on aws EC2 (t2.large w/ 16 GiB volume) with several connections going to snowflake. They have different cadences of backup which have at time lined up with one another. When these syncs align, the sync times out and eventually fails, putting the EC2 in an unhealthy state. These are the failure logs:

2023-06-10 00:59:20 INFO i.a.w.i.HeartbeatTimeoutChaperone(runWithHeartbeatThread):111 - thread status... heartbeat thread: true , replication thread: false
...
2023-06-10 04:39:06 destination > WARN c.z.h.p.HikariPool$HouseKeeper(run):804 HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=49s970ms985µs729ns).

In posts I've found, upgrading to t2.large solved these issues, but we're already there. Any ideas?

Not sure if this is the right channel, please guide me if there is a better venue for the question. How can we install 2 "environments" in windows? I want to run 1 version that has all of my connections, let's call that "local main installation", and another version that is always "from scratch" for client demos, etc. When I clone airbyte it to a new directory and run

bash run-ab-platform.sh

, somehow docker is aware of my config, etc and it shows me all of my config from my "local main installation" version. Maybe it is some simple file I need to change some setting (or filename?) to tell it to build it "new" airbyte... but I don't know how it all works. Please advise.