https://linen.dev logo
Join Slack
Powered by
Today I decided to upgrade airbyte from 0.40 to 0....
# ask-ai
h
Today I decided to upgrade airbyte from 0.40 to 0.44.5 and I have run into two significant issues. Having fought with airbyte for a year and never gotten it to work stable I am quite ready to throw in the towel. Can you please. please. put some of the effort you put into redesigning the UI into building a stable product?
k
A message from kapa.ai
I understand your frustration with the issues you've encountered while upgrading Airbyte. As an AI, I can't directly contribute to the development of Airbyte, but I can provide you with some guidance based on the information extracts I have. To upgrade Airbyte, you can follow the instructions in the [Upgrading Airbyte on Docker](https://docs.airbyte.com/operator-guides/upgrading-airbyte#upgrading-on-docker) or [Upgrading Airbyte on K8s](https://docs.airbyte.com/operator-guides/upgrading-airbyte#upgrading-on-k8s-0270-alpha-and-above) sections of the documentation, depending on your deployment method. If you still encounter issues after following the upgrade instructions, I recommend checking the [Troubleshooting & FAQ](https://docs.airbyte.com/troubleshooting) guide for possible solutions. Additionally, you can reach out to the Airbyte community on [Slack](https://airbytehq.slack.com) or the [community forum](https://discuss.airbyte.io) for assistance. Airbyte is an open-source project, and the development team is continuously working to improve its stability and features. Your feedback is valuable, and I encourage you to share your experiences and suggestions with the community to help make Airbyte better.
m
@Huib what are the actual problem you’re having?
h
s
I went from 0.39.x to 0.44.4 and I had to scrap the PVC for the postgres because of breaking changes of the DB internal tables.
also another thing I noticed is that I had to introduce
Copy code
MAX_NOTIFY_WORKERS: "10"
  MAX_SYNC_WORKERS: "10"
  MAX_SPEC_WORKERS: "10"
  MAX_CHECK_WORKERS: "10"
  MAX_DISCOVER_WORKERS: "10"
h
and in the past year: syncs that fail because the worker “forgot” what resource requests to pass, meaning it couldn’t schedule (due to policy) helm charts that have “latest” tags that then break when that image is updated missing requests on pods that go heywire
@Simon Thelin where did you set those?
those sound exactly like something I’d need 😉
s
I currently set these on the temporal and worker
I run airbyte in vanilla k8s, reversed engineered the helm chart.
I had to run the server as a STS as well
h
and yeah, I have completely torched the installation because of the second issue I mentioned
s
at the time when I deployed it first time I took kustomize and changed to vanilla since I like to setup up locally in a specific way
now I want to move to helm at some point but I learn how it works now at least.
h
we’re using helm, but be aware, the helm chart is quite incomplete
so be prepared to download and modify yourself (which is insane)
s
yeah I don’t find a lot of docs to what to set in the values
so I just went with what I know, vanilla k8s lol
but I had massive issues with the temporal until I introduced the envs above.
h
😉 I’m super close to just give up. Every time it’s something new that breaks. Every update seems to fix an issue I’m having but introduce 3 new bugs and things that break
m
@Huib I asked one platform engineer to take a look in the 
CheckConnectionWorkflow
problem and see if there anything we’re working to fix it
s
and I had massive issues with the PG so be careful, I had to reset 2 years of syncs due to it
every update is quite breaking change yes.
also to mention how slow the sync is with the destination and source pods, they can’t distribute so relatively small tables takes forever to sync.
but the tool is versatile, but you need to fiddle a ot
no idea how the hosted version works, but I doubt it is any better to what I do now.
a
if you're using the helm chart, I highly recommend using a values.yaml where you specify the application version. that way, you can upgrade the helm chart separately from the application. (but, agreed with many comments above about how constructing a good values.yaml is tricky)
s
do you happen to know any good production like example of this?
h
I have had to go in and modify the templates themselves because missing resource requests…
because you can’t set them in the values.yaml
s
Right, I don’t have this issue since I run it all myself without helm.
h
same for tolerations/selectors
a
@Huib I have the opposite experience - I find this chart to be very flexible in what it lets you set (limits/resources included). It just isn't well documented
s
this is what I thought was the case
a
We have node selectors set for all airbyte components, as well as limits on some
s
I tried to find some good examples but so far it has been quicker for me to just do it all from scratch
h
does that work though, only updating the app? I feel like there’s so many moving parts it’s going to be tricky to not-forget one?
s
I guess that is always the case either way
they changed the architecture quite a bit since I deployed it first time
but I want to try to move to helm so I can upgrade easier, will need to find some good examples of it…
h
@Adam Bloom quite a lot, yeah, but not all. For instance, there’s no requests (that you can set) on airbyte-db, nor on minio
and we have a policy in place that rejects pods without requests, so I can not deploy the vanilla helm chart
a
ah - I don't use either of those. we run in AWS, so we use RDS for the database and S3 instead of minio
h
ah! yeah that explains, I could indeed deploy the rest 😉
s
This is all envs I use @Huib , with this it has been stable, example from my local setup. However I then inject this on deployment/STS level. I set my resources on my own deployments.
example-config.yml
RDS is probably a good idea, I was wondering if the helm chart supported to config that, could not find anything, I might looked in the wrong place.
However I still use minio for the logs
a
Copy code
postgresql:
  enabled: false

externalDatabase:
  host: ${rds_host}
  user: ${rds_user}
  password: ${rds_password}
  database: ${rds_database}
  port: ${rds_port}

minio:
  enabled: false

cron:
  enabled: false
✅ 1
h
in my deployment that cron thing is new, what does it do? any idea?
s
I have been trying to understand that as well, there is no docs for the cron ,name server, the builder seem to be just for the source/dest connectors but I am not sure.
Copy code
airbyte/cron:0.44.4
airbyte-server-container
I did have the airbyte server failing over and over again as I did the upgrade to 
0.44.4
, so I moved it to STS and then bumped up the resources for the server-container, logs don’t say much either.
h
what is STS in this context?
s
statefulset
Open in Slack
PreviousNext
Powered by