https://linen.dev logo
Join Slack
Powered by
Hi everyone :wave: ! We have currently multiple ...
# feedback-and-requests
c
Hi everyone 👋 ! We have currently multiple connections running since a few months. From time to time, we need to update them to add one or more new table to the integration and to do so, we export the Airbyte config files and make updates on some 
.yaml
files before re-uploading it. Problem here is that these yaml files are getting bigger and bigger (especially 
JOBS.yaml
and 
ATTEMPTS.yaml
) and it's getting complicated to browse easily each to make my changes. What could be a solution to trim these files safely without causing side effects ? For instance, I would have deleted all inactive connection from files, as well as old job but I don't want to break or to loose my existing data sync. Is there any existing tool/solution for cleaning / trimming these config files ?
Hey doesn't it make sense to take the backup of the db since we moved all of this to the database?
Thanks for the reply @Harshith (Airbyte) 🙏 ! Hm, not sure how to do this. For the record, I'm running Airbyte on Kubernetes (version 0.33.XX). If I understand correctly, you mean that, by accessing the database from my 
airbyte-db
pod and by updating directly rows from the different tables (I guess 
airbyte.attempts
, 
<http://airbyte.jobs|airbyte.jobs>
and 
airbyte.airbyte_configs
in my case) , this will have the same consequences as changing the 
.yaml
files from the exported config ?
Hey I mean since it is a postgres Db I think you can do a postgres dump and upload to s3 or I you have a managed service postgres which handles it for you
Ok thanks for the info ! And, just to find out, If I need to at some point to re-load this dump in the postgres db, will it be working ?
Yeah it should be
2 Views
Open in Slack
PreviousNext
Powered by