Trying to migrate on production database. Does it work? Don’t understand what to do

what is happening now?

It’s finished with an error that I have no space left 😄 trying again after volume extending

cool! let us know if an error happens!

it’s eating the space like crazy. 1GB per minute

@Davin Chia (Airbyte) is there any way to decrease the size of export configuration? is it exporting ALL for every upgrade version?

no just the one you are at

I'm guessing there is a large job history

@Chris (deprecated profile) I vaguely remember us chatting about pruning job history - did we implement that?

@Jenny Brown?

it’s used about 100GB already and not finished, just upgraded from 0.26 to 0.27

so the question is how to purge logs manually?

I haven't tested this, but if you are okay losing job history, it should be possible to connect to the jobs db drop all records older than a certain timestap

how to purge logs manually?

1. Purge in exported file. a. You can export the configs in the old Airbyte version. b. Look for the

ATTEMPTS.yaml

under

airbyte_db

, and remove old job attempt entries from that file. c. Import the configs back. 2. Purge in the database. a. Connect to the airbyte internal database:

docker exec -it airbyte-db -U docker -d airbyte

assuming you are using the default one. b. Look into the

jobs

table, and delete old records.

Before purging, please always export the configs and keep a backup.

We are deprecating the old file-based migration system. Once you upgrade to Airbyte 0.29.x, the migration will run directly in the database, so there won’t be any disk space issue.

what is it? my tmp folder in airbyte-server is full of this kind of files

they are eating the space

where is the migration at?

the automatic migration mechanism will export the database to a tmp folder so it can import it in the next version. I'm guessing those files are it. @Subodh (Airbyte) can you confirm?

@Davin Chia (Airbyte) going from 0.28 to 0.29

but it’s exporting the database for every version, one version takes ~50GB

if you aren't going to run out of space, I would wait till it finishes - you are almost there. it is possible to delete the files for the versions < 0.28 now but we can't reliably tell what is for what version

the migration only runs for minor versions so this is the last one

@Ivan Fokeev, what is inside the

schemasxxxxxxx

directories? If they are old database dumps, you can delete the old ones. I checked the codebase but I cannot verify the content of these directories in the code. Our best guess is that it is the intermediate auto migration dumps.

@Subodh (Airbyte), do you know which class generates these

schemas<timestamp>

directories?

noooo

it’s still has 47GB free

it might be the disk space given to the docker agent?

no

I would give Liren's option 2 suggestion a shot and see if the job history can be trimmed

the bad thing also that I need to migrate again from the beginning

Job history purging went in a few weeks ago. It won’t keep much job history. Not sure if that’s related here or not, just answering the question that was asked earlier in the thread.

I deleted all airbyte volumes and see my sources, destinations and connections in the listings. So may be it will work fine

why I can’t cancel sync jobs?

@Davin Chia (Airbyte) @Jenny Brown @Liren Tu could you please explain, why I can’t cancel attempts? It’s creating “pending” job again and again after clicking on “cancel”

It’s configured to schedule a job every 30 mins

If you want to control when jobs are scheduled, you can set the timing to manual