Hi all, we are evaluating Airbyte at the moment for some transfers and we really like it at start, but now having some setbacks and would like to ask for advice:
1. extending the raw tables with default Airbyte columns breaks our ETL, we can’t insert rows because column list doesn’t match (is there a workaround there or we would also need to map this new fields in every ETL script) - that doesn’t sound really user friendly
2. when trying to move large tables (it even started breaking with tables with 2M raws, so we didn’t even test with 2G or more) the job fails since after 3 attempts (trying in 50k batches and fail) - does using a larger machine in container solves this or we are missing some setup - we use GCS Stage for BigQuery upload as recommended
any advice on these two things