Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Quick question if you guys happen to have the answer already but for the Postgres destination, do you truncate the table for full refreshes or drop it and recreate it? Wondering if I can modify the table's schema to have an index and have it persist.

This happens in `PostgresDestination` -&gt; `AbstractJdbcDestination` -&gt; `JdbcBufferedConsumerFactory:141`

On the final normalized table it s handled by dbt, i'm not so sure what s happening there...

Oh good point. Are you actually using normalization for this Roshan?

Looks like `dbt` bashes the table and recreates it each time. Is there a thing I can do to retain an index on the finished result? Or even to just re-index each time? It's O(n²) to do that each time it syncs but I'll just slow down the sync.

Alternatively, I could drop the sync to once a day and just also run `CREATE INDEX` once a day but that's solidly in megahax territory.

Once we have better DBT integration you would be able to set up a post hook: see <https://github.com/fishtown-analytics/dbt/issues/804#issuecomment-398868392>

That would allow you to do the expensive re-index.

We also have a project on our backlog to do more incremental dbt stuff, and I’m not sure if any of those changes will impact the table bashing. Chris might know?

Great, good stuff. I'll slow this sync and re-index periodically for now.

Actually, this makes incremental copies not very useful to me at all. A full-table re-index is very slow. I will implement this sync myself manually.

That’s unfortunate. We basically have to wait for incremental DBT support for this.

I’m honestly surprised this hasn’t been raised as a concern before.