Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Is it possible to set partition field when using full refresh append? This way it would be possible to set partition expiration and automatically get rid of old data.

<@U0239L8MA76> do you mean stream slices?

if so then yes they are also supported in the full refresh sync mode and will be passed into `read_records` and other methods

it mostly about BigQuery <https://cloud.google.com/bigquery/docs/managing-partitioned-tables#partition-expiration|tables partition>, when create a table its possible to specify field for partition and expiration time. As they wrote in doc,
&gt; When a partition expires, BigQuery deletes the data in that partition.
It could be useful for large data

I believe the bigquery connector doesn’t currently support writing partitioned tables but it is on the roadmap.

See here: <https://github.com/airbytehq/airbyte/issues/2579>