We have a lot of custom python ingestion code (that runs in airflow python operators) and does things like “connect to this ftp server and download a subset of the files, using a list of desired filenames / file types / whatever criteria” - we’re interesting in switching to airbyte as much as possible and are wondering: has anyone figured out a way to customize data ingestion in airbyte. to fit this sort of finicky stuff?

Hi. we use airflow with airbyte. if you are using airflows airbyte operators - this should be enough to make it work- the only thing you need to worry about is the source on airbyte- for files - depends on the file type - we create a source and use it. if you have csv for example you would want to create as much as dynamic source as possible for most types of csvs and in the UI specify what is the type or if you are hard core then create a guessing mechanism in the source it self (I dont like it since you can always look at the data before you pull it so why guess). let me know the use case and maybe i can help

Hey Karl and thanks for the input, Eli! Here's a doc on using the Airbyte Airflow operator: https://airbytehq.github.io/operator-guides/using-the-airflow-airbyte-operator/ Not quite sure if this is applicable to your usecase, but we also have a Dagster integration: https://airbytehq.github.io/operator-guides/using-dagster-integration Let me know if this helps!