Hi, we're building a SaaS platform for automotive logistics. As part of the daily routines we need a way to • import files in misc formats, e.g. XML, csv or industry-standard formats, such as EDI (EDIFACT, ODETTE, X12) • perform some field/value mappings • Send the result as a JSON payload to our Web API for processing. A key feature is being able to know when processing of a file has failed and having a mechanism to either retry or cancel the import Do you think Airbyte could be a good fit for what we are trying to do? My areas of concern are not having connectors for every format we want to support. Is there a way to import parsers that we could implement ourselves when we miss a format, e.g. XML? Would it be feasible to do the data mapping/transforms in a easy way for non-developers? I have not seen http as possible destination, is this something that you would plan to do? if not, could we add a plugin so we can perform those http requests? thank you!

we don’t support transforms inside Airbyte at the moment… but we leave it to specialized tools like

dbt

to do so (which works on limited number of destinations)

I didn't know about Airflow, it does seem to be a better fit for what we are trying to do

Airflow seems to be fitting better if you have quite some custom and specific logic to implement in your script, but if you need something more common/standard on the source/destination api/endpoint, that’s where airbyte factor efforts from the community into the connectors and share the burden of maintaining them

that makes a lot of sense, thanks very much for the tips Chris

One point to add, it's possible to use Airflow with Airbyte, so one potential solution is to use Airflow for orchestration and Airbyte for the actual syncs

That's a good point and could have been a possibility. Unfortunately the majority of the syncs we want to use are not supported. Might be worth exploring the use of Airflow with something else?