Hi here, I could not find a lot of content about ingesting streaming data in an ELT fashion to your DWH. So I wrote about my experience here: https://blog.netfondstech.de/streaming-ingestion/ If you have similar use cases and would like to exchange, send me a message.

this is great!! nice share and yeah we like microbatches over streaming as well, just need the ingest infrastructure to be incremental and reliable

Really looking forward to replace the EL with airbyte soon 🙂

whats motivating the replacement? i.e. what are the pain points you are solving

First EL is currently not 100% separated from T, Second there is a Target dependency in the EL right now and finally connectors 🙂