Hi here, I could not find a lot of content about ingesting streaming data in an ELT fashion to your DWH. So I wrote about my experience here: https://blog.netfondstech.de/streaming-ingestion/ If you have similar use cases and would like to exchange, send me a message.
💯 1
🔥 1
octavia loves 1
fiesta parrot 1
airbyte rocket 1
s
Shawn Wang (Airbyte)
07/12/2022, 2:53 PM
this is great!! nice share and yeah we like microbatches over streaming as well, just need the ingest infrastructure to be incremental and reliable
j
Jan Erik Herrmann
07/12/2022, 2:55 PM
Really looking forward to replace the EL with airbyte soon 🙂
s
Shawn Wang (Airbyte)
07/12/2022, 5:28 PM
whats motivating the replacement? i.e. what are the pain points you are solving
j
Jan Erik Herrmann
07/13/2022, 7:06 AM
First EL is currently not 100% separated from T, Second there is a Target dependency in the EL right now and finally connectors 🙂