Hello guys, we are thinking of building a real-time BI system that could be used by any of our employees. Currently we have a Power BI system that is linked directly to our production database, so any complex queries significantly increase the resource usage on the server, and we would like to separate the BI side from ops. We are thinking of using Apache Druid for that, but the only problem is that the db schema is very decoupled because it is auto-generated by our ERP and often changes, so it would be hard to use Debezium with CDC for our db (SQL Server) as source. Maybe someone faced such a problem and can recommend something? The architecture we currently have in mind is: SQL Server -> Airbyte -> dbt -> Apache Druid -> Metabase

Hi Alex, I'm not sure I completely understand your question. Druid can handle fluid schemas on incoming data, either through batch or streaming. Is the issue that you want to be able to change retroactively reformat all of the historical data to conform to the new schema? Or just accept schema changes on the new data as it comes in? Thanks. John

@John Kowtko Hello, John. As far as I understand our current issue is how to get the data from sql server to druid. I know that druid can handle evolving schema, but I have been advised against using Debezium in this case, and was suggested Airbyte. Or maybe you can offer some other tool?

Hi Alex, I am not experienced with Debezium or Airbyte so unfortunatey not of much help there. My one suggestion would be to keep Kafka integration open for consideration as an option, since the Kafka->Druid integration is strong. Maybe trying to solve SQL Server -> Kafka data migration will open up other options.