Hello all! I have a perhaps weird question. We currently have Datahub deployed on our org, on AWS. We deploy Datahub on containers on ECS, and use the managed services for the storage layer (MSK, OpenSearch and RDS). AWS MSK and AWS Opensearch are the most costly components. Since we are not a big org, we were considering the possibility that the ‘default’ arquitecture of Datahub is a bit ‘overkill’ for us. We were wondering, did anyone ever succeed in running Datahub without these components somehow? Would this even be possible? For example, could we use AWS SQS to exchange events between components and then store them somewhere else? We are are aware of Datahub Lite, but keeping the UI, lineage and Rest API is important for us. Thank you 🙂

What were the reasons to use managed services by AWS? Have you considered to use the native services as defined by DataHub? Per default, DataHub spins up it's own Apache Kafka instance, etc.

We thought the managed services would be easier to manage, since in our team we are not experts in Kafka/Elastic Search. Just seemed a bit safer. But, maybe we could save some costs by hosting these services ourselves for sure.

That's a valid concern and I fully understand your thoughts her. When discussing cost savings in terms of AWS bills, it's essential to consider that the potential savings here may not be negligible. Depending on the size of your organization, you can operate the services with relatively small container configurations, resulting in significantly lower costs compared to using managed services. However, as you rightly pointed out, this is only one aspect of the equation. Self-hosting does come with its own responsibilities and requires some effort on your part. That said, I can share some of our experiences with our setup. We've been self-hosting all DataHub dependencies, with the exception of Kafka, as we already had MSK in place. Over the past 9 months, we haven't encountered any issues with these components; they have been performing reliably. It's worth noting that there may be situations where you need to become familiar with the components, particularly when it comes to tasks like securing the traffic between Elasticsearch nodes. Hope this helps you for your considerations. Perhaps replacing the most costly service with a self-hosted solution could be a viable option?

Thanks for sharing your experience! We did some match yesterday, and it unfortunately does not seem that we would save much cost (or any at all), by self hosting the services instead of using the managed services. We are still evaluating this though, so maybe we will reach a different conclusion. My original question was more if we could replace the more expensive components of the arquitecture (kafka and EL) with other less expensive (but also probably less powerful) components. It does not seem to be the case.

I apologize for steering the discussion in a different direction; I believed it might be a more achievable approach. However, when it comes to replacing the components with other services, I lack the expertise or experience to provide any assistance.

No need to apologize at all 🙂 , your comment was more than welcome. I just think I was chasing for something that was really maybe not that realistic.