Hello everyone, We are thinking of starting out with EC2 instance (4 cores, 16GB) as @incalculable-ocean-74010 advised. What if later on, we would like to migrate to the K8s solution, would we be able to do so without losing data? How difficult would that process be? Someone has ever done this kind of migration? It makes sense that just other data sources, datahub could be one source and another datahub could be the sink. - does this feature exists? Thank you!

Hello Harel, Data in DataHub is stored in 2 places. The relational DB which contains the metadata model you see and ElasticSearch which contains temporal & ephemeral metadata information as well as search indices. The first can be backed up and restored fairly easily. It’s a SQL dump and restore essentially. Note that there is some information in the database that is encrypted using some configs in GMS. If these configs differ in new GMS instances (that you migrate to) then that metadata will not be readable. These include DataHub Secrets, Access tokens and the like. The latter is trickier. On one hand, you can dump and restore elasticsearch indicies, see https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html. Never tried it personally. On the other you can run datahub’s restore indicies job to generate elastic search indices from the data in the relational DB. This does not include timeseries information, that you will lose, but since it is only worth for a small period of time, you should be able to generate it or simply let be invalid as new timeseries information gets added. Some examples of this data include: https://datahubproject.io/docs/graphql/interfaces/#timeseriesaspect

It makes sense that just other data sources, datahub could be one source and another datahub could be the sink. - does this feature exists?

This does not exist but it is in my personal backlog, no ETA. Most likely it will only focus on the relational DB side of things.