Hi, Our company is very interested in integrating DataHub in our system. Is there anything required ? What is the sequentially steps in deploying DataHub ? How can we backup metadata ? If we can deploy by docker, what is the persistent part to backup metadata when docker restarts ? Can you share the experience in deploying ? We need a doccument how to do that and we really need your support. It would be nice , if we could contact in detail. Here is my email address: phamminhsyhcmus@gmail.com

Hi there! Welcome to DataHub. We'll follow up soon 🙂

Hi! We depend on a few storage layers: mysql (or other relational db), elasticsearch, neo4j, and kafka. If you have a version of these components running in your environment, it will make deployment easier. Once these storage layers have been deployed, you can deploy the datahub containers that uses these storage services to start up datahub. Please find more details https://datahubproject.io/docs/datahub-kubernetes

Once we are running DataHub, how can we update version if new one is released on git

We will do a helm version upgrade every time we release a new version! You can also set the image tags to the released version like v0.8.3

We’ve decided to deploy DataHub with Kubernetes. But we have some prolems: 1 . We have already Kafka, Elasticsearch available in our system. The thing is how to integrate these things to DataHub. How can we do that ? Can you give me an example ? 2. How can we backup mysql to get persistent data with Kubernetes? What is the steps to do so ?

It might be better to jump on a call here

For 1, check out this section in the AWS guide https://datahubproject.io/docs/deploy/aws/#use-aws-managed-services-for-the-storage-layer Ignoring the provisioning part, it shows you which part of the values.yaml to change!

I found this in datahub/datahub-kubernetes/datahub/values.yaml file. Are we going to use your mysql setup job configuration ? And it happens to lack of the name and mount path things in the volume mounts part that is exactly what we need to backup data. How can we change the configuration to do that ?

I don’t fully understand. This is just a setup job that runs once on upgrade. Where are you putting the backup logic?

Can you show me where i can put the backup logic if there is something i want to change ?

What is the backup logic you have in mind? Let’s see if it fits in any of our existing jobs

I want to set up where the bound persistent volume should be mounted within our MySQL Kubernetes container

ah interesting

not really ! We want to setup somewhere to mount data to persistent it. Is there any way to do that ? Could you share we how you can backup data ?

For us, we use the s3 backup file in parquet and read it to restore. We don’t have support for mysql dumps yet, but we will work on it soon.

We don't have AWS available yet. How long does it take you to support for mysql dumps. Is there any solution for us right now ?

what kind of persistent store can you use? The dumps need to be stored somewhere outside kubernetes

Could it be be on file or MySQL on local ? Can you give me some suggestion ?

Ah so didn’t ask yet. Are you using kubernetes for deploys or docker-compose?

We are using Kubernetes for deploys