Hi， Do we support using the ES cluster as a DataHub component？

you can change it to point to a k8s Service

Now I don't have a suitable ES cluster to test, and it will create indexes and data in it after connecting to ES

Datahub supports ES cluster as xL mentioned using a managed service, k8s service url or some other stable url. DataHub also requires that ES be properly configured. For that we provide an elasticsearch setup docker image that applies the needed changes. In DataHub helm it is part of the deployment as a one-off job.

May I ask whether this place can use multiple ES IP addresses？ Such as es1,es2,es3. When one ES connection is disconnected, the other ES connection is automatically used. My tests are not going well so far

Hello @mysterious-nail-70388 DataHub requires a static host, which can be a load balanced url. In kubernetes this is typically a service endpoint. It automatically does connection management for you.

Hi, so I have been doing some tests and in Docker it can be done too. Instead of having 3 services of master as usual, the thing is to create just 1 services with replication 3. With this, by putting the name of the service in

ELASTICSEARCH_HOST

you would be referring all 3 master nodes.

When we don't have the Kubernetes environment, we can't use the first method. When the number of services in ES is greater than the number of replication, our second way should have a problem.