Hello all, is there a way to deploy dedicated nodes types with Datahub helm chart for elasticsearch? E.G 3 master nodes ,and 3 data nodes. There are a lot of shards created by Datahub which makes ES queries unresponsive since 3 nodes share both master, data and ingest roles. Or do you recommend us to set up our own cluster? Thanks @big-carpet-38439 @dazzling-judge-80093

@brainy-tent-14503 Would love your help here!

The Elasticsearch helm chart is not maintained by DataHub, we simply point to the elastic.co chart here. Any configuration supported by their upstream chart will work. The configuration of ES depends on your scale and reliability requirements. Sharding can be controlled using the datahub helm chart configuration here. I’d also consider the cpu and memory resources, you may simply need additional resources instead of a more complex ES configuration.

so you mean it would be OK if we dont use the elasticsearch values provided by datahub-helm, instead we spin up our own ES cluster using bitnami charts with 3 master and 6 data nodes say, then also the datahub setup would work ?

You can definitely use your own ES cluster, the helm datastores are examples to get someone started.

@brainy-tent-14503 It seems like some components like Datahub GMS have hard coded values for IP resovle like:

hostAliases:

- ip: "192.168.0.104"

hostnames:

- "broker"

- "mysql"

- "elasticsearch"

- "neo4j"

How are you running DataHub? The helm chart has values here to configure the ES connection. If you are using docker-compose then there are environment variables to set here.

We are running in kubernetes with helm but now a seperate cluster with ES since the are a lot of shards needed for datahub ingestion. The sample set up cannot handle that since all the nodes are master, data node , ingest and coordinator at once. But thanks will check!

There shouldn’t be any need to modify the

hostAliases

and you can drop the elasticsearch host name here along with the port and other settings.

Use a service name that is not covered by the hostAliases, or set the hostAliases without

elasticsearch

if the bitnami service is also called

elasticsearch

@brainy-tent-14503 thanks, will try!

We are seeing 999 of 1000 shards in use on our OpenSearch cluster. We’ve never modified those settings either. Are other people setting explicit values based on the AWS recommendations for sizing OpenSearch?

DataHub definitely doesn’t require that many shards but it would depend on how you’re configuring the default sharding. Out of the box DataHub uses 1 shard per index with ~60 indices.

Cool, also I see this in the values file for the /prerequisites/charts/elasticsearch/values.yaml I'm guessing this is where we would make our changes to specify our OpenSearch Endpoint, ReplicaCount, and max number of shards, etc...? lifecycle: {} # preStop: # exec: # command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"] # postStart: # exec: # command: # - bash # - -c # - | # #!/bin/bash # # Add a template to adjust number of shards/replicas # TEMPLATE_NAME=my_template # INDEX_PATTERN="logstash-*" # SHARD_COUNT=8 # REPLICA_COUNT=1 # ES_URL=http://localhost:9200 # while [[ "$(curl -s -o /dev/null -w '%{http_code}\n' $ES_URL)" != "200" ]]; do sleep 1; done # curl -XPUT "$ES_URL/_template/$TEMPLATE_NAME" -H 'Content-Type: application/json' -d'{"index_patterns":['\""$INDEX_PATTERN"\"'],"settings":{"number_of_shards":'$SHARD_COUNT',"number_of_replicas":'$REPLICA_COUNT'}}' Not at the time we are setting up OpenSearch.

In case anyone finds this later. That is an example for configuring an index template. The number of shards is by default 1 per index. Configuration can be added to increase the shard count as needed per index. As of today and depending on the # of custom entities there are just under 60 indices. There is by default 1 shard and 1 replica for a minimum instance.