Hi All, We are running , Datahub with version : `0...
# troubleshoot
w
Hi All, We are running , Datahub with version :
0.10.1
Running Datahub on AWS EKS and we are facing very high latency (~20 seconds) in our Datahub searches. AWS's Elastic search service with version 7.10 and 3 data nodes and no dedicated master node Cache is not enabled on Datahub. We have are using RDS for Mysql We are not seeing any high memory or CPU utilisation on any Datahub components and even Elastic Search data metrics seems to looks good. Datahub GMS config 2 replicas with 16G each and the rest of pods are on 4G memory. We have 10 - 13 k Datasets. Can some one please guide on how to tune the Datahub search.
šŸ” 1
āœ… 1
šŸ“– 1
l
This content can't be displayed.
w
I am also trying to enable cache using https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-gms/values.yaml but its not enabling the cache
r
hey @orange-night-91387 ! would it be possible to get a hand here?
a
@brainy-tent-14503 might be able to provide some insight here!
w
I have managed to enable hazelcast cache on GMS yesterday and frequent search on same key word was improved. But the new keyword searches are still slow (> 15 seconds) . AWS EKS Elastic Search config, Version : Elasticsearch 7.10 3 Datanodes and no dedicated master Availability Zone(s) : 2 AZ Instance type : r6g.xlarge.search EBS volume type : General Purpose (SSD) - gp3 General Purpose (SSD) - gp3 : 100 G Provisioned IOPS : 3000 IOPS Provisioned Throughput (MiB/s) : 125 MiB/s I have also set ELASTICSEARCH_INDEX_BUILDER_REFRESH_INTERVAL_SECONDS to 5 seconds to improve the performance. • All the cluster health metrics are green. • Key performance indicators looks good • Data nodes CPU & memory are not too high
Search config for datahub,
Copy code
## Search related configuration
      search:
        ## Maximum terms in aggregations
        maxTermBucketSize: 20

        ## Configuration around exact matching for search
        exactMatch:
          ## if false will only apply weights, if true will exclude non-exact
          exclusive: false
          ## include prefix exact matches
          withPrefix: true
          ## boost multiplier when exact with case
          exactFactor: 2.0
          ## boost multiplier when exact prefix
          prefixFactor: 1.6
          ## stacked boost multiplier when case mismatch
          caseSensitivityFactor: 0.7
          ## enable exact match on structured search
          enableStructured: true
Graph service Dao config,
Copy code
## Configuration for graph service dao
        graph:
          ## graph dao timeout seconds
          timeoutSeconds: 50
          ## graph dao batch size
          batchSize: 1000
          ## graph dao max result size
          maxResult: 10000
Search cache config is like,
Copy code
cache:
      search:
        ## Enable general search caching
        enabled: true
        ## Configuration for the primary cache
        primary:
          ttlSeconds: 600
          maxSize: 10000
        ## Configuration for homepage cache
        homepage:
          entityCounts:
            ttlSeconds: 600
        ## Lineage specific caching options
        lineage:
          ## Enables in-memory cache for searchAcrossLineage query
          enabled: false
          ttlSeconds: 86400
          lightningThreshold: 300
a
Can you examine the largest indices? It might be time to shard the larger ones using this in order to split the load across your multiple data nodes.
b
I see
We have 10 - 13 k Datasets.
=>
(> 15 seconds)
seems very wrong. Are there other indices besides the dataset indices which are very large? For example run this
GET _cat/indices
perhaps an index which contains
datasetusagestatisticsaspect
in the name?