wide-afternoon-79955
04/17/2023, 3:35 PM0.10.1
Running Datahub on AWS EKS and we are facing very high latency (~20 seconds) in our Datahub searches. AWS's Elastic search service with version 7.10 and 3 data nodes and no dedicated master node
Cache is not enabled on Datahub.
We have are using RDS for Mysql
We are not seeing any high memory or CPU utilisation on any Datahub components and even Elastic Search data metrics seems to looks good.
Datahub GMS config 2 replicas with 16G each and the rest of pods are on 4G memory.
We have 10 - 13 k Datasets.
Can some one please guide on how to tune the Datahub search.lively-cat-88289
04/17/2023, 3:35 PMwide-afternoon-79955
04/17/2023, 7:13 PMrapid-hamburger-95729
04/18/2023, 9:55 AMastonishing-answer-96712
04/18/2023, 5:29 PMwide-afternoon-79955
04/19/2023, 8:47 AMwide-afternoon-79955
04/19/2023, 8:47 AM## Search related configuration
search:
## Maximum terms in aggregations
maxTermBucketSize: 20
## Configuration around exact matching for search
exactMatch:
## if false will only apply weights, if true will exclude non-exact
exclusive: false
## include prefix exact matches
withPrefix: true
## boost multiplier when exact with case
exactFactor: 2.0
## boost multiplier when exact prefix
prefixFactor: 1.6
## stacked boost multiplier when case mismatch
caseSensitivityFactor: 0.7
## enable exact match on structured search
enableStructured: truewide-afternoon-79955
04/19/2023, 8:48 AM## Configuration for graph service dao
graph:
## graph dao timeout seconds
timeoutSeconds: 50
## graph dao batch size
batchSize: 1000
## graph dao max result size
maxResult: 10000wide-afternoon-79955
04/19/2023, 8:49 AMcache:
search:
## Enable general search caching
enabled: true
## Configuration for the primary cache
primary:
ttlSeconds: 600
maxSize: 10000
## Configuration for homepage cache
homepage:
entityCounts:
ttlSeconds: 600
## Lineage specific caching options
lineage:
## Enables in-memory cache for searchAcrossLineage query
enabled: false
ttlSeconds: 86400
lightningThreshold: 300aloof-gpu-11378
04/22/2023, 12:44 AMbrainy-tent-14503
04/22/2023, 12:48 AMWe have 10 - 13 k Datasets. => (> 15 seconds) seems very wrong. Are there other indices besides the dataset indices which are very large? For example run this GET _cat/indices perhaps an index which contains datasetusagestatisticsaspect in the name?