Hi team, I saw this weird behavior in my local dat...
# troubleshoot
b
Hi team, I saw this weird behavior in my local datahub. Not sure if anyone encountered this behavior before or could share some idea. ๐Ÿ™‡ After starting my local datahub with
./docker/dev-without-neo4j.sh
, I do a first ingestion using
datahub ingest -c ~/Desktop/recipe1.yml
(it ingests from json file with MCEs for Dataset), and then do the second ingestion using
datahub ingest -c ~/Desktop/recipe2.yml
(it ingests from json file with MCEs for Dashboard), The
getSearchResultsForMultiple
gql query with query variable โ€œ_types_โ€ set to the first entity type (in this case, Dataset) return expected results. However, it returns no result when โ€œ_types_โ€ is set to the second entity type (in this case, Dashboard). Supposedly, there should be some entities returned as Dashboard entities are ingested in the second run. I checked the ES, the entities exist in the
dashboardindex_v2
index. If I do the reverse, ingest Dashboards first then Datasets, I am not able to get Datasets from
getSearchResultsForMultiple
. Does this have anything to do with Elasticsearch cache?
And if I put all MCEs for Dataset & Dashboard into one file and ingest within one run,
getSearchResultsForMultiple
returns expected results when types is set to
DATASET
or
DASHBOARD
b
Digesting ๐Ÿ™‚
Hmm it seems that this indeed could be caching related - if the same query is issued twice in a row datahub may cache the results for up to 10 minutes... However I believe there is a recently introduced flag to disable this behavior cc @early-lamp-41924
e
I donโ€™t think itโ€™s related to caching. Mainly because you used multiple filters to search. Do you still see this issue? It may take some time after the ingest command finishes for the search index to get updated
In the latest release we are disabling caching as a whole with a flag to turn it on if you see performance issues
b
yup i think it just took some time for search to function probably! However, itโ€™s flaky - occasionally it works immediately after ingest from CLI. Thanks John & Dexter, let me monitor for some while ๐Ÿ™‡