Hello, we have an issue on advanced filtering from the ui. in any filter type (platform,tags..) we are not able to search for a free keyword. because the search is returning always 5 fixed elements, that are showed also as default values. any valid tag or platform is not found if its not in the default list. does anyone encounter this issue before? we are on version 0.9.6 but it was not working also on 0.9.0 for us

hey Francesco! so even if you start typing in that search box there for a tag you know exists it doesn't show up?

Hey, yes we tested on your demo page and its working perfectly, so we don't understand what can be causing the issue on our side. maybe the elastic search index? but we runned the setup job correctly. here for example i search for "ela" platform should return elastic search which is ingested in our datahub. same query on the demo page works.

yeah that would definitely be my best guess as well - something going on in elastic search. do you have any issues searching for the platforms you expect from the tab search bar on the search page?

we actually have only 7 platforms inside dataplatformindex_v2. 3 of them are custom sources developed by us, and only databricks, salesforce, delta-lake and vertica from datahub connectors (we are not using any of them). entries in this index are matching with the one we can search in the filter. lot of other sources we ingested are not in this index. 🤔

hmm yeah that would definitely explain it. I would suggest restoring your indices in this situation! https://datahubproject.io/docs/how/restore-indices/

we runned the restore indices jobs and it fixed for platform. but we still have same issue on tags. we are able to search for every tag that is in index tagindex_v2. but not all the tags are there. any suggestion about why restore indices is not putting everything there?

hmm that's very weird.. did you see any errors output from your restore indices job? and can you confirm that all the tags you expect are in your database? the way restore indices job works is it works through the entries in your document store and creates metadata changelog events that get picked up by a hook which upserts items into elasticsearch indices

no error logs from the restore indices job. tags should be in the db because we can see them both in entities page and in the side bar tag list: but not in the index and in the advanced search 🤔

yeah so technically they don't need to be their own entities in the db in order to see them in the filter bar and on entity pages - they just need to be referenced by another entity as having been tagged with this "tag" (whether the tag actually "exists" in its own right with its own key aspect and everything). do you know how you tagged all these entities? and how you created your tags in the first place?

oh ok i see, so we should check better in mysql. so the majority of our tags are added with transformers on ingestion pipelines. and only some of them are manual

ahh yup okay I think this might be a shortcoming with our transformers - if you apply tags on transformers that don't already exist, we set the reference and it shows up on entity pages and you can filter by it, but the tag itself never gets truly "created" so it doesn't show up in search as itself

ok that makes more sense now! i will try to find a way to create them in ingestion on our side. thanks for taking the time on it 🙂

perfect. and no problem glad we figured it out!