Hi guys. I have a weird thing with some of my Hive table, som of them are not displayed in dataset/datasourse wiew, but if I search them by name - they are found in search results. They exhist in datahub, they can be accesed by direct link, but they are not displayed in a list of tables when I open the data container (schema) they are in. It happens accross different data schemas, for dosens of tables. This seems like a bit issue to me, since it affect the basic fucbtion of a catalog - to provide a way to browse through all datasets and see what kind of tables are there. But now it looks like a significant amount of tables can be found in Datahub only I I already have information about their name or content. It started since we firt installed Datuhub and didn't disapper with any upgrades we had. We're now on 0.9.6. Our DevOps suggested soft-deleting such tables when we found them, and after the next ingest they appear normally, visible when browsing a dataset. But I don't see this as an effective solution, cause we have to manually find all problematic tables (and they can be endless), and also soft-deleting erases data descriptions and links with glossaries etc (I guess? correct me if it isn't so). Any idea what else should we look into to find what cause some tables to not display in dataset view, and solve the issue that's causing it, not just single consequences?

Firstly, this should not happen. I agree this is concerning. Hive Databases and Schemas should correctly list the assets within them. A couple questions: 1. Are ANY DBs / Schemas showing the correct contents inside, or are all of them facing this corruption? 2. What does your ingestion recipe look like? Are you using stateful ingestion? And some response on your points: • Soft-deletion does NOT change data descriptions, links with glossaries, etc. It only hides the entities from browse experiences (search, browse, recommendation, lineage) • It's not clear to me why soft deletion is helping in this case at all. When you manually soft delete an entity (e.g a dataset), and re-ingest, you are saying that it becomes visible in its parent container? Any and all screenshots / videos to demonstrate the problem will be very useful for us. I'm trying to nail down whether this is a hive connector issue, an index out of sync issue (reindexing would address), or something else altogether. cc @dazzling-judge-80093

@big-carpet-38439 thank you for sharing my concern. 1. So far every schema that I checked in Hive databse is showing that, but we have a lot of schemas so I I wasn't able to check them all manually, and I'm not sure how to automize it. 2. We're not using stateful ingestions yet. Here's out ingestion recipe

sink:
    type: datahub-rest
    config:
        server: '<https://datahub>.***.me/gms'
source:
    type: hive
    config:
        host_port: '***'
        env: PROD
        username: default
pipeline_name: 'urn:li:dataHubIngestionSource:a***'

@elegant-salesmen-99143 can you paste here the urn of this dataset?

The urn was fine, the path was there. The only place where it was missing was that place on the screenshot