Hi team, QQ I'm trying to retrieve all of our bigquery datasets, even those that we didn't directly ingest (i.e they were created from lookml lineage or something) I'm using the below snippet:

search(
      input:
          {
              type: DATASET,
              query: "",
              count: 6000,
              filters: [
                  {
                      field: "platform",
                      value: "urn:li:dataPlatform:bigquery"
                  }
              ]
          }
  )

However this is only returning datasets that I have directly ingested, and not those brought in "indirectly" through lineage or other means. Is this supposed to happen? And if so, can I modify the search params to include these results?

So I’m guessing you mean the unresolved pointers in the system? We cal these ‘ghost nodes’ - and yes they don’t formally exist as real entities inside DataHub until we have more information (since we can’t render much of use given only an urn)

Yeah those! So I can query their URNs directly, but they won't be returned in a search query?

Yeah you can query for those nodes, but the system considers them "virtual" -- that's why you don't get much information back

The problem is that the ghost nodes are not being returned at all in the above search query

I see