Hello guys! :smile: Hope you are well. I have a ...
# troubleshoot
f
Hello guys! πŸ˜„ Hope you are well. I have a question about graphQL searches, specifically related to filtering datasets by URNs. I'm using the following query to filter datasets by a list of URNs.
Copy code
{
  search(
    input: {
      type: DATASET, 
      query: "*", 
      start: 0, 
      count: 100,
      orFilters: {
        and: [
        	{
        	field: "urn",
        	values: ["urn:li:dataset:(...)", "urn:li:dataset:(...)"],
        	condition: EQUAL,
        	},
        ]
      }
    }
  ) {
    start
    count
    total
    searchResults {
      entity {
        ... on Dataset {
          urn
          name
          platform {
            urn
          }
          dataPlatformInstance {
            urn
          }
          domain {
            domain {
              urn
            }
          }
          properties {
            description
          }
          ownership {
            owners {
              associatedUrn
            }
          }
          tags {
            tags {
              associatedUrn
            }
          }
          glossaryTerms {
            terms {
              associatedUrn
            }
          }
          schemaMetadata {
            fields {
              description
            }
          }
        }
      }
    }
  }
}
The problem is that it returns me empty results, always. The same does not happen when I try to filter for tags, platform, etc. It all works, except for filtering by URNs. Am I doing something wrong here? My intention is to fetch information about ALL datasets that we have. However, this is not possible due to ES limitations when query is above 10k results (https://github.com/datahub-project/datahub/issues/4575), so my plan is to create batches of URNs and filter for it, one batch at a time. Appreciate your help! πŸ˜ƒ
πŸ“– 1
πŸ” 1
βœ… 1
l
Hey there πŸ‘‹ I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: βœ… There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? βœ… It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react βœ… to your original message to let us know!
f
I'm using datahub
v0.10.0
a
hi, this is likely a bug, @big-carpet-38439 may be able to help you here next week
b
Hey @fancy-crayon-39356 we recently rolled a scrollAcrossEntities GraphQL Query that allows for deep pagination to more than 10k entities - you can use this to list all of your Datasets batch by batch without requiring a separate process for retrieving urns!
I do not believe that "urn" is actually a filter we support in the filters set (unfortunately). @astonishing-answer-96712 Let's mark this as something to document clearly
e
@big-carpet-38439 sorry for butting in, but since I started learning to use GraphQl in Datahub I have a lot of questions about it, and I follow other users' posts here in order to get more info. The documentation is indeed not very clear for GraphQL queries, it's hard to get the whole picture when the guide bifurcates in so many ways, separate oblects and arguments, and they are not grouped by type or function, but by alphabet... I beleive having examples of whole queries, or at least whole filters would be useful. Or for examples if I go to SearchAcrossEntitiesInput documentation - it is so not obvious for me from it what kinds of queries or filters are possible within it (or where should I click to see what is possible)😒
f
Hey @big-carpet-38439! Thanks for the response. Will definitely check this one out. What I'm doing as a workaround in the mean time is to use the
dataset
query that supports the
urn
argument to get the aspects that I need. But this is bad since I need to make N queries where N is my total number of datasets 😒 Will try now with the
scrollAcrossEntities
πŸ˜„
a
@elegant-salesmen-99143, thanks for the docs feedback! We’re currently working on building out more robust API guides with examples for each usecase- you can check them out in docs under β€œGetting Started: SDK’s and APIs”
b
Great feedback Nadia! I totally agree with you. We need some much better examples here. We will add this to our docs backlog!
@fancy-crayon-39356 - Yea I would definitely recommend scrolling πŸ™‚
f
I've implemented scrolling and it works nicely πŸ˜„ thank you!
s
@elegant-salesmen-99143 HI ! Did you manage to generate a request in Graphql? Also faced with the fact that there is not enough documentation.
e
@steep-needle-64409 I've managed to put together some queries, yes, and I keep getting a hand of more. What kind of queries do you need?
h
@big-carpet-38439 @elegant-salesmen-99143 - Need some recommendation for searching datasets by strings and using this : https://datahubproject.io/docs/api/graphql/getting-started#search . Currently using 0.10.1 version. Sample GraphQL Query I am using :
Copy code
{
  searchAcrossEntities(input: {
    types: [DATASET], 
    query: "*out*",
    orFilters: [
      {
        and: [
          {
            field: "tags",
            values: ["urn:li:tag:tag1","urn:li:tag:tag2"],
            condition: EQUAL
          },
          {
            field: "domains",
            values: ["urn:li:domain:fde75cdc-69f5-4cba-8d48-c1234475ddf"],
            condition: EQUAL
          }
        ]
      }
    ]
  }) {
    searchResults {
      entity {
        urn
        ... on Dataset{
          urn
          glossaryTerms{
            terms{
              term{
                urn
                
              }
            }
          }
          
        }
        
        
      }
      
    }
  }
}
e
@handsome-football-66174 I'll try to help if I can, but I don't understand what exactly the question is. Does this sample query not work?
h
@elegant-salesmen-99143 - trying to filter datasets using multiple keywords Also the above query does not work, though using https://datahubproject.io/docs/api/graphql/getting-started#search as reference.
Only this works : query: "out"
e
I think for multiple tags condition should be IN, not EQUAL. What kinds of errors does this query return?
I tried writing it like this (code below) and it worked for me And when I tried
condition: IN, values: ["urn:li:tag:Tag1", "urn:li:tag:tag2"]}
it gave me "DataFetchingException" and error in results. I don't know, maybe multiple values in one condition are not supported in filters?
Copy code
{
  searchAcrossEntities (
  input: { types: DATASET,
    query: "*",
    orFilters: [{and: 
      [
        {field: "domains", condition: EQUAL, values: ["urn:li:domain:XXX"]}
        {field: "tags", condition: EQUAL, values: ["urn:li:tag:Tag1"]}
        {field: "tags", condition: EQUAL, values: ["urn:li:tag:Tag2"]}
      ]
    }]
  } )
    {
    searchResults {
      entity {
        type
        ... on Dataset {
          urn
          properties {name}
        }
      }
    }
  }
}
h
Giving multiple tags is working
its multiple keywords , datasets names is what is the issue
e
query: "raw, search"
worked for me (raw and search being keywords). How do you write multiple dataset names?
h
which version of Datahub are you using ?
Copy code
{
  searchAcrossEntities(input: {
    types: [DATASET], 
    query: "out, schema",
    orFilters: [
      {
        and: [
          {
            field: "tags",
            values: ["urn:li:tag:tag1","urn:li:tag:tag2"],
            condition: EQUAL
          },
          {
            field: "domains",
            values: ["urn:li:domain:fde75cdc-69f5-4cba-8d48-c1234475ddf"],
            condition: EQUAL
          }
        ]
      }
    ]
  }) {
    searchResults {
      entity {
        urn
        ... on Dataset{
          urn
          glossaryTerms{
            terms{
              term{
                urn
                
              }
            }
          }
          
        }
        
        
      }
      
    }
  }
}
this does not work for me
e
I use 10.1. It would help to understand if you add the error message you get for the query
h
We are also using 0.10.1, unfortunately there is no error message , just that I dont get any results
e
You mean the results say null?
h
Copy code
{
  "data": {
    "searchAcrossEntities": {
      "searchResults": []
    }
  },
  "extensions": {}
}
e
I don't know why, but if I add * like this
query: "*raw*, *search*"
it also returns empty for me, but when I remove one * at the end -
query: "*raw*, *search"
- it gives me results alright
alse you are sure that you have results that match both out and schema keywords at once?
h
I need results that match out or schema , not out and schema
even the results that match keywords out and schema , do not show up in the results
e
okay after trying some variations this worked for me:
Copy code
input: { types: DATASET
    query: "keyword1"
    query: "keyword2"
  } )
but it also gave me lots of results that don't even fully match either condition, so I don't know. Maybe it also searched through field names as well as dataset names?
@bulky-soccer-26729 the
browsePaths:
query doesn't work in GraphQL for me anymore. It worked in 9.6.1, Is it no longer supported in 10.1?
h
@elegant-salesmen-99143 - for me unale to give multiple query in the input
e
@handsome-football-66174 I don't really understand how we have diffrerent capabilities in GraphQL if we're both on 10.1. I can't write multiple arguements in Filters, but you can, I can write multiple queries, but you can't... Very strange. I hope someone from Datahub Team will be able to help then.
h
@bulky-soccer-26729 @big-carpet-38439 - Could you help on this ?
e
@handsome-football-66174 ok I think I fround a way to search for multiple words in dataset names:
input:{query: "/q browsePaths: *user* OR browsePaths: *search*"}
h
@elegant-salesmen-99143 - Thank you ! That worked and solved it for me :
Copy code
query: "/q urn: *Group* OR urn: *Provider*",
    orFilters: [
@big-carpet-38439 @astonishing-answer-96712 - The query : input{query "/q browsePaths: user OR browsePaths: *search*"} works to filter multiple words in dataset names, but fails in a few cases. Do we have query on the name of the dataset ? This link seems to indicate , we can query on the name of the dataset.
b
@big-carpet-38439 @astonishing-answer-96712 -I have the below query, { "query":"{search(input: { type: DATASET, query: \"*\", start: 0, count: 10, orFilters: [{ and: [{field: \"domains\",values: [\"urnlidomain:b8449fa4-4ae8-43c2-93c1-d673f7da62d7\"],condition: EQUAL}]}]}) {total count searchResults {entity {type...on Dataset {name ownership { owners { owner { ... on CorpUser { urn type username properties{displayName email}} ... on CorpGroup { urn type name} } } }domain { associatedUrn domain { urn properties { name } } }tags {tags {tag {name urn } } }glossaryTerms {terms {term {urn glossaryTermInfo { name description } } } } }}}}}", "variables":{} }
r
Hey there! πŸ‘‹ Make sure your message includes the following information if relevant, so we can help more effectively! 1. Which DataHub version are you using? (e.g. 0.12.0) 2. Please post any relevant error logs on the thread!
b
I want to pass urn as a variable dynamically, how can this be acheived
0.12.0
r
Hi! Make sure to create your own threads for future questions rather than posting on year old threads, this makes it easier for us to see the message and respond to it. You can do this using GraphQL variables: https://hygraph.com/docs/api-reference/content-api/variables if you look in your network tab (developer tools) while browsing the DataHub UI, you should see lots of examples of these.