https://datahubproject.io logo
Join Slack
Powered by
hi folks! I have been trying to ingest lineage dat...
# troubleshoot
n
hi folks! I have been trying to ingest lineage data using the datahub-cli (awesome tool btw), and I was facing an error when the cli does a request to retrieve the upstreamLineage from the GMS.
Copy code
com.linkedin.restli.server.RestLiServiceException [HTTP Status:400]: Invalid value type for parameter aspects
looking deeper into it, that error is thrown when the header X-RestLi-Protocol-Version set to 2.0.0, if I remove or change it to i.e. 1.0.0, the request looks good the header is added in https://github.com/acryldata/datahub/blob/master/metadata-ingestion/src/datahub/cli/cli_utils.py#L163 any hint why this is causing the backend to crash?
b
Hmm thank you for reporting this. Are you using any type of customer transformer? What CLI command are you running?
n
I am using the cli as a library
Copy code
from datahub.ingestion.run.pipeline import Pipeline
pipeline = Pipeline.create({
        "source": {
            "type": "datahub-lineage-file",
            "config": {
                "file": "lineage.yaml"
            }
        },
        "sink": {
            "type": "datahub-rest",
            "config": {
                "server": "myserver",
                "token": "mytoken"
            }
        }
    })
pipeline.run()
pipeline.pretty_print_summary()
b
Got it. So we do want to be using Rest.li 2.0.0. Does the dataset you're attempting to update already have any lineage?
n
no afaik, it is a auto ingested dataset from glue
without any lineage
m
@numerous-camera-74294: which version of the cli are you using?
n
Copy code
$ datahub --version
acryl-datahub, version 0.8.27.1
$ pip3 freeze | grep datahub
acryl-datahub==0.8.27.1
b
My guess is that something in the source itself 
datahub-lineage-file
is causing this issue
n
Copy code
---
version: 1
lineage:
  - entity:
      name: database1.table1
      type: dataset
      env: PROD
      platform: glue
    upstream:
      - entity:
          name: database2.table1
          type: dataset
          env: PROD
          platform: glue
      - entity:
          name: database2.table2
          type: dataset
          env: PROD
          platform: glue
      - entity:
          name: database2.table3
          type: dataset
          env: PROD
          platform: glue
      - entity:
          name: database2.table4
          type: dataset
          env: PROD
          platform: glue
that is my lineage.yaml
b
After some digging, my hypothesis is that this line may be causing it: https://github.com/linkedin/datahub/blob/e2f8db7926fe50624738ccaa670128f0a0be13e0/metadata-ingestion/src/datahub/cli/cli_utils.py#L400. If we request only 1 aspect, I think the list argument may be malformed. cc. @mammoth-bear-12532 But it doesn't make sense why this source has been working for some folks. can you try setting the preserve_upstream flag to False in the config, and see if anything changes it? Also, can you confirm that you're running the latest DataHub server? This will appear in the UI on the right side user dropdown
n
sure, give me 5 minutes and I will come back
actually working with that flag at False
and I can confirm that the error is when getting:
Copy code
<https://myendpoint.com/api/gms/entities/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aglue%2Cdatabase1.table1%2CPROD%29?aspects=List(upstreamLineage)>
the version of the datahub is v0.8.27
for both frontend and gms
this is the curl of the faulty request
Copy code
curl --location --request GET '<https://myendpoint.com/api/gms/entities/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aglue%2Cdatabase1.table1%2CPROD%29?aspects=List(upstreamLineage)> ' \
--header 'Authorization: Bearer myToken' \
--header 'User-Agent: python-requests/2.25.1' \
--header 'Accept-Encoding: gzip, deflate' \
--header 'Accept: */*' \
--header 'Connection: keep-alive' \
--header 'X-RestLi-Protocol-Version: 2.0.0' \
--header 'Content-Type: application/json'
if I remove the X-RestLi-Protocol-Version, it works as expected
b
thanks let me try this out!
n
thank you!!
b
Okay strangely this curl works for me 😞
But i was calling my backend server directly
Let me try to proxy from frontend
n
it can be something related to my entity? or to my gms?
this is deployed on AWS on EKS, if that helps
b
Aha!
I also see the issue when proxying
Let's see here...
n
good news!! I guess
b
It's good - this is a bug!
we like [finding] bugs
n
hahaha, I can relate
do you want me to open a formal issue or something?
b
You can, but I'm going to see right now if its a quick fix
n
ok! I’ll wait then 🙂
b
Hi Alberto. So I've triaged this and found that its a problem in our proxying to the backend server - we are URL encoding the path in a way that the backend doesn't accept. I'm trying to determine the best path forward, but unfortunately it doesn't seem like a quick fix. Do you have a way to proxy to the DataHub backend (GMS) instead? Ie can you expose that to your ingestion source for now?
i'll be creating a github issue to track this
n
what I end up doing was editing the cli code to not send that header and it seems to be working ok, at least for my poc
so dont worry about me, I dont have dh in production (just yet!!)
b
Okay. Thanks Alberto. We will separately look into how to address this. We are currently working on a new non-restli based server that will hopefully alleviate this problem entirely (next ~2-4 weeks)
cc @orange-night-91387 for visibility
👀 1
n
greate, thank you John
2 Views
Open in Slack
PreviousNext
Powered by