https://datahubproject.io logo
Join Slack
Powered by
Hi team, I’m facing the same issue like <this> Exe...
# troubleshoot
p
Hi team, I’m facing the same issue like this Executing helm on my namespace, but not sure if upgraded is relevant here GMS helm chart:
Copy code
apiVersion: v2
appVersion: v0.9.3
description: A Helm chart for LinkedIn DataHub's datahub-gms component
name: datahub-gms
type: application
version: 0.2.121
Also - how can I found, from the namespace, what’s the GMS version?
m
I think there is a 
/config
endpoint in the gms that tells you the info.
p
@modern-artist-55754 hi WDYM?
m
You want to know the version of gms, there is an endpoint in the gms that return the its version. Not sure if my answer is helpful.
p
Can you elaborate @modern-artist-55754? What endpoint? Technical steps will assist 🙏
I’m running in K8s
m
Sorry i might have misunderstood your question. Did you want to know which version your gms is right now?
p
YUp
🙂
BTW - did you faced this error too?
m
You can use postman to request the /config endpoint from gms Http://gms-ip:port/config
Or write python code like this
Copy code
import requests

res = requests.get("<http://ip:port/config|http://ip:port/config>")
print(request.json())
p
Can I see this on the pod/UI/exec ?
m
I am not sure, i am not that familiar with k8s. Can you ssh to the gms port, maybe ssh to that and just curl http://localhost:8080/config.
p
Copy code
"linkedin/datahub" : {
      "version" : "v0.9.6.1",
      "commit" : "xxxxxx"
    }
m
Is that the information you after?
p
Sorry?
m
Is that the version information you need?
p
Not sure
I tying to understand the RC for my error
Based on previous comment on this channel, the gms version was the issue I'm trying to figure out if my gms chart.yaml is correct
m
You have duplicate ingestion?
p
Nope
I have several ingestions,but each for a different resource
m
I am not sure how gms version related to the problem. He had a set up that running multiple event consumer in and out of gms
g
@orange-night-91387 might help you
@incalculable-ocean-74010 Could you please have a look here ?
a
Hello, Using Kubernetes you can see the version of DataHub that got deployed by checking the resource manifest and searching for the 
image
property.
If GMS is already deployed then a curl command to GMS endpoint with the 
/config
endpoint will return a JSON akin to:
Copy code
{
  "models": {},
  "patchCapable": true,
  "versions": {
    "linkedin/datahub": {
      "version": "v0.10.3",
      "commit": "a29b576daa2fffcdd356250ca8a60ea9d40a4e11"
    }
  },
  "managedIngestion": {
    "defaultCliVersion": "0.10.3",
    "enabled": false
  },
  "statefulIngestionCapable": true,
  "supportsImpactAnalysis": true,
  "timeZone": "GMT",
  "telemetry": {
    "enabledCli": true,
    "enabledIngestion": false
  },
  "datasetUrnNameCasing": false,
  "retention": "true",
  "datahub": {
    "serverType": "prod"
  },
  "noCode": "true"
}
☝️ is for our demo
p
@incalculable-ocean-74010 hi This is mine:
Copy code
{
  "models": {},
  "patchCapable": true,
  "versions": {
    "linkedin/datahub": {
      "version": "v0.9.6.1",
      "commit": "xxxxxxx"
    }
  },
  "managedIngestion": {
    "defaultCliVersion": "0.9.6",
    "enabled": true
  },
  "statefulIngestionCapable": true,
  "supportsImpactAnalysis": true,
  "telemetry": {
    "enabledCli": true,
    "enabledIngestion": false
  },
  "datasetUrnNameCasing": false,
  "retention": "true",
  "datahub": {
    "serverType": "prod"
  },
  "noCode": "true"
}
@modern-garden-35830 FYI to this issue ☝️
@incalculable-ocean-74010 @astonishing-answer-96712 @orange-night-91387 can someone pls. assist & tell me what & how to run the upgrade? This is our prod env & we need this resolution pls. 🙏
i
What upgrade? If you are using the community Helm chart to deploy DataHub a simple Helm upgrade should be enough
By this I mean, update your datahub helm-chart dependency in the Chart.yaml like so:
Copy code
apiVersion: v2
name: demo-datahub
description: A Helm chart to deploy DataHub for 
type: application
version: 0.0.1
dependencies:
  - name: datahub-prerequisites
    version: 0.0.12
    repository: <https://helm.datahubproject.io/>
    condition: datahub-prerequisites.enabled
  - name: datahub
    version: 0.2.132 <--- Update this to whatever version you want.
    repository: <https://helm.datahubproject.io/>
    condition: datahub.enabled
p
@incalculable-ocean-74010 hi I’m using FORK for your repo & DH community Helm chart, but it’s not upgrading my versions The change you mentioned above is for the prerequisites chart or DH chart?
Ok, I’ve deployed to my staging environment with 
0.2.165
Where can I see that I’m using this version?
i
My apologies, I don’t understand the question.
p
Hi @incalculable-ocean-74010 I've set my chart with 
0.2.165
but I'm still facing the same error in the ingestion
@astonishing-answer-96712 I need inputs here pls.
g
Hi @incalculable-ocean-74010 / @dazzling-yak-93039 Could you please check this issue
d
Hi @powerful-cat-68806, I took a look and it looks like the error is
Copy code
Provided urn urn:li:corpGroup:sccm-Amadeus Altea INV (RIT HD\\\\\\" is invalid
which is coming from this line of code: https://github.com/datahub-project/datahub/blob/8f9a23fb2e129cace52e7adc42978120b8[…]/src/main/javaPegasus/com/linkedin/common/urn/UrnValidator.java I believe the validation check that is failing specifically is this one: https://github.com/datahub-project/datahub/blob/8f9a23fb2e129cace52e7adc42978120b8[…]tils/src/main/javaPegasus/com/linkedin/common/urn/TupleKey.java which checks for matching parentheses in an Urn. Can you see where this Urn is coming from and exclude it? Or make sure it is not being truncated? We use parentheses as a special character in Urns to indicate tuples of entities, so the Urn you're trying to store should not have parentheses if it is not representing this concept.
p
@dazzling-yak-93039 hi 10x for the input. I’m not fully understand, how this is relevant to the issue I’m trying to resolve….
From all that I’ve read, this is related to the GMS version I’m running Trying to upgrade, didn’t work for me
n
For some time I'm having the same issue: I run it with docker compose. From what I see. I see this error in first ingestions of same source. It stops after the first one. It shouts error when I ingest postgres, and also business glossary yaml file. I use postgres as datahub db. Error Code: 
failed to write record with workunit xxx with ('Unable to emit metadata to DataHub GMS: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?', {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'message': 'javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?', 'status': 500, 'id': 'urn:li:glossaryNode:pii.impact.levels'}) and info {'exceptionClass': 'com.linkedin.restli.server.RestLiServiceException', 'message': 'javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?', 'status': 500, 'id': 'urn:li:glossaryNode:pii.impact.levels'}
Copy code
{
  "models": {},
  "patchCapable": true,
  "versions": {
    "linkedin/datahub": {
      "version": "v0.10.4",
      "commit": "3de94c52230ff2ea25de6afd1cf42c7fd85b2375"
    }
  },
  "managedIngestion": {
    "defaultCliVersion": "0.10.4",
    "enabled": true
  },
  "statefulIngestionCapable": true,
  "supportsImpactAnalysis": true,
  "timeZone": "GMT",
  "telemetry": {
    "enabledCli": true,
    "enabledIngestion": false
  },
  "datasetUrnNameCasing": false,
  "retention": "true",
  "datahub": {
    "serverType": "dev"
  },
  "noCode": "true"
}
p
@nutritious-yacht-6205 on my side, this error pops when ingesting Redshift cluster I’m also using Postgres as default DH db
@astonishing-answer-96712 hi Can someone assist here pls. ?
g
@orange-night-91387 Could you please check this ?
o
The workaround for this is to reduce the worker count on the ingestion to 1
p
@orange-night-91387 it’s not helping
Open in Slack
PreviousNext
Powered by