im using a mix of GraphQL and datahubGraph to query and archive implemented policies on my instance. I wanted to archive my implemented policies. I noticed that when using:

policy = graph.get_aspect_v2(
  entity_urn=policy_urn,
  aspect = "dataHubPolicyInfo",
  aspect_type=DataHubPolicyInfoClass
)

I will inevitably encounter an AvroException: ('Datum union type not in schema %s', 'filter') for policies that are created in UI and have a filter condition. (and 1st time created without edits) It is somehow resolved if i add more entities to the filter, then reverting to the original filter. Not sure why it is happening.

Hi Everyone, Mistakenly I have deactivated root user policy in Datahub, Now I am trying to revert it but getting error 'Unauthorized' . Is there any way out? How can I revert it and enable privileges for root user?

Hi all, i am trying to delete container by CLI and i get this error: (version 0.8.38)

datahub delete --urn "urn:li:container:b2f754cac6699f0bd0688b5573eb2156" --dry-run
[2022-07-21 14:56:35,407] INFO     {datahub.cli.delete_cli:134} - DataHub configured with <http://localhost:8080>
This urn was referenced in 5 other aspects across your metadata graph:
+----------------+---------------------------------------------------------------------------------------------------------------+-----------+
| relationship   | entity                                                                                                        | aspect    |
+================+===============================================================================================================+===========+

{i removed this lines}


Do you want to delete these references? [y/N]: y
Failed to execute operation
java.lang.RuntimeException: Could not deserialize ByteString(length=4,bytes=6e756c6c) for aspect container

Hi folks, we're trying to upgrade our metadata service from 0.8.35.x -> 0.8.41. We are running the metadata service on kubernetes (there's around 10 replicas as we've not broken out the MCE and MAE consumers). When we rolled out the 0.8.41 upgrade, we have noticed the canary instance trying to bootstrap on 0.8.41 seems to be getting stuck in the

IngestPoliciesStep

. I think it seems to be related to @early-lamp-41924’s PR - https://github.com/datahub-project/datahub/pull/4733 but for some reason we're still seeing the bootstrap fail (from what I understand that should allow us to be able to bootstrap). Some details in: 🧵

Hello all, I have a small question regarding docker upgrade I was reading https://datahubproject.io/docs/docker/datahub-upgrade/ As far as i understood I just need to run ./datahub-upgrade.sh -u NoCodeMigration and it should update all the images as I'm running a local setup. Or did I miss something ? I just want to upgrade from 0.8.40 to 0.8.41. In short it didn't upgrade.

Hey all - bumping my issue from a few days ago to see if anyone has any pointers. https://datahubspace.slack.com/archives/C029A3M079U/p1658191371824019

Hey all, for those who are having

- Docker doesn't seem to be running. Did you start it?

issue, here's how I managed to quickstart docker. The main reason why I am getting that is because I'm not using docker-desktop (this is due to licensing issues for larger corporations) The way to start is, clone datahub and then run this command on datahub folder:

docker-compose -p datahub -f docker/quickstart/docker-compose-without-neo4j-m1.quickstart.yml up -d

PS: I'm on Macbook M1, so that's why I'm using the one that has

neo4j-m1

adjust that accordingly based on your device

Hey all, during ingest, trying to connect to redshift database fails because I'm using redirected ports (tunnel) on my local machine. How do I start datahub docker quickstart with -p for ports as a pass through?

We encountered the same problem using k8s in aws, upgrading datahub 0.8.33 to 0.8.41

hi ,team How to switch login authentication to internal employee authentication。What is the way to replace datahub login with custom authentication。

Hi all, I try to query glossary group with graphiql ,and I need result query glossary node and show glossary term in this node . please advise me.

Hi all, I need your help about how to connection datasource with SSL certificate to encrypt?

hi, I'm having trouble getting lineage from a superset chart to a postgres database. i'm looking here https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/superset.py#L175 and

database_name

is actually getting set to the display name thats in superset (see attached screenshots). is this intentional? i would have thought DataHub would want to grab the actual name of the database.

Hello, i’m trying to set authentificate login with my own oAuth apis is there any way to setup this?

Hello With v0.8.41 we are still getting intermittent 401 errors. PFA error screenshot. Can someone point me to the PR that has fixed this issue.

Hi Team, Trying to enable Glue Stats for our Datasets. Tried ingesting the Dataset , but the stats were empty. Found this where it is asking to use update-table https://github.com/datahub-project/datahub/issues/5156 to enable stats. What needs to be updated via aws command

Hello one of my users is trying to do a search for a piece of text that is contained in a field description and it seems to not be working. Take the example from the demo site below On this page https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:bigquery,bigq[…]italy.data_by_province,PROD)/Schema?is_lineage_mode=false There are multiple columns with the term "Province" int he description. But a search for "editedFieldDescriptions: Province OR fieldDescriptions: Province" yields no results https://demo.datahubproject.io/search?page=1&query=editedFieldDescriptions%253A%2520Province%2520OR%2520fieldDescriptions%253A%2520Province Any idea what's going on here?

Hi All, when I build datahub on windows I am getting an error that says

Caused by: java.io.IOException: Cannot run program "C:\Program Files\jdk8\bin\java.exe" (in directory "C:\Users\sarat.lingamarl\Documents\datahub\datahub\metadata-models"): CreateProcess error=206, The filename or extension is too long

Is there an easy way to shorten the command line? I found a solution online but I dont know if this works: https://blog.jetbrains.com/idea/2017/10/intellij-idea-2017-3-eap-configurable-command-line-shortener-and-more/ The JetBrains Blog IntelliJ IDEA: Configurable command line shortener | The IntelliJ IDEA Blog In IntelliJ IDEA, it’s possible to configure a command-line shortener. Let’s take a closer look at when and how to use this feature. Configurable command line shortener When the classpath gets too l https://blog.jetbrains.com/idea/2017/10/intellij-idea-2017-3-eap-configurable-command-line-shortener-and-more/ [11:14 AM] I was able to build datahub easily on Mac

Hi all, I was wondering if it was possible to filter a graphQL query by its

timestampMillis

?

I'm trying to enable OIDC in datahub via helm charts and I'm running into this error, log in thread

Hi, everyone, how to use GE? Even if it's not.Yaml, it's the same alarm

Error Msg: Could not find checkpoint 'xxxxxx' (or its configuration is invalid)

Hi team, I am currently facing an error that I have seen is quite common when using OIDC Authentication (in my case I am using Keycloak). Such error is the following:

ERROR auth.sso.oidc.OidcCallbackLogic - Unable to renew the session. The session store may not support this feature

. I have seen that the origin of this error used to be that the PLAY_TOKEN cookie was rejected due to userprofile exceeding 4096 chars. I have checked the length of this cookie and in my case is lower than 4096 chars. The thing is that I get this error both when I login and when I try to logout (I say try to logout as it doesn't logout, it just keeps redirecting to the main page of Datahub, the one where you can see the datasets ingested,...) I have also tried limiting the claims of the

AUTH_OIDC_SCOPE

but haven't been lucky. Any help would be appreciated!

Hi everyone, i have a problem with launching the command "datahub docker quickstart". Untile yesterday everything was fine, but now this error message appears. I've tried restarting containers, restarting my VM, starting healing via "willfarrell/autoheal" but nothing seems to be working. Does somebody know something about this ? thanks

Hello, can someone please confirm the quickest way to reset the default datahub account password on an AWS/K8/Helm deployment of DataHub?

hello, I am running docker container locally by using docker-compose; and providing

docker-compose-without-neo4j-m1.quickstart.yml

as the file. Wondering how can I enable debug mode?

Hi all! I've been trying to change the UI and I've been coming up an error. Here was my process: To compile the changes I made

./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint

To create the image

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker build -t linkedin/datahub-frontend-react:debug -f docker/datahub-frontend/Dockerfile .

Build and start datahub

datahub docker quickstart --build-locally

Replacing and running the local changes

(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate datahub-frontend-react)

I did it in this order and at the log in screen it was fine but after that, I received error messages like these: Validation error of type InvalidFragmentType Failed to Complete Search Any help would be appreciated, and let me know if my process for updating the source code is wrong, as I'm pretty inexperienced with development on containers, thanks!

Hi everyone, I'm not sure if this is the right channel or if this question is too low level, but here goes: where are the requests sent from

RestliEntityClient

received ?

Hi everyone!! Currently, I am testing OIDC integration locally. I used Google Authentication and it was possible to log in. However, when logging out, error occurs and the user is redirected to the original page (http://localhost:9002) without logging out. Post error messages and settings. Please help me. I am using datahub:v0.8.41 Error Message:

datahub-frontend-react    | 08:23:23 [application-akka.actor.default-dispatcher-35] ERROR auth.sso.oidc.OidcCallbackLogic - Unable to renew the session. The session store may not support this feature

datahub/docker/datahub-frontend/env/docker.env:

AUTH_OIDC_ENABLED=true
AUTH_OIDC_CLIENT_ID=my-client-id
AUTH_OIDC_CLIENT_SECRET=my-client-secret
AUTH_OIDC_DISCOVERY_URI=<https://accounts.google.com/.well-known/openid-configuration>
AUTH_OIDC_BASE_URL=<http://localhost:9002>

AUTH_OIDC_SCOPE=openid profile email
AUTH_OIDC_USER_NAME_CLAIM=email
AUTH_OIDC_USER_NAME_CLAIM_REGEX=([^@]+)

docker-compose:

docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml up

Hello all, trying to enable OIDC on azure for datahub running on docker. My yaml is correctly parsed and contain: oidcAuthentication: enabled: true provider: azure clientId: XXXX clientSecret: XXXX azureTenantId: XXXX based on values from https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-frontend/values.yaml after running helm --debug upgrade pointing to yaml i see: 250 oidcAuthentication: 251 azureTenantId: XXXXX 252 clientId: XXXXX 253 clientSecret: XXXXX 254 enabled: true 255 provider: azure which is perfectly fine and correct; however, when i access https://my-url/callback/oidc i get:

Failed to perform SSO callback. SSO is not enabled for protocol: oidc

do i have to specify env. vars to be added to a datahub front end in addition to options specified in datahub-frontend/values.yaml? Specifically, i see uri reference but no options for it in the values.yaml file. Pointers/ideas? thank you, Dima.

Hi Team, Trying to use GraphQL to search for entities. Is there a way we can search and just get a list of databases ?