still not working can someone point me at a reference

Hello, I am ingesting users and their groups to my datahub instance and I have some problems, since according to this: https://datahubproject.io/docs/graphql/objects/#corpgroupinfo corpgroupinfo is deprecated and I understand GroupMembership aspect of corpuser entity should be used to assign user to groups. I wonder if there is a way to assign admin status to some of the groups user? Previously it was achievable via admins property of corpgroupinfo.

Hey! I am trying to ingest some custom

com.linkedin.metadata.snapshot.DataJobSnapshot

using rest api, but get an error on validating the

com.linkedin.datajob.DataJobInfo

aspect:

[HTTP Status:400]: Parameters of method 'ingest' failed validation with error 'ERROR :: /entity/value/com.linkedin.metadata.snapshot.DataJobSnapshot/aspects/1/com.linkedin.datajob.DataJobInfo/type :: union type is not backed by a DataMap or null

I can’t exclude

type

as it’s not optional. If I exclude the entire aspect altogether ingestion works, but the task looks ugly in the UI. Running

0.8.17

. What am I missing here?

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [https://opensearch:9200], URI [/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"datahub_usage_event","node":"IuZ7a2BGSmSWZ4jN_vU8IA","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.","caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}},"status":400}

Is there a way to delete an aspect for all urns?

Hi everyone! I have a problem wish

datahub docker quickstart

elasticsearch-setup faild:

2021/12/09 10:00:33 Received 503 from <http://elasticsearch:9200>. Sleeping 1s
2021/12/09 10:00:34 Timeout after 2m0s waiting on dependencies to become available: [<http://elasticsearch:9200>]

then elasticsearch:7.9.3 Up 9 minutes (unhealthy) Frontend didnt work with error:

Caused by: java.lang.RuntimeException: Failed to generate session token for user

VM: 8 cpu 8 ram 30 GB Debian 10. Help plz)

Not sure if this is a bug or a feature, but when your run

datahub ingest rollback --run-id some-run-id

, all the aspects (like

editableDatasetProperties

and I assume all the other ones that were entered by users) of the urns from that run get deleted. The way I see it they were not ingested with any run and it can also potentially be data that people don’t want to lose.

Hey everyone! I used to launch DataHub with the

datahub docker quickstart

command, but am now trying to get my Okta OIDC authentication for the datahub-frontend set up so that new logins can create new DataHub user accounts. I've configured everything correctly on Okta's side, but to the best of my understanding, I need to launch DataHub in a different way to get the authentication working properly. I've done

docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml  up datahub-frontend-react

(the port for Neo4j is currently occupied on the server I'm running on) after executing the quickstart, but upon trying to access DataHub in a browser, I am met with a vague error. Could someone help me out with the steps needed to have a valid auth and new user flow?

Hello everyone!! I have a problem trying to import SQL Profile from Postgres to Datahub. When I execute the command

datahub ingest -c ./datahub_postgres_local.yml

I get an error:

psycopg2.errors.UndefinedFunction: operator does not exist: json = unknown
LINE 68: ...count(*) AS element_count, sum(CASE WHEN (address IN (NULL) ...
                                                              ^
HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.

Seems like

json

type is not supported. Any idea to skip this? Thanks a lot!

Hi Everyone, I am trying to clear all the dataset from Datahub. What all do I need to clear ?

Hi Everyone! I've deployed datahub on AWS Fargate using AWS RDS, AWS OpenSearch, AWS Glue and AWS MSK to host datahub prerequistes (a.k.a staging). I wrote an integration test to validate that data ingestion works properly (inspired from example_to_datahub_rest.yml. The test asserts that the proper data is ingested using the graphql endpoint. The test is successful when targeting a datahub environment provisioned with datahub quickstart (i.e. local datahub environment running via docker compose) as well as when targeting the datahub environment deployed on AWS Fargate. However when I use the UI to load the

/browse/dataset

endpoint, the page shows 0 entities for the staging environment while the same endpoint for the quickstart environment lists 7 datasets. Do you have any pointers on further debugging this issue?

I followed the following steps: 1. First, I removed the mysql container from the yml file, since I would be using my own. 2. Then I changed the credentials under environment for both mysql-setup, datahub gms (ebean fields) in the yml. 3. Then I ran datahub docker quickstart --quickstart-compose-file file.yml Here's the file (I have used <> wherever I supplied my own credentials and edited). ``````

Hey team, what’s the purpose of this platform_name.isalpha() check? When the platform_name is

s3

, it logs warning messages like below.

..
..
WARNING: improperly formatted data platform: s3
WARNING: improperly formatted data platform: s3
WARNING: improperly formatted data platform: s3
..
..

I just learned that datahub now supports redshift lineage. How do I update my current datahub to make use of this feature?

V0.8.18 redshift error: I just updated to the latest version, I got the error when ingesting my previous yml file for redshift.

File "/opt/miniconda3/lib/python3.8/site-packages/datahub/emitter/rest_emitter.py", line 107, in test_connection
    102  def test_connection(self) -> None:
    103      response = self._session.get(f"{self._gms_server}/config")
    104      response.raise_for_status()
    105      config: dict = response.json()
    106      if config.get("noCode") != "true":
--> 107          raise ValueError(
    108              f"This version of {__package_name__} requires GMS v0.8.0 or higher"
    ..................................................
     self = <datahub.ingestion.graph.client.DataHubGraph object at 0x7fc6fc416280>
     response = <Response [200]>
     self._session.get = <method 'Session.get' of <requests.sessions.Session object at 0x7fc6fc4163a0> sessions.py:534>
     response.raise_for_status = <method 'Response.raise_for_status' of <Response [200]> models.py:918>
     config = {'compatibilityLevel': 'BACKWARD'}
     response.json = <method 'Response.json' of <Response [200]> models.py:874>
    ..................................................

---- (full traceback above) ----
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/entrypoints.py", line 102, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
File "/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
File "/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/opt/miniconda3/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 141, in wrapper
    res = func(*args, **kwargs)
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 76, in run
    pipeline = Pipeline.create(pipeline_config, dry_run, preview)
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 143, in create
    return cls(config, dry_run=dry_run, preview_mode=preview_mode)
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 103, in __init__
    self.ctx = PipelineContext(
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/ingestion/api/common.py", line 38, in __init__
    self.graph = DataHubGraph(datahub_api) if datahub_api is not None else None
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/ingestion/graph/client.py", line 39, in __init__
    self.test_connection()
File "/opt/miniconda3/lib/python3.8/site-packages/datahub/emitter/rest_emitter.py", line 107, in test_connection
    raise ValueError(

ValueError: This version of acryl-datahub requires GMS v0.8.0 or higher

Hi community! Any recommended actions following the recent Log4j disaster?

Hey community! I have a problem trying to Ingest Google Groups via OIDC. ⚠️*Problem*: Groups not ingested into Datahub Context: Datahub Version: 0.8.17 OIDC configuration:

extraEnvs:

- name: AUTH_JAAS_ENABLED

value: "false"

- name: AUTH_OIDC_ENABLED

value: "true"

- name: AUTH_OIDC_CLIENT_ID

valueFrom:

secretKeyRef:

name: datahub

key: OIDC_CLIENT_ID

- name: AUTH_OIDC_CLIENT_SECRET

valueFrom:

secretKeyRef:

name: datahub

key: OIDC_CLIENT_SECRET

- name: AUTH_OIDC_DISCOVERY_URI

value: "<https://accounts.google.com/.well-known/openid-configuration>"

- name: AUTH_OIDC_BASE_URL

value: "<https://XXXXXXX>"

- name: AUTH_OIDC_SCOPE

value: "openid email profile"

- name: AUTH_OIDC_USER_NAME_CLAIM

value: "email"

- name: AUTH_OIDC_USER_NAME_CLAIM_REGEX

value: "([^@]+)"

- name: AUTH_OIDC_JIT_PROVISIONING_ENABLED

value: "true"

- name: AUTH_OIDC_PRE_PROVISIONING_REQUIRED

value: "false"

- name: AUTH_OIDC_EXTRACT_GROUPS_ENABLED

value: "true"

- name: AUTH_OIDC_GROUPS_CLAIM

value: "groups"

Note: The Login using OIDC is working as expected. Any help here? Thanks! 🤶

Hey community, recently i am getting follow errors when running

./gradlew build

, i have checked that my python version is 3.8. Any ideas how to fix this? Thanks.

Hi General - Trying to use GraphQL for creating Policy using this link https://datahubproject.io/docs/graphql/mutations#createpolicy but unable to add actors & resources. How do I add these ( came up with this so far ) ?

mutation createPolicy{
     createPolicy (
      input: {
         type: PLATFORM,
         name: "TestPolicy",
         state: ACTIVE,
         description: "Testing Policy via Graphiql",
         #resources:ResourceTypeFilterInput(resources:allResources),
         privileges:["MANAGE_POLICIES",
        						"MANAGE_USERS_AND_GROUPS",
        						"VIEW_ANALYTICS"]
        
        
      }
    )
  }

Hi all! I saw this (the thread shared below) and was wondering if the framework for restoring all of the DataHub storage components had been released yet? I couldn't seem to find any guides in the DataHub documentation besides the Search & Graph Index Restoration EDIT: I discovered that when I had DataHub running and executed

mysql --host=127.0.0.1 --port=3306 -u datahub -p datahub

, I was able to enter the mysql container for DataHub. It contained two tables,

metadata_aspect_v2

and

metadata_index

. Would these following steps be all that I need to backup in order to restore DataHub completely in case of dire circumstances where current volumes are removed or corrupted? Backup:

mysqldump --host=127.0.0.1 --port=3306 -u datahub -p --all-databases --no-tablespaces  > metadata.sql

Restore:

mysql --host=127.0.0.1 --port=3306 -u datahub -p < metadata.sql

I tried to reinstall datahub, and after running

datahub docker quickstart

Both my mac and linux box return this error:

CalledProcessError: Command '['docker-compose', '-f', '/var/folders/_6/ql7t0n_j2zxd7r_wbrwsgptc0000gq/T/tmphpw6pyai.yml', '-p', 'datahub', 'pull']' returned non-zero exit status 1.

I’ve ingested a bunch of hive datasets and when I browse in the UI my gms container logs are full of:

12:38:44.067 [qtp1504109395-479] INFO  c.l.m.filter.RestliLoggingFilter - GET /aspects/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2Csome_db.some_table%2CPROD%29?aspect=subTypes&version=0 - get - 404 - 1ms
12:38:44.067 [qtp1504109395-479] ERROR c.l.m.filter.RestliLoggingFilter - null

It does not seem to affect anything - everything works fine as far as I can tell. Running

0.8.17

. Is this a known issue or am I doing something wrong?

we use lineage error ,Such as: But sometimes it works. datahub-gms error log:

Hey, we are having some issues because we have an (actually multiple) entities which is loading slow because it is fetching too much lineage. Basically this query

query getDataset {
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:redshift,mydataset,PROD)") {
    downstreamLineage {
        entities {
          entity {
            ... on Dataset {
                downstreamLineage {
                  entities {
                    entity {
                      urn
                    }
                  }
                }
                upstreamLineage {
                    entities {
                      entity {
                        urn
                      }
                  }
                }
              }
            }
          }
        }
    }
}

Is really slow (10+ seconds). Is there something that might improve this? We use Neo4j on the backend, so we were wondering if switching to the ES backend is faster or not? Or if you have any tips in debugging this. (Neo4j is not hitting capacity limits, GMS only very rarely.)

Hey I have a bunch of

com.linkedin.metadata.snapshot.DataJobSnapshot

that ties together a bunch of hive tables. Some of the data jobs have a lot of

inputDatasets

( > 100). Yet somehow in the UI only 99 are showed although I can see through dev tools that graphql query returns two arrays.

Hello all, I am trying to get redshift-lineage working but there seems to a problem I am using an superuser for database ingestion. I do see this warning if it can help troubleshooting :

WARNING: dev.pg_catalog.stv_wlm_query_state missing table
WARNING: dev.pg_catalog.stv_wlm_classification_config missing table
WARNING: dev.pg_catalog.stv_wlm_service_class_config missing table
WARNING: dev.pg_catalog.stv_wlm_service_class_state missing table

Thanks in advance!

Hello, I am trying to get started with Datahub following these instructions https://datahubproject.io/docs/quickstart Unfortunately, when I run "datahub docker quickstart" I get:

I still have the above issue.

Hi all, I am having a hard time understanding what is wrong here. I am trying to configure the frontend to use oidc and Azure AD as its OIDC provider. Here is the frontend log:

15:57:20 [main] INFO  play.core.server.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9002
16:00:50 [application-akka.actor.default-dispatcher-16] WARN  p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check
16:00:50 [application-akka.actor.default-dispatcher-9] WARN  p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check
16:00:50 [application-akka.actor.default-dispatcher-30] WARN  p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check
16:00:51 [application-akka.actor.default-dispatcher-20] WARN  p.api.mvc.LegacySessionCookieBaker - Cookie failed message authentication check
16:00:54 [application-akka.actor.default-dispatcher-9] WARN  o.p.o.profile.creator.TokenValidator - Preferred JWS algorithm: null not available. Using all metadata algorithms: [RS256]
16:00:54 [application-akka.actor.default-dispatcher-9] ERROR auth.sso.oidc.OidcCallbackLogic - Unable to renew the session. The session store may not support this feature
16:00:57 [application-akka.actor.default-dispatcher-22] ERROR auth.sso.oidc.OidcCallbackLogic - Unable to renew the session. The session store may not support this feature

If I start from an incognito window (with no cookies), I don't get the `Cookie failed message`errors. I am using the latest helm chart and deploying on Azure AKS. Here are my

extraEnvs

values:

datahub-frontend:
    ingress:
      enabled: true
      hosts:
        - host: <http://datahub.mydomain.com|datahub.mydomain.com>
          paths:
            - "/"
      tls:
        - secretName: mydomain-tls
          hosts:
            - <http://datahub.mydomain.com|datahub.mydomain.com>
    extraEnvs:
        # Required Configuration Values for OIDC:
      - name: AUTH_OIDC_ENABLED
        value: "true"
      - name: AUTH_OIDC_CLIENT_ID
        value: "..."
      - name: AUTH_OIDC_CLIENT_SECRET
        value: "..."
      - name: AUTH_OIDC_DISCOVERY_URI
        value: "<https://login.microsoftonline.com/><tenantID>/v2.0/.well-known/openid-configuration"
      - name: AUTH_OIDC_BASE_URL
        value: "<https://datahub.mydomain.com>"

and the .

well-known/openid-configuration

from Azure:

{
  "token_endpoint": "<https://login.microsoftonline.com/><tenantID>/oauth2/v2.0/token",
  "token_endpoint_auth_methods_supported": [
    "client_secret_post",
    "private_key_jwt",
    "client_secret_basic"
  ],
  "jwks_uri": "<https://login.microsoftonline.com/><tenantID>/discovery/v2.0/keys",
  "response_modes_supported": [
    "query",
    "fragment",
    "form_post"
  ],
  "subject_types_supported": [
    "pairwise"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ],
  "response_types_supported": [
    "code",
    "id_token",
    "code id_token",
    "id_token token"
  ],
  "scopes_supported": [
    "openid",
    "profile",
    "email",
    "offline_access"
  ],
  "issuer": "<https://login.microsoftonline.com/><tenantID>/v2.0",
  "request_uri_parameter_supported": false,
  "userinfo_endpoint": "<https://graph.microsoft.com/oidc/userinfo>",
  "authorization_endpoint": "<https://login.microsoftonline.com/><tenantID>/oauth2/v2.0/authorize",
  "device_authorization_endpoint": "<https://login.microsoftonline.com/><tenantID>/oauth2/v2.0/devicecode",
  "http_logout_supported": true,
  "frontchannel_logout_supported": true,
  "end_session_endpoint": "<https://login.microsoftonline.com/><tenantID>/oauth2/v2.0/logout",
  "claims_supported": [
    "sub",
    "iss",
    "cloud_instance_name",
    "cloud_instance_host_name",
    "cloud_graph_host_name",
    "msgraph_host",
    "aud",
    "exp",
    "iat",
    "auth_time",
    "acr",
    "nonce",
    "preferred_username",
    "name",
    "tid",
    "ver",
    "at_hash",
    "c_hash",
    "email"
  ],
  "kerberos_endpoint": "<https://login.microsoftonline.com/><tenantID>/kerberos",
  "tenant_region_scope": "NA",
  "cloud_instance_name": "<http://microsoftonline.com|microsoftonline.com>",
  "cloud_graph_host_name": "<http://graph.windows.net|graph.windows.net>",
  "msgraph_host": "<http://graph.microsoft.com|graph.microsoft.com>",
  "rbac_url": "<https://pas.windows.net>"
}

The error message

Unable to renew the session. The session store may not support this feature

is not helping me in this case... Any idea what is wrong with my setup?

Quick question: How’s the progress of Data Profiling and Dataset Previews?