Hello all, I was wondering if it is possible to bring in Okta groups without using the ingestion piece? My team was trying to do it directly from the app, but that has not been successful so far.

Hi DH team, When deploying prerequisites, from the helm chart, does it mount PV automatically? When executing

kubectl get pv/pvc

I found PV for several services that I’m deploying, but I don’t see any code reference for this, except:

# # Request smaller persistent volumes.
# volumeClaimTemplate:
#   accessModes: ["ReadWriteOnce"]
#   storageClassName: "standard"
#   resources:
#     requests:
#       storage: 100M

Hi team, I am having trouble with

@RelationShip

I am trying to get data from One to Many Entity Relationship inside Java.

Hello Team, I'm trying to build custom frontend image. However the build returns a below error Can someone help me ?

=> ERROR [prod-install 1/5] COPY ./datahub-frontend.zip /                                                 0.0s
------
 > [prod-install 1/5] COPY ./datahub-frontend.zip /:
------
failed to compute cache key: "/datahub-frontend.zip" not found: not found

For all the change events in datahub ..i want to capture all the schema changes and lineage changes into a kafka topic in confluent cloud.. i was able to create a actions framework app ..but the messages were not populated in kafka topics

Hi. I'm trying to set up datahub on AWS EKS as per the documentation here: https://datahubproject.io/docs/deploy/aws/ I've deployed the load balancer, got a certificate, and re-deployed the datahub install with updated values.yaml as per documentation. I get a timeout whenever I try to access datahub via the host address, although I can access the web front end (over the public internet) via the service specific url for datahub-datahub-frontend. I would be grateful for any suggestions for troubleshooting. Further details regarding logging, etc, contained in thread. The only thing that stands out to me is that kubectl describe ingress datahub-datahub-frontend returns the following: "ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>) ". Thank you, Ben

Getting this error in gms service

ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.22023/03/21 05:03:33 Received signal: terminated
2023/03/21 05:03:34 Command exited with error: exit status 143

any idea?

Hello, I had working deployment of Datahub in EKS. I did nodes update for cluster. After that I have redeployed Datahub via helm, as not all pods were correctly running after nodes upgrade. Now I can login to Datahub via SSO like earlier, but I don't see any users from sso that I can give permissions. Looks like new users are there to select, but users which existed before upgrade are not visible. Evan my own user which I'm logged in is not displayed. As well I cannot search old data (before nodes upgrade) in search bar. Only new things appear there. Any clue what that might be ?

Hi, Im running into this error, when trying to access PowerBI: HTTPError: 401 Client Error: Unauthorized for url: https://api.powerbi.com/v1.0/myorg/groups I searched a bit and I do not have a personal access token yet, as I wouldnt know where to put it. Could you please help me?

Hi. after an upgrade from

v0.8.44

->

v0.10.0

gms stopped working. Details in 🧵

Hi, I'm trying to get data from PowerBI. I connected succesfully but ingestion fails. I see following warnings:

Non relational data-source found = {'datasourceType': 'Extension', 'connectionDetails': {'path': '...', 'kind': 'Snowflake'}, 'datasourceId': '...', 'gatewayId': '...'}

This looks strange

[2023-03-21 12:37:24,095] INFO     {datahub.ingestion.reporting.file_reporter:52} - Wrote UNKNOWN report successfully to <_io.TextIOWrapper name='/tmp/datahub/ingest/3865c265-8540-4f4c-be96-f59e1d2056af/ingestion_report.json' mode='w' encoding='UTF-8'>

and in the end:

[2023-03-21 12:37:24,385] ERROR    {datahub.entrypoints:213} - Command failed: Failed to fetch reports from power-bi for

When I connected to the workspace without datasets with DirectQuery it finished succesfully

I am having trouble with ingesting athena tables; specifying tables in table pattern but trying to ingest everything and failing on ingesting column information ...

is the Athena recipe working at all?

Hey, guys I got Mac M1 , tried

datahub docker quickstart
datahub docker quickstart --arch m1
datahub docker quickstart --version v0.9.6.1

I am still getting

Unable to run quickstart - the following issues were detected:
- schema-registry is not running
- datahub-gms is still starting
- mysql-setup is not running
- elasticsearch is running by not yet healthy

my docker version is:

Docker version 20.10.23, build 7155243
Docker Compose version v2.15.1
Docker Desktop version v4.17.0

I really need your help

👋 Hello team! I’m struggling to come up with a graphql query to retrieve dataset with some health problem… has someone already used this kind of query?

Hi All - Is there a way to get the FineGrainedLineage out of a GraphQL query? I can see the lineage when looking at the column in the UI, so I'm assuming GraphQL is being used to create that, but when I use

searchAcrossLineage

and pass in a starting

schemaField

urn, I only get the datasets back. It doesn't tell me which columns in those datasets are linked to the starting

schemaField

.

Hello Team, Any idea why this error shows up ? Datahub version - v0.9.6.1 running on EC2 instance

Hello Team, How do I get the task name to show up instead of urn ?

datajob_input_output_mcp = MetadataChangeProposalWrapper(

entityUrn=builder.make_data_job_urn(

orchestrator=job_platform, flow_id=job_name, job_id=task_name

),

aspect=datajob_input_output,

)

hello team, if i want to add a particular field of an entity as filter/facet on search page, how would I do so?

Hey! We've run into an issue with v0.10.0 and graphQL. When running the

searchAcrossLineage

on datasets that have a lot of downstream links we end up getting a

DataFetchingException

which seems to be related to

{
  "error": {
    "root_cause": [
      {
        "type": "max_bytes_length_exceeded_exception",
        "reason": "max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 45576"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "datasetindex_v2_1668526466773",
        "node": "W7nwRsS-SlOu01ySlY3N-w",
        "reason": {
          "type": "max_bytes_length_exceeded_exception",
          "reason": "max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 45576"
        }
      }
    ]
  },
  "status": 500
}

Since the upgrade required a reindexing in elasticsearch we're not exactly sure what we should be doing to address this new issue. Wondering if anyone has any ideas! It only pops up for datasets that have a lot of downstream lineage links.

Hi Team, Trying to use graphql query to search datasets using keywords , tags and domains using below query. How to filter using multiple keywords ?

{
  searchAcrossEntities(input: {
    types: [DATASET], 
    query: "*edi*",
    orFilters: [
      {
        and: [
          {
            field: "tags",
            values: ["urn:li:tag:phi"],
            condition: EQUAL
          },
          {
            field: "domains",
            values: ["urn:li:domain:hsdjfsdf"],
            condition: EQUAL
          }
        ]
      }
    ]
  }) {
    searchResults {
      entity {
        urn
        type
      }
    }
  }
}

Hi All, when I run pipeline for oracle getting system tables as well. Is it possible to add any flag to pull only user tables?

Hi guys, I’m trying to run

start:mock

for web-ui but I face to this issue. Any glues?

validate.mjs:87 Uncaught Error: Unknown type "UpdateLineageInput". Did you mean "UpdateNameInput", "UpdateViewInput", "LineageInput", "UpdateEmbedInput", or "UpdateMediaInput"?

Unknown type "EntityPrivileges". Did you mean "Privileges"?

Unknown type "EntityPrivileges". Did you mean "Privileges"?

Unknown type "EntityPrivileges". Did you mean "Privileges"?

Unknown type "EntityPrivileges". Did you mean "Privileges"?

Unknown type "EntityPrivileges". Did you mean "Privileges"?

Unknown type "EntityPrivileges". Did you mean "Privileges"?
    at assertValidSDL (validate.mjs:87:1)
    at buildASTSchema (buildASTSchema.mjs:29:1)
    at Module.<anonymous> (schema.ts:14:1)
    at ./src/graphql-mock/schema.ts (schema.ts:15:1)
    at __webpack_require__ (bootstrap:856:1)
    at fn (bootstrap:150:1)
    at Module.<anonymous> (schema.ts:15:1)
    at ./src/graphql-mock/server.ts (server.ts:84:1)
    at __webpack_require__ (bootstrap:856:1)
    at fn (bootstrap:150:1)
    at Module.<anonymous> (createServer.ts:8:1)
    at ./src/graphql-mock/createServer.ts (createServer.ts:13:1)
    at __webpack_require__ (bootstrap:856:1)
    at fn (bootstrap:150:1)
    at Module.<anonymous> (verticalogo.png:1:1)
    at ./src/index.tsx (index.tsx:18:1)
    at __webpack_require__ (bootstrap:856:1)
    at fn (bootstrap:150:1)
    at 1 (index.ts:21:1)
    at __webpack_require__ (bootstrap:856:1)
    at checkDeferredModules (bootstrap:45:1)
    at Array.webpackJsonpCallback [as push] (bootstrap:32:1)
    at main.chunk.js:1:87

Hi, DataHub Team 👋 https://datahubproject.io/docs/lineage/airflow/#using-datahubs-airflow-lineage-plugin I have completed the airflow integration as above link. As shown in thread, I can see the Airflow card in Platforms, but not the Pipelines card in “Explore your Metadata”. I didn’t find any error logs in GMS/ FRONTEND. Where should I look? The version I am using is v0.9.2. Thanks

hi, team, getting started with DataHub, is there any way to process file like 'ibe-order-channel+2+0027146031.json.gzip'?

What is the recommended version of jdk, node and yarn for building gms and frontend. Is there a way to check this for each release ? Commands used for build

./gradlew  :datahub-frontend:docker
./gradlew :metadata-service:war:build

# Datahub version: v0.9.6.1 Problem: yarn tests are failing with node v14.15.5 and yarn 3.5.0. Also if i download the source code of a release and execute frontend build then is see this error/warning. Can this be ignored ?

> Configure project :metadata-auth:auth-api
Caused by: java.lang.IllegalArgumentException: Cannot find '.git' directory

Hey is there any chance to update name of the already created group using graphql? From UI point of view we can do it but when Im trying to do this using API we can only change

description

slack

email

for reference

Hi team. I'm running the quickstart image, and after a restart, datahub-gms is went unhealthy, and it is in that state ever since. In the logs I found this: at java.base/java.lang.Thread.run(Thread.java:829) Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/datahubpolicyindex_v2/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true], status line [HTTP/1.1 400 Bad Request] {"error":{"root_cause":[{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":943}],"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [must]","caused_by":{"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [should]","caused_by":{"type":"x_content_parse_exception","reason":"[1:943] [bool] failed to parse field [should]","caused_by":{"type":"parsing_exception","reason":"[term] query does not support [case_insensitive]","line":1,"col":943}}}},"status":400} Is it possible to fix this somehow?

Hi everyone! I'm new to this so keep that in mind! 😄 I have set up an Azure VM that i've installed docker and K8s on to run datahub locally on. Everything has been going well until i try to run: helm install datahub datahub/datahub My guess that there is something blocking in the network but i figured i'd post here to see if anyone has ran in to the same issue! I get this error when running: PS C:\Windows\system32> helm install datahub datahub/datahub --debug install.go194 [debug] Original chart version: "" install.go211 [debug] CHART PATH: C:\Users\ADMCHR~1\AppData\Local\Temp\helm\repository\datahub-0.2.160.tgz client.go477 [debug] Starting delete for "datahub-elasticsearch-setup-job" Job client.go133 [debug] creating 1 resource(s) client.go703 [debug] Watching for changes to Job datahub-elasticsearch-setup-job with timeout of 5m0s client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: ADDED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0 client.go731 [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED client.go770 [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 2, jobs succeeded: 0 Error: INSTALLATION FAILED: failed pre-install: timed out waiting for the condition helm.go84 [debug] failed pre-install: timed out waiting for the condition INSTALLATION FAILED main.newInstallCmd.func2 helm.sh/helm/v3/cmd/helm/install.go:141 github.com/spf13/cobra.(*Command).execute github.com/spf13/cobra@v1.6.1/command.go:916 github.com/spf13/cobra.(*Command).ExecuteC github.com/spf13/cobra@v1.6.1/command.go:1044 github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.6.1/command.go:968 main.main helm.sh/helm/v3/cmd/helm/helm.go:83 runtime.main runtime/proc.go:250 runtime.goexit runtime/asm_amd64.s:1571 PS C:\Windows\system32>