Hey guys, hope you all are doing great there Here are 2 questions I am currently concerned with: 1. Just deployed datahub in k8s using https://github.com/acryldata/datahub-helm and I was wondering where are the credentials to login. 2. How am I supposed to install libraries? (for instance, let’s say I would like to install this module https://datahubproject.io/docs/generated/ingestion/sources/delta-lake/) Thanks in advance!

Bit of background - With our implementation, we need our deployment workflow to publish the avro schema to schema registry and are unable to use the GMS publish of the schema when it detects a difference. We recently were having an issue where the schema published by the workflow was different than what the GMS was publishing, so the GMS kept getting an error trying to publish the schema. The cause for the difference was special characters in the following files that the Github Actions runner was not generating correctly: MLModelFactors.pdl (’) MLFeatureDataType.pdl (—, 4 instances) EthicalConsiderations.pdl (–) We were able to get the runner to properly generate the characters by adding a file encoding flag to the Gradle build to use UTF-8 and are no longer having the issue, though it was a little confusing working through the issue. To prevent this from happening to others, can I create a PR replacing the special characters with standard ones?

Hi, I've created a view but am having trouble saving the filter match setting. When I change to "Any Filter" and click save, it doesn't actually save my setting. When I go back into the view after saving, it will be reverted back to "All filters". Maybe a bug?

Hi Team, Can we customized view for different users on datahub? thanks

Hello team. In the context of upgrade tests on datahub, I have to restore the mysql data (67Mb), which works without any problem. But as indicated here, https://datahubproject.io/docs/how/restore-indices/, I also need to restore the indices, with the datahub-restore-indices-adhoc job. On a node with 50GB of RAM, this job fails, with the following error:

java.lang.OutOfMemoryError: Java heap space

I am using datahub-helm 0.2.89 and I have to start from this version. Any idea ? Thanks.

Hello Team , we are trying to analyze few necessary KPI like "weekly active users" and was wondering is there anyway that i can get "weekly active users per country " ?

Hello Release v0.9.6 docker build is failing with below error: Commands Executed

cd datahub
git clone <https://github.com/datahub-project/datahub.git> .
git checkout tags/v0.9.6 -b datahub_v0.9.6

DATAHUB_VERSION=v0.9.6 COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f docker/docker-compose.yml -f docker/docker-compose.override.yml -p datahub build datahub-frontend-react

Error

Hi team, I’m trying to ingest usage data from snowflake through the UI unfortunately without any success. While looking at snowflake UI I can see the datahub’s queries and the produced rows but eventually get 0 assets. I looked up the logs and found a few connection retries (see the attached warning). I looked up solutions on the internet but couldn’t find anything. I’m using datahub version: 0.9.6 I would like to get your help. Thanks

Hello team, I have a small question about DataHub upgrades. Does the migration job need to be run every time we upgrade beyond version 0.8? If not, will it ever be needed again? I find it hard to find a definitive answer on that. Moreover, do we need to disable token authentication before running the job?

Hey, so I've been using

datahub

for local dev on my

M1

mac (using

--version=v0.9.3

with docker quickstart), and it's all running fine. However, I just got my hands on a rather lovely SBC from khadas (an Edge 2 pro - 16GB ram 8 cores arm64), and even though this is well within the docker specs, it fails to run (looks like mysql-setup and kafka-setup failed), any ideas? Logfile in thread

Hi we're having troubles with Postgres ingests, they fail with error

"ImportError: cannot import name 'Distribution' from 'pkg_resources' (unknown location)\n",

. Any idea why that might be happening? They used to ingest successfully before, and the failes started about two weeks ago or so

Hi, We have deployed the Datahub on GKE and flagged log4j vulnerability on the Neo4j : The library

org.apache.logging.log4j:log4j-core

version

2.14.0

was detected in

Maven library manager

located at

/var/lib/neo4j/lib/neo4j-logging-4.2.4.jar

and is vulnerable to

CVE-2021-45105

, which exists in versions

>= 2.13.0, < 2.17.0

. The vulnerability was found in the Github Security Advisory with vendor severity:

High

(NVD severity:

Medium

). This vulnerability has a known exploit available. Source: Github. The vulnerability can be remediated by updating the library to any of the following versions

2.3.1, 2.12.3, 2.17.0

or higher, using

mvn versions:use-latest-releases -Dincludes=org.apache.logging.log4j:log4j-core

. The library is associated with the technology

Apache Log4j

. Is there a fix as part of deployment ?

Hi, We are pushing our presto and hive tables on datahub. Is there any way by which we will get an idea how many tables are accessed a lot on presto. Can we push access information of presto or hive table on datahub. Thanks

Hi Folks, I am getting an error with a GraphQL mutation that has worked historically. I tested a few other queries without issue. This is the error that is returned

{
  "errors": [
    {
      "message": "An unknown error occurred.",
      "locations": [
        {
          "line": 4,
          "column": 3
        }
      ],
      "path": [
        "createGlossaryTerm"
      ],
      "extensions": {
        "code": 500,
        "type": "SERVER_ERROR",
        "classification": "DataFetchingException"
      }
    }
  ],
  "data": {
    "createGlossaryTerm": null
  },
  "extensions": {}
}

This is the query and variables

#Create glossary term

mutation CreateGlossaryTerm($name:String!, $description: String, $urn: String){
  createGlossaryTerm(input: {name: $name, description: $description, parentNode: $urn})
}

#Variables
{
  "name":"firstcustkey",
  "description": " Initial CustKey for a Line.",
  "urn": "urn:li:glossaryNode:821e6ed3-756b-43da-805c-ce7ed1d02ee8"
}

This same query was working fine yesterday so I am not sure what changed. I have confirmed that the parentnode exists.

Hi team. I’m having trouble working with

@Relationship

in datahub. Please see thread for details. Please advise.

Hi team, I am editing the Kafka topics inside the kafka-setup and schema-registry but the bottom 4 topics are not being correctly set (using the default values). I am following the naming procedure that appears in the datahub-gms section of https://datahubproject.io/docs/how/kafka-config/#metadata-service-datahub-gms Note that the schema-registry extra-envs are identical to these ones:

kafkaSetupJob:
  enabled: true
  image:
    repository: linkedin/datahub-kafka-setup
    # tag: "v0.9.6" # defaults to .global.datahub.version
  resources:
    limits:
      cpu: 500m
      memory: 1024Mi
    requests:
      cpu: 300m
      memory: 768Mi
  securityContext:
    runAsUser: null
  podSecurityContext:    
    fsGroup: null
    runAsUser: null
  podAnnotations: {}
  extraEnvs:
    - name: DATAHUB_USAGE_EVENT_NAME
      value: "DataHub_UsageEvent_JSON_v1"
    - name: PLATFORM_EVENT_TOPIC_NAME
      value: "DataHub_PlatformEvent_JSON_v1"
    - name: METADATA_CHANGE_EVENT_NAME
      value: "DataHub_MetadataChangeEvent_JSON_v4"
    - name: METADATA_AUDIT_EVENT_NAME
      value: "DataHub_MetadataAuditEvent_JSON_v4"
    - name: FAILED_METADATA_CHANGE_EVENT_NAME
      value: "DataHub_FailedMetadataChangeEvent_JSON_v4"
    - name: METADATA_CHANGE_PROPOSAL_TOPIC_NAME
      value: "DataHub_MetadataChangeProposal_JSON_v1"
    - name: METADATA_CHANGE_LOG_VERSIONED_TOPIC_NAME
      value: "DataHub_MetadataChangeLog_Versioned_JSON_v1"
    - name: FAILED_METADATA_CHANGE_PROPOSAL_TOPIC_NAME
      value: "DataHub_FailedMetadataChangeProposal_JSON_v1"
    - name: METADATA_CHANGE_LOG_TIMESERIES_TOPIC_NAME
      value: "DataHub_MetadataChangeLog_Timeseries_JSON_v1"

Hey folks! Is there a specific version of Python we should use when building datahub locally i.e.

./gradlew build

?

Hi team, I have caught this error while testing the running of quickstart/docker-compose.quickstart.yml compose file. Log from container named broker said that the cluster id does not match with what is stored in Zookeeper. I think this id is internally generated because it changes each time I re-run the compose file. I would like to ask for help about fixing this issue. Please advice.

Hi team, I got this 'schema-registry is not running' error when running the quickstart. Please any idea as to why this is happening and how can it can be resolved?

Hi. I'm having troubles with my Hive ingestion, in does not ingest views, even though I was told it should do so by default. I'm on 0.9.6. Any ideas why that might be happening?

Hi team, another question for me. Is it necessary to run the docker containers with volume mounted ? I am having an argument about whether volumes is ultimately required for the deployment on docker. I don't really think volumes are that important since log files are stored mostly.

Hello, would anyone be able to assist me in the steps to go from my localhost to my staging environment? I did a datahub init with my url and token, but it says its unable to connect. Is there something I missed? And a way to check my connection without ingesting data?

Hi all, I'm experiencing problems with managing groups and authenticating over OIDC with Azure AD. So the first thing I did was create an ingestion recipe to ingest users from AD as well as map respective groups to each user. After this I set up the sso through the docker environment file and restarted the container. When I sign in through sso however, the AD groups that were mapped to the user and ingested into datahub are removed. I've made a bug report on github but I was wondering if anyone else has seen this before. I've tested this on v0.9.6, v0.9.5, and v0.9.3. (v0.9.4 has a looping error where you cannot autheticate through OIDC)

Hi all, I upgraded to v0.9.6 using helm I can only get it to work when actions pod pointing to version 0.8.43, and I can’t get it to work with any greater version. can anyone assist please? also what if I change it in UI under advanced and set custom cli version to be 0.9.6?

Hi Team.. I believe so many of you used Azure Kubernetes for deploying datahub.. i am struggling to create a secret on cluster.. I am automating the datahub instance creation using terraform.. Idea is to create a module.. Kubectl commands works with AAD but not with Service Principal.. Jenkins job is setup with Service Principal.. Any idea how you guys resolved the issue

Hi Team, I tried upgrading to the latest version and the permissions broke. I use to ingest Azure groups and assign permissions to these groups. The

groups

is blank now. I tried using datahub user to assign my user Admin and then login using my user. I still did not have admin privileges.

Hi Team, datahub-gms is not starting up with

datahub docker quickstart

on M1 Mac OS. is this already a known issue? Any suggestion or work around?

Not sure if this is known but trying to roll forward to 0.9.6.1 and the postgresql docker setup image isnt available for the patch release https://hub.docker.com/r/acryldata/datahub-postgres-setup/tags. thanks for help in advanced. Mysql for reference

Hi guys. I have a weird thing with some of my Hive table, som of them are not displayed in dataset/datasourse wiew, but if I search them by name - they are found in search results. They exhist in datahub, they can be accesed by direct link, but they are not displayed in a list of tables when I open the data container (schema) they are in. It happens accross different data schemas, for dosens of tables. This seems like a bit issue to me, since it affect the basic fucbtion of a catalog - to provide a way to browse through all datasets and see what kind of tables are there. But now it looks like a significant amount of tables can be found in Datahub only I I already have information about their name or content. It started since we firt installed Datuhub and didn't disapper with any upgrades we had. We're now on 0.9.6. Our DevOps suggested soft-deleting such tables when we found them, and after the next ingest they appear normally, visible when browsing a dataset. But I don't see this as an effective solution, cause we have to manually find all problematic tables (and they can be endless), and also soft-deleting erases data descriptions and links with glossaries etc (I guess? correct me if it isn't so). Any idea what else should we look into to find what cause some tables to not display in dataset view, and solve the issue that's causing it, not just single consequences?

Just posted this issue. Not sure if this has bit anyone else: https://github.com/datahub-project/datahub/issues/7107