Does DataHub use Kafka for anything internally, or is it just for ingestion? Maybe a dumb question, but wondering if we can get away without it and integrate directly via HTTP for all calls, since I see both the MCE & MAE consumers are optional.

hi,team.About blood Lineage display In the acquisition of consanguinity, although my relationship has dataset&dashbarod&dataflow, can I only acquire the consanguinity of a single resource. eg: Scenario A -I only show the blood relationship between datasets。 Scenario B -I only show the blood relationship between dataflow Scenario C- dataset+dataflow Does the current blood relationship query interface support？

This message was deleted.

Hello. After deploying Datahub to AWS using the provided helm charts and exposing the endpoint through a load balancer I am curious are the IP addresses that the k8s-default-datahubd-……. address resolves to static or dynamically generated?

Does anyone using google sso know how exactly to grant specific people an admin role?

re: storing state, I take it both Elastic/Open Search and a relational database need to be snapshotted before updating?

Hello everyone, I'm having a problem with the datahub actions pod. I've noticed that the ingestion pipelines weren't running so I checked my kubernetes pods by doing

kubectl get pods

and noticed that the actions pod had the status ImagePullBackOff. After deleting the pod to try to rebuild it I've checked the logs and got the error Error from server (BadRequest): container "acryl-datahub-actions" in pod "datahub-acryl-datahub-actions-7667977f8b-v6vhp" is waiting to start: image can't be pulled. Does anyone know how to fix this issue?

Hello all! I’m currently deploying Datahub using EKS and RDS for Postgres as DB. Due to compliance reasons we can’t use the default

public

schema, and changing the default schema is not viable for the same reason. Is there a way to easily change the schema setting in datahub.?

hello team, I used minikube to deploy datahub, and got some errors, does anyone know how to solve it? I refer to the official documentation for deployment, minikube version: v1.20.0, the deploy command:

kubectl create secret generic mysql-secrets --from-literal=mysql-root-password=datahub
kubectl create secret generic neo4j-secrets --from-literal=neo4j-password=datahub
helm repo add datahub <https://helm.datahubproject.io/>
helm install prerequisites datahub/datahub-prerequisites
helm upgrade datahub datahub/datahub

The error messages:

Hello, quick question: Would it make sense to do Hive profiling jobs on Spark instead of doing them via GE, in order to make them more efficient and less time consuming?? I think it would be a great thing, but don't know if I am missing some key points behind the actual reasoning.

I set

METADATA_SERVICE_AUTH_ENABLED

in the helm chart now UI-based ingestion fails (CLI/API based works just fine). How do I add an API token to the interface? (if that’s even needed?)

next question, logged in as the root user to see if I can give my personal account authorization to write comments on DataHub entities. I don’t see my personal account or the root account, am I missing something? Thanks,

Under the AWS elastic search service setup for datahub it says If you have access control enabled with IAM auth, enable AWS auth signing in Datahub and says to add _OPENSEARCH_USE_AWS_IAM_AUTH=true_ but doesn’t specify where to add this in the values.yaml file or under which service/heading? Also, is it supposed to be _OPENSEARCH_USE_AWS_IAM_AUTH*:“true”*_ ??

Can anyone help me understand if the Google IAP provided jwt is going to work with DataHub without a code change? https://cloud.google.com/iap/docs/signed-headers-howto has some docs describing the token format

When i try to build Datahub on step

./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint

i get

build failed

. I get the same error on a othere ubuntu machine. Its allways a problem to start yarn:

Execution failed for task ':datahub-web-react:yarnInstall'.
> A problem occurred starting process 'command 'yarn''

To Reproduce Steps to reproduce the behavior: 1. Go to 'datahub' 2. Execute

./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint

3. See error Expected behavior Gradle should build datahub without error. Desktop (please complete the following information): • OS: Ubuntu 20.04.5 LTS • Version 0.9.3 (build 11.0.17+8-post-Ubuntu-1ubuntu220.04, mixed mode, sharing) : here a link to the closed issue: https://github.com/datahub-project/datahub/issues/6677 Thank u all here :)

Hi @brainy-intern-50400, is your yarn updated to the most recent version??

you can turn on

--stacktrace

or

--debug

for more detailed outputs that we can help with here

@astonishing-answer-96712 Hi Paul, can you please let me know what the steps are to reset the DataHub root users password after the Quick Start (self-hosted) install?

I am trying to run the gms service on my laptop but getting below bean creation failure Also is there any documentation which I can go through to setup the code development through Intellij that would be great. Basically I would want to make changes, compile/build and deploy to local. Additionaly if I can point my gms service to external resources like Kafka, MySQL, Elastic, etc that would be great. I tried by setting the environment variables but it didn't work

Hi all, I have deployed datahub in gcp using gce and docker. I want to change de default admin password but do not find any tips on the documentation. Can someone help me please ?

Has anybody tried to add custom integration/ingestion for a data source that is not currently supported? Any documentations and examples?

Hi everyone, I am having issues running DataHub with Kubernetes. It seems that anytime I run an install of the helm charts it causes all of the existing PATs to stop working (even though they still appear to be valid in the UI). This means that anytime we uninstall/reinstall the helm charts we have to regenerate all of the PATs, which is less than ideal. Is this the intended behavior? Or is there potentially something that we are not including which would prevent this from happening?

Hey folks, We are running into some issues with our EKS deployment where after a few hours (and a few ingestion jobs), all data regarding Users, Variables/Secrets, Ingestion Jobs, etc disappear. The actual data for entities (datasets, charts, etc. persists). This screenshot of pod health is from shortly after nuking and redeploying. Anybody see something amiss or encountered a similar problem?

Good morning, quick question: Is it possible to obtain via API all the information of a Datahub deployment regarding the tags, domains, glossary terms and in which datasets they are all added in order to add the same semantic to another Datahub deployment where all of the things mentioned exist but the dataset in this second deployment lack of such semantic??

Good Evening, quick question: Can we use PostgreSql instead of MySQL for persisting the metadata?

Hi everyone, I've been trying to deploy Datahub to Kubernetes using helm chart, but

datahub-gms

pod fails to start. The only error messages I can see are:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation

But I doubt this is the issue. And these messages:

│ ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Too │
│ l version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ERROR StatusLogger │
│  Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...                                                                            │
│ ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.2ANTLR Too │
│ l version 4.5 used for code generation does not match the current runtime version 4.7.2ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7.22022/12/15 15:28:2 │
│ 2 Command exited with error: exit status 1

I've tried increasing pod memory limits but that didn't help either (according to this issue it has been resolved in newer version, just wanted to rule it out)

Good morning, just discovered that secrets stored in Datahub can be obtained (their respective value) in plaintext via GraphQL API. I know this probably is intended but, isn't the fact of being able to see this kind of things by all users a security problem? Are there any plans on restricting the API access depending on the user's roll?? For example only admins could query secrets, editors can't access GraphQL (as info can't be edited there as far as I know) and readers all the things but secrets and other things regarding security aspects. Thanks in advance!!

Hello everyone ! Maybe someone have guide how to implement keycloak with datahub? Actually struggling here 😕 Thanks in advance !

hey all, has anyone successfully deployed (all the way to production) datahub services on Google Cloudrun?

Hello, all. We tried to upgrade datahub to 0.9.3 version and we are getting some problems with k8s (GCP GKE).

helm upgrade --install datahub datahub/datahub --debug
history.go:56: [debug] getting history for release datahub
upgrade.go:142: [debug] preparing upgrade for datahub
upgrade.go:524: [debug] copying values from datahub (v6) to new release.
upgrade.go:150: [debug] performing update for datahub
upgrade.go:322: [debug] creating upgraded release for datahub
client.go:310: [debug] Starting delete for "datahub-elasticsearch-setup-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job datahub-elasticsearch-setup-job with timeout of 5m0s
client.go:568: [debug] Add/Modify event for datahub-elasticsearch-setup-job: ADDED
client.go:607: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:607: [debug] datahub-elasticsearch-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:607: [debug] datahub-elasticsearch-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-elasticsearch-setup-job: MODIFIED
client.go:310: [debug] Starting delete for "datahub-kafka-setup-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job datahub-kafka-setup-job with timeout of 5m0s
client.go:568: [debug] Add/Modify event for datahub-kafka-setup-job: ADDED
client.go:607: [debug] datahub-kafka-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:607: [debug] datahub-kafka-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:607: [debug] datahub-kafka-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:607: [debug] datahub-kafka-setup-job: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-kafka-setup-job: MODIFIED
client.go:310: [debug] Starting delete for "datahub-mysql-setup-job" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job datahub-mysql-setup-job with timeout of 5m0s
client.go:568: [debug] Add/Modify event for datahub-mysql-setup-job: ADDED
client.go:607: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:607: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:607: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:607: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:607: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
client.go:568: [debug] Add/Modify event for datahub-mysql-setup-job: MODIFIED
client.go:607: [debug] datahub-mysql-setup-job: Jobs active: 1, jobs failed: 1, jobs succeeded: 0
upgrade.go:434: [debug] warning: Upgrade "datahub" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:84: [debug] pre-upgrade hooks failed: timed out waiting for the condition
UPGRADE FAILED
main.newUpgradeCmd.func2
        <http://helm.sh/helm/v3/cmd/helm/upgrade.go:200|helm.sh/helm/v3/cmd/helm/upgrade.go:200>
<http://github.com/spf13/cobra.(*Command).execute|github.com/spf13/cobra.(*Command).execute>
        <http://github.com/spf13/cobra@v1.4.0/command.go:856|github.com/spf13/cobra@v1.4.0/command.go:856>
<http://github.com/spf13/cobra.(*Command).ExecuteC|github.com/spf13/cobra.(*Command).ExecuteC>
        <http://github.com/spf13/cobra@v1.4.0/command.go:974|github.com/spf13/cobra@v1.4.0/command.go:974>
<http://github.com/spf13/cobra.(*Command).Execute|github.com/spf13/cobra.(*Command).Execute>
        <http://github.com/spf13/cobra@v1.4.0/command.go:902|github.com/spf13/cobra@v1.4.0/command.go:902>
main.main
        <http://helm.sh/helm/v3/cmd/helm/helm.go:83|helm.sh/helm/v3/cmd/helm/helm.go:83>
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_amd64.s:1581

It seems that datahub-mysql-setup-job failed to run Any help to solve it?