Small question: have a setup of datahub (self-hosted in cloud) up and running, but would like to ingest on-prem SQL server metadata. Of course, we could open up these connections but security-wise, we'd rather not have to resort to this, so we figured these are our options:

Hello team. I have successfully deployed DataHub on eks with aws storage layer. I need to add properties below security.protocol=SASL_SSL sasl.mechanism=AWS_MSK_IAM sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler for MSK kafka IAM role-based authentication Enabled How can I do this? Thanks in advance .

Hi team, We are currently deploying datahub via helm on k8s. In this, we have setup Google Auth layer for onboarding users onto Datahub. Is it possible to pre-assign the Reader role to every incoming user? Currently, everybody is onboarding in the "No Role" role

Hello all. I have questions regarding graph_service_impl using Neo4j vs ES. 1. The value of graph_service_impl changed from

neo4j

to

elasticsearch

starting from v0.10.0, while the default value in subcharts are still

neo4j

. Is there any reason of the change? Also, it is mentioned in docs that

neo4j

is still the default because of backward compatibility. What is the recommended graph_service_impl from now on and going forward? 2. In what situation will Neo4j have advantage over ES? Which specific features and scenario where Neo4j will be more beneficial? Another question, what is the recommended schema registry between internal and kafka? What is the advantage and disadvantage between them?

Hi there! As a policy at the company I work for we try to stay away from Kubernetes if possible. Even still, we would like to host DataHub and thus were wondering if it is realistic to host DataHub on a more managed service such as Azure Container Apps (similar to AWS ECS). Thanks in advance!

Hello support. I am following up this document https://datahubproject.io/docs/authentication/guides/add-users In this document it is mentioned that enable SSO first but i can not see it on setting. Managed DataHub Single Sign-On can be configured and enabled by navigating to Settings SSO > OIDC. Note that a user must have the Manage Platform Settings >Platform Privilege in order to configure SSO settings.

can anyone answer this please https://datahubspace.slack.com/archives/CV2UVAPPG/p1686226468322889

Anyone aware of this issue https://datahubspace.slack.com/archives/CV2UVAPPG/p1686229227423299 Few details: I have setup datahub helm chart on my AWS EKS environment, now i am trying above link to setup the Single Sign On Azure-AD. I am using the Datahub-UI but when i go to settings i do not found Settings > SSO > OIDC. Anyone can tell me am i missing something?

I have seen a recent changes were made on https://github.com/acryldata/datahub-helm/commits/master/charts/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml Looks like some issue while phrasing yaml file.

Hi, I’m having a problem with creating datahub for the first time using the new helm chart, using

INTERNAL

schema registry. The system update job and the gms pod fail. System update job log:

2023-06-09 07:16:51,099 [main] INFO  c.l.d.u.impl.DefaultUpgradeReport:16 - Executing Step 4/5: DataHubStartupStep...
org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: java.io.IOException: No schema registered under subject!

GMS pod cannot connect to itself:

2023-06-09 07:21:57,361 [R2 Nio Event Loop-1-1] WARN  c.l.r.t.h.c.c.ChannelPoolLifecycle:139 - Failed to create channel, remote=localhost/127.0.0.1:8080
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.base/java.lang.Thread.run(Thread.java:829)

x-post from troubleshoot https://datahubspace.slack.com/archives/C029A3M079U/p1686322783003879 wondering if people have tip on how to fix this

Trying to connect to Redshift but getting 'Unable to emit metadata to DataHub GMS'. What would be the cause for this?

hi,team.I have two questions。 【1】I currently have a scenario where certain datasets will be applied to some interface APIs. Is there a model targeting this type of API and supporting the blood relationship between datasets and APIs 【2】If I extend a model to define API interfaces, does the extended model support lineage

Hi we are having CVE-2022-42889 vulnerability in

/usr/local/lib/python3.10/site-packages/pyspark/jars/commons-text-1.6.jar

Hi, We have been running v10.3 since last week without issues. Today we decided to do a re-install of 10.3 just as a weekly maintenance activity. The deployment failed due to chart issues in the datahub cleanup job. I believe that fix is made and PR open - https://github.com/acryldata/datahub-helm/pull/328 I see last working chart version was 0.2.165 so we used the same. The deployment for GMS pod failed with the message -

2023-06-13 12:29:26,953 [ThreadPoolTaskExecutor-1] ERROR o.s.k.l.KafkaMessageListenerContainer:149 - Consumer exception

java.lang.IllegalStateException: This error handler cannot process 'SerializationException's directly; please consider configuring an 'ErrorHandlingDeserializer' in the value and/or key deserializer

at org.springframework.kafka.listener.DefaultErrorHandler.handleOtherException(DefaultErrorHandler.java:151)

at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.handleConsumerException(KafkaMessageListenerContainer.java:1815)

at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1303)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: org.apache.kafka.common.errors.SerializationException: Error deserializing key/value for partition MetadataChangeLog_Versioned_v1-0 at offset 11251397. If needed, please seek past the record to continue consumption.

Caused by: org.apache.kafka.common.errors.SerializationException: Error retrieving Avro unknown schema for id 5

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)

at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)

at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)

at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)

We installed prerequisites with latest chart and kafka 3.4.0 and tried installing 10.3 again, but still failed with same message. Please advise on what is the correct working combination currently. Our application is down currently.

Is there any way to change MySQL service type? I tried adding a parameter in Values.yaml, creating a new service and revising the original service by apply -f command but all the trials failed.

hi,team .custom metadata model can be used in lineage?

Hi team, We are trying to deploy Datahub 0.10.4 via helm charts. For ES, we are using AWS. Version of ES is 7.9 Can someone confirm if this version is enough because we are getting errors similar to https://datahubspace.slack.com/archives/C029A3M079U/p1680690599338069 PS. The doc says 7.9 is enough https://datahubproject.io/docs/deploy/aws#elasticsearch-service

Hello team! We already have datahub

v0.10.2

running in production (k8s deployment) for quite a while now. However, we are implementing basic auth to schema registry and I would like to know if datahub supports that? I've tried setting the following, on `values.yaml`:

credentialsAndCertsSecrets:
  name: my-secret
  secureEnv:
   <http://schema.registry.basic.auth.user.info|schema.registry.basic.auth.user.info>: schema_registry_basic_auth_user_info

And it resulted in the creation of the

SPRING_KAFKA_PROPERTIES_SCHEMA_REGISTRY_BASIC_AUTH_USER_INFO

env variable in every datahub component, pointing to the right secret and secret-key. However, I still get 401's from our Schema Registry - meaning that basic auth was not implemented. If it is supported, how can we define it correctly?

Hello everyone . Is there any update on https://datahubspace.slack.com/archives/CV2UVAPPG/p1686261643646089 I just checked and got same error (AWS EKS)

Hi Team, We trying to build arm64 Images for frontend in my local MAC M1 chip. Build was succcessful using the command

./gradlew build

. When I tried to build frontend image using the below dockerfile the Image build is getting stuck in the middle at gradle build command and it’s not moving forward. I had check the logs using

--debug

mode but didn’t figure out the actual issue. I had also attached the logs for your reference. Could you please help me out with the issue.

docker buildx build . -t datahub --platform=linux/arm64

hey, im trying to deploy the official helm chart on our on premise openshift server and keep getting the following error: unsadesfideddependencyexpection creating bean with name ive got all the dependencies and the logs say it manages to connect to some of the DBs but this error keeps popping up and im not sure what it means or why its happening, does anyone know or can help?

hi team, I have deployed datahub version 0.9.5 using docker container in the AWS VM. I have done all the setup required. But when I run the docker compose file it prints the logs on stdout/terminal, but when I go inside the gms container, I am unable to see the logs at path /tmp/datahub/logs/gms. can someone suggest me the solution for this.

Hi, I have deployed datahub through helm charts. I set up ingress and wanted to route like <domain.com>/datahub but that way it is not able load everything it is giving blank page. Any solution for this ?

hi,team。I have enabled config->ENABLE_ PROMETHEUS but when i When I publish a service, i prompts error Unable to find jar jmx_prometheus_javaagent.jar

Hi team What does this topic datahubupgradehistory_v1use for ?

hi all, i'm trying to create the

datahub-rest-default

connection from Airflow to the GMS deployed to Kubernetes. I'm facing the following error (likely to do with proxy ingress?), any guidance would be appreciated!

raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for '{my domain:gms service is populated here}:8080/aspects?action=ingestProposal'

👋 are there plans to upgrade the tag used for

acryldata/datahub-actions

in the helm chart? It’s currently on v0.0.11 which has the old

0.10.0.6

CLI. Ref: https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L47

So far every version I've upgraded to has downtime with a reindexing in the

datahub-system-update-job

. That is

v0.9.3

->

v0.10.0

->

v0.10.1

->

v0.10.3

. Is this expected? https://datahubproject.io/docs/how/updating-datahub/ This page says only

v0.10.0

should have caused downtime. Is this issue related to the now-fixed retention bug in the

DataHubUpgradeHistory_v1

topic? What is confusing me is that when I made changes but kept the Datahub version the same, it wouldn't trigger a reindex even if it was past the previous

DataHubUpgradeHistory_v1

default retention period of 7 days.

Hello, My dev innstance is installed by the quickstart command but i need to installm my produciton instance of datahub inside a docker with docker compose inside aws EC2 and i need to know wich docker compose file can i use to do that, Knowing i use the managed services for elasticsearch kafka and Mysql 🙏