Hello everyone, we just deployed Datahub to AWS EKS for evaluation and we wonder if there are any good practices in respect to volume sizing. Of course it depends on the use case, but first thing that strikes me as a potentially way too big storage allocation is zookeeper claiming an 8Gi PV. Do I understand correctly, that Zookeeper is only used to let Kafka nodes find each other? In such a case I would give it rather a few Mi instead. Is there something like a dimensioning guide in general? Thanks!

Hey team! I am trying to set up Datahub on AWS, using managed services (EKS, OpenSearch, RDS MySQL, MSK, Glue). I would like to configure authentication to MSK and Glue using IAM. I have configured a serviceAccount to assume a role using OIDC (docs). This works well with the kafka-setup-job and I see the topics are created as expected. However, when running the system-upgrade-job there seems to be a problem, the java SDK is using the instance profile (role attached to the EKS node) instead of the configured serviceAccount. This causes the pod to fail since the instance profile role does not have permissions to Glue. I have followed the steps described in this guide, and verified the serviceAccount works with other pods. Is there anything I can configure on the Java SDK level to use the correct role? The other option is to ditch Glue and move to cp-schema-registry, but then I need to authenticate to MSK from cp-schema-registry, which is again not possible without modifying the image…. Thank you very much!!

Hi Team, I am trying to build datahub frontend. I am getting error while running below command.

./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint --debug

Error:

Process 'command 'yarn'' finished with non-zero exit value 137

Below are the version for java , nodejs and yarn

[root@ip-10-224-*-* ec2-user]# java --version
openjdk 11.0.18 2023-01-17 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-1.amzn2.0.1) (build 11.0.18+10-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-1.amzn2.0.1) (build 11.0.18+10-LTS, mixed mode, sharing)
[root@ip-10-224-*-* ec2-user]# node -v
v16.19.1
[root@ip-10-224-*-* ec2-user]#  npm -v
8.19.3
[root@ip-10-224-*-* ec2-user]# yarn -v
3.5.0

Can anyone help on this? Please find the log in attachment.

Flink integration with datahub is on cards? what it takes to contribute?

This message was deleted.

hi,team.about lineage->Whether there are official cases of large data volume lineage? I want to estimate the maximum number of downstream nodes that our lineage can support. I found that there are currently more than 1000 downstream nodes, and the response is starting to slow down

I have done a MySQL dump manually from my Datahub old version before redeploying to the newest version. Now, how do I restore all my data from my SQL backup? Thank you. FYI: I’m deploying by my custom Docker Compose

This message was deleted.

Hi Team, We had recently deployed datahub on eks and wanted to move kafka from kube to confluent cloud managed service. While setting it up i noticed that the kafka setup job by default uses a single partition for all topics. Does this mean that we need to make some sort of changes from producer/consumer side for a performance upgrade (im guessing there is a single mae/mce consumer). New to kafka, so any help is appreciated

Hey team, can we attach airflow task to domain? Thanks in advance

I am getting this error when gms starts up:

2023-03-30 09:06:02,014 [main] INFO  c.d.event.PlatformEventProcessor:37 - Creating Platform Event Processor
2023-03-30 09:06:02,023 [main] INFO  c.l.g.f.k.s.AwsGlueSchemaRegistryFactory:40 - Creating AWS Glue registry
2023-03-30 09:06:03,147 [main] WARN  c.l.metadata.entity.EntityService:807 - Unable to produce legacy MAE, entity may not have legacy Snapshot schema.
java.lang.UnsupportedOperationException: Failed to find Typeref schema associated with Config-based Entity
	at com.linkedin.metadata.models.ConfigEntitySpec.getAspectTyperefSchema(ConfigEntitySpec.java:80)
	at com.linkedin.metadata.entity.EntityService.toAspectUnion(EntityService.java:1510)
	at com.linkedin.metadata.entity.EntityService.buildSnapshot(EntityService.java:1459)
	at com.linkedin.metadata.entity.EntityService.produceMetadataAuditEvent(EntityService.java:1269)
	at com.linkedin.metadata.entity.EntityService.sendEventForUpdateAspectResult(EntityService.java:803)
	at com.linkedin.metadata.entity.EntityService.ingestAspectIfNotPresent(EntityService.java:767)
	at com.linkedin.gms.factory.telemetry.TelemetryUtils.createClientIdIfNotPresent(TelemetryUtils.java:39)
	at com.linkedin.gms.factory.telemetry.TelemetryUtils.getClientId(TelemetryUtils.java:25)
	at com.linkedin.gms.factory.telemetry.DailyReport.<init>(DailyReport.java:40)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory.dailyReport(ScheduledAnalyticsFactory.java:26)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$9b09a49a.CGLIB$dailyReport$0(<generated>)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$9b09a49a$$FastClassBySpringCGLIB$$55e22012.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
	at com.linkedin.gms.factory.telemetry.ScheduledAnalyticsFactory$$EnhancerBySpringCGLIB$$9b09a49a.dailyReport(<generated>)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
	at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)
	at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)
	at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1073)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1449)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1414)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:423)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:387)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
	at org.eclipse.jetty.runner.Runner.run(Runner.java:519)
	at org.eclipse.jetty.runner.Runner.main(Runner.java:564)

Then, datahub-gms refuses to connect to itself. What is this error? Could it be that this is causing the issue? Any ideas? Thanks

Hi team, Good day! Can I know how I can backup Datahub deployed on Kubernetes and then restore it in a fresh instance?

Hello Guys, I saw this article about Graph Service Implementation https://datahubproject.io/docs/how/migrating-graph-service-implementation/ . But I cannot find en explanation on how to access this Graph Representation of Datahub data. Could u guys help me? Thanks !

Hello, for a custom data source, how can stateful-ingestion be enabled. Whats the best way to create checkpoints, are there some examples in git for this?

Hi team, I apologize if this is the wrong channel. We are wondering how to implement simple quality rules with datahub, we have seen that you can use other tools like dbt or great expectations. We wanted something that simply identifies basic controls without having to install any other solution, with datahub there is a way to visualize if a table meets some formatting quality controls for example.

How could Great Expectations + dataHub be deployed on Kubernetes?

Heyo, From nowhere my gms pod started failing and logs are showing:

Caused by: com.linkedin.r2.RetriableRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
│ 2023-04-02 10:21:19,146 [pool-14-thread-1] ERROR c.d.m.ingestion.IngestionScheduler:244 - Failed to retrieve ingestion sources! Skipping updating schedule cache until next refresh. start: 0, count: 30                                                                     ││ com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://localhost:8080/entities>
2023-04-02 10:21:19,143 [R2 Nio Event Loop-1-1] WARN  c.l.r.t.h.c.c.ChannelPoolLifecycle:139 - Failed to create channel, remote=localhost/127.0.0.1:8080

My upgrade job [1] (deployed with helm chart actual version is 0.2.160 / 10.0.0 (app version) My first thought was to upgrade the helm version to 0.2.161 but does anyone know the problem’s source? On the instance, there were several ingestions nothing big and I was playing with graphql api + custom python script for groups and policies I found similar cases:

<https://datahubspace.slack.com/archives/C029A3M079U/p1678354501041149>
<https://datahubspace.slack.com/archives/CV2UVAPPG/p1678293436288999>

This message was deleted.

Hello Team. could you please help me with datahub message, that I received after installing datahub on windows.: datahub-gms is running by not yet healthy, windows datahub-upgrade is still running . I see datahub web application, but I cannot log in to it. Thank you for attention

Does the lineage created automatically when we import postgres data using primary and foreign key relationships

hi,team。Asked about the issue of lineage loading. version:v0.10.1 1. I found that although I have more than 500+downstream lineages, only 100 downstream lineages are displayed by default. 2. When I modify the parameter to obtain 2000 lineages, I prompt "Async processing failed"; nested exception is java.lang.StackOverflowError Is there a better plan or distributed loading plan for loading a large amount of data.

👋 just noticed that our MCE service started spewing things like

2023-04-03 13:21:27,918 [R2 Nio Event Loop-1-1] WARN  c.l.r.t.h.c.c.ChannelPoolLifecycle - Failed to create channel, remote=localhost/127.0.0.1:8080
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-04-03 13:21:27,919 [ThreadPoolTaskExecutor-1] ERROR c.l.m.k.MetadataChangeEventsProcessor - MCE Processor Error
com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://localhost:8080/entities>
----
2023-04-03 13:21:27,919 [ThreadPoolTaskExecutor-1] INFO  c.l.m.k.MetadataChangeEventsProcessor - Error while processing MCE: FailedMetadataChangeEvent - {error=com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://localhost:8080/entities>

Why would the MCE trying to call

<http://localhost:8080/entities>

? I checked, and GMS_HOST is correct.

Hi Team, I am trying to create frontend image using below command:

./gradlew :datahub-frontend:docker

Image creation got stuck at datahub-frontend:cleanLocalDockerImages from last 4 hour. Could you please help on this.

May I ask the advised way to upgrade DataHub properly? For the context, we’ve managed to rebase the project from v0.8.35 to v0.10.1 and are thinking about the rollout plan with migration on both MySQL and ES sides, which is expected to take ~18h. (due to full table scan in MySQL to move policies into ES). And it will be great to get some insights from you. • We are super happy to see the migration job datahub-upgrade is decoupled from GMS Spring bootstrap step, but are wondering what’s the best way to deploy it. For example, say our services are running in k8s without Helm, should we deploy datahub-upgrade as a separate k8s workload resource, or within GMS as a sidecar/init container, or there are other recommended ways? • According to our understanding of how ES re-index works, we expect there will be data loss if we don’t turn off mae consumer. Is that a correct assessment? And if that’s the case and the migration is going to take long, is there a way to reduce the system downtime during the upgrade? • Any other tips?

Hi team. My DevOps tells me there is no Kubernetes Chart for 10.1 yet, and those charts are usually released some time after the release itself, And since we deploy Datahub in Kubernetes, we can't upgrade to 10.1 which we really want to. Is there a reason to why Kubernetes Chart is not realesed along with the release itself?

Hey team, I have created multiple actions pipeline specific to multiple use case and I am deploying them as a cron job but my cron job is always running and it’s not getting completed. Can suggestion on this ?

Hi team, having some trouble with my EKS deployment. I'm not able to initially deploy datahub with my AWS services configured. I'm having to deploy the prereqs unaltered, then datahub unaltered, then go back and do a helm upgrade install with my custom values. When I try to deploy with my services I'm getting an error with the system update job and it says it can't find the 'datahub-auth-secrets'

hi,team.I found that migrate about neo4j and es. https://datahubproject.io/docs/how/migrating-graph-service-implementation/ I would like to know how we choose neo4j or es, and what are their performance advantages and disadvantages？ And we are currently facing optimization of query performance for bloodline queries We are internally customized based on official assignments, so we are more concerned about changes. https://datahubspace.slack.com/archives/CV2UVAPPG/p1680514403407399

read tcp 192.168.100.6053804 >104.18.125.25443: read: connection timed out . Unable to run quickstart - the following issues were detected: - quickstart.sh or dev.sh is not running

Hi Team, How can I rename

group.id

= generic-platform-event-job-client in GMS? i use helm, datahub v10.0 I tried config, but it doesn't work

extraEnvs:
  - name: KAFKA_CONSUMER_GROUP_ID
    value: "my-apps-mae-consumer"

Thank you for attention