https://datahubproject.io logo
Join Slack
Powered by
# all-things-deployment
  • w
    @early-lamp-41924 I’m going to start playing around with the ES env vars for the system to try and get the retries to handle throughput. Am I setting these just on the MAE Consumer or should they be set on the GMS as well? I was just going to include them as 
    extraEnvs
    entries in the charts. Also what does 
    ES_BULK_REQUESTS_LIMIT
    and 
    ES_BULK_FLUSH_PERIOD
    set? * ES_BULK_REQUESTS_LIMIT * ES_BULK_NUM_RETRIES * ES_BULK_RETRY_INTERVAL * ES_BULK_FLUSH_PERIOD
    e
    • 2
    • 8
  • m
    Hello , i'm working to deploy datahub on k8s using HelmRelease , i've successfully deployed the prerequisites and got an error while deploying datahub
    Copy code
    Warning  FailedReleaseSync  5s (x2 over 27s)  helm-operator  synchronization of release 'datahub-frontend' in namespace 'datahub' failed: installation failed: template: datahub-frontend/templates/deployment.yaml:118:12: executing "datahub-frontend/templates/deployment.yaml" at <include "datahub-jmxexporter.container" .>: error calling include: template: no template "datahub-jmxexporter.container" associated with template "gotpl"
    e
    b
    • 3
    • 26
  • m
    and have 2 questions : 1 i git this error from datahub-upgrade-job `
    Copy code
    2021-06-17T14:31:48.873793757Z ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
    e
    • 2
    • 7
  • m
    2 - this is my deployment did i miss some component
    Copy code
    NAME                                                READY   STATUS      RESTARTS   AGE
datahub-datahub-frontend-866cb6c97b-tss6f           1/1     Running     0          110s
datahub-datahub-gms-84d8f8599b-9l7pn                1/1     Running     0          110s
datahub-datahub-upgrade-job-6ktd7                   1/1     Running     0          18s
datahub-elasticsearch-setup-job-gxv6d               0/1     Completed   0          108s
datahub-kafka-setup-job-97jfc                       0/1     Completed   0          106s
datahub-mysql-setup-job-w6jdt                       0/1     Completed   0          110s
elasticsearch-master-0                              1/1     Running     0          21m
elasticsearch-master-1                              1/1     Running     0          21m
elasticsearch-master-2                              1/1     Running     0          21m
flux-68bdf85d98-gmblt                               1/1     Running     0          15h
prerequisites-cp-schema-registry-598fd5876b-lrjcq   2/2     Running     1          21m
prerequisites-kafka-0                               1/1     Running     1          21m
prerequisites-neo4j-community-0                     1/1     Running     0          21m
prerequisites-zookeeper-0                           1/1     Running     0          21m
    e
    l
    • 3
    • 25
  • e
    the logs mysql | 2021-06-17 091348+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091348.737417Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091348.743779Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091348.743821Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091419+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091419+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091419+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091419+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091419.915462Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091419.916688Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091419.916797Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091448+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091448+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091448+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091448+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091448.916118Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091448.917661Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091448.917726Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091519+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091519+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091519+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091520+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091520.036442Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091520.038114Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091520.038183Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091549+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091549+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091549+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091549+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091549.299410Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091549.300939Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091549.301042Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091618+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091618+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091618+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091618+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091618.651448Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091618.653242Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091618.653469Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091650+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091650+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091650+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091650+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091650.468225Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091650.474352Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091650.474418Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091816+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091816+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091817+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091817+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091817.121322Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091817.123295Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091817.123372Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091848+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091848+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091848+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091848+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091848.586271Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091848.587760Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091848.587844Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091917+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091917+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' mysql | 2021-06-17 091917+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. mysql | 2021-06-17 091917+00:00 [Note] [Entrypoint]: Initializing database files mysql | 2021-06-17T091917.967146Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). mysql | 2021-06-17T091917.970734Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting. mysql | 2021-06-17T091917.970803Z 0 [ERROR] Aborting mysql | mysql | 2021-06-17 091948+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.34-1debian10 started. Desktop (please complete the following information):
    e
    • 2
    • 5
  • s
    When trying to run the frontend in K8s I am now having issues with requests
    l
    e
    g
    • 4
    • 26
  • p
    @early-lamp-41924 do we have a published helm chart version compatible with the 0.8.2 release?
    e
    • 2
    • 2
  • b
    This message was deleted.
    b
    e
    • 3
    • 2
  • c
    Copy code
    2021/06/29 08:13:35 Waiting for: <tcp://prerequisites-mysql:3306>
2021/06/29 08:13:35 Waiting for: <tcp://prerequisites-kafka:9092>
2021/06/29 08:13:35 Waiting for: <http://elasticsearch-master:9200>
2021/06/29 08:13:35 Waiting for: http:
2021/06/29 08:13:35 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:35 Connected to <tcp://prerequisites-mysql:3306>
2021/06/29 08:13:35 Connected to <tcp://prerequisites-kafka:9092>
2021/06/29 08:13:35 Received 200 from <http://elasticsearch-master:9200>
2021/06/29 08:13:36 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:37 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:38 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:39 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:40 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:41 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:42 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:43 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:44 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2021/06/29 08:13:45 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
    l
    e
    +2
    • 5
    • 30
  • s
    Is there a helm repository for the charts? https://datahubproject.io/docs/datahub-kubernetes asks to use the git repository folder itself
    • 1
    • 2
  • s
    do the helm chart contain any option of adding 
    nodeSelector
    ? I could not find any
    e
    • 2
    • 2
  • r
    Copy code
    datahub-upgrade]$ cat ./nocode/run_upgrade.sh
#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
cd $DIR && ../datahub-upgrade.sh -u NoCodeDataMigration -a batchSize=1000 -a batchDelayMs=100 -a dbType=POSTGRES

datahub-upgrade]$ ./nocode/run_upgrade.sh
privateregistry/datahub/datahub-upgrade:head
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.4.RELEASE)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
ERROR SpringApplication Application run failed
 org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'upgradeCli': Unsatisfied dependency expressed through field 'noCodeUpgrade'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanAspectDao' defined in class path resource [com/linkedin/gms/factory/entity/EbeanAspectDaoFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.entity.ebean.EbeanAspectDao]: Factory method 'createInstance' threw exception; nested exception is java.lang.NullPointerException
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643)
        at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:116)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1422)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
        at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139)
        at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:13)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ebeanAspectDao' defined in class path resource [com/linkedin/gms/factory/entity/EbeanAspectDaoFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.entity.ebean.EbeanAspectDao]: Factory method 'createInstance' threw exception; nested exception is java.lang.NullPointerException
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:484)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1338)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:557)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:310)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1287)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1207)
        at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640)
        ... 25 more
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.entity.ebean.EbeanAspectDao]: Factory method 'createInstance' threw exception; nested exception is java.lang.NullPointerException
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
        ... 42 more
Caused by: java.lang.NullPointerException
        at io.ebean.datasource.pool.ConnectionPool.notifyDataSourceIsDown(ConnectionPool.java:406)
        at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:535)
        at io.ebean.datasource.pool.ConnectionPool.createUnpooledConnection(ConnectionPool.java:524)
        at io.ebean.datasource.pool.ConnectionPool.createConnectionForQueue(ConnectionPool.java:766)
        at io.ebean.datasource.pool.PooledConnectionQueue.ensureMinimumConnections(PooledConnectionQueue.java:227)
        at io.ebean.datasource.pool.ConnectionPool.initialise(ConnectionPool.java:301)
        at io.ebean.datasource.pool.ConnectionPool.<init>(ConnectionPool.java:246)
        at io.ebean.datasource.core.Factory.createPool(Factory.java:15)
        at io.ebeaninternal.server.core.DefaultContainer.getDataSourceFromConfig(DefaultContainer.java:273)
        at io.ebeaninternal.server.core.DefaultContainer.setDataSource(DefaultContainer.java:217)
        at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:103)
        at io.ebeaninternal.server.core.DefaultContainer.createServer(DefaultContainer.java:35)
        at io.ebean.EbeanServerFactory.createInternal(EbeanServerFactory.java:109)
        at io.ebean.EbeanServerFactory.create(EbeanServerFactory.java:70)
        at com.linkedin.metadata.entity.ebean.EbeanAspectDao.createServer(EbeanAspectDao.java:594)
        at com.linkedin.metadata.entity.ebean.EbeanAspectDao.<init>(EbeanAspectDao.java:65)
        at com.linkedin.gms.factory.entity.EbeanAspectDaoFactory.createInstance(EbeanAspectDaoFactory.java:24)
        at com.linkedin.gms.factory.entity.EbeanAspectDaoFactory$$EnhancerBySpringCGLIB$$10d70657.CGLIB$createInstance$0(<generated>)
        at com.linkedin.gms.factory.entity.EbeanAspectDaoFactory$$EnhancerBySpringCGLIB$$10d70657$$FastClassBySpringCGLIB$$68ed9ed1.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
        at com.linkedin.gms.factory.entity.EbeanAspectDaoFactory$$EnhancerBySpringCGLIB$$10d70657.createInstance(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
        ... 43 more
    l
    • 2
    • 1
  • r
    While upgrading from v0.7.1 to 0.8.3 following the no code upgrade guide, we are getting this exception: https://datahubproject.io/docs/advanced/no-code-upgrade Please suggest if we are missing something.
    e
    • 2
    • 5
  • g
    Hi, I want to use Postgresql for database of GMS. Can you release a version of postgresql setup job to docker hub just like mysql setup job ?
    b
    • 2
    • 1
  • s
    I am trying to run in K8s with the latest updates and I am seeing an odd behavior. I did helm uninstall and then helm install and now MCE is trying to process old messages that do not exist anymore. I am confused as to where the MCE is getting these old messages since the topics are empty.
    m
    e
    • 3
    • 10
  • c
    I am facing a frustrated problem. I tried to deploy DataHub on k8s using my own dependency services (I just clone helm charts from github and replace corresponding repository by the path to helm chart local instead of link http as default in 
    prerequisites/Chart.yml
    ). Everything worked well but i got this error 
    Exception while fetching data (/browse) : java.lang.RuntimeException: Failed to execute browse: entity type: DATASET, path: [], filters: null, start: 0, count: 10
    when i browsed on UI
    l
    e
    • 3
    • 4
  • a
    hi everyone, we’ve got a usecase to open up the RestAPI to access outside the k8s cluster. I noticed the current helm chart didn’t have a template for an ingress on the datahub-gms service. I’ve opened this PR, which adds the template for the ingress. Its my first PR in the repo, so let me know if it doesn’t meet the required structures and standards. Thanks!
    e
    m
    • 3
    • 6
  • b
    Hi there, I'm going through the EKS/AWS setup steps and I'm not seeing the mce/mae-consumer pods coming up- I'm probably missing a configuration step here as this is all new territory for my team.
    l
    e
    +2
    • 5
    • 14
  • s
    datahub docker nuke
    is not working in 0.8.5
    Copy code
    docker.errors.APIError: 409 Client Error for <http+docker://localhost/v1.41/volumes/datahub_mysqldata>: Conflict ("remove datahub_mysqldata: volume is in use - [95df02e070a6a5685c6c56fd59de4b4c3d482963dadd72e9138742ac0f2411fe]")
    It seems the order of removal is wrong and it is trying to remove volumes before containers
    g
    b
    • 3
    • 5
  • f
    Hi We need to point to external kafka, mysql and elastic search from datahub, currently we point to dockerized versions of them. Please let us know what all changes we need to do in order to achieve this.
    e
    g
    • 3
    • 12
  • s
    Is there any doc on how to setup kafka on confluent cloud (https://confluent.cloud/) with datahub? I read https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub and it just mentions the URLs under 
    global.kafka.*
    settings. There is no mention of any API key to authenticate with confluent cloud
    m
    s
    b
    • 4
    • 15
  • c
    Hi, I am deploying DataHub on minikube and i tried to set up ingress for datahub-frontend to expose the 9002 port to the public. I had set enabled of ingress to true in the 
    datahub/charts/datahub-frontend/values.yaml
    file. But when i check 
    kubectl get ingress
    , there is no resource. How can i expose the 9002 port to the public ?
    e
    • 2
    • 2
  • s
    the helm chart gives the following error in 
    datahub-datahub-gms
    Copy code
    secret "neo4j-secrets" not found: CreateContainerConfigError
    but my 
    neo4j
    is disabled. This is datahub chart 
    0.2.1
    with app version 
    0.8.3
    . I guess the helm chart was not updated on the helm repository
    Copy code
    helm search repo datahub --versions
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                      
datahub/datahub                 0.2.1           0.8.3           A Helm chart for LinkedIn DataHub                
datahub/datahub                 0.2.0           0.8.1           A Helm chart for LinkedIn DataHub                
datahub/datahub                 0.1.2           latest          A Helm chart for LinkedIn DataHub                
datahub/datahub-prerequisites   0.0.1                           A Helm chart for packages that Datahub depends on
    e
    • 2
    • 2
  • s
    Does 
    indexPrefix
    for elasticsearch work? I see there is an option for that https://github.com/linkedin/datahub/blob/eb60da975dce1f855883e628541cdbcef89717af/datahub-kubernetes/datahub/quickstart-values.yaml#L51 as well as in https://datahubproject.io/docs/deploy/aws/ I have this value in my kubernetes cluster
    Copy code
    global:
  graph_service_impl: elasticsearch
  datahub_analytics_enabled: true
  datahub_standalone_consumers_enabled: false

  elasticsearch:
    host: "elasticsearch-master.apps.svc.cluster.local"
    port: "9200"
    indexPrefix: datahub
    The elasticsearch setup job is complete. But I am seeing this in my indices. There is no "datahub" prefix for the index names. Did I understand this option incorrectly?
    Copy code
    health status index                    uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   datasetdocument          lHs5ppPkTgWYBud82GEKvw   1   1          0            0       416b           208b
green  open   dashboarddocument        84GsR1ZORtyRaX7yMdgOxg   1   1          0            0       416b           208b
green  open   glossaryterminfodocument GqaUw-bESp69Hp1tiNNXRA   1   1          0            0       416b           208b
green  open   chartdocument            wcR3nE1mT3ut8f9yVb_VKA   1   1          0            0       416b           208b
green  open   datajobdocument          Y4HSYjrHQbmcYx5ioFK-yg   1   1          0            0       416b           208b
green  open   dataflowdocument         tYOFqQYZRYuFd3z6vhr1Kg   1   1          0            0       416b           208b
green  open   dataprocessdocument      QMiZcLs6SR6aEp0d6s3wyA   1   1          0            0       416b           208b
green  open   mlmodeldocument          Yr9TxaRHTvWiDDGpRNiwCA   1   1          0            0       416b           208b
green  open   glossarynodeinfodocument _vo5wMgKRAqOQWkDSS-nVA   1   1          0            0       416b           208b
green  open   corpuserinfodocument     zSJp34coQSyB7vjJBj4BFQ   1   1          0            0       416b           208b
green  open   tagdocument              F2EqZ8cTTl-xd5g3eode7w   1   1          0            0       416b           208b
    e
    • 2
    • 2
  • a
    Hi, I’m Jensen and I just deployed datahub on k8s with the guides here.https://github.com/acryldata/datahub-helm/tree/master/charts/prerequisites this might be a silly question, if I want to connect aws athena & redshift on the prerequisites charts how can i do it? I just wanna test how it will look like after connecting datahub with aws. https://datahubproject.io/docs/metadata-ingestion The guide here seems to only explain how to ingest it through docker or locally. is there a fast way to deploy an ingestion with k8s?
    e
    • 2
    • 15
  • b
    Hi again, I'm trying to split the k8s deployment to use separate AWS services for mysql, kafka and elasticsearch. Pointing the config to my AWS elasticsearch endpoint is showing 403 errors in the datahub-elasticsearch-setup-job logs
    e
    • 2
    • 3
  • n
    Can I use Postgresql instead of MySQL for datahub-database on k8s?
    👀 1
    • 1
    • 4
  • s
    seems there is some bug with ingress. I have following in my values of helm chart
    Copy code
    datahub-frontend:
  enabled: true
  image:
    repository: linkedin/datahub-frontend-react
    tag: "v0.8.5"
  # Set up ingress to expose react front-end
  ingress:
    enabled: false
    but the ingress is still being created. Same for 
    datahub-gms
    This is with
    Copy code
    chart_ref: datahub/datahub
chart_version: 0.2.6
    • 1
    • 1
  • s
    Facing issues with gms access to kafka in confluent cloud
    b
    e
    a
    • 4
    • 33
  • c
    Copy code
    Type     Reason       Age    From               Message
  ----     ------       ----   ----               -------
  Normal   Scheduled    9m48s  default-scheduler  Successfully assigned default/datahub-datahub-mce-consumer-5775f5c487-8z9dr to minikube
  Warning  FailedMount  9m48s  kubelet            MountVolume.SetUp failed for volume "datahub-datahub-mce-consumer-token-8s9sh" : failed to sync secret cache: timed out waiting for the condition
  Normal   Pulling      9m47s  kubelet            Pulling image "linkedin/datahub-mce-consumer:v0.8.6"
  Normal   Pulled       8m29s  kubelet            Successfully pulled image "linkedin/datahub-mce-consumer:v0.8.6" in 1m18.086509993s
  Normal   Created      8m29s  kubelet            Created container datahub-mce-consumer
  Normal   Started      8m28s  kubelet            Started container datahub-mce-consumer
    g
    s
    • 3
    • 11
12345...53Latest