https://datahubproject.io logo
Join Slack
Powered by
# all-things-deployment
  • b
    Access should be open to all roles on the account so I'm not sure where my config is going wrong here?
  • b
    Untitled
    Untitled
  • b
    This is what I ended up with following the AWS services setup docs.
  • c
    Is that the DataHub of version 0.8.6 doesn't support to use elasticsearch as graph service yet ? There is some wrong with mce component. I had checked the logs of mce and it turned out okay with all INFO things. Everything worked well when using neo4j as graph service. I am deploying DataHub on k8s
  • f
    gms service is not starting. it starts and then exits bxxxxxxx8  linkedin/datahub-gms:head                  "/bin/sh -c /datahub…"  30 minutes ago  Exited (255) 7 seconds ago                                                     datahub-gms b
  • c
    @chilly-wall-98962 has left the channel
  • c
    Copy code
    datahub-datahub-mae-consumer-8dcb6df97-69vs5       0/1     Error       3          2m18s
  • c
    Copy code
    Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.io.FileNotFoundException: /datahub/datahub-gms/resources/entity-registry.yml (No such file or directory)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651)
	... 106 common frames omitted
  • o
    Question If I set 
    global.datahub_standalone_consumers_enabled: false
    when upgrading to DataHub v0.8.8 with the 
    helm
    chart, will I also need to change my metadata ingestion approach to hit the GMS REST API instead? (details in thread)
  • b
    Putting this here as I don't see a Docker channel. I've spent some time getting a basic proof of concept version of the Dockerised Datahub working in an AWS environment. I thought I post my working quickstart file here in case it helps anyone else out trying the same. This is connecting to AWS managed services for the database, kafka and elasticsearch.
  • s
    @stale-jewelry-2440 has left the channel
  • r
    We've successfully deployed DataHub on k8s in DFDS (www.dfds.com). We've open sourced the configuration (CICD pipelines, terraform and configuration files), maybe it's interesting to the community? https://github.com/dfds-data/datahub-infrastructure
    👍 4
  • c
    kubectl logs datahub-datahub-gms 082614.234 [qtp544724190-117] INFO c.l.m.r.entity.EntityResource - GET SEARCH RESULTS for chart with query conan 082614.250 [pool-11-thread-1] INFO c.l.metadata.filter.LoggingFilter - POST /entities?action=search - search - 200 - 16ms 082614.261 [qtp544724190-80] INFO c.l.m.r.entity.EntityResource - GET SEARCH RESULTS for dataset with query conan 082614.266 [qtp544724190-126] INFO c.l.m.r.entity.EntityResource - GET SEARCH RESULTS for corpuser with query conan 082614.275 [qtp544724190-127] INFO c.l.m.r.entity.EntityResource - GET SEARCH RESULTS for dashboard with query conan 082614.323 [pool-11-thread-1] INFO c.l.metadata.filter.LoggingFilter - POST /entities?action=search - search - 200 - 57ms 082614.354 [qtp544724190-119] INFO c.l.m.r.entity.EntityResource - GET SEARCH RESULTS for dataFlow with query conan 082614.358 [pool-11-thread-1] INFO c.l.metadata.filter.LoggingFilter - POST /entities?action=search - search - 200 - 83ms 082614.364 [pool-11-thread-1] INFO c.l.metadata.filter.LoggingFilter - POST /entities?action=search - search - 200 - 103ms 082614.438 [pool-11-thread-1] INFO c.l.metadata.filter.LoggingFilter - POST /entities?action=search - search - 200 - 84ms
  • c
    I want something like 2021-09-06 082614 [ ] INFO... @early-lamp-41924 ^^
  • p
    has anybody tried to integrate the logs from the pods to something external like datadog?
  • b
    @better-afternoon-19270 has left the channel
  • b
    Hello
  • b
    I want deploy the frontend as cluster ip
  • b
    datahub-datahub-frontend
  • b
  • b
    use this values files, set ingress as false,
  • b
    somehow, I still get as Lb deployment
  • b
    where do i miss?
  • b
    figured out, need to update the service type
  • b
    ok, then. I use helm install the datahub completely. backend service running in healthy state. now I set up ingestion with cron job: datahub-datahub-ingestion-cron
  • b
    if I use sinktype: console, croj job run completion.
  • b
    if I use datahub:
  • b
    source: _# source configs_sink: type: “datahub-rest” config: server: “http://localhost:8080”
  • a
    Hi All, Is possible if we can use IAM based authentication in Datahub app instead of using hard coded password for mysql ?
    d
    o
    a
    • 4
    • 4
  • c
    hi,team。when i release from docker. [0.9.6.1] My main version of Java is Java11, and some of the code is Java8. When I publish it, I report an error。
    Copy code
    Could not find tools.jar. Please check that /usr/lib/jvm/java-1.8-openjdk contains a valid JDK installation.
    Copy code
    ull recompilation is required because no incremental change information is available. This is usually caused by clean builds or changing compiler arguments.
Compiling with toolchain '/usr/lib/jvm/java-1.8-openjdk'.
Starting process 'Gradle Worker Daemon 1'. Working directory: /root/.gradle/workers Command: /usr/lib/jvm/java-1.8-openjdk/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Xmx512m -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -cp /root/.gradle/caches/6.9.2/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Worker Daemon 1'
Successfully started process 'Gradle Worker Daemon 1'
Started Gradle worker daemon (0.624 secs) with fork options DaemonForkOptions{executable=/usr/lib/jvm/java-1.8-openjdk/bin/java, minHeapSize=null, maxHeapSize=null, jvmArgs=[], keepAliveMode=SESSION}.
Compiling with JDK Java compiler API.
:li-utils:compileMainGeneratedDataTemplateJava (Thread[Execution worker for ':' Thread 46,5,main]) completed. Took 1.604 secs.
[91m
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':li-utils:compileMainGeneratedDataTemplateJava'.
> Could not find tools.jar. Please check that /usr/lib/jvm/java-1.8-openjdk contains a valid JDK installation.

* Try:
Run with --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':li-utils:compileMainGeneratedDataTemplateJava'.
    ✅ 1
    d
    • 2
    • 1
1...484950...53Latest