Hello channel, I wonder to know if there is any chance to deploy datahub on k8s (with helm charts) but without

kafka

?

Datahub helm chart does not have option to add resources in ingestion-cron helm chart.

Hello everybody, can you tell me please what is the main reason that DataHub uses 2 kafka streams in back groud one for ingestion another one for elastic and graph ? If it takes info from mysql which was generated by another information from another kafka stream, why it is not consuming directly from first kafka topics?

Hi All, I'm trying to setup DataHub on AWS EKS using helm charts, using AWS managed services for the storage. Having issue with the

kafkaSetupJob

, getting below error

[kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.common.utils.AppInfoParser - App info kafka.admin.client for adminclient-1 unregistered
[kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
[main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.
org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1657820045949, tries=1, nextAllowedTryMs=-9223372036854775709) timed out at 9223372036854775807 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata

Which datahub services are communicating to gms ?

Can we use latest images for datahub prerequisites which has less number of vulnerabilities? Like zookeeper and cp schema and jmx exporter I tested and deployed it is working fine .

This message was deleted.

We are following datahub aws guide to setup. We are running following service in the datahub 1. datahub-acryl-datahub-actions 2. datahub-datahub-frontend 3. datahub-datahub-gms 4. datahub-schema-registry We have following aws components 1. Kafka 2. RDS(Mysql) 3. Opensearch To make connection with elasticsearch (opensearch) datahub services

datahub-datahub-gms

not able to connect with elasticsearch. Getting following error in the log

2022/07/16 11:25:03 Received 403 from <https://vpc--datahubes-vsqu3edgv5tsiyshvnq.ap-south-1.es.amazonaws.com:443>. Sleeping 1s
2022/07/16 11:25:04 Received 403 from <https://vpc-datahubes-vsqu3edgv5tsiyshvnq.ap-south-1.es.amazonaws.com:443>. Sleeping 1s

While we have attached the serviceAccount with the pods and the AWS IAM role is also whitelisted in the resource policy with proper permission. I want to confirm that does datahub support AWS signature version 4 to make connection with AWS elasticsearch.

Hey guys, can you tell me please if we need to instal somehow neo4j with aws setup or no? In AWS deployment it says that graph DB will be soon available with Aws Neptune but what about now? How it works now?

I have my broker service running why still get No resolvable bootstrap urls given in bootstrap.servers.

8f1001b0ddb2 confluentinc/cp-kafka:5.4.0 "/etc/confluent/do..." 25 minutes ago Up 17 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->29092/tcp broker

Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer

Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers

Couldn't resolve server broker:29092 from bootstrap.servers as DNS resolution failed for broke

Hello, can someone point me to the datahub python client source code? I need to know the setup.py file specifically.

Hi All, I'm trying to deploy DataHub with Postgres.

postgresqlSetupJob

is failing to start, the username(dev_datahub_app_user) is passed as database name. Below is the error

psql: error: connection to server at "<http://dev-datahub.cluster-cbcjxpl7jfef.us-east-1.rds.amazonaws.com|dev-datahub.cluster-cbcjxpl7jfef.us-east-1.rds.amazonaws.com>" (10.26.203.144), port 5432 failed: FATAL:  database "dev_datahub_app_user" does not exit

psql: error: connection to server at "<http://dev-datahub.cluster-cbcjxpl7jfef.us-east-1.rds.amazonaws.com|dev-datahub.cluster-cbcjxpl7jfef.us-east-1.rds.amazonaws.com>" (10.26.203.144), port 5432 failed: FATAL:  database "datahub" does not exist
2022/07/18 19:14:57 Command exited with error: exit status 2

When I am trying to configure OIDC in datahub-frontend. I'm getting this error. Can someone please help?

👋 Hello, team! Pls reconfirm the query whether still schema registry is a mandate prerequisite (or) we can skip it during helm chart install of prerequisites? PS: Thread https://datahubspace.slack.com/archives/CV2KB471C/p1643033131052800

How can I add the authentication info into ingestion yaml file?

Hi teams, does DataHub team have a plan for introducing visualization tools like tree map, stack map and so on?

Hello I'm having trouble setting up oidc in GKE, so I'm asking for help. As stated in the documentation, I want to set the following in gke, but I don't know how. https://datahubproject.io/docs/how/auth/sso/configure-oidc-react-google#4-configure-datahub-frontend-to-enable-oidc-authentication

AUTH_OIDC_ENABLED=true
AUTH_OIDC_CLIENT_ID=your-client-id
AUTH_OIDC_CLIENT_SECRET=your-client-secret
AUTH_OIDC_DISCOVERY_URI=<https://accounts.google.com/.well-known/openid-configuration>
AUTH_OIDC_BASE_URL=your-datahub-url
AUTH_OIDC_SCOPE="openid profile email"
AUTH_OIDC_USER_NAME_CLAIM=email
AUTH_OIDC_USER_NAME_CLAIM_REGEX=([^@]+)

So, i have two question. First, many yaml file exist In GKE. Which yaml file do i need to edit in GKE menu? service in "Services and Ingress" or deployment in "Workloads"? Second, where should i add "extraEnvs" in yaml file? Thanks

hi ,team How to switch login authentication to internal employee authentication

When I use docker/dev.sh to start, and log in with datahub datahub, then, it apprears 'Failed to log in! SyntaxError: Unexpected token < in JSON at position 1'

hi, local docker/dev.sh, 'atahub-actions_1 | 2022/07/19 123745 Problem with request: Get "http:///health": http:'

Hi, esError: Could not find or load main class "-Xms256m

Caused by: java.lang.ClassNotFoundException: "-Xms256m

Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]

output:

error:

at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:126)

at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)

at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)

at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:137)

at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)

Hi guys, in which format datahub stores info in kafka? Avro or json?

When I tried to run

docker/dev.sh

, I got the following error. It looks like the

debug

tag no longer exists for the image.

Error response from daemon: manifest for linkedin/datahub-kafka-setup:debug not found: manifest unknown: manifest unknown

https://hub.docker.com/r/linkedin/datahub-kafka-setup/tags?page=1&amp;name=debug I found that the

debug

tag is also missing for

datahub-frontend-react

and

datahub-elasticsearch-setup

images. Does anyone know how to work around this?

Hi team I have enable the Prometheus and jmx port for monintoring How can I see the metrics ?

ŸÍXDCEFVRGBTHNY/P;'[-PT5FR4E3SW2Q1A`2W36P-[=\

Hi, local run, it appreas, 'datahub-actions_1 | 2022/07/20 032818 Problem with request: Get "http:///health": http: no Host in request URL. Sleeping 1s', what happened?