Hi @early-lamp-41924 I found a weird mysql issue again 😭

Datahub supports SqlAlchemy data sources. Do we have any example of connecting kerberos+hive with GE and emitting assertions in Datahub.

I am looking for the best way to deploy datahub on a remote free server to ensure that all my users can access it ? ps : currently I use datahub on my local Windows machine following the quickstart guide

H, I think there is an error on the datahub-kafka-setup v0.8.31 image. Maybe a windows line ending inside the code. v0.8.29 works perfect. (sorry for the image, can’t copy text from VDI)

hi! I am trying to update my DH to 0.8.32, but the upgrade job is failing 😞

Hi guys, I have 2 simple questions: I installed datahub following the quickstart guid, I ingested my metadata and everything is fine, now I want to deploy datahub to be used in production on a remote server. I plan to use AWS or GCP for this. My first question is : to do this deployment, do I need to clone the project locally? My 2nd question is : when I finish the deployment and I access the datahub to use it, do I find all my data and configurations that I made before the deployment? Thanks for the help

Hello everyone, I want to use git and datahub for version control, But I have no idea how to structure this. The point is to have a git repository so I can know what recipes have been ingested and how(whether it was through the REST API or something else). I need to keep track of what metadata is being ingested in our office datahub and how. Eg: I have some glossary_term.yml file that I ingested into datahub but I would like to add CI/CD so that this happens automatically whenever it is updated. Any suggestions?

Hi, is there anyone who tried to deploy datahub on heroku ?

Hi All, I have a newb question that I'm sure someone on here will be able to answer. I am trying to follow the instructions to create users using the user.props file as per https://datahubproject.io/docs/how/auth/add-users however, for some reason I get a "The \"HOME\" variable is not set. Defaulting to a blank string." warning when I run datahub docker quickstart. I've looked in my home folder and there is a .datahub directory but the only file in there is telemetry-config.json. So, I created the C:\Users\user\.datahub\plugins\frontend\auth directory and put a user.props file in there, and restarted my datahub container. No luck with adding users this way. Can someone please shed some light on this? How can I set the /HOME/ variable, I am assuming this will fix the issue? Thanks.

Hi All, again... 🙂 What is the scope of the datahub docker quickstart command? If I run this command do I risk overwriting the local data in the existing deployed containers? What is a general strategy for managing deployed containers? Thanks in advance.

Hi folks, We're exploring DataHub as a data catalog service and we were wondering if we could simplify the deployment / dependency footprint a bit. From what I understand, we need to deploy the frontend, gms services as well as Kafka, a database (e.g MySQL / Postgres), Search svc (Elasticsearch) and a graph service (e.g Neo4j). Curious if we can skip some of these as it's a bit cumbersome for us to spin up all these dependencies upfront. Im wondering if we can: • Skip Kafka if we're ok using Rest based ingest and don't have any MAE consumers at the start • Store the graph index in ES like suggested in the docs - not sure how much of a risk this is perf wise

Hi team I have deployed data hub on kubernetes My gms and front end pods are running But Mae-consumer and Mce-consumer are not getting deployed

Just went through upgrading to the most recent helm release (v0.8.32 and I don't seem to have the lineage impact assessment? Shouldn't this have been included in v0.8.28?

Hi All, I have ingested some databases from a micrsoft sql server. Can anyone point me in the right direction as to how I enable the lineage tab on the database tables? I watched the lineage 101 seminar run by the datahub team but it only references Snowflake and BigQuery. Am I right in thinking perhaps this feature isn't yet available for MSSQL? Thanks in advance.

hihi penguin Hello, DataHub Community! We have been getting questions on building DataHub images and deploying them in an air-gapped environment, and we would love to gather knowledge from the Community teamwork If you have either built images in a completely air-gapped environment or you deployed inside one, please share your experiences/best practices! • Building in an air-gapped env: ◦ What are the steps you took to make sure all DataHub dependencies are downloaded from a custom artifactory? ◦ Were there any issues you ran into that other people would likely also face? • Deploying in an air-gapped env: ◦ Which deployment methods did you use to deploy in the air-gapped env? (Docker-compose, Kubernetes, etc.) ◦ What steps did you take to make sure the containers are available in the environment (if you did not build it yourself) ◦ Any gotchas that you faced while deploying?

what is PE_consumer_enabled in gms env settings?

I’m in a situation where I can’t use docker, and I’d like to know how to install datahub without using docker.

I have deployed Datahub with quick start and created a recipe in GCP VM. But when i run it throws an error Any idea how can make Datahub to use underlying VM service account for bigquery?

Hi, can i confirm that https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-gms/templates/config-jmx-exporter.yaml This JMX exporter is only for prometheus consumption? If i want to expose my JMX remotely, i should use

-Dcom.sun.management.jmxremote.port

and put in

JMX_OPTS

env variable? In my scenario, I have to expose the port for datadog agent to consume.

Hello everyone, if i want to use lineage of Spark Integration , but my spark version is 3.1.2 ,do i need to recompile the datahub with spark 3.1.2 ? thanks a lot

Hello everyone , I have deployed the datahub acryl action image pulled from public ecr having datahub version 0.8.28 But when I am deploying that the pods is crashed and giving this error:

Hi team, I am doing deployment of data hub on kubernetes One thing I noticed acryl datahub action image is from acryl data and other images are from LinkedIn Any reason for this ?

<!here> calling all Centos users - we have been seeing build and deploy errors on Centos and the core DataHub team doesn't have bandwidth to resolve all of them 🧵

Hi team, I have enable table lineage and view lineage still I am unable to see anything in my objects Please help @early-lamp-41924 and @big-carpet-38439

Hi team, I am trying to enable ingress for frontend and backend by following https://datahubproject.io/docs/deploy/aws/#expose-endpoints-using-a-load-balancer I want to enable ingress only for use inside the company and do not want to expose the IP outside. Should I change the

<http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>

to

internal

while deploying the chart? Also what do I do for the certificate-arn?

Does anyone know what the

datahub-actions

container is for. I’m trying to install it manually, but I can’t find the corresponding information。

@square-activity-64562 : I have deployed the datahub on kubernetes but when I am trying look into schema it is giving unknown error occurs How can I check logs as I am ingesting the recipe through cron job?

But the same code is working in my different dev environment

I want to look at deploying DataHub on AWS for existing resources. Can I use Postgres instead of MySQL for our relational database and what changes would be needed? We have another existing ElasticSearch cluster(on aws) would it be possible/easy to share the same resource that we have with another application? Is there anything we need to know?

https://datahubproject.io/docs/metadata-jobs/mce-consumer-job/ Hi all, I found out that helm deployment doesn’t expose any JMX port for MCE consumer but do expose on MAE… Is there really nothing to monitor on MCE? 🤔 https://github.com/acryldata/datahub-helm/blob/71072cbb0550823e5c10e1f7c6d214bb579[…]atahub/subcharts/datahub-mce-consumer/templates/deployment.yaml Happy to create a PR if monitoring on MCE is important too.