Good day I had the local version of datahub up and running, but deleted it to try to restart. But now I'm getting this error:

datahub-graphql-core:compileJava FAILED

or this

failed to create parent directory

Has anyone ever had this error before? Could it be linked to my jdk?

What does datahub suggest to deal with non-structural data like zip files. Is datahub able to manage them?

Hi everyone! I am new to DataHub! We have a use-case to solve for which we are looking into DataHub. We have our raw data coming from Kafka to GCS in Hudi format (raw layer). Next we want to create derived tables/layer after proper Schema validation check. We will be using Spark to move/process data from raw layer to derived layer. What we want to achieve via DataHub: • Store schema of tables in DataHub • Get the schemas in Spark from DataHub • Apply the schema to the data PS: We don't only want to use DataHub as Schema Registry but this one of the use-case we want to solve.

I am trying to use Python Emitter to emit databricks notebook details to Datahub. Any Examples?

Hi, Im on RHEL OS and Im getting some errors when I try to deploy datahub with quickstart. Im using latest datahub version. Errors: - kafka-setup is still running - schema-registry is not running - broker is not running - datahub-frontend-react is running but not healthy - datahub-gms is still starting - elasticsearch-setup is still running - elasticsearch is running but not healthy HELP! Anyone knows how can I fix it? Thanksss.

Good day people! I have more questions 😄 With the bigquery-usage feature, is it possible to get the most used datasets/tables?

Hey guys, is there a way to restrict access to specific assets to users/groups. Thanks in advance.

I’m trying to load run_results information from dbt using the run_results.json, but my validation tab leads to a blank page (essentially an about:blank), am I missing something here?

For anyone else having the error with building psutil while installing the datahub cli I found that the python you're using matters (not just the version) - switching to the python from python.org solved the problem (my default python3 was coming from a MinGW installation, causing the issue)

re: custom Actions, the docs say this about installing a custom action:

The easiest way to do this is to just place it in the same directory as your configuration file, in which case the module name is the same as the file name - in this case it will be

custom_action

.

rn my file is

my_action.py

, in the same dir as my

my_action.yaml

config file. however, having

type: my_action:ActionClassInside

doesn’t work. what should the correct format be?

Hi All, I have CSV files in SFTP and Azure blob store. How should I configure the profiling? It appears that there is only an option for local and S3 ( data-lake-files) but not for SFTP or Azure blob storage. Is there a recommendation?

Is it possible to send metadataChangeProposals via the java kafka emitter to topic

MetadataChangeEvent_v4

instead of

MetadataChangeProposal_v1

?

how do you run the quickstart when behind a company proxy? I'm getting a SSL self signed certificate error when running

datahub docker quickstart

This message was deleted.

I know this question regarding use of OpenSearch(1.2) as an alternative to a hosted ElasticSearch have been asked before. But it doesn’t fully work for us -

....
elasticsearchSetupJob:
      enabled: true
      image:
        repository: linkedin/datahub-elasticsearch-setup
        tag: "v0.8.40"
      extraEnvs:
        - name: USE_AWS_ELASTICSEARCH
          value: "true"    
....

it triggers a job that finishes successfully The data ingestion works fine updating ES index. But analytics fail with

type	:	illegal_argument_exception
reason	:	Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [browserId] in order to load field data by uninverting the inverted index. Note that this can use significant memory.

now I can successfully send POST to -

<https://opensearch-domain-endpoint:443/datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true>

Because of above GMS k8s pod is getting evicted periodically. Which affects all datahub setup. Any suggestions?

sos

How can I use the function of modeling schemaMetadata and profiling for s3 objects without ingesting it into datahub (without doing 'datahub ingest -c'). I found the ability to extract schemaMetadata and profile on each column very helpful, but want to manage them myself.

Hi,everyone,I installed the datahub with python3. In which directory do I save docker_compose files at startup? Thanks in advance for your help. https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/cli/docker.py

hi guys! What is the difference between inherits and contains in glossary ?

hi everyone, what's the default Grafana user/passwd for minitor installation

Hey friends, anyone using Key Pair Authentication for Snowflake ingestion? I have my initial Datahub deployed in an AWS EC2 using docker-compose but I am not sure where should I store the public key in the EC2. Should I create a specific volume, and for which service/container

datahub-gms

?

Hi everyone! We are on process to upload the OpenAPI to Microsoft API Manager and, our colleages on the team has asked us to upload the JSON export of the Swagger UI, is possible to access the JSON format somehow? Thanks in advance!

Hello everyone. Am trying to deploy datahub on aws kubernetes . I am new to kubernetes as well. Is there a way we can deploy datahub as a service? Rather than using kubectl to run helm?

Hello Everyone, Is there any document available for Role Based Access In Datahub? I tried Applying some Privileges to a Policy but when I am logging in back I can see all the tabs. FYI I have applied 'View Analytics' Privileges only and I can see other tabs too.

hi everyone. Anyone using DataHub with Data Mesh? Was wondering what DataHub Entity type you use to catalog a data product?

Hello guys, I've set the Glossary Terms via file ingestion. Then, I had to change the name of some of the glossary terms and, instead of reingesting the glossaries via the yaml file, I've renamed it directly via the UI. However, the URN stayed the same name as before. How should I correct this ? Thanks !

I have deployed datahub from frontend and gms containers and i see after sometime both the containers are going in unhealthy state and gms container is going into exited state as well. when i checked the log of datahub frontend container, i see below exception error as below java.net.UnknownHostException: broker at java.net.InetAddress.getAllByName0(InetAddress.java:1282) at java.net.InetAddress.getAllByName(InetAddress.java:1194) at java.net.InetAddress.getAllByName(InetAddress.java:1128) at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:104) at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403) at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363) at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151) at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:943) at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:68) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1114) at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1005) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:537) at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:331) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:238) at java.lang.Thread.run(Thread.java:748) 073519 [kafka-producer-network-thread | datahub-frontend] WARN o.apache.kafka.clients.NetworkClient - [Producer clientId=datahub-frontend] Error connecting to node broker:29092 (id: 1 rack: null)

My team is evaluating options for Data Lineage and Catalog solutions. We have done a POC with Apache Atlas. And want to compare this with DataHub which provides a lot more connectors.(Particularly interested in RDBMS(Postgres and MySQL)) I am curious does DataHub support real time updates for Postgres or is it based on batch jobs.

Hello everyone. For datahub on aws production deployment, did everyone steps as in document or created any images or cdk for it