I am trying DataHub first time and I do not see AZURE STORAGE ( BLOB or ADLS GEN 2) as a source.

Will that be available in future?

Hello everyone! Quick question: is it possible to use the business logic only and create my own UX/UI of the tool?

Hi Guys, I am new to Datahub. I have tried the quick-start guide already but I need a customise way to deploy. Any tutorial on how to deploy datahub to an on-prem machine using Docker compose NOT the quickstart method.

Hi Team, I want to create my own custom metadata model...is there any easy way to do so?

This message was deleted.

Hi everyone! I'm trying to deploy datahub using helm chart, specify postgresql database:

values.yaml

host: "deps-postgresql.deps.svc.cluster.local:5432"
      hostForpostgresqlClient: "deps-postgresql.deps.svc.cluster.local"
      port: "5432"
      url: "jdbc:<postgresql://deps-postgresql.deps.svc.cluster.local:5432/dh02?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>"
      driver: "org.postgresql.Driver"
      username: "postgres"
      password:
        secretRef: postgresql-secrets
        secretKey: postgres-password
      # --------------OR----------------
      #   value: password

But postgresql setup job creates database

datahub

and ignores database

dh02

specified in connection string. Any minds?

2023/05/12 15:11:11 Waiting for: <tcp://deps-postgresql.deps.svc.cluster.local:5432>
2023/05/12 15:11:11 Connected to <tcp://deps-postgresql.deps.svc.cluster.local:5432>
CREATE DATABASE
-- create metadata aspect table

After job finished database

datahub

appears on the postgresql instance and tables present there, but db

dh02

is not.

Hi guys,

I'm trying to run the quickstart using datahub docker quickstart from Windows. It starts fine, but then gets stuck repeating this message:

[+] Running 0/0
 - Container mysql  Creating                                                                                       0.0s
Error response from daemon: invalid volume specification: 'C:\Users\janko\.datahub\mysql\init.sql:/docker-entrypoint-initdb.d/init.sql:rw': invalid mount config for type "bind": bind source path does not exist: c:\users\janko\.datahub\mysql\init.sql

I don't have a mysql folder in .datahub. Any ideas why this could be?

Hi Guys, I am getting the below error while executing ./gradlew quickstart

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Configure project dockermysql-setup

fullVersion=v0.10.2-147-g0fa983a.dirty cliMajorVersion=0.10.2 version=0.10.3-SNAPSHOT SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Task dockermysql-setup:docker

FAILED unknown flag: --load See 'docker --help'.

FAILURE: Build completed with 3 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task 'dockerkafka-setup:docker'.

Process 'command 'docker'' finished with non-zero exit value 125

* Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task 'dockerelasticsearch-setup:docker'.

Process 'command 'docker'' finished with non-zero exit value 125

* Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 3: Task failed with an exception. ----------- * What went wrong: Execution failed for task 'dockermysql-setup:docker'.

Process 'command 'docker'' finished with non-zero exit value 125

* Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.9.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 1m 10s 88 actionable tasks: 52 executed, 36 up-to-date (venv) Apples-MacBook-Pro-2:datahub apple$

Hi team, Looking to understand the "type of owners" concept in datahub? I see four different kinds of owners that datasets can have (Technical Owner, Business Owner, Data Steward and None). Can you tell me what impact does adding each of these types of owners have on the underlying asset? Will a user (with Reader role) be able to edit datasets by virtue of being an owner on any of these datasets

Hi there. What resources do you recommend for brand new DataHub users (like me!) who want to learn how to use DataHub to find stuff? Everything I've found at datahubproject.io, the DataHub blog on medium.com, and the DataHubProject channel on YouTube is for people who already know how to use DataHub or for people who are setting up DataHub for others to use. It's discouraging.

Hi, I am trying to build custom metadata model as per documentation but I was confused what should I mention in the extraVolumeMounts in the datahub-gms container?

Hi there, I have upgraded to the latest version of datahub (0.10.2.3). However I can no-longer get the broker service to start. I get.. ERROR Fatal error during KafkaServer startup... kafka.common.InconsistentClusterIdException. The Cluster ID ... doesn't match stored clusterId Some(...) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.

Hi! Having a look at DataHubSystemAuthenticator, doc mentions that:

This authenticator also looks for a "delegated actor urn" which can be provided by system callers using the 'X-DataHub-Actor' header.

However, the current logic does not match that, how is that? https://github.com/datahub-project/datahub/blob/master/metadata-service/auth-impl/[…]ub/authentication/authenticator/DataHubSystemAuthenticator.java

Hi team, A very naive question but how can everything done via Datahub UI be persisted over Git? Is it possible to back up everything to Git via yaml?

Hi all! Can I track how many users do search > click result > click link to dashboard? Can I track how many users actually use datahub? it’s like tracking the funnel of the datahub and monitoring whether it is really used, and whether it is really helpful.

Hi, Is it possible to run a full production datahub instance in a Minikube or are you advising not to run production in Minikube? Best Regards Chris

Hi All, I want to know how I can customize the logos and some parts of the react app, I have deployed datahub via docker and dont see the datahub web-react component. Any advise on this is appreciated.

Hi, can we add new custom aspect to a entity through the yaml configuration file ?

Hi team, I have successfully ingested hive datasets from hive metastore in mysql using presto-on-hive recipe. I was able to ingest a few datasets that I wanted for testing using

database_pattern

and

table_pattern

. Additionally, I'm trying to do the following things, and I need some advice on it. Any help would be appreciated. 1. Now I'm trying to ingest most of the datasets from

hive metastore

, and I was wondering if there's a way to do pattern filtering for other items in

hive metastore

as well: • DBS.DB_LOCATION_URI (e.g. allow only the pattern "hdfs://cluster1/dsc/.*") • DBS.OWNER_NAME (e.g. deny those with accounts ".*test") • If it is impossible via recipes, would there be any other possible ways? 2. Our organization manages additional metadata for hive's DB, table, and column in other mysql DB (say

our_meta

) which is separated from

hive metastore

. For example, for a table named

customer.cust_mst

on the hive, this table exists in the

hive metastore

, and a separate mysql DB also manages information about this table. Given the situation, I'd like to ingest the metadata of

our_meta

into datahub. What should be the best way to do it? • Some of the managed items in

our_meta

(mainly technical meta) seems to be managed as custom properties, and this should be synced in batch mode. • Some of the managed items in

our_meta

(mainly business meta) can be managed as business glossary or tags, and both batch sync and API calls should possible. • I am looking into custom ingestion source and metadata ingestion transformer, but I am not sure how to approach. I hope I can handle these without forking the source if possible. Thanks in advance.

Dear community, I am trying to start up datahub through the terminal using the command “datahub docker quickstart” but the process continues to run endlessly with the following message being displayed (attached in the picture). What exactly causes this error message? Any help is greatly appreciated Thanks.

Hi all, I am new to Datahub I am trying to work on a use-case to test spark lineage, with various data sources. I created a simple example where you read two files from hdfs join them and save it back to hdfs. this works on my local setup of datahub it shows spark lineage as shown in image below. When I point this to the datahub server deployed on cloud it doesn't show any upstream and downstream dataset lineage, as shown in first image and idea where I am going wrong?

Hi All, Could you please recommend a way how can I limit access for the user or for the group to platform resources? For example, I have resources: BigQuery, MetaBase, SQL Server, Kafka. • User A must have access to BigQuery, Metabase • User B must have access to MetaBase, SQL Server, Kafka

Hello, I'm wondering if the open source version of Datahub supports any kind of single sign on? I saw somewhere in the docs it mentioned only the cloud version supported OIDC but other docs didn't mention this so I wasn't sure..

Hi All! I am ingesting data from s3 using the datahub pipeline (

datahub.ingestion.run.pipeline

). Then to update properties, tags, owners etc: 1. I search for the uploaded dataset urn using the resli api with endpoint

/entities

because I do not know the urn beforehand 2. Update the urn The problem that I face with this approach is that urn creation takes some time depending on the dataset. And usually it is not ready when I try searching for it. Is there some way to know when the urn is created after the pipeline is run? Or there a better way to do this?

Dear community,I can see the lineage at the table level but not at the field level. Do I need additional configuration?