How do people specify foreign keys in datahub? Adding descriptions would be too cumbersome

Suggestion: Maintain a page with breaking changes like apache superset does. https://github.com/apache/superset/blob/master/UPDATING.md. While github releases are good, the way apache superset has been doing it https://github.com/apache/superset/tree/master/RELEASING/release-notes-1-2 is great

Hi guys! Any chance to see the recording of the last Townhall?

In the townhall new CLI commands were shown which could do rollbacks. In a kubernetes deployment where would we run those commands? Inside gms pod? Will that rollback the profiling stats? Asking because I would like to enable the profiling stats. But the

sample values

is something that I am not sure I would like to have in the UI. So wanted to try to ingest it once. But in case wanted to revert it should be able to do that

Hi, I'm new DataHub but very in its capabilities. I'm looking some guidance or support on deploying GKE. Any guidance or support much appreciated

Not sure what's the right channel but... In The super awesome feature of dataset stats seems to rely on time series based datastore. It sounds like that data is not persisted in MySQL but instead is only stored in ES... In case of some issues and a need for running reindexing of ES how is that data preserved? Or would we need to fully re-ingest this data after reindexing?

When is the next release going to happen? Looking forward to some of the new things which are already in.

Is there any way to search for datasets with/without owners? I can see "21.94% have owners assigned!". I am trying to get a breakdown for this so I can see something like

users  -> owner of number of datasets
groups -> owner of number of datasets

Dear Datahub team - can we have another group for installation and infrastructure setup please

Is there a ping endpoint for gms? I would like to test connectivity from a different cluster to see if I able to reach gms

Welcome @little-smartphone-52405 and @late-pizza-91254!

will the rbac feature timeline be shifted back further? i think the last ETA communicated was in mid Aug?

Welcome @chilly-wall-98962! You are member 1000 🙂

Welcome @full-refrigerator-38116! Just because you are 1001 doesn't mean you're not special for us 🙂

Hi … is there a documentation available detailing the differentiators from Amundsen ?

Is there an example of View

DatasetLineageTypeClass

https://github.com/linkedin/datahub/blob/5eee818a619e27300f8ae3cf749d3f82aa23f43e/metadata-ingestion/src/datahub/metadata/schema_classes.py#L2279 Does it show up differently in the UI?

Dear team. Does anyone know, how to clean metadata correctly (both mysql and elastic)? Not as big as

docker/nuke.sh

, just drop meta P.S. It is about dev env, running from

docker/dev.sh

, so not related to any production cases

<!here> 📣 The big July release is here! We released DataHub 0.8.7 earlier today along with the

acryl_datahub

PyPi package (version 0.8.7.0). Looking forward to hearing all the awesome things you all will do with this!

Hi I am trying to run update

docker pull acryldata/datahub-upgrade:head && docker run --env-file ~/setup/docker/datahub-upgrade/env/docker.env acryldata/datahub-upgrade:head -u NoCodeDataMigration

but getting below exception to build docker image

ERROR SpringApplication Application run failed
 org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'upgradeCli': Unsatisfied dependency expressed through field 'noCodeUpgrade'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.linkedin.gms.factory.entityregistry.EntityRegistryFactory': Unsatisfied dependency expressed through field 'configEntityRegistry'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configEntityRegistry' defined in class path resource [com/linkedin/gms/factory/entityregistry/ConfigEntityRegistryFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.io.FileNotFoundException: ../../metadata-models/src/main/resources/entity-registry.yml (No such file or directory)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:643)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:116)
	at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1422)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:879)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:316)
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139)
	at com.linkedin.datahub.upgrade.UpgradeCliApplication.main(UpgradeCliApplication.java:13)
...
Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'com.linkedin.gms.factory.entityregistry.EntityRegistryFactory': Unsatisfied dependency expressed through field 'configEntityRegistry'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configEntityRegistry' defined in class path resource [com/linkedin/gms/factory/entityregistry/ConfigEntityRegistryFactory.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.models.registry.ConfigEntityRegistry]: Factory method 'getInstance' threw exception; nested exception is java.io.FileNotFoundException: ../../metadata-models/src/main/resources/entity-registry.yml (No such file or directory)

good afternoon. Would datahub discover the SSIS packages for pipelines?

In terms of observability, is DataHub exposing metrics such us number of entities?

Hi folks..... I'm evaluating datahub as a potential data catalog for our org. I ingested some test metadata but now want to delete it from datahub. I trying to follow the delete instructions here: https://datahubproject.io/docs/how/delete-metadata but my datahub server is remote. How can i specify the server url using the datahub cli ? Cheers -- Simon

Does anyone know if its possible to restrict the metadata users can see by using role based access control in datahub. Our org wants to restrict what particular users can see within the datahub catalog.

Hello! I've been looking at extending the built-in metadata using the [No Code Metadata](https://datahubproject.io/docs/advanced/no-code-modeling) instructions, but it seems the [MCP/MCL stuff](https://datahubproject.io/docs/advanced/mcp-mcl) might make some changes to that process. Do the steps remain the same? Also, are all the old topics (MCE, MAE, FCE) and new topics (MCP, FMCP, MCLV, MCLT) required for a new deployment? Thanks!

PSA: github seems to be having some stability issues. Please write code slowly 🐢

Are there currently any efforts or discussion to enable dynamic/runtime Entity definition?

Hi all.... I'm trying to use curl to retrieve details on a dataset I added to datahub. No matter what I try, I always receive the message "You need to enable JavaScript to run this app". I'm running the following curl command:

curl -H 'X-RestLi-Protocol-Version:2.0.0' -H 'X-RestLi-Method: get'  '<https://datahub.xxxxxxxx-nonprod-1.yyyyyyyy-datalake-nonprod.com.au/dataset/urn%3Ali%3Adataset%3A(urn%3Ali%3AdataPlatform:redshift,datalake.kayo_datalake_transform_current.ares_masterindex_daily_snapshot_view,DEV)/schema>'

I thought maybe I'm using the wrong endpoint, but curl for the config works fine:

curl '<https://datahub.xxxxxxxx-nonprod-1.yyyyyyyy-datalake-nonprod.com.au/config>'

Does anyone have any ideas ?

This message was deleted.

<!here> 📣 Release 0.8.9 just dropped! along with the companion pip package

acryl_datahub

v0.8.9.0. Release Highlights: • Support for nested structs, union types and key-value schemas in Kafka • Support for JDBC Connector based sources in Kafka Connect • Support for Okta as a source for User and Group metadata • Support for using AWS Glue schema registry Detailed notes are here. Enjoy! 🔥

Is there support for classifications. Currently in the demo site can see tag "classification:confidential" , is this the structure ? Could not find much documentation regarding same or PII fields , apologize if I missed it.