can anyone tell me how to use Datahub Action integration with Big Query and Great Expectations. @big-carpet-38439

Just a note on deleting: you can use

stateful_ingestion

and configure it to delete datasets that don't show up in the latest ingestion. For example, see the

stateful_*

configuration options on sources like hive. Loving datahub!

TIL the hard way that the

simple_add_dataset_tags

transformer will replace any existing tags you have on a dataset. Keep an eye out for that!

@big-carpet-38439 I see all the Ingestion sources that was added through the frontend Ingestion has been running twice for the same scheduled time. As far as the entry in the table I see only 1 entry per source

select * from metadata_aspect_v2 where urn like 'urn:li:dataHubIngestionSource:%' and version = 0 and aspect = 'dataHubIngestionSourceInfo'

Do you know what else could trigger this duplicate execution?

Hi all 👋 I’m seeing some weird behaviour when trying to delete all dataset for dbt using the command below I can see the right number of entities to delete is being detected correctly however when checking the UI the data is still available (thought this could be down to timing or cache but is doesn’t look like) — Screenshots in thread

datahub delete --env PROD --entity_type dataset --platform dbt --hard

can i ingest business glossary through datahub UI

Hi all, can anyone tell me what all informations can be ingested using the Java/ Python emitter provided by Datahub?

hi,team,!I want to initialize our users to the datahub in the ingestion module. How should I build it with corpUser,Are there any encapsulation methods in metadata-ingestion module

Hey guys, I am having trouble creating the lineages. I want to create a lineage between my S3 tables and my Redshift tables which have the same name. Do you guys have an example of how I could do that ? The dataset-dataset lineage example they give on Git is too simple... I can't make it work. Thank you so much!

Hi all. I can ingest from snowflake using UI but it failed using CLI. Anyone has any idea? Thank you.

Hi DH community! I have a couple of use cases where I require to apply a function to all datasets (eg set some aspect). One option could be to define such a function as a transform and apply the transform in all connectors and ingestions. However, that would left out all datasets ingested via push. Also, in some use case I may require some aggregated view of the datasets. So: Is DH providing some interface to do such batch processing? or any plan for that? Has anyone here solved this use case before? Thanks!

Hi all, there is a command/api to check if dataset exist in datahub?

Hi all, Need the help from the folks here as I am not very familar with restli. Was trying to ingest some dataset by calling the method "ACTION_INGEST" in (https://github.com/datahub-project/datahub/blob/master/metadata-service/restli-ser[…]java/com/linkedin/metadata/resources/entity/EntityResource.java). However, the code now is not able to capture the actor/user making the call to the method. Where should we make the necessary changes to include auditStamp information?

I ingest data from hive and debug log of datahub-action below. I run datahub in company with proxy. What should i do? Tks

What permissions do I need to open before I can see the schema, table and column in the dataset

No matter which one is selected, it is always loading why,Some plug-ins failed to load successfully????

We are trying to ingest Oracle database,

sudo python3 -m pip install cx_Oracle --upgrade
sudo python3 -m pip install cx_Oracle --upgrade --user

hi,team.I would like to ask if glossary_terms only two levels are supported? I would like to ask whether it supports multi-level

So, looking at datahub I was wondering what would be the best way to handle the following scenario with ingestion: I have a kafka deployment on kubernetes through helm charts. I am declaring it like so:

apiVersion: <http://kafka.strimzi.io/v1beta2|kafka.strimzi.io/v1beta2>
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    <http://strimzi.io/cluster|strimzi.io/cluster>: my-cluster
    <http://infra.mycompany.net/dev_team|infra.mycompany.net/dev_team>: dev_team_a
    <http://infra.mycompany.net/ops_team|infra.mycompany.net/ops_team>: ops_team_a
    <http://infra.mycompany.net/app_id|infra.mycompany.net/app_id>: app-0001
spec:
  partitions: 10
  replicas: 3
  config:
    <http://retention.ms|retention.ms>: 604800000
    segment.bytes: 1073741824

What would be the best way that I can either get this data into the datahub system? I dont know if it would be best to go and make commits to the kafka integration that supports connecting to kubernetes and getting the metadata, or if there's a better way? Kind of a huge edge case considering the metadata is in the kubernetes layer, but that's the only way we found to manage these kinds of metadata

Hello, Anybody was successful connecting to Synapse using MSSQL plugin? I could not get it working. I am using “ODBC Driver 17 for SQL Server”. I would appreciate if you could share your experience. Thanks, Mahbub

I am looking to be able to add the ability to insert a custom property as part of ingestion. Could someone point me towards the code for adding a config option to ingestion?

General question about ingestion and tagging - is the idea that the tags like owner, domain, documentation etc are all manually added in the datahub UI?

During oracle ingestion we are getting below errors. Please suggest.

used by: java.lang.RuntimeException: Failed to validate entity URN '
'urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.getUrnFromProposal(EntityKeyUtils.java:33)\n'
'\tat com.linkedin.restli.internal.server.RestLiMethodInvoker.doInvoke(RestLiMethodInvoker.java:177)\n'
'\t... 78 more\n'
'Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields '
'do not have same length\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:97)\n'
'\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)\n'
'\tat java.lang.Thread.run(Thread.java:748)\n'
'Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException\n'
'\tat com.datahub.util.RecordUtils.invokeProtectedMethod(RecordUtils.java:355)\n'
'\tat com.datahub.util.RecordUtils.getRecordTemplateField(RecordUtils.java:258)\n'
'\tat com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:30)\n'
'\t... 84 more\n'
'Caused by: java.lang.reflect.InvocationTargetException\n'
'\tat sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)\n'
'\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n'
'\tat com.datahub.util.RecordUtils.invokeProtectedMethod(RecordUtils.java:353)\n'
'\t... 90 more\n'
'Caused by: com.linkedin.data.template.TemplateOutputCastException: Invalid URN syntax: Invalid number of '
'keys.: urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.common.urn.DatasetUrn$1.coerceOutput(DatasetUrn.java:78)\n'
'\tat com.linkedin.data.template.RecordTemplate.obtainCustomType(RecordTemplate.java:366)\n'
'\t... 94 more\n'
'Caused by: java.net.URISyntaxException: Invalid number of keys.: '
'urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.common.urn.DatasetUrn.createFromUrn(DatasetUrn.java:49)\n'
'\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:918)\n'
'\tat java.lang.Thread.run(Thread.java:748)\n'
'Caused by: java.lang.RuntimeException: Failed to validate entity URN '
'urn:li:dataset:(urn:li:dataPlatform:oracle,b0225565.PORTIN_,MAR22,PROD)\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.getUrnFromProposal(EntityKeyUtils.java:33)\n'
'\tat com.linkedin.restli.internal.server.RestLiMethodInvoker.doInvoke(RestLiMethodInvoker.java:177)\n'
'\t... 78 more\n'
'Caused by: java.lang.IllegalArgumentException: Failed to convert urn to entity key: urns parts and key fields '
'do not have same length\n'
'\tat com.linkedin.metadata.utils.EntityKeyUtils.convertUrnToEntityKey(EntityKeyUtils.java:97)\n'

Could you please help me to answer questions? how to display mssql icon in new ingestion source page,This command has been executed

sudo python3 -m pip install 'acryl-datahub[mssql]'

Hello Members, I do not see any plugin for Azure Databricks. Anyone has any experience of connecting to Databricks using out-of-the-box or custom plugin? Thanks, Mahbub

Hi all, I’ve read about Snowflake column lineage support here, but I don’t see it even with

include_table_lineage: True

on Column Stats tab. Is it a feature exclusive to Acryl data?

Also I’m having difficulty showing lineage from s3 to snowflake table though all the lineage between snowflake tables are shown fine. Can DataHub reflect the ‘copy into’ query to show its lineage? Thanks in advance.

Hi there! Quick question about ingesting metadata from Looker into DataHub. We would need to ingest additional metadata from Looker than DataHub extracts (specifically we need dashboard

view_count

,

updated_at

,

last_updater_name

, etc.). How could we go about ingesting that additional metadata? Would we have to write a transformer?

Hi Team, I would like to pull the default datahub credentials that is currently available thru

user.props

to be more secure, Either pulled in via ENV var or otherwise...Could you help on this?

Hello, I am running datahub in Azure Kubernetes Services(AKS) using Helm chart. I can access the UI but I am not able to ingest the data. How do I find out REST endpoint for ingestion? Here is my gms pod and service up and running.