Hello, for a custom data pipeline, how can stateful-ingestion be enabled. Whats the best way to create checkpoints, are there some examples in git for this?

👋 everyone! I've got a query that returns the first degree downstream lineage for a given urn (see below). How would I enhance this to get all downstream relationships, regardless of degree?

{
    dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:dbt,database_name.schema_name.table_name,PROD)"){
        urn
        name
        lineage(input: { direction: DOWNSTREAM, start:0, count:10 }){
            relationships{
                type
                entity {
                  urn
                  type
                }
                degree
            }
        }
    }
}

Hi All,

Hi, I am getting below error while running datahub instance locally: root@fc5e309734ee:/# datahub docker quickstart No ~/.datahubenv file found, generating one for you... [2023-03-30 175733,278] INFO {datahub.cli.quickstart_versioning:144} - Saved quickstart config to /root/.datahub/quickstart/quickstart_version_mapping.yaml. [2023-03-30 175733,278] INFO {datahub.cli.docker_cli:638} - Using quickstart plan: composefile_git_ref='master' docker_tag='head' Docker doesn't seem to be running. Did you start it?

Hello eveyone! I am checking the column lineage . But I don't get how to create it, I am using lineage using file base lineage but I cannot see how to implement the column based lineage via file or via any other API. I have checked this API https://datahubproject.io/docs/lineage/lineage-feature-guide/ and there is no documentation. How is the Datahub column lineage supported? Thanks!

Hi @Everyone , new to datahub need help to disable rootuser in helm chart can anyone try to help me out

Looking at the DataHub API Comparison page, I don’t see any support listed for adding new custom properties to a dataset (without deleting existing properties). Is this operation supported?

Using the python SDK are you able to retrieve all datasets with a given platform (e.g, get all datasets in bigquery)?

Hey 👋 we are evaluating datahub and I was wondering if it’s possible to add information about reports/dashbaords from systems like ServiceNow or Salesforce ?

Hello everyone! Is it possible to add a dashboard manually (like using API)? I mean not to ingest a dashboard description from some BI system instance, but load it from some sort of a file (json, csv, whatever).

#getting-started Hello, when adding an 'owner' to an entity there is the option to choose an 'owner type' (technical owner, business owner, data steward, none). Is there a way to configure this and add types relevant to your organization, such as product owner, or remove types from the list?

Hello everyone!! Is the DataHub lineage row-level feature released? Where can I find the documentation?

Hi folks, two questions - 1. This page (https://datahubproject.io/docs/how/delete-metadata) talks about the ability to delete metadata from Datahub. However, the only option it talks about is deletion via the cli. Is there no API endpoint that can be used to delete metadata? 2. I want to understand the concept of URNs better. Can you please refer me to a guide around this. Essentially, I am looking to find the URN of any entity in Datahub. I would like to know about how it is constructed as well as how to find the value of any entity's URN. cc: @astonishing-answer-96712

Hi team. I run profiling ingestion on Hive, it shows error

"failed to write record with workunit {} with ('Unable to emit metadata to DataHub GMS', {'message': '413 Client Error: Request Entity Too Large for url: ', 'id': 'urn:li:dataset) and info {'message': '413 Client Error: Request Entity Too Large for url".

How to reduce the size or maximum request size limit

I have a question. Is it not possible to deactivate the 'readers view all asset' policy on the manager permissions page in the Datahub?

This message was deleted.

Hi all, hope you guys all well.. I just encountered a trouble...

"ConfigurationError: hive is disabled; try running: pip install 'acryl-datahub[hive]'\n"
           '[2023-04-07 12:23:15,107] INFO     {datahub.entrypoints:161} - DataHub CLI version: 0.8.33 at '
           '/tmp/datahub/ingest/venv-9768a823-ed62-4fae-ac1e-ca86febf3fca/lib/python3.9/site-packages/datahub/__init__.py\n'
           '[2023-04-07 12:23:15,107] INFO     {datahub.entrypoints:164} - Python version: 3.9.9 (main, Dec 21 2021, 10:03:34) \n'

but, I already have it installed and enabled...

$ pip install 'acryl-datahub[hive]'
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: acryl-datahub[hive] in /home/datahub/.local/lib/python3.9/site-packages (0.10.1.1)
Requirement already satisfied: docker in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (5.0.3)
Requirement already satisfied: mypy-extensions>=0.4.3 in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (0.4.3)
Requirement already satisfied: cached-property in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (1.5.2)
Requirement already satisfied: aiohttp<4 in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (3.8.4)
Requirement already satisfied: ratelimiter in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (1.2.0.post0)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (0.4)
Requirement already satisfied: toml>=0.10.0 in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (0.10.2)
Requirement already satisfied: expandvars>=0.6.5 in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (0.9.0)
Requirement already satisfied: packaging in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (23.0)
Requirement already satisfied: jsonref in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (1.1.0)
Requirement already satisfied: pydantic!=1.10.3,>=1.5.1 in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (1.10.7)
Requirement already satisfied: jsonschema in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (4.17.3)
Requirement already satisfied: typing-extensions>=3.10.0.2 in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (4.5.0)
Requirement already satisfied: click-spinner in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (0.1.10)
Requirement already satisfied: tabulate in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (0.8.9)
Requirement already satisfied: avro-gen3==0.7.10 in /home/datahub/.local/lib/python3.9/site-packages (from acryl-datahub[hive]) (0.7.10)
Requirement already satisfied: termcolor>=1.0.0 in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (1.1.0)
Requirement already satisfied: progressbar2 in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (4.0.0)
Requirement already satisfied: Deprecated in /usr/local/lib/python3.9/site-packages (from acryl-datahub[hive]) (1.2.13)
...................
      Successfully uninstalled jupyter_server-2.5.0
  WARNING: The script jupyter-server is installed in '/home/datahub/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  Attempting uninstall: great-expectations
    Found existing installation: great-expectations 0.16.6
    Uninstalling great-expectations-0.16.6:
      Successfully uninstalled great-expectations-0.16.6
  WARNING: The script great_expectations is installed in '/home/datahub/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
jupyter-events 0.6.3 requires traitlets>=5.3, but you have traitlets 5.2.1.post0 which is incompatible.
comm 0.1.3 requires traitlets>=5.3, but you have traitlets 5.2.1.post0 which is incompatible.
Successfully installed databricks-dbapi-0.6.0 great-expectations-0.15.50 ipykernel-6.17.1 jupyter-client-7.4.9 jupyter-core-4.12.0 jupyter-server-1.23.6 nbclient-0.6.3 traitlets-5.2.1.post0

$ datahub check plugins
Sources:
athena         (disabled)
azure-ad
bigquery       (disabled)
clickhouse     (disabled)
clickhouse-usage (disabled)
csv-enricher
datahub-business-glossary
datahub-lineage-file
datahub-stream
dbt            (disabled)
dbt-cloud
delta-lake     (disabled)
demo-data
druid          (disabled)
elasticsearch  (disabled)
feast          (disabled)
file
glue           (disabled)
hana
hive
iceberg        (disabled)
json-schema
kafka

Can someone tell me the solution? Thank you in advance.

Hello. I’m starting to explore datahub and I haven’t found yet a way to define some files (for instance excel files in a Sharepoint) as a dataset. What is the right way to do it? Use the CLI to create a custom platform (MS Sharepoint)? And define the custom datasets also programaticaly? Thanks in advance.

Hi team. I have a quick question about lineage emitting. I know that

dataset to dataset

lineage emit can be done by simple yaml file, is there (current or future) support for configuring

dataset -> datajob -> dataset

lineage with yaml file? Since I want my other teammates who are not familiar with datahub to easily configure lineage of there pipeline using yaml file, I was wondering if I have to make yaml parser that parses file and create lineage emitter using python sdk. Thanks in advance 😄

Hi, team. I am trying to configure using datahub docker image in private cloud environment. But looking at the list below, I'm not sure which one to choose. Could you please let me know the datahub docker image list of the most recent version required for new configuration? https://hub.docker.com/search?q=datahub&type=image&operating_system=linux

hii Team , i am not able to add secrets and glue catalog in datahub when i click on add then suddenly disappear from ui

Hello guys nice to meet you! I'm new to the datahub project, is there any material on good practices modeling domain and subdomains for metadata? I've already looked at the docs and the demo but it doesn't go deep on best practices Best regards Carlos

Hi, We have used datahub now for 6 months or so.. v0.10.1 was first time I see image tags not fixed. linkedin/datahub-frontend-react:v0.10.1 and linkedin/datahub-gms:v0.10.1 have changed yesterday although v0.10.1 of datahub has been out for 3 weeks? If we can't trust to tags how should we fix the deployments to certain versions?

Hi, I'm new to Datahub. Is it possible to deploy DataHub to EKS and use AWS managed services such as Elasticsearch and MSK but replace them with AWS RDS (Aurora MySQL) as the only managed service? I don't want use MSK, Elasticsearch (on aws).

Hi team, I have a use-case like the one mentioned below. Is there any way to achieve this currently?

We are going to use the tags feature of Datahub to store domain specific consumption of data products. In this we have a use-case where any time a tag / glossary item is added to an entity, can we trigger a workflow to run grant permissions on Snowflake?

Hello team , 1. i have already running kafka, zookeeper ,mysql, schema-registry on my server, Can these replace into datahub docker yaml file ? 2. can i use postgres RDS instance instead of mysql maria-db ? please let me know if anyone have any idea. thanks

How to set schema-registry:8081 authentication?

Hi all! We are trying out the Java emitter API, however we cannot find any documentation. Do you have links to that?

Hello team, Is there someone know how to change backend db from mysql to postgresql in configuration helm chart? I tried to add this under datahub-gms but it seems like always error

extraEnvs:
    - name: EBEAN_DATASOURCE_USERNAME
      value: ""
    - name: EBEAN_DATASOURCE_PASSWORD
      value: ""
    - name: EBEAN_DATASOURCE_HOST
      value: ""
    - name: EBEAN_DATASOURCE_URL
      value: ""
    - name: EBEAN_DATASOURCE_DRIVER
      value: org.postgresql.Driver

Anyone know how to connect backend db using cloudsql postgres? Notes • I’m using version 0.10

Hello Team! I have installed datahub via quickstart, and when I am trying to hit the gms-service I am getting this error from the docker container:

2023-04-12 10:52:17 io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
2023-04-12 10:52:17 Caused by: java.net.ConnectException: Connection refused
2023-04-12 10:52:17     at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
2023-04-12 10:52:17     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
2023-04-12 10:52:17     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
2023-04-12 10:52:17     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
2023-04-12 10:52:17     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
2023-04-12 10:52:17     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
2023-04-12 10:52:17     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
2023-04-12 10:52:17     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
2023-04-12 10:52:17     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2023-04-12 10:52:17     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2023-04-12 10:52:17     at java.base/java.lang.Thread.run(Thread.java:829)
2023-04-12 10:52:19 2023-04-12 08:52:19,961 [R2 Nio Event Loop-1-2] WARN  c.l.r.t.h.c.c.ChannelPoolLifecycle:139 - Failed to create channel, remote=localhost/127.0.0.1:8080
2023-04-12 10:52:19 io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
2023-04-12 10:52:19 Caused by: java.net.ConnectException: Connection refused
2023-04-12 10:52:19     at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
2023-04-12 10:52:19     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
2023-04-12 10:52:19     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
2023-04-12 10:52:19     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
2023-04-12 10:52:19     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
2023-04-12 10:52:19     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
2023-04-12 10:52:19     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
2023-04-12 10:52:19     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
2023-04-12 10:52:19     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2023-04-12 10:52:19     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2023-04-12 10:52:19     at java.base/java.lang.Thread.run(Thread.java:829)
2023-04-12 10:52:24 2023-04-12 08:52:24,165 [R2 Nio Event Loop-1-3] WARN  c.l.r.t.h.c.c.ChannelPoolLifecycle:139 - Failed to create channel, remote=localhost/127.0.0.1:8080
2023-04-12 10:52:24 io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
2023-04-12 10:52:24 Caused by: java.net.ConnectException: Connection refused
2023-04-12 10:52:24     at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
2023-04-12 10:52:24     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
2023-04-12 10:52:24     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
2023-04-12 10:52:24     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
2023-04-12 10:52:24     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2023-04-12 10:52:24     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2023-04-12 10:52:24     at java.base/java.lang.Thread.run(Thread.java:829)
2023-04-12 10:52:24 2023-04-12 08:52:24,167 [pool-14-thread-1] ERROR c.d.m.ingestion.IngestionScheduler:244 - Failed to retrieve ingestion sources! Skipping updating schedule cache until next refresh. start: 0, count: 30
2023-04-12 10:52:24 com.linkedin.r2.RemoteInvocationException: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://localhost:8080/entities>
2023-04-12 10:52:24     at com.linkedin.restli.internal.client.ExceptionUtil.wrapThrowable(ExceptionUtil.java:135)
2023-04-12 10:52:24     at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponseImpl(ResponseFutureImpl.java:130)
2023-04-12 10:52:24     at com.linkedin.restli.internal.client.ResponseFutureImpl.getResponse(ResponseFutureImpl.java:94)
2023-04-12 10:52:24     at com.linkedin.common.client.BaseClient.sendClientRequest(BaseClient.java:51)
2023-04-12 10:52:24     at com.linkedin.entity.client.RestliEntityClient.list(RestliEntityClient.java:365)
2023-04-12 10:52:24     at com.datahub.metadata.ingestion.IngestionScheduler$BatchRefreshSchedulesRunnable.run(IngestionScheduler.java:220)
2023-04-12 10:52:24     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
2023-04-12 10:52:24     at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
2023-04-12 10:52:24     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
2023-04-12 10:52:24     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2023-04-12 10:52:24     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2023-04-12 10:52:24     at java.base/java.lang.Thread.run(Thread.java:829)
2023-04-12 10:52:24 Caused by: com.linkedin.r2.RemoteInvocationException: Failed to get response from server for URI <http://localhost:8080/entities>
2023-04-12 10:52:24     at com.linkedin.r2.transport.http.common.HttpBridge$1.onResponse(HttpBridge.java:67)
2023-04-12 10:52:24     at com.linkedin.r2.transport.http.client.rest.ExecutionCallback.lambda$onResponse$0(ExecutionCallback.java:64)
2023-04-12 10:52:24     ... 3 common frames omitted
2023-04-12 10:52:24 Caused by: com.linkedin.r2.RetriableRequestException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
2023-04-12 10:52:24     at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.onError(ChannelPoolLifecycle.java:142)
2023-04-12 10:52:24     at com.linkedin.r2.transport.http.client.common.ChannelPoolLifecycle.lambda$create$0(ChannelPoolLifecycle.java:97)
2023-04-12 10:52:24     at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
2023-04-12 10:52:24     at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
2023-04-12 10:52:24     at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
2023-04-12 10:52:24     at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
2023-04-12 10:52:24     at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
2023-04-12 10:52:24     at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
2023-04-12 10:52:24     at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
2023-04-12 10:52:24     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
2023-04-12 10:52:24     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
2023-04-12 10:52:24     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2023-04-12 10:52:24     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2023-04-12 10:52:24     ... 1 common frames omitted
2023-04-12 10:52:24 Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:8080
2023-04-12 10:52:24 Caused by: java.net.ConnectException: Connection refused
2023-04-12 10:52:24     at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
2023-04-12 10:52:24     at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)
2023-04-12 10:52:24     at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
2023-04-12 10:52:24     at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
2023-04-12 10:52:24     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
2023-04-12 10:52:24     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2023-04-12 10:52:24     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2023-04-12 10:52:24     at java.base/java.lang.Thread.run(Thread.java:829)

Could you help me figure out what could be the problem! Thank you in advance!