Hello Team. Can I setup lineage between two different dags within datahub? I would want to set the outlet of my last task to point to the first task in one of my dags. Is this possible?

Hi, everyone I liked the data hub, but i could not run quick start to test, return "Unable to run quick start", i checked the docker service, its ok, running. I run the datahub version, returned the "DataHub CLI version: 0.8.39 Python version: 3.10.4 (main, Apr 2 2022, 090419) [GCC 11.2.0]" Can you help me?

Hello, everyone I'm having problemns with MAE Consumer. Can someone help me?

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
18:27:10.629 [main] ERROR o.s.b.d.LoggingFailureAnalysisReporter - 

***************************
APPLICATION FAILED TO START
***************************

Description:

Parameter 1 of constructor in com.linkedin.metadata.kafka.hook.siblings.SiblingAssociationHook required a bean named 'entityAspectDao' that could not be found.

The injection point has the following annotations:
        - @javax.annotation.Nonnull(when=ALWAYS)


Action:

Consider defining a bean named 'entityAspectDao' in your configuration.

Maybe @bulky-soccer-26729 could help here... I deleted one of the Glossary Nodes urn using the CLI but after that I am facing issues where frontend is not completely loading and I get the below error in GMS. Tagged Chris as I thought he was the one working closely with Glossary. Error stack in 🧵

Hello, having trouble while mssql ingestion with table description or column description, with next error:

ProgrammingError: (pyodbc.ProgrammingError) ('ODBC SQL type -150 is not yet supported.  column-index=2  type=-150', 'HY106')

DataHub CLI version: 0.8.39
Python version:      3.8.0 
SQLAlchemy           1.3.24
pyodbc               4.0.32
sqlalchemy-pytds     0.3.4

Can someone help me?

Morning all, I am trying to restore the indices via the restore-indices-job. I am getting logs like the following:

2022-06-30 10:56:22.625 ERROR 1 --- [           main] c.l.m.dao.producer.KafkaEventProducer    : Failed to emit MCL for entity with urn urn:li:corpuser:<username removed>
org.apache.kafka.common.errors.TimeoutException: Topic MetadataChangeLog_Versioned_v1 not present in metadata after 60000 ms.

I am not sure that it is actually doing anything and the process is running for more than 20 hours. So, I am getting failed to emit MCL for entity for all the entities. Is that normal? How can I fix that?

Hello everyone! I'm trying to make alerting on failed Great Expectations assetions with Actions Framework. I noticed that Kafka Event Souches only receives assertion DELETE or UPSERT events, but not assertion status changing. Is there any way to catch such event?

Hello, I got the docker images running on mac by switching manuallly to the m1 yml file. I have a strange other issue with the database image.

2022-06-30 11:56:04+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.

2022-06-30 11:56:06+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

2022-06-30 11:56:06+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.

2022-06-30 12:03:02+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.

2022-06-30 12:03:04+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'

2022-06-30 12:03:04+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.5.8+maria~focal started.

2022-06-30 11:56:08 0 [Note] mysqld (mysqld 10.5.8-MariaDB-1:10.5.8+maria~focal) starting as process 1 ...

2022-06-30 11:56:08 0 [Note] InnoDB: Using Linux native AIO

2022-06-30 11:56:08 0 [Note] InnoDB: Uses event mutexes

2022-06-30 11:56:08 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2022-06-30 11:56:08 0 [Note] InnoDB: Number of pools: 1

2022-06-30 11:56:08 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions

2022-06-30 11:56:08 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)

2022-06-30 11:56:08 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728

2022-06-30 11:56:08 0 [Note] InnoDB: Completed initialization of buffer pool

2022-06-30 11:56:08 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2022-06-30 11:56:08 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MySQL 5.7.38.

2022-06-30 11:56:08 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error

2022-06-30 11:56:08 0 [Note] InnoDB: Starting shutdown...

2022-06-30 11:56:08 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-06-30 11:56:08 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

2022-06-30 11:56:08 0 [Note] Plugin 'FEEDBACK' is disabled.

2022-06-30 11:56:08 0 [ERROR] Could not open mysql.plugin table: "Unknown storage engine 'InnoDB'". Some plugins may be not loaded

2022-06-30 11:56:08 0 [ERROR] Unknown/unsupported storage engine: InnoDB

2022-06-30 11:56:08 0 [ERROR] Aborting

2022-06-30 12:03:06 0 [Note] mysqld (mysqld 10.5.8-MariaDB-1:10.5.8+maria~focal) starting as process 1 ...

2022-06-30 12:03:06 0 [Note] InnoDB: Using Linux native AIO

2022-06-30 12:03:06 0 [Note] InnoDB: Uses event mutexes

2022-06-30 12:03:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2022-06-30 12:03:06 0 [Note] InnoDB: Number of pools: 1

2022-06-30 12:03:06 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions

2022-06-30 12:03:06 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)

2022-06-30 12:03:06 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728

2022-06-30 12:03:06 0 [Note] InnoDB: Completed initialization of buffer pool

2022-06-30 12:03:06 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2022-06-30 12:03:06 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MySQL 5.7.38.

2022-06-30 12:03:06 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error

2022-06-30 12:03:06 0 [Note] InnoDB: Starting shutdown...

2022-06-30 12:03:06 0 [ERROR] Plugin 'InnoDB' init function returned error.

2022-06-30 12:03:06 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

2022-06-30 12:03:06 0 [Note] Plugin 'FEEDBACK' is disabled.

2022-06-30 12:03:06 0 [ERROR] Could not open mysql.plugin table: "Unknown storage engine 'InnoDB'". Some plugins may be not loaded

2022-06-30 12:03:06 0 [ERROR] Unknown/unsupported storage engine: InnoDB

2022-06-30 12:03:06 0 [ERROR] Aborting

Hello everyone! I'm using 2 environments to work with DataHub. After an upgrade to version 0.8.36 in one of them, we have noticed that we can't see in the frontend the glossary terms loaded. Has anyone experienced something similar? I'm not sure if I need to add any additional step. Thank you in advance!

Hey everyone - We are trying to update our datahub plugins from version 32 -> 40. Seeing the below dependency issue when trying to get things updated with pip

Could not find a version that matches traitlets<5.2.2,>=4.2,>=4.2.1,>=5.0,>=5.1.0,>=5.2.2 (from acryl-datahub[bigquery,bigquery-usage,datahub-rest,looker,lookml]==0.8.40->-r

Has anyone run into/resolved this issue before?

Hello guys, I'm trying to work with standalone consumers to take some of the burdens of the GMS. However, the MAE consumer is always breaking and I can't identify the reason. I'll pass on this thread the error that usually starts to appear when some entity is created or changed. DataHub version: v0.8.39 Deployment: GKE

Hi, datahub team! I was playing with the locally set up datahub and found an issue which I can not resolve. Starting from this week I'm not able to see the term's "custom_properties" when I go through the termName link on the web page. It just shows me the content for half of a second and then refreshes automatically to the white screen. I'm not sure why it stopped working. I've already reinstalled the datahub once again and it still shows the same behavior. Also the issue doesn't disappear when switching to another browser. May you please #troubleshoot ? Here is my testing glossary and how it looks (The first term has "custom_properties" in it):

Hi, so I have recently upgraded to version v0.8.40 (since, if I am not mistaken, I heard that the problem (the one related with the ebeans) on previous versions was solved here). My problem is that when I log in as user datahub I get the following: (view image) Has it changed in this version that the user datahub is the default user ?? Thanks in advance!!

Hi All - we've updated to 0.8.39 to get all the awesome new DBT stuff. It looks amazing! One question though, which I don't know if is an issue on Datahub's side or whether we're doing something incorrectly ourselves (for example wrong settings or recipes for example). In particular, we're having some challenges showing the DBT tests (assertions). I have attached a video here to show the behaviour. We search for a dataset called

ai_review_estimate

. You see there are 3 entries as this dataset is present on 3 data platform instances (Athena in 3 different regions). This dataset is created 3 times by running DBT in the respective regions. The thing is, when we click on one of the datasets, it goes to the DBT "Version" of the dataset by default (why so? can we make it so it goes to the athena "version"?). Also, on the DBT version, the validation tab is empty. Instead, when we click on the Athena version, the validations are there. Any clue to as what is happening? Thanks!

Hello, we've updated to 0.8.40 to checkout the new dbt sibling stuff + also the validation tab. We've ingested dbt with tests. When I click the validation tab, the web UI goes blank. Also tried nuking the whole datahub down and starting everything from scratch with the same error. Any clue what might be causing it? Posting logs in thread

Hi everyone, I started the test with redshift ingestion, but i'm not getting enable Lineage and filter pattern tables. My ingestion config and log:

'[2022-07-01 14:20:22,315] INFO     {datahub.cli.ingest_cli:99} - DataHub CLI version: 0.8.40\n'
           '1 validation error for RedshiftConfig\n'
           'table_pattern.allow\n'
           '  extra fields not permitted (type=value_error.extra)\n',
           "2022-07-01 14:20:25.906577 [exec_id=f4d6c21c-0ee4-466a-bdb0-d87f29a7585d] INFO: Failed to execute 'datahub ingest'",
           '2022-07-01 14:20:25.907014 [exec_id=f4d6c21c-0ee4-466a-bdb0-d87f29a7585d] INFO: Caught exception EXECUTING '
           'task_id=f4d6c21c-0ee4-466a-bdb0-d87f29a7585d, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 121, in execute_task\n'
           '    self.event_loop.run_until_complete(task_future)\n'
           '  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 89, in run_until_complete\n'
           '    return f.result()\n'
           '  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
           '    raise self._exception\n'
           '  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
           '    result = coro.send(None)\n'
           '  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 115, in execute\n'
           '    raise TaskError("Failed to execute \'datahub ingest\'")\n'
           "acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"]}
Execution finished with errors.

Hi, I have a problem with duplicates. I don't know why but I see twice a snowflake database (and its schemas and tables) as you can see in the picture. What should I do? This can lead problems with related lineage. Thanks

hi, is there a workaround for this issue? I'm facing the same problem even when using the new method for specifying Params

What is the set of permissions that is needed for a user to create a new tag? I am getting the following error

Failed to create & add tag

. I have given the user the following permissions: They can

Edit Tags

and

Edit Dataset Column Tags

on Resources

[Tags, Datasets]

.

Hello While benchmarking datahub browse endpoint with 200users we are getting "HTTP 401" for random requests. Out of 0.3million requests 12k fail with this error. Script link: https://datahubspace.slack.com/archives/C029A3M079U/p1656676936490069?thread_ts=1656361928.525129&cid=C029A3M079U datahub version: v0.8.38 Let me know if this requires a github issue. @magnificent-notebook-88304 Error & Locust screenshot

Dear Team, I am a newbie to metadata management and docker. I am trying to connect MS SQL server running in a container from the datahub custom source ingestion option. However, I am keep getting error as "connection refused". I am able to connect to my sql server through telnet, ssms and grafana. I tried filling "host.docker.internal" / localhost / container name / ip in the recipe file and all leads to same error. Can you please suggest?

Hi all, I tried to start datahub after nuking but I got this error. May i know how to fix this?

No Datahub Neo4j volume found, starting with elasticsearch as graph service.
To use neo4j as a graph backend, run
`datahub docker quickstart --quickstart-compose-file ./docker/quickstart/docker-compose.quickstart.yml`
from the root of the datahub repo

Fetching docker-compose file <https://raw.githubusercontent.com/datahub-project/datahub/master/docker/quickstart/docker-compose-without-neo4j.quickstart.yml> from GitHub
[2022-07-04 20:24:14,340] ERROR    {datahub.entrypoints:184} - File "c:\users\sdekng\appdata\local\programs\python\python36\lib\site-packages\datahub\entrypoints.py", line 149, in main
    146  def main(**kwargs):
    147      # This wrapper prevents click from suppressing errors.
    148      try:
--> 149          sys.exit(datahub(standalone_mode=False, **kwargs))
    150      except click.exceptions.Abort:
    ..................................................
     kwargs = {}
     datahub = <Group datahub>
     click.exceptions.Abort = <class 'click.exceptions.Abort'>
    ..................................................

Hi, guys. I send invite code to my colleague and he register Datahub. When he register DataHub, he use Korean in name line. After that I check his profile at Users&Groups page. But his name is indicated as "???". I think I need utf8 encoding to input Korean. What should I do?

How do I upgrade the databricks cli? I already upgraded the datahub with just "datahub docker quickstart" and its on 0.8.40. Getting an error when trying the new csv-enricher ingestion.

Hi everyone! Until yesterday, we had our datahub crossplatform lineages properly connected. We set some "outlets" in our Airflow tasks, so then Datahub linked the task urns with downstream dataset urns. After reingesting some snowflake and dbt metadata, this connection has disappeared. I mean: If you go to the Runs details of an airflow task, the output is properly stated (snowflake table as shown in the image). If you then click on that snowflake table, and go to the lineage, there is not a single trace of the airflow task or DAG. And it was there a couple of days ago. At the same time, some snowflake and dbt urns referring to the same models/tables have been combined in one sibling association, which is nice, but... could this be related with the absence of link between airflow and dbt/snowflake in the lineages? Should we configure our outlets in a different way to be properly recognized? Thanks in advance! Dani

Hi guys, we have some problems with the dbt tests shown in the UI. If we search a snowflake table generate by dbt and look at the validations, it seems that no assertions are present. The urn is something like this: /dataset/urnlidataset:(urnlidataPlatform:dbt,pre_production.aos.iap,PROD)/Validation?is_lineage_mode=false . We have noticed that if we change the platform in the urn to snowflake (/dataset/urnlidataset:(urnlidataPlatform:snowflake,pre_production.aos.iap,PROD)/Validation?is_lineage_mode=false) the assertions appear.

Hello I'm on a linux server system and I want to install datahub and I get an error it seems that it wants to install version 0.8.90dev when I just type in pip3 install datahub - shouldn't this install a stable version ? - sorry for the noise the package is acryl-datahub.

Hello, I am trying to implement a custom authenticator by adding a class that implements com.datahub.authentication.Authenticatorin the Metadataservice java process classpath. As I need to add this custom Authenticator to "Authentication Chain" by editing application.yml configuration file for the Metadata Service, instead of updating the application.yml file in Medataservice code, is there a way that I can pass the application.yml file as an argument to the java process which can load the configuration from the file that I have passed instead of the one from the code base?

Hello everyone, I tried to dump the metadata_aspect_v2 and restore it back to the database. However, the data did not appear on the DataHub UI. Did i miss any steps for the back up?

Good morning all, I'm currently working on setting up an sso with our keycloak installment. I have datahub redirecting to keycloak to sign in, but once I sign in it will just cycle through loading the page. On the backend, I'm getting an unsupported SSL message error (I'll provide screenshots of the error soon). I've looked at the documentation but I haven't seemed to be able to find an answer to my problem. Any help or insight is greatly appreciated! Also, if this isn't the correct spot for this question, let me know and I'll post in the correct channel.