Good morning, everyone! I'm facing the same problem that @happy-island-35913 reported earlier. I thought it was a version problem, upgraded to the latest version, and the bug is still there. Thanks in advance!

Not sure if this is the right channel for this question - but gonna send it What's the difference between an entity and a snapshot? I found the collection of defined entities here: https://github.com/linkedin/datahub/tree/ebdaa0e359048f92a8d79ff4f990ab3c1bd662df/metadata-models/src/main/pegasus/com/linkedin/metadata/entity But I see that doesn't include certain items, such as mlfeaturetable, which when queried looks like:

"urn":"urn:li:mlFeatureTable:(urn:li:dataPlatform:sagemaker,fraud-prediction)"

. In fact I even see "Feature Table" listed alongside other entities in the "Explore your metadata" section And then, what is in the urn? Is it the snapshot name, or the entity name? Finally, specifically for mlfeaturetable, which entity does that actually belong to?

Hi! Im having a problem with a AWS Elasticsearch I followed official documentation, but every time is the same error

Received 401 from <https://vpc-datahub-rtixgcnstthsm6uosewxuafrpy.us-west-2.es.amazonaws.com:443>. Sleeping 1s

I test with AWS Opensearch 1.1 and Elasticsearch 7.10, and same error This my part of the

values.yaml

elasticsearch:
    host: <http://vpc-datahub-rtixgcnstthsm6uosewxuafrpy.us-west-2.es.amazonaws.com|vpc-datahub-rtixgcnstthsm6uosewxuafrpy.us-west-2.es.amazonaws.com>
    port: "443"
    useSSL: "true"
    auth:
      username: root
      password:
        secretRef: elasticsearch-secrets
        secretName: elasticsearch-password

elasticsearchSetupJob:
  enabled: true
  image:
    repository: linkedin/datahub-elasticsearch-setup
    tag: "v0.8.23"
  extraEnvs:
    - name: USE_AWS_ELASTICSEARCH
      value: "true"

EKS cluster where datahub im trying to install is same VPC as AWS Elasticsearch/Opensearch cluster, and same for Security Group Secret is with the correct value, and confirm testing from browser is working the domain Could anyone please help me, if im doing something bad?

Hi, team. Something changed with platforms ingestion? It throws me some strange errors like this:

Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry 'urn:li:dataPlatform:clickhouse-dataPlatformKey-0' for key 'PRIMARY'\n\tat com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)

And not ingesting custom platforms. This was working fine in previous releases (I'm at 0.8.23 atm) Any idea how to fix that?

Hi guys, UI shows some Airflow entities but I can't delete them via CLI. What's wrong? The followings are UI and CLI results. (my) user@user:~/mywork/datahub_cli$ datahub delete --platform bigquery [2022-01-27 135827,504] INFO {datahub.cli.delete_cli:192} - datahub configured with [...my datahub url...] [2022-01-27 135828,034] INFO {datahub.cli.delete_cli:205} - Filter matched 104 entities. Sample: [...sample entities...] This will delete 104 entities. Are you sure? [y/N]: (my) user@user:~/mywork/datahub_cli$ datahub delete --platform airflow [2022-01-27 135911,408] INFO {datahub.cli.delete_cli:192} - datahub configured with [...my datahub url...] [2022-01-27 135911,731] INFO {datahub.cli.delete_cli:205} - Filter matched 0 entities. Sample: [] This will delete 0 entities. Are you sure? [y/N]:

Hi guys, probably someone already reported this but the demo environment (https://demo.datahubproject.io/) is acting weird in terms of Lineage. The “+” button is not working properly (sometimes not showing any dependencies on click) and also some “+” are missing where they supposed to be. One example is this : https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:dbt,jaffle_shop.dbt_jaffle.customers,PROD)?is_lineage_mode=true I do not see this behaviour in our own instance. If you need help or a better description please feel free to reach out 🙂

Hi all! Importing Oracle views with TEXT definition >32767 chars is giving me the known error:

cx_Oracle.DatabaseError: DPI-1037: column at array position 0 fetched with error 1406

Searching for DPI-1037 in here showed that this is a recurrent issue, usually solved by denying import of affected views. This is, unfortunatelly, not an option in our use case. It seems that the issue is due to SQLalchemy lacking cursor.outputtypehandler option when called by the ingest script. I already passed options to SQLalchemy create_engine part, but this does not seem to be feasible as is. Do you have an idea how I could tweak the cursor invoked by the pipeline.run()?

I have a pipeline that finishes with some failures because of lack of permissions for doing the profile on some table (see logs below) My questions are: • Even if the pipeline finishes with failures, are metadata events published? Seems so:

'records_written': 220

• I’m running the ingestion command from a k8s cronjob. Since the ingestion finishes with failures, the pod fails and so it is retried. Does it mean we are republishing the events in every retry? Does that make sense?

Hello! datahub get --urn all -- not work i will try get all url (datasets) through UI wtf )) :8080/api/graphql

javax.servlet.ServletException: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.UnsupportedOperationException: GraphQL gets not supported.

:8080

{"exceptionClass":"com.linkedin.restli.server.RestLiServiceException","stackTrace":"com.linkedin.restli.server.RestLiServiceException [HTTP Status:404]\n\tat

Please help!! doc https://datahubproject.io/docs/api/graphql/getting-started

Hello! datahub ingest show --run-id no-run-id-provided Failed to execute operation no-run-id-provided is a default run-id provided for non labeled ingestion runs. You cannot delete using this reserved run-id

Hi, i just trying out the datahub graphql, my objective is to retrieve the dataset lineage through the graphql i landed in "Dataset" query, i choose Relationships as the output but it needs Relationships input, any idea what is the value for "Types" in Relationships input ?

this what i currently tried in graphql and the actual lineage that i'm going to retrieve

Hi, team Datahub! I wanna do some trouble shooting about Datahub. Can I ask someting about datahub on this channel?

Hi team, for spark data hub lineage version released 0.8.23 and 0.8.24 we are receiving NullPointerException from DataHubSparkListener class. We are working with spark 2.4.0 and Scala 2.11.12. and python 2.7.5. Can you please help?

Airflow jobs failing when GMS is down We have configured our Airflow instance to emit metadata to DataHub GMS. That was working well up until I had to bring down DataHub for some reason. While GMS was down, all our Airflow jobs failed because they couldn't emit to GMS:

datahub.configuration.common.OperationalError: ('Unable to emit metadata to DataHub GMS', {'message': "HTTPConnectionPool(host='datahub-datahub-gms.datahub.svc.cluster.local', port=8080): Max retries exceeded with url: /entities?action=ingest (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known'))"})

I now have the following questions: 1. Is it possible to disable that emitting from Airflow on-demand? 2. Is it possible to make Airflow return a warning instead of a failure when it can't reach GMS? In this case, my DAG primary task got completed, it's only "the reporting" to DataHub that failed... In other words, how can I break the dependency of Airflow on GMS...? 3. Is using Kafka as an endpoint the only way to break that dependency?

Hello all! We're seeing weird lineage bugs in latest version of datahub, and I'm wondering if anyone else has been experiencing issues? After refreshing a couple times names of tables seem to move randomly, and lineage nodes are lost. The database seems to contain the right info, so I'm thinking some translation layer is broken somewhere. Open to contributing here, just need a nudge in the right direction to look 🙂 Sample code to reproduce and screenshots in this Git ticket: https://github.com/linkedin/datahub/issues/3965

Hi Team-We see the below intermittent error while logging into Datahub. Usually the error goes away after few attempts or by clearing cookies. Not sure what is causing this error and how to resolve this.

Hi data ingestion was successful but i'm not able to see data in ui.

Hey folks, Did we find a approach to mitigate the

MSG_SIZE_TOO_LARGE

Kafka issue? We’re facing this problem while ingesting data from Redshift to DataHub.

---- (full traceback above) ----
File "/usr/local/lib/python3.8/site-packages/datahub/entrypoints.py", line 102, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/datahub/telemetry/telemetry.py", line 174, in wrapper
    res = func(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/datahub/cli/ingest_cli.py", line 82, in run
    pipeline.run()
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/run/pipeline.py", line 167, in run
    self.sink.write_record_async(record_envelope, callback)
File "/usr/local/lib/python3.8/site-packages/datahub/ingestion/sink/datahub_kafka.py", line 83, in write_record_async
    self.emitter.emit_mcp_async(
File "/usr/local/lib/python3.8/site-packages/datahub/emitter/kafka_emitter.py", line 152, in emit_mcp_async
    producer.produce(
File "/usr/local/lib/python3.8/site-packages/confluent_kafka/serializing_producer.py", line 176, in produce
    super(SerializingProducer, self).produce(topic, value, key,

KafkaException: KafkaError{code=MSG_SIZE_TOO_LARGE,val=10,str="Unable to produce message: Broker: Message size too large"}

He Team, I am trying to connect to my local MSSQL and I used the below Ingestion source details source:     type: mssql     config:         host_port: 'localhost:1433'         database: AdventureWorksDW2019         username: sa         password: P@ssw0rd         use_odbc: 'True'         uri_args: null         driver: 'ODBC Driver 17 for SQL Server' sink: null I got the below error

Hi! Could anybody help me? I’m trying to deploy datahub on AWS, using AWS services rds, kafka, elastic, and I’m facing the error in GMS service datahub-datahub-gms. When I’m open it in browser (********.us-east-1.elb.amazonaws.com) i’m see this:

{"exceptionClass":"com.linkedin.restli.server.RestLiServiceException","stackTrace":"com.linkedin.restli.server.RestLiServiceException [HTTP Status:404]\n\tat

Also, while installing

helm install datahub datahub/datahub

I see the error in this job

pod/datahub-datahub-upgrade-job-**** :

ERROR: Cannot connect to GMSat host datahub-datahub-gms port 8080. Make sure GMS is on the latest version and is running at that host before starting the migration.
java.net.ConnectException: Connection refused (Connection refused)

BUT after this the same job executing one more time w\o errors

pod/datahub-datahub-upgrade-job-***                   0/1     Completed   0          2m8s
pod/datahub-datahub-upgrade-job-***                   0/1     Error       0          3m6s

Hi, I have another question, regarding my previous thread. I expect to use load balancer (ingress) like it described in documentation here https://datahubproject.io/docs/deploy/aws#expose-endpoints-using-a-load-balancer but in fact ingress does not created! I see the ingress k8s object, but there is no public address, and I can’t see it in EC2 LoadBalancer’s list

kubectl get ing
NAME                       CLASS    HOSTS                     ADDRESS   PORTS   AGE
datahub-datahub-frontend   <none>   <http://datahub.mydomain.com|datahub.mydomain.com>             80      2m37s

Hi guys, i think i found a problem on datahub ingestion for bigquery while profiling is True. The error sound like this

'Cannot access field colors on a value with type ARRAY<STRUCT<colors BOOL>> at '

Datahub try to do select

hide_product_relations

.

colors

but its not possible because without unnest.

'[SQL: SELECT count(*) AS `element_count`, sum(CASE WHEN '(`hide_product_relations`.`colors` IN (NULL) OR `hide_product_relations`.`colors` '

Hello everyone. Would be possible to create an authentication in Datahub by email? Is there any documentation about it?

Hey. Team, got one question, if I want to issue one restli request to gms, is there any recommended existing library to use? How about GraphQL?

Hey everyone! I had a question about Snowflake ingestion jobs: I haven't run an ingestion probably since December of last year, and I think some things may have updated in my system since then. What used to originally work for me now yields the following error:

DatabaseError: (snowflake.connector.errors.DatabaseError) 390190 (08001): Failed to connect to DB: [our_account_id].<http://snowflakecomputing.com:443|snowflakecomputing.com:443>, The specified authenticator is not accepted by your Snowflake account configuration.  Please contact your local system administrator to get the correct URL to use.
(Background on this error at: <http://sqlalche.me/e/13/4xp6>)

We have MFA enabled for Snowflake through Duo Mobile, and was wondering if my troubles had something to do with this. I used to be able to verify my Snowflake login for ingestion jobs through Duo, but can no longer do that - I'm wondering if the ingestion process for DataHub slightly changed and I need a different auth flow. I tried to update my recipe by modifying the parameter for

authentication_type

from its default to

"EXTERNAL_BROWSER_AUTHENTICATOR"

, and that changed the error a bit:

DatabaseError: (snowflake.connector.errors.DatabaseError) 390190 (08001): Failed to connect to DB: [our_account_id].<http://snowflakecomputing.com:443|snowflakecomputing.com:443>, There was an error related to the SAML Identity Provider account parameter. Contact Snowflake support.
(Background on this error at: <http://sqlalche.me/e/13/4xp6>)

From some searching online about Snowflake authentication troubles, I tried to add my email domain to the end of my Snowflake username, and this yielded an extra notice:

DatabaseError: (snowflake.connector.errors.DatabaseError) 390190 (08001): Failed to connect to DB: [our_account_id].<http://snowflakecomputing.com:443|snowflakecomputing.com:443>, There was an error related to the SAML Identity Provider account parameter. Contact Snowflake support.
(Background on this error at: <http://sqlalche.me/e/13/4xp6>)
Initiating login request with your identity provider. A browser window should have opened for you to complete the login. If you can't see it, check existing browser windows, or your OS settings. Press CTRL+C to abort and try again...

Any thoughts on what might be going on?

Hi, I am trying to build docker image with a few cosmetic changes for the front end. Any guidance on where I might be going wrong ? Attached below if the build with debug logging.

Hi all. I'm running into some odd issues with the datahub cli, where I receive the following error any time I try to use the cli:

Traceback (most recent call last):
  File "/home/shivan/data-dict/env/bin/datahub", line 5, in <module>
    from datahub.entrypoints import main
  File "/home/shivan/data-dict/env/lib64/python3.6/site-packages/datahub/entrypoints.py", line 11, in <module>
    from datahub.cli.delete_cli import delete
  File "/home/shivan/data-dict/env/lib64/python3.6/site-packages/datahub/cli/delete_cli.py", line 17, in <module>
    from datahub.telemetry import telemetry
  File "/home/shivan/data-dict/env/lib64/python3.6/site-packages/datahub/telemetry/telemetry.py", line 138, in <module>
    telemetry_instance = Telemetry()
  File "/home/shivan/data-dict/env/lib64/python3.6/site-packages/datahub/telemetry/telemetry.py", line 37, in __init__
    self.update_config()
  File "/home/shivan/data-dict/env/lib64/python3.6/site-packages/datahub/telemetry/telemetry.py", line 50, in update_config
    with open(CONFIG_FILE, "w") as f:
PermissionError: [Errno 13] Permission denied: '/home/shivan/.datahub/telemetry-config.json'

I am reverted to version 0.8.23 due to MFA issues with my Snowflake ingestion, and this didn't seem to be a problem that appeared a month ago. Any ideas what could be the cause of this?

Trying to use Trino with acryldata/airflow-datahub:latest , that we are using for a datahub setup. I see below error in the Trino logs. Attached detailed log from the airflow webserver. Wanted to check if this usecase is supported as part of this image or I am using an incorrect format ? I am able to connect to kafka using similar syntax, so wondering if Trino is supported for this airflow image.

Hi All, I am using the below code to post to graphql endpoint. this is resulting in 401 error. can someone point me in the right direction?     Python Code:   accessToken= <generated-access-token>  endpoint= <frontend proxy details> headers={"Authorization": f"Bearer {accessToken}","Content-Type": 'application/json'}   query="""query{  search(input: { type: DATASET, query:"%cats", start: 0, count: 11 }) {    start    count    total    searchResults {      entity {         urn         type         ...on Dataset {            name         }      }    }  } }""" r = requests.post(endpoint, json={"query": query}, headers=headers)