DataHub #troubleshoot

some-car-9623

01/12/2023, 3:01 PM

Hello Everyone, we are facing the issue in the Lineage where only partial linage is showed up in the UI. like for example there are 4 up streams for a Dataset but in UI it displayed only 2 other 2 is missing . In the Sql it shows 4 upstreams . Any Idea what is causing this ?

average-hydrogen-40851

01/12/2023, 4:23 PM

Hi all, simple thing - on page https://datahubproject.io/docs/lineage/sample_code/ there's a e404 on link Datahub Snowflake Lineage. (i'm not sure is it right place to report it).

👀 1

sparse-australia-70466

01/12/2023, 4:37 PM

Hi all! - Probably a silly question but how would I go about enabling debugging through an environment variable for the

datahub-frontend

? I'm having some issues integrating authentication with our internal OIDC provider and I wanted to see some more verbose logs.

✅ 1

red-waitress-53338

01/13/2023, 2:35 AM

Hi All, Is there a way to use GCP SQL Auth Proxy with the datahub GMS? Based on this link we can do that but I am confused how can I integrate Cloud SQL Auth proxy with the GMS service. Can someone please help? https://cloud.google.com/sql/docs/mysql/connect-instance-private-ip

✅ 1

abundant-television-56673

01/13/2023, 9:33 AM

Can anyone shed any light on why this regex pattern may not work in the Azure Ingestion?

Copy code

[A-z]+\.[A-z0-9-]+@company\.com

I get zero users returned when there are clear matches in the ingestion output

Copy code

'urn:li:corpuser:<http://ZMuraaaa_x.com#EXT#@company.onmicrosoft.com.*|ZMuraaaa_x.com#EXT#@company.onmicrosoft.com.*>',
'urn:li:corpuser:zoe.a1@company.com.*',
'urn:li:corpuser:zoe.a2@company.com.*',
'urn:li:corpuser:<http://Zoe.a3_south.gov#EXT#@company.onmicrosoft.com.*|Zoe.a3_south.gov#EXT#@company.onmicrosoft.com.*>',
'urn:li:corpuser:<http://Zoe.Chase_test.com#EXT#@company.onmicrosoft.com.*|Zoe.Chase_test.com#EXT#@company.onmicrosoft.com.*>',

✅ 1

salmon-psychiatrist-4013

01/13/2023, 11:13 AM

Hello Team, I am trying to do data profiling on a Vertica Table. If I enable profiling in my recipe, the ingestion via cli gives error as such:

Copy code

File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python39\lib\site-packages\datahub\ingestion\source\sql\vertica.py", line 1014, in loop_profiler_requests
    for projection in inspector.get_projection_names(schema):  # type: ignore
AttributeError: 'PGInspector' object has no attribute 'get_projection_names'

With little bit of debugging i was able to figure out that the inspector being used is sqlalchemy.dialects.postgresql.base.PGInspector and this inspector infact doesn't have function get_projection_names. Question : is it the correct function being used in loop_profiler_requests call and in general did anybody have success with profiling Vertica ?

👀 1

✅ 1

melodic-dress-7431

01/13/2023, 1:43 PM

Hi Team ...have datahub v0.9.5 deployed in kubernetes...while ingesting airflow i get the following error

melodic-dress-7431

01/13/2023, 1:44 PM

Copy code

Did not find a registered class for airflow

✅ 1

melodic-dress-7431

01/13/2023, 1:44 PM

how do I add the plugin

elegant-state-4

01/13/2023, 4:06 PM

I updated the DataPlatform entity and added DATA_MESH as a value. I am unable to run the command

./gradlew :gms:impl:build -Prest.model.compatibility=ignore

as recommended in the developer guide. I am getting the following error:

Project 'gms' not found in root project 'datahub'.

Is the guide out of date? Also when I try to build the metadata-service I get the following error:

Copy code

idl compatibility report:
Incompatible changes:
  1) /collection/actions/batchIngest/parameters/entities/type: new enum added symbols DATA_MESH
  2) com.linkedin.entity.Entity/value/com.linkedin.metadata.snapshot.Snapshot/ref/union/com.linkedin.metadata.snapshot.DataPlatformSnapshot/aspects/array/items/com.linkedin.metadata.aspect.DataPlatformAspect/ref/union/com.linkedin.dataplatform.DataPlatformInfo/type/com.linkedin.dataplatform.PlatformType/symbols: new enum added symbols DATA_MESH, breaks old readers
  3) com.linkedin.entity.Entity/value/com.linkedin.metadata.snapshot.Snapshot/ref/union/com.linkedin.metadata.snapshot.DataPlatformSnapshot/aspects/array/items/com.linkedin.metadata.aspect.DataPlatformAspect/ref/union/com.linkedin.dataplatform.DataPlatformInfo/type/com.linkedin.dataplatform.PlatformType/symbols: new enum added symbols DATA_MESH, breaks old readers

[RS-COMPAT]: false
[MD-COMPAT]: false
[RS-I]:/collection/actions/batchIngest/parameters/entities/type: new enum added symbols DATA_MESH
[MD-I]:com.linkedin.entity.Entity/value/com.linkedin.metadata.snapshot.Snapshot/ref/union/com.linkedin.metadata.snapshot.DataPlatformSnapshot/aspects/array/items/com.linkedin.metadata.aspect.DataPlatformAspect/ref/union/com.linkedin.dataplatform.DataPlatformInfo/type/com.linkedin.dataplatform.PlatformType/symbols: new enum added symbols DATA_MESH, breaks old readers
[MD-I]:com.linkedin.entity.Entity/value/com.linkedin.metadata.snapshot.Snapshot/ref/union/com.linkedin.metadata.snapshot.DataPlatformSnapshot/aspects/array/items/com.linkedin.metadata.aspect.DataPlatformAspect/ref/union/com.linkedin.dataplatform.DataPlatformInfo/type/com.linkedin.dataplatform.PlatformType/symbols: new enum added symbols DATA_MESH, breaks old readers

busy-dream-36363

01/13/2023, 5:17 PM

Hello everyone. I was wondering, is it the case that DataJob's do not support column level (fine-grained) lineages?

✅ 1

busy-dream-36363

01/13/2023, 5:17 PM

The model itself has the finegrained lineage property (atleast in the Java SDK)

busy-dream-36363

01/13/2023, 5:18 PM

Dataset-Dataset finegrained lineage work fine

busy-dream-36363

01/13/2023, 5:18 PM

But Dataset-DataJob-Dataset shows no column level lineages when I pass it the same FineGrainedLineageArray that I do to datasets

busy-dream-36363

01/13/2023, 5:19 PM

(Neither are input/output fields shown, despite being able to set them)

faint-actor-78390

01/13/2023, 5:25 PM

Hi Team, Trying to install datahub actions , not found ?

DataHub CLI version: 0.9.5.2

Python version: 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)]

✅ 1

average-dinner-25106

01/16/2023, 4:48 AM

Hi, how can I update the version of datahub quickstart? Recently, the version is v0.9.2. But I want to upgrade latest version. I tried "datahub docker quickstart" command one more time, but it didn't change version.

✅ 1

bland-orange-13353

01/16/2023, 1:12 PM

This message was deleted.

powerful-cat-68806

01/16/2023, 2:03 PM

Hi Datahub team 🙂 OIDC authentication fails. I tried 1st to apply k8s secret, but it didn’t work Then, I’ve added explicitly the keys-values pair for adding all needed Deployment fails. From the output. it looks like the configuration is messed up(it’s working for part of these values) Pls. assist

Copy code

datahub-frontend:
    enabled: true
    oidcAuthentication:
      enabled: true
      provider: okta
#      clientId:
#        secretRef: datahub-okta-client-secret
#        secretKey: clientid
#      # only needed if you would like to store the client secret in secret
#      clientSecretRef:
#        secretRef: datahub-okta-client-secret
#        secretKey: clientsecret
      oktaDomain: <http://jfrog.okta.com|jfrog.okta.com>
    extraEnvs:
      - name: AUTH_OIDC_ENABLED
        value: true
      - name: AUTH_OIDC_CLIENT_SECRET
        value: xxxxxxxxxxxx 
      - name: AUTH_OIDC_BASE_URL
        value: xxxxxxxxxxxx
      - name: AUTH_OIDC_DISCOVERY_URI
        value: <https://my_okta_domain/.well-known/openid-configuration>
      - name: AUTH_OIDC_SCOPE
        value: <mailto:xxxxx@jxxxx.com|xxxxx@jxxxx.com>
      - name: AUTH_OIDC_BASE_URL
        value: xxxxxxxxxxxxxxx
      - name: AUTH_OIDC_CLIENT_ID
        value: xxxxxxxxxxxxxxxx

fancy-rose-64137

01/16/2023, 3:54 PM

Hello guys, iam trying to ingest Athena metadata with Datahub v0.9.5 and i catch this error, when occur ingesting stuck running forever. Anyone can help me?

Copy code

Caused by: java.sql.SQLException: Incorrect string value: '\\xE2\\x8F\\xB3 Pi...' for column 'metadata' at row 1

exception.txt

polite-activity-25364

01/17/2023, 6:08 AM

Hi Team, Is there a way to clear all ingestion history? above image, bigquery ingesiton status is pending , but it actually succeeded. (Other sources’ histories look good) if delete the history from the UI and ingest it again, it will appear in the pending state. env is eks with helm chart

✅ 1

limited-refrigerator-50812

01/17/2023, 10:12 AM

Hi Team, My question is hopefully simple. Is there any way to manipulate the order of link posts (global announcements) on the front page? Right now, the order appears arbitrary to me.

rich-pager-68736

01/17/2023, 11:25 AM

After upgrading from v0.9.5 to v0.9.6 existing secrets cannot be decrypted anymore using UI ingestion:

Copy code

"Failed to resolve secret with name {match}. Aborting recipe execution."

After recreating those secrets it works again.

✅ 1

gifted-knife-16120

01/17/2023, 12:04 PM

Hi, for custom lineage in UI. Can I set the policy to permit specific groups only able to edit?

brainy-piano-85560

01/17/2023, 1:03 PM

Hi, when tagging (Tags/Glossary terms/Domain) a container entity (for example, a postgres schema that has some tables under it), is there a way that the tags will be applied to all tables? If all tables in the schema has the same tags, it's weird manually tagging X tables and not applying it to them automatically by tagging their container

✅ 2

better-fireman-33387

01/17/2023, 1:14 PM

Hi all, I’m trying to wipe down all metadata after dropping mysql tables and deleting elastic indexes I can still see the datasets from ui. I’m running datahub with helm can you assist please?

✅ 1

👀 1

faint-hair-91313

01/17/2023, 1:40 PM

Hey guys, maybe this was asked, I just upgraded to the latest version and wanted to see Tableau preview, but I get this failure to load the picture. Is there anything that needs to be done at browser level to enable it. I've tried both Edge and Chrome.

elegant-state-4

01/17/2023, 3:44 PM

Failing to run

./gradlew :metadata-ingestion:installDev

. I get the following error.

Copy code

In file included from src/cxoApiType.c:11:
      src/cxoModule.h:14:10: fatal error: 'Python.h' file not found
      #include <Python.h>

👀 1

cold-city-21137

01/17/2023, 4:08 PM

is there a configuration that needs to happen to allow external calls to port 9002? I can curl localhost:9002 and get info back but can't access from laptop. All ports open to laptop in firewall. Can even hit the elastic search 9200 port with a browser from the laptop. Internal on VM hosting the containers, I cannot use the <local IP>:9002 it times out. Only localhost:9002 works?

👀 1

✅ 1

creamy-machine-95935

01/17/2023, 6:52 PM

Hi! Is there workaround to work with a Looker Repository hosted on Gitlab and LookML? 😬

✅ 1