DataHub #troubleshoot

polite-ghost-91039

10/27/2022, 2:33 PM

Hi team, I am using datahub 0.9.0 and trying to ingest data using the cli from trino but the ingestion is failing with HttpError: error 400: b'<html>\r\n<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>\r\n<body>\r\n<center><h1>400 Bad Request</h1></center>\r\n<center>The plain HTTP request was sent to HTTPS port</center>\r\n</body>\r\n</html>\r\n' Can someone help me with the issue is and how I can fix it. TIA

witty-microphone-40893

10/27/2022, 2:48 PM

Hi All. Is there a guide somewhere for integrating Piicatcher with DataHub? DataHub is awesome but I'm still relatively new to it and now need to label PII data fields.

proud-lamp-13920

10/28/2022, 5:19 AM

Hi All I don't know how to run datahub cli in Google kubernetes environment when i type datahub on cloud shell terminal i got

datahub: command not found message

help me please

high-gigabyte-86638

10/28/2022, 7:13 AM

Hi all, Do somebody know how i can add data container to a dataset? https://datahubproject.io/docs/generated/metamodel/entities/dataset#container

plus1 1

helpful-london-56362

10/28/2022, 11:26 AM

hello, I deployed datahub to AWS EKS using the official helm charts. I'm trying to set up datahub actions. How do I get the

bootstrap-server endpoint

to use in the actions yaml file. I used the values specified in my

values.yaml

file, but those couldn't be resolved. any help will be appreciated. Thank you!

handsome-football-66174

10/28/2022, 7:01 PM

Hi everyone , is there a way to upload a logical model in Datahub ( apart from adding external links ) ?

clever-garden-23538

10/28/2022, 9:50 PM

i think i've found a glitch on v0.8.45, if you're looking at a dataset, you can't go back one level in the hierarchy by clicking on it in the explorer chain Ex, if i have a dataset baz which is located at Datasets -> foo -> bar -> baz, when im on baz's page i'll see "Datasets -> foo -> bar" in the top left. Clicking on "bar" doesnt do anything.

billowy-pilot-93812

10/30/2022, 9:22 AM

Hi all, i'm installing datahub via docker, but these two job has run for nearly an hour now and havent finished yet, any idea? Thank you

✅ 1

astonishing-pager-27015

10/30/2022, 8:57 PM

nevermind! restarted the containers and everything looks good

✅ 1

proud-lamp-13920

10/31/2022, 6:23 AM

Hi all I found a way to delete other metadata such as tables (through cli, UI, graphql or openapi) I have another question When a column in a table is deleted, can it be automatically deleted from the datahub, or can you guide how to manually delete it through the api?

✅ 1

helpful-london-56362

10/31/2022, 8:05 PM

just re-posting my earlier question here, I'm a bit new to DataHub. I've deployed DataHub on K8 using the default helm charts. From what it seems, I will have to configure a LoadBalance to access Kafka from outside the cluster. Is this the recommended approach?. I want to set up DataHub action. Any help will be appreciated. Thank you https://datahubspace.slack.com/archives/C029A3M079U/p1666956408664309

careful-france-26343

11/01/2022, 12:01 AM

Getting this error on EKS, private subnet:

Copy code

2022/10/31 23:12:01 Problem with request: Get <http://elasticsearch-master:9200>: dial tcp 10.100.139.159:9200: connect: connection refused. Sleeping 1s
2022/10/31 23:12:01 Timeout after 2m0s waiting on dependencies to become available: [<http://elasticsearch-master:9200>]

Does anyone know what's causing it?

microscopic-room-90690

11/01/2022, 6:27 AM

Hi guys, I got a trouble when I ingest metadata from S3. The recipe I use is attached. Firstly, it worked very well and I got datasets in table granularity. While after I set profiling and set it back, I only got some containers which are in tables' upper level. No more datasets in table granularity. Any help will be appreciated. Thank you!

Copy code

source:
  type: s3
  config:
    platform: s3
    profiling:
      enabled: false
      profile_table_level_only: false
    path_specs:
       - include: "<s3://path/cluster=dev/datatype={table}/year={partition[0]}/month={partition[1]}/day={partition[2]}/*.parquet>"
    aws_config:
      aws_region: us-east-1

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

lemon-cat-72045

11/01/2022, 9:48 AM

Hi everyone, I am suddenly seeing this error and don't know what is causing it. Could anyone help? Thanks. Now when I try to access Domain and Glossary details page, this shows up.

full-shoe-73099

11/01/2022, 10:16 AM

Hi people. Tell me please how I can choose role of the owner for TermGroup?

bland-orange-13353

11/01/2022, 12:45 PM

This message was deleted.

witty-microphone-40893

11/01/2022, 5:54 PM

Hi. Does anyone here have experience with exporting

piicatcher

tags from the sqlite database into Datahub. I'm struggling to get the config correct and keep seeing the message

Copy code

{datahub.entrypoints:195} - Command failed: 
        (sqlite3.OperationalError) no such table: sources

My config file is as follows:

Copy code

source:
  type: dbcat.datahub.CatalogSource
  config:
    database: main
    source_names:
      - test_cat
sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

Command

$ datahub ingest -c ./export.yml

witty-television-74309

11/01/2022, 8:39 PM

Reposting the message from ingestion tab. This could be very basic question. I am ingesting snowflake tables. In the recipe profiling is enabled. but I am not able to see row level statistics. Even the stats tab is disabled in the UI. Do you know which steps am i missing ? I see table schema.

ancient-apartment-23316

11/01/2022, 10:38 PM

Hello! I’m trying to update the datahub using helm (v0.8.24 to v0.9.1) and i’m facing the

datahub-elasticsearch-setup-job

error. The credentials seem to be correct, I can login to opensearch via browser using the username

datahub-dev-1

and the password from the k8s

elasticsearch-secrets

Copy code

kubectl logs job.batch/datahub-elasticsearch-setup-job

2022/11/01 22:02:31 Waiting for: <https://vpc-datahub-opensearch-dev-1-qweqwe.us-east-1.es.amazonaws.com:443>
2022/11/01 22:02:32 Received 401 from <https://vpc-datahub-opensearch-dev-1-qweqwe.us-east-1.es.amazonaws.com:443>. Sleeping 1s
2022/11/01 22:04:31 Timeout after 2m0s waiting on dependencies to become available: [<https://vpc-datahub-opensearch-dev-1-qweqwe.us-east-1.es.amazonaws.com:443>]

values.yaml:

Copy code

elasticsearch:
    host: "<http://vpc-datahub-opensearch-dev-1-qweqwe.us-east-1.es.amazonaws.com|vpc-datahub-opensearch-dev-1-qweqwe.us-east-1.es.amazonaws.com>"
    port: "443"
    useSSL: "true"
    auth:
      username: "datahub-dev-1"
      password:
        secretRef: elasticsearch-secrets
        secretKey: elasticsearch-password

modern-garden-35830

11/02/2022, 9:56 AM

Hi guys, Anyone knows if there is a way to extract what we did in DataHub into XML file or JSON or something similar? Is there a database that can be copied in case we change servers for examples?

microscopic-mechanic-13766

11/02/2022, 1:08 PM

Good afternoon, so I have found a behaviour that I think should be fixed. If you enter a dataset's page and you want to go to the previously visited page by click the brower's arrow (<-), you have to double click it to be able to go to such page. The problem might be of the URLs used as it changes from https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,fct_users_deleted,PROD)/Schema?is_lineage_mode=false&schemaFilter= to https://demo.datahubproject.io/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,fct_users_deleted,PROD)/Schema?is_lineage_mode=false. Also, in the first URL the name of the tab is like "<dataset_name>|Dataset" , but in the second one it is just "Datahub" I have noticed this behaviour in Looker, Hive, dbt, BigQuery, PostgreSQL, S3, HDFS and Kafka (in other words, when you access a page of a dataset ) Note: This behaviour is not specific to the 0.9.1, it also happens in versions like 0.9.0

bright-diamond-60933

11/02/2022, 1:45 PM

Reposting for visibility: i am trying to enable iam auth for kafka using an msk cluster. I tried setting these properties:

SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL

SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION=/tmp/kafka.client.truststore.jks

SPRING_KAFKA_PROPERTIES_SASL_MECHANISM=AWS_MSK_IAM

SPRING_KAFKA_PROPERTIES_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;

SPRING_KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

and also set the

KAFKA_BOOTSTRAP_SERVER

to the MSK public endpoint and the

KAFKA_ZOOKEEPER_CONNECT

in kafka_setup to the zookeper TLS endpoint in msk cluster. I did this within docker.env inside gms, mce-consumer, mae-consumer and kafka-setup, but its not connecting to the msk cluster. I am getting this error:

Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: No LoginModule found for software.amazon.msk.auth.iam.IAMLoginModule

kafka-setup               |     at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:172)

kafka-setup               |     at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)

kafka-setup               |     at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)

kafka-setup               |     at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)

kafka-setup               |     at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)

clever-garden-23538

11/02/2022, 7:19 PM

java.lang.UnsupportedOperationException: ChangeType not supported: CREATE for aspect corpUserInfo

why is CREATE not supported for users/groups 😞

lively-dusk-19162

11/02/2022, 9:14 PM

Hi all, is there any way to write code to find column level lineage out of sql queries?

powerful-breakfast-81119

11/03/2022, 12:02 AM

Hello, we're trying to use the "download as csv" feature on search results. We filter results using the left sidebar, then click the "Download" link from the "..." menu at the top right. We've found that it works fine if there are only a few hundred results, but if we get up to a thousand or more results, it never returns a result. Even if we leave it for several hours, it still shows "Downloading" under the "..." menu. Should this feature work for 1000+ results? And if not, is there any other way to get this out of DataHub?

many-piano-52097

11/03/2022, 3:29 AM

Hello, have you encountered such mistakes? Signed JWT rejected: Another algorithm expected, or no matching key(s) found

polite-ghost-91039

11/03/2022, 10:47 AM

Hello i am trying to connect airflow to datahub 0.9.1 , I am new to airflow and done till step 4(adding airflow hook) mentioned here- https://datahubproject.io/docs/lineage/airflow/ but I am not seeing anything on the airflow> admin> plugins. Can someone help with what could be going wrong here?

ancient-apartment-23316

11/03/2022, 11:35 AM

Hello, I updated the Datahub using helm (v0.8.24 to v0.9.1), and now after the update I have problems with login via Okta, I did not change any Okta settings, I just did the update

helm upgrade --install datahub datahub/datahub --values values.yaml --debug

Now I see this error when I try to go to my main datahub page:

Copy code

Failed to perform post authentication steps. Error message: Failed to provision user with urn urn:li:corpuser:Slava.S@qwerty.com.

hallowed-spring-18709

11/03/2022, 1:47 PM

Hi guys! I need help deploying datahub using Docker desktop with kubernetes enabled. I cloned the datahub-helm repo and after the front end is finally working I type the username and password ("datahub") but no success at login. I also tried modifying the

values.xml

files for both datahub and prerequisites with no avail. I will post my

values.xml

as a thread. I also added env values obtained from the datahub quickstart docker-compose files but doesn't seem to help either.

bumpy-manchester-97826

11/03/2022, 2:16 PM

Not sure this is the correct channel to ask in, but I’ll ask anyways. Is there a way to see the role of a user via the GraphQL API? I’ve been using listUsers to get all the users and can see everything about each apart from their role. Thanks!