DataHub #troubleshoot

Join Slack

great-account-95406

09/16/2022, 3:23 AM

Could anyone helm me with that?

little-dinner-99970

09/16/2022, 8:45 PM

I accessed the docker log and found this error below:

few-carpenter-93837

09/19/2022, 6:00 AM

Hey, there seems to be an issue with showing Vertica under the Platforms even if there are existing datasets available.

salmon-angle-92685

09/19/2022, 7:01 AM

Hello everyone, I've found this error when ingesting redshift data but I don't know what it's happening. It seems to have quotes somewhere where it shouldn't be, but not clear. Do you guys would know why this happens ?

microscopic-mechanic-13766

09/27/2022, 8:11 AM

Good morning everyone, so I have just updated my datahub deployment to the version v0.8.45 and found a persistent error that I am unable to know the source of. As I have updated my front container (that internally uses the image

linkedin/datahub-frontend-react

) to the v0.8.45, I have found the persistent error shown in the file. At first I thought the source of the error was could be of some misconfiguration, but I have double checked and that configuration for the v0.8.44 works perfectly fine. This error occurs as I try to access the OIDC login page. Is there any new configuration needed for the front in v0.8.45?? Thanks in advance!

Error.txt

eager-oil-39220

09/27/2022, 2:52 PM

Hi Team, may I ask where do you set up or pass in the value for the version here?

cuddly-butcher-39945

09/27/2022, 3:44 PM

datahub_quickstart_debug_trace.txt

ancient-library-85500

10/13/2022, 7:21 PM

When creating a new custom entity (that is not just an extension of another entity), how is that data taken from the backend and then displayed on the front end? After using a CLI PUT command to insert data, we have been unable to locate that data in the local instance of datahub. Is there a default visualization for custom entities?

gifted-bird-57147

10/19/2022, 12:18 PM

I'm currently running into the same -f error... As far as I understand it has to do with docker compose (v2) not supporting the -f flag. They are folding docker-compose (with hyphen) into docker core command. might be good to document this in the quickstart troubleshooting section. And ideally fix in the quickstart python package?

bright-diamond-60933

10/28/2022, 4:31 PM

i am trying to enable iam auth for kafka using an msk cluster. I tried setting these properties:

SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL

SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION=/tmp/kafka.client.truststore.jks

SPRING_KAFKA_PROPERTIES_SASL_MECHANISM=AWS_MSK_IAM

SPRING_KAFKA_PROPERTIES_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;

SPRING_KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

and also set the

KAFKA_BOOTSTRAP_SERVER

to the MSK public endpoint and the

KAFKA_ZOOKEEPER_CONNECT

in kafka_setup to the zookeper TLS endpoint in msk cluster. I did this within docker.env inside gms, mce-consumer, mae-consumer and kafka-setup, but its not connecting to the msk cluster. I am getting this error:

Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: No LoginModule found for software.amazon.msk.auth.iam.IAMLoginModule

kafka-setup               |     at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:172)

kafka-setup               |     at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)

kafka-setup               |     at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)

kafka-setup               |     at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)

kafka-setup               |     at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)

boundless-sundown-7757

10/31/2022, 4:16 PM

I have similar issue after upgrading to 0.9.0, but it worked fine on Friday when I upgraded though

Copy code

16:11:05.080 [qtp1125736023-167] WARN  c.d.a.a.AuthenticatorChain:70 - Authentication chain failed to resolve a valid authentication. Errors: [(com.datahub.authentication.authenticator.DataHubSystemAuthenticator,Failed to authenticate inbound request: Provided credentials do not match known system client id & client secret. Check your configuration values...), (com.datahub.authentication.authenticator.DataHubTokenAuthenticator,Failed to authenticate inbound request: Authorization header missing 'Bearer' prefix.)]

microscopic-room-90690

11/01/2022, 6:57 AM

Hi @mammoth-bear-12532 I got the same error. I only got some containers without any datasets. When I send the events to a file sink and the output I got is attached.

Copy code

"fieldPath": "p95",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "p96",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "p97",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "p98",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "p99",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "region",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "rowKey",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "ts",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            },
                            {
                                "fieldPath": "year",
                                "jsonPath": null,
                                "nullable": false,
                                "description": null,
                                "created": null,
                                "lastModified": null,
                                "type": {
                                    "type": {
                                        "com.linkedin.pegasus2avro.schema.StringType": {}
                                    }
                                },
                                "nativeDataType": "string",
                                "recursive": false,
                                "globalTags": null,
                                "glossaryTerms": null,
                                "isPartOfKey": false,
                                "isPartitioningKey": null,
                                "jsonProps": null
                            }
                        ],
                        "primaryKeys": null,
                        "foreignKeysSpecs": null,
                        "foreignKeys": null
                    }
                }
            ]
        }
    },
    "proposedDelta": null,
    "systemMetadata": {
        "lastObserved": 1667285121057,
        "runId": "s3-2022_11_01-06_44_04",
        "registryName": null,
        "registryVersion": null,
        "properties": null
    }
},
{
    "auditHeader": null,
    "entityType": "dataset",
    "entityUrn": "urn:li:dataset:(urn:li:dataPlatform:s3,path/cluster=dev/joinvoip,PROD)",
    "entityKeyAspect": null,
    "changeType": "UPSERT",
    "aspectName": "container",
    "aspect": {
        "value": "{\"container\": \"urn:li:container:55b31a9bf2521237914e6ad52ccb5f4f\"}",
        "contentType": "application/json"
    },
    "systemMetadata": {
        "lastObserved": 1667285121089,
        "runId": "s3-2022_11_01-06_44_04",
        "registryName": null,
        "registryVersion": null,
        "properties": null
    }
}

full-shoe-73099

11/01/2022, 7:06 PM

John, thank you for the answer. Now I am calm 😅

gifted-bird-57147

11/02/2022, 10:02 AM

Hi Team, I did some more testing and there is definitely something wrong with associating mixed case GlossaryTerms to datasets. I tested with GraphQL just now and

Copy code

mutation addTerms {
  addTerms (input: {termUrns: ["urn:li:glossaryTerm:Areaaldata begrippenkader.bordDRIS"],
  	resourceUrn: "urn:li:dataset:(urn:li:dataPlatform:postgres,postgres.public.av_borddris_p,PROD)"}
  )
}

runs fine, but the relationship does not show up in the UI. As long as my terms are lowercase everything works as expected. Can anyone shed some light on this?

ancient-apartment-23316

11/02/2022, 6:01 PM

should I set the same port in elasticsearchSetupJob block?

swift-farmer-36942

11/07/2022, 8:45 PM

additionally, can someone tell me why the datahub root user cannot reset a users password? the option appears grayed out

microscopic-mechanic-13766

11/08/2022, 8:27 AM

Thanks xL! I think something in the logOut could have been modified too as the AUTH_SESSION_ID cookie in this newer version doesn't appear, ergo I don't think it is erased and you are able to keep log in without having to specify user/pass. The first photo is what the log out looks like in v0.9.0 and prior versions, and the second in version 0.9.2. (This could be wrong as I am not quite versed in cookies matter, but it does ring a bell as previous problems with not logging out had to do with cookies; so please correct me if I am mistaken) Thanks!

colossal-laptop-87082

11/08/2022, 10:00 AM

what @hundreds-photographer-13496?? Is there any way that this issue can be overcome because my main use case is CSV Data Observability, what do with this, I'm stuck totally now???

green-lion-58215

11/09/2022, 3:46 PM

Hello, Was there a fix for this issue? I am also facing the same issue. I am using cli version 0.8.44 @hundreds-photographer-13496

gifted-rocket-7960

11/17/2022, 6:22 AM

@dazzling-judge-80093 In this link https://datahubproject.io/docs/generated/metamodel/entities/dataset/#fine-grained-lineage defined transformOperation = "myfunc" in UI where this information is displayed ?

Copy code

FineGrainedLineage(
        upstreamType=FineGrainedLineageUpstreamType.FIELD_SET,
        upstreams=[fldUrn("bar3", "c2")],
        downstreamType=FineGrainedLineageDownstreamType.FIELD,
        downstreams=[fldUrn("bar", "c2")],
        confidenceScore=0.8,
        transformOperation="myfunc",

happy-baker-8735

11/18/2022, 3:19 PM

Yes! That's it! Thank you @bulky-soccer-26729!

microscopic-mechanic-13766

11/21/2022, 11:55 AM

The steps would be: •

pip install 'acryl-pyhive[kerberos]'

•

pip install kerberos

•

pip install thrift == 0.13.0

The downgrade to v0.13.0 thrift I think is neccesary due to an issue related with cookies (I think it appears in 0.15.0 with the

sendHeaders

method and it is solved in 0.17.0, which is not published yet)

👍 1

ancient-apartment-23316

11/23/2022, 10:03 PM

Could anyone help me with this issue?

microscopic-mechanic-13766

12/14/2022, 11:19 AM

Okey so apparently it is not a bug but an error from changing the actions version from 0.0.8 to 0.0.9 having trouble with the

executor.yaml

damp-greece-27806

12/20/2022, 3:34 PM

Hi, I’m encountering this exact same problem on datahub-frontend, but datahub-gms is logging as expected. We’re using Nomad to orchestrate the containers, deployed on aws. I’ve tried a variety of setting changes and have had no luck. This is preventing us from figuring out what’s causing datahub-frontend to 500.

silly-ability-65278

12/21/2022, 7:54 AM

Copy code

'[2022-12-21 07:52:22,520] ERROR    {datahub.entrypoints:207} - Command failed: Invalid IPv6 URL\n'

this is the error message that I ggot

powerful-cat-68806

12/21/2022, 11:06 PM

@orange-night-91387 any idea? I’ve changed here(values.yaml) the comment for the

sqlDbType

& ran

helm upgrade xxxx xxx

, but no change

Copy code

datahubUpgrade:
  enabled: true
  image:
    repository: acryldata/datahub-upgrade
    tag: "v0.9.3"
  batchSize: 1000
  batchDelayMs: 100
  noCodeDataMigration:
#    sqlDbType: "MYSQL"
     sqlDbType: "POSTGRES"

rhythmic-lock-29204

12/28/2022, 10:03 PM

datahub plugin list.txt,odbc config.txt,Error log,txt.log

odbc config.txt Error log,txt.log datahub plugin list.txt

powerful-cat-68806

12/29/2022, 9:35 AM

Hi team ☀️ Can someone pls. help with the above? ☝️

✅ 1

melodic-dress-7431

01/03/2023, 4:41 PM

Screenshot 2023-01-03 at 10.10.28 PM.png