DataHub #troubleshoot

blue-zoo-89533

10/16/2021, 5:00 PM

Copy code

DataHub CLI version: 0.8.15.7
Python version: 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0]

blue-zoo-89533

10/16/2021, 6:05 PM

Copy code

datahub ingest -c bq.yml
[2021-10-16 18:04:11,101] INFO     {datahub.cli.ingest_cli:57} - Starting metadata ingestion
[2021-10-16 18:04:14,418] INFO     {datahub.cli.ingest_cli:59} - Finished metadata ingestion

Source (bigquery) report:
{'failures': {},
 'filtered': [],
 'tables_scanned': 5,
 'views_scanned': 1,
 'warnings': {},
 'workunit_ids': [--],
 'workunits_produced': 6}
Sink (datahub-rest) report:
{'failures': [], 'records_written': 6, 'warnings': []}

blue-zoo-89533

10/16/2021, 6:10 PM

Copy code

source:
  type: "bigquery"
  config:
    env: "DEV"
sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

witty-keyboard-20400

10/18/2021, 12:49 PM

When I try the following GraphQL query:

Copy code

{
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:cg,kv_entity,PROD)") {
    schemaMetadata {
      fields {
        fieldPath
      }
    }
  }
}

The response is an error message:

Copy code

{
  "errors": [
    {
      "message": "An unknown error occurred.",
      "locations": [
        {
          "line": 5,
          "column": 5
        }
      ],
      "path": [
        "dataset",
        "schemaMetadata"
      ],
      "extensions": {
        "code": 500,
        "type": "SERVER_ERROR",
        "classification": "DataFetchingException"
      }
    }
  ],
  "data": {
    "dataset": {
      "schemaMetadata": null
    }
  }
}

However, my GMS is running fine which I verified using the same dataset urn, but queried on ownership info, which worked. Could anyone help me understand what is wrong in my query. Basically I'm looking for GraphQL queries wherein: 1. Given a dataset URN and field path mention, the response gives list of all the field names. 2. field name is specified in query, and the response returns the names of all the datasets which have field with the matching name. 3. Glossary term is specified in the query and the response returns the names of all the fields (and datasets) to which the glossary term is tagged.

high-notebook-40979

10/20/2021, 4:46 AM

When i try to call with grap API then response code 500

Copy code

curl --location --request POST '<http://10.110.120.10:9002/api/v2/graphql>' \
--header 'Cookie: <my_cookie>' \
--data-raw '{
    "operationName": "listGroups",
    "variables": {
        "input": {
            "start": 0,
            "count": 25
        }
    },
    "query": "query listGroups($input: ListGroupsInput!) {\n  listGroups(input: $input) {\n    start\n    count\n    total\n    groups {\n      urn\n      type\n      name\n      info {\n        displayName\n        description\n        email\n        __typename\n      }\n      memberCount: relationships(\n        input: {types: [\"IsMemberOfGroup\"], direction: INCOMING, start: 0, count: 1}\n      ) {\n        total\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n"
}'

Response

Copy code

{
  "errors": [
    {
      "message": "An unknown error occurred.",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "listGroups"
      ],
      "extensions": {
        "code": 500,
        "type": "SERVER_ERROR",
        "classification": "DataFetchingException"
      }
    }
  ],
  "data": {
    "listGroups": null
  }
}

modern-nail-74015

10/20/2021, 6:53 AM

where Can I see these logs?

modern-nail-74015

10/20/2021, 6:55 AM

image.png

modern-nail-74015

10/20/2021, 7:09 AM

image.png

modern-nail-74015

10/21/2021, 1:19 AM

Which endpoint will be used for logging out for OIDC? I configured

revocation_endpoint

in my own

.well-known/openid-configuration

, but it seems not been called？

modern-nail-74015

10/21/2021, 8:26 AM

image.png

handsome-football-66174

10/22/2021, 3:49 PM

General- Has anyone successfully integrated with Ping OIDC ?

red-pizza-28006

10/26/2021, 6:48 PM

Untitled.txt

brief-lock-26227

10/29/2021, 4:19 PM

I'm trying to build from source (using @busy-dusk-4970’s recent fork for M1 support). But the

gradlew build

is failing:

Copy code

> Task :docs-website:generateGraphQLDocumentation FAILED
yarn run v1.22.0
$ docusaurus docs:generate:graphql
internal/modules/cjs/loader.js:883
  throw err;
  ^

Error: Cannot find module './fp/_baseConvert'
Require stack:
- /Users/patrick/src/datahub/docs-website/node_modules/lodash/fp.js
- /Users/patrick/src/datahub/docs-website/node_modules/wait-on/lib/wait-on.js
- /Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/webpack/plugins/WaitPlugin.js
- /Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/webpack/server.js
- /Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/commands/build.js
- /Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/index.js
- /Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/bin/docusaurus.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._resolveFilename (/Users/patrick/src/datahub/docs-website/node_modules/module-alias/index.js:49:29)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/Users/patrick/src/datahub/docs-website/node_modules/lodash/fp.js:2:18)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/patrick/src/datahub/docs-website/node_modules/lodash/fp.js',
    '/Users/patrick/src/datahub/docs-website/node_modules/wait-on/lib/wait-on.js',
    '/Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/webpack/plugins/WaitPlugin.js',
    '/Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/webpack/server.js',
    '/Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/commands/build.js',
    '/Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/lib/index.js',
    '/Users/patrick/src/datahub/docs-website/node_modules/@docusaurus/core/bin/docusaurus.js'
  ]
}
error Command failed with exit code 1.

I tried googling this but didn't find a solution. Any suggestions?

busy-dusk-4970

10/29/2021, 4:22 PM

I had it working until I got latest and now everything broke again and trying to figure out how to get it working again well at least running the ui cause it never actually worked 100%

curved-magazine-23582

11/01/2021, 1:47 PM

Follow up on externalUrl, I got charts, dataflow, datajobs and datasets all ingested with externalUrl. the

view in powerbi

buttons are working for charts, dataflow, datajobs, but not datasets. I am not sure what I am doing wrong here with datasets. Below is snippet of dataset json,

Copy code

{
  "entity": {
    "value": {
      "com.linkedin.metadata.snapshot.DatasetSnapshot": {
        "urn": "urn:li:dataset:(urn:li:dataPlatform:powerbi,Data Architecture,PROD)",
        "aspects": [
          {
            "com.linkedin.common.Ownership": {
              "owners": [
                {
                  "owner": "urn:li:corpuser:dataservices",
                  "type": "DATAOWNER"
                }
              ],
              "lastModified": {
                "actor": "urn:li:corpuser:dataservices",
                "time": 0
              }
            }
          },
          {
            "com.linkedin.dataset.DatasetProperties": {
              "description": "",
              "tags": [],
              "customProperties": {
                "Refreshable": "True"
              },
              "externalUrl": "<https://app.powerbi.com/groups/a4998c5d-fab6-46b0-99cc-490d3df28981/datasets/91a530c4-d39a-44ac-84b4-08e2942153a7/details>"
            }
          },
          {
            "com.linkedin.common.Status": {
              "removed": false
            }
          },
          {
            "com.linkedin.schema.SchemaMetadata": {
              "schemaName": "Data Architecture-schema",
              "platform": "urn:li:dataPlatform:powerbi",
              "version": 0,
              "created": {
                "actor": "urn:li:corpuser:dataservices",
                "time": 0
              },
              "lastModified": {
                "actor": "urn:li:corpuser:dataservices",
                "time": 0
              },
              "hash": "",
              "platformSchema": {
                "com.linkedin.schema.KafkaSchema": {
                  "documentSchema": "{\"type\":\"record\",\"name\":\"Data Architecture-schema\",\"namespace\":\"com.linkedin.dataset\",\"doc\":\"\",\"fields\":[]}"
                }
              },
              "fields": []
            }
          },
          {
            "com.linkedin.common.BrowsePaths": {
              "paths": [
                "/prod/powerbi/samtec - pbi test environment/data architecture"
              ]
            }
          }
        ]
      }
    }
  }
}

faint-hair-91313

11/03/2021, 4:23 PM

image.png

faint-hair-91313

11/03/2021, 4:23 PM

image.png

adamant-van-40260

11/08/2021, 2:59 AM

anw please suggest a solutions 😞

curved-magazine-23582

11/08/2021, 5:37 AM

hello, I upgraded to latest with docker compose, but now UI won't work, and I assume there is something wrong with GMS, since GMS container status is now marked as unhealthy. Anyone reporting similar issues? 🤔

refined-apple-6340

11/08/2021, 9:04 PM

any ideas on the state question in docker env?

refined-apple-6340

11/08/2021, 9:59 PM

past the state parameter now getting

refined-apple-6340

11/08/2021, 9:59 PM

org.apache.shiro.crypto.CryptoException: Unable to execute 'doFinal' with cipher instance [javax.crypto.Cipher@65185d4a]

refined-apple-6340

11/08/2021, 11:06 PM

i have now setup a client in google, configured the docker-compose correctly, used ngrok to get a public ip, when logging into datahub on localhost:9002 i get the redirect and login from google but still get the state not consistent error i will ask in office hours tomorrow

teamwork 1

mysterious-park-53124

11/11/2021, 4:30 AM

I run 'datahub ingest -c kafka_to_datahub_ex.yml' and fail anyone know this error, please?

red-pizza-28006

11/12/2021, 2:48 PM

Any ideas or if anyone could share an example of a Kafka ingestion using confluent? Thread in Slack Conversation

nice-planet-17111

11/15/2021, 2:41 AM

I was trying restore-indices via kubectl, (`kubectl create job --from=cronjob/datahub-datahub-restore-indices-job-template UnknownFormatConversionException`error during job process (and it fails). I didn't really changed anything, maybe only difference is that i use external CloudSQL as storage instead of Mysql Pod. helm upgrade or all the other operation works fine. Any ideas on what this means? 😞 (full log on thread) cc. @square-activity-64562

billions-tent-29367

11/17/2021, 12:15 AM

I'm curious: why do you folks choose to check in generated json/avsc/pdl files?

refined-apple-6340

12/01/2021, 7:23 PM

what is the setting for ELASTIC_CLIENT_SSL_SECURE_RANDOM_IMPLEMENTATION. (can you give an example)

refined-apple-6340

12/02/2021, 2:58 PM

thanks

refined-apple-6340

12/03/2021, 7:33 PM

Attributes: Requested Extensions: X509v3 Subject Alternative Name: DNS:opensearch