Hi All, I am in process to integrate the datahub with great expectations on composer 2 for bigquery. I can see the bigquery table is available on datahub however when transmitting it to datahub for bigquery getting below, but it doesnt reflect on the bigquery table on datahub, any suggestions: The table name for the urn at bigquery is "FBT_Diff" however when transmitting it lowercases the table table name "fbt_diff"

Calculating Metrics:   0%|          | 0/15 [00:00<?, ?it/s]
Calculating Metrics:   0%|          | 0/15 [00:00<?, ?it/s]
Calculating Metrics:  13%|█▎        | 2/15 [00:00<00:01, 12.15it/s]
Calculating Metrics:  13%|█▎        | 2/15 [00:00<00:01, 12.15it/s]
Calculating Metrics:  13%|█▎        | 2/15 [00:00<00:01, 12.15it/s]
Calculating Metrics:  27%|██▋       | 4/15 [00:02<00:08,  1.34it/s]
Calculating Metrics:  27%|██▋       | 4/15 [00:02<00:08,  1.34it/s]
Calculating Metrics:  27%|██▋       | 4/15 [00:02<00:08,  1.34it/s]
Calculating Metrics:  47%|████▋     | 7/15 [00:02<00:05,  1.34it/s]
Calculating Metrics:  47%|████▋     | 7/15 [00:02<00:05,  1.34it/s]
Calculating Metrics:  80%|████████  | 12/15 [00:05<00:01,  2.14it/s]
Calculating Metrics:  80%|████████  | 12/15 [00:05<00:01,  2.14it/s]
Calculating Metrics:  80%|████████  | 12/15 [00:05<00:01,  2.14it/s]
Calculating Metrics: 100%|██████████| 15/15 [00:07<00:00,  1.86it/s]
Calculating Metrics: 100%|██████████| 15/15 [00:07<00:00,  1.86it/s]
Calculating Metrics: 100%|██████████| 15/15 [00:07<00:00,  1.86it/s]
Calculating Metrics: 100%|██████████| 15/15 [00:07<00:00,  1.86it/s]
Calculating Metrics: 100%|██████████| 15/15 [00:07<00:00,  1.93it/s]
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Finding datasets being validated
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Datasource my_bigquery_datasource is not present in platform_instance_map
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - GE expectation_suite_name - demo, expectation_type - expect_column_values_to_not_be_null, Assertion URN - urn:li:assertion:6f56acc887e38af0561eaeb8d41b0bdb
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - GE expectation_suite_name - demo, expectation_type - expect_column_values_to_be_between, Assertion URN - urn:li:assertion:aa04dc0fc98f145d01ae9fcd5f7f4ee3
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Sending metadata to datahub ...
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Dataset URN - urn:li:dataset:(urn:li:dataPlatform:bigquery,project.dataset.fbt_diff,PROD)
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Assertion URN - urn:li:assertion:6f56acc887e38af0561eaeb8d41b0bdb
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Assertion URN - urn:li:assertion:aa04dc0fc98f145d01ae9fcd5f7f4ee3
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Metadata sent to datahub.
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Validation succeeded!
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - 
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - Suite Name                                   Status     Expectations met
[2023-02-08, 01:07:57 EST] {subprocess.py:89} INFO - - demo                                       ✔ Passed   2 of 2 (100.0 %)
[2023-02-08, 01:07:59 EST] {subprocess.py:93} INFO - Command exited with return code 0

passing Datasource my_bigquery_datasource is not present in platform_instance_map value as :
action_list:
  - name: store_validation_result
    action:
      class_name: StoreValidationResultAction
  - name: store_evaluation_params
    action:
      class_name: StoreEvaluationParametersAction
  - name: update_data_docs
    action:
      class_name: UpdateDataDocsAction
      site_names: []
  - name: datahub_action
    action:
      module_name: datahub.integrations.great_expectations.action
      class_name: DataHubValidationAction
      server_url: <http://ip_address:8080>
      platform_instance_map:
         datasource_name: my_bigquery_datasource
      parse_table_names_from_sql: true

Hey @quaint-barista-82836, there is no workaround possible on GX side, however, you can convert bigquery ingestions urns to lowercase by updating recipe for bigquery ingestion:

convert_urns_to_lowercase: True

Hey @hundreds-photographer-13496, so will this put both dataset and table name in lower case or only tables ? As the urn which we get from GX has

urn:li:dataset:(urn:li:dataPlatform:bigquery,lowercase_project_name.FBT.fbt_diff,PROD)

Also if I re-ingest the metadata in bigquery, with convert_urns_to_lowercase: True, will it overwrite existing urns or make a new entry ?

Hi @quaint-barista-82836, was this resolved, or is this the issue you recently reposted?

Hi @astonishing-answer-96712, the issue still persist so opened a new thread: https://datahubspace.slack.com/archives/C029A3M079U/p1675878525432749

Hey @quaint-barista-82836, let's continue discussion on this thread, as it has all the context related to this problem.

if I re-ingest the metadata in bigquery, with convert_urns_to_lowercase: True, will it overwrite existing urns or make a new entry ?

The ingestion with that

convert_urns_to_lowercase

changed, would create new URNS(and so new dataset). The old dataset (with old URN) will have to be deleted. If you were using stateful ingestion in recipe, delete will happen automatically.

Hi @hundreds-photographer-13496 and Team any other suggestions, looks like the datahub lowercase complete urn and GX does onky tablename from urn which makes them incompatible

This looks like an issue. Can you please create a github issue mentioning the Dataset URN from GX and Bigquery Dataset URN with and without using

convert_urns_to_lowercase: True

?

Hi @hundreds-photographer-13496, created issue at github : https://github.com/datahub-project/datahub/issues/7300

if I re-ingest the metadata in bigquery, with convert_urns_to_lowercase: True, will it overwrite existing urns or make a new entry ?

The ingestion with that

convert_urns_to_lowercase

changed, would create new URNS(and so new dataset). The old dataset (with old URN) will have to be deleted. If you were using stateful ingestion in recipe, delete will happen automatically. This has created 2 records on datahub with "stateful_ingestion":{"enabled": True },

did you have stateful ingestion enabled when not using convert_urns_to_lowercase before? How stateful ingestion works is - it stores seen datasets for every run. If the same pipeline (

pipeline_name

config) runs again and the dataset from last run is not seen, ingestion soft-deletes that dataset. In case you are not able to get this to work using stateful ingestion, you can start bigquery ingestion afresh by soft deleting bigquery datasets (data is not permanently deleted, just marked as removed) using datahub cli command.

datahub delete --platform bigquery --soft

and then run bigquery ingestion again to surface only the required datasets.

Hi @hundreds-photographer-13496 following up on the fix of the request.

created this PR with config to lowercase urns, if required. https://github.com/datahub-project/datahub/pull/7472