A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hello everyone, in the Google Cloud there is the possibility to define labels for resources. So far we have not been able to query these labels via the API and save them in datahub. Is there a suitable solution to the problem here? Information on this is very welcome. Thank you very much.

Hello there <@U02K4CXMY3W>! That's an interesting use case. To make sure I understand your question properly- is the problem that you can't read the labels out of google cloud or that you can't save them in Datahub?

Datahub does have some user-friendly APIs for applying tags to datasets :slightly_smiling_face:

Hello <@U01JUBAPZPS>, the plan is to store the label as a tag in datahub. Tagging seems pretty easy with GrapgQL or RestAPI like

`urn:li:dataset:(urn:li:dataPlatform:TestProject,/TestDataset/TestTable1,PROD)`

The problem is how to get the Labels from a resource like GCS, BigQuery or Dataflow,

I see- have you looked into the public APIs these sources supply? Each source is going to be slightly different

it looks like Google Cloud provides a full rest API here: <https://cloud.google.com/resource-manager/reference/rest>

I'm sure there will be one endpoint to let you fetch the labels applied to your resources

over time, folks have contributed different connectors they have written to extract metadata from various systems

this means that everyone in the community won't have to duplicate work!

unfortunately, it doesn't look like anyone has contributed a google cloud source yet, so you would need to write that yourself. If you did, we'd love for you to contribute it back to open source!

that way other folks could easily extract tags from GCS

we do have a bigquery source but it looks like it is not currently extracting tags: <https://datahubproject.io/docs/metadata-ingestion/source_docs/bigquery>

I would also encourage you to contribute that extension to the existing bigquery source if you are looking into it!

OK I see, here is an example python code from google how to extract labels from bigquery:
<https://github.com/googleapis/python-bigquery/blob/35627d145a41d57768f19d4392ef235928e00f72/samples/get_dataset_labels.py>

Sure thing <@U02K4CXMY3W> - let me know if you have any questions while you work on this integration! I'd love to help you contribute back any progress you make as well :slightly_smiling_face:

Also I would welcome any feedback you have about the experience- what could be simplified, clarified, etc!