numerous-refrigerator-15664
05/16/2023, 11:16 AMdatabase_pattern
and table_pattern
.
Additionally, I'm trying to do the following things, and I need some advice on it. Any help would be appreciated.
1. Now I'm trying to ingest most of the datasets from hive metastore
, and I was wondering if there's a way to do pattern filtering for other items in hive metastore
as well:
• DBS.DB_LOCATION_URI (e.g. allow only the pattern "hdfs://cluster1/dsc/.*")
• DBS.OWNER_NAME (e.g. deny those with accounts ".*test")
• If it is impossible via recipes, would there be any other possible ways?
2. Our organization manages additional metadata for hive's DB, table, and column in other mysql DB (say our_meta
) which is separated from hive metastore
. For example, for a table named customer.cust_mst
on the hive, this table exists in the hive metastore
, and a separate mysql DB also manages information about this table. Given the situation, I'd like to ingest the metadata of our_meta
into datahub. What should be the best way to do it?
• Some of the managed items in our_meta
(mainly technical meta) seems to be managed as custom properties, and this should be synced in batch mode.
• Some of the managed items in our_meta
(mainly business meta) can be managed as business glossary or tags, and both batch sync and API calls should possible.
• I am looking into custom ingestion source and metadata ingestion transformer, but I am not sure how to approach.
I hope I can handle these without forking the source if possible. Thanks in advance.astonishing-answer-96712
05/16/2023, 8:29 PMnumerous-refrigerator-15664
05/16/2023, 11:22 PM