https://datahubproject.io logo
Join Slack
Powered by
Hello again :slightly_smiling_face: is it possible...
# ingestion
n
Hello again 🙂 is it possible to customize folder structure of entities on web browser when ingesting from mysql? For example, currently entities is saved under 
prod/mysql/{schema_name}/{schema_name}.{table_name}
. But this has some problems.. 1. it does not represent instance name - therefore make it hard to search 2. When i ingest from multiple instances, all the schemas will be under the same mysql. Is there a way to set path like prod/mysql/{instance_nam}/{schema_name}/{table_name} or something like that?
✔️ 1
b
n
That would be a way, but i have to edit data after ingestion.. I was wondering if i could set path in the ingestion stage.
b
There is a transformer method in the thread as well
n
Oh sorry, i missed that part. thank you so much 🙂
For anyone who will be interested later : you can set transformers like this at the end of the recipe yaml file
Copy code
transformers:
  - type: "set_dataset_browse_path"
    config:
      replace_existing: True
      path_templates:
        - /ENV/PLATFORM/my-custome-file/DATASET_PARTS
plus1 1
s
There is a guide https://datahubproject.io/docs/metadata-ingestion/transformers/ for these transformers with examples. But these seem to be under "Dev guides". Raised this PR https://github.com/linkedin/datahub/pull/3591 which will move it under metadata-ingestion and make it easier to find
❤️ 2
s
Old threat but I was wondering... Is there any transformer, or any other way, to set a different urn at ingestion? My urns look like this: urnlidataset:(urnlidataPlatform:databricks,.DATABRICKSWORKSPACEURL.prod.conformed.v_dim_bil,PROD) While I need them to look like this: urnlidataset:(urnlidataPlatform:databricks,prod.conformed.v_dim_bil,PROD) So that the lineage between PowerBI and Databricks will be conserved...
Open in Slack
PreviousNext
Powered by