Hello all, I’ve started to ingest data from Kafka ...
# ingestionp
polite-flower-25924
08/13/2021, 9:08 AMHello all, I’ve started to ingest data from Kafka & Hive. All dataset origin is set to “PROD”. Is it possible to adjust that in ingestion recipes and how can I change that after ingestion? Thank you
b
better-orange-49102
08/13/2021, 9:11 AMi assume u meant, remove "prod" from the UI browsing hierachy to get to the dataset?
for instance: prod > hive > your-DB > your_table?
there is a transformer you can apply in the ingest recipe to remove it such that it becomes
hive > your-DB > your_table
s
square-activity-64562
08/13/2021, 9:25 AMp
polite-flower-25924
08/14/2021, 7:53 AMThank you for the answers! I will do that and let you know the results. Thank you @better-orange-49102 @square-activity-64562
polite-flower-25924
08/14/2021, 8:13 AMIs it possible to update the dataset after ingestion? @mammoth-bear-12532 @big-carpet-38439
b
better-orange-49102
08/14/2021, 9:08 AMYes absolutely. Ingesting the data again from the same dataset will create a new "version". You can see the versions in the UI or by accessing the MySQL database that datahub uses.
p
polite-flower-25924
08/14/2021, 9:18 AMThank you @better-orange-49102!
m
mammoth-bear-12532
08/14/2021, 2:25 PM
@polite-flower-25924 what is the issue with the env that you previously set? Is it incorrect?
mammoth-bear-12532
08/14/2021, 2:26 PM
The dataset urn includes the origin (env), so you can’t really change the dataset’s identifier, however you CAN change how you navigate to it by customizing the browse path aspect.
p
polite-flower-25924
08/14/2021, 7:16 PM@mammoth-bear-12532 actually, I didn’t set anything. It’s automatically set to Prod
polite-flower-25924
08/14/2021, 7:17 PMI just ingest the metadata from dev environment, but it’s populated with “PROD” even though I didn’t specify that
m
mammoth-bear-12532
08/14/2021, 7:40 PM
right, the default env for the connectors is PROD
thankyou 1
mammoth-bear-12532
08/14/2021, 7:42 PM
you will have to specify
env: DEVthankyou 1