Hi everyone, first of all, thank you all for devel...
# ingestionh
hallowed-spring-18709
10/06/2022, 1:23 PMHi everyone, first of all, thank you all for developing such a great tool.
I have been trying to ingest a simple recipe using the datahub's ui but I want this recipe to ingest properties as well. I get a success message on ingest but no properties get added to my ingested files. I will post my recipe on the thread. I've seen there is a post already suggesting to start with the mce files from the examples but I am talking specifically on how to achieve the ingestion via the ui in accordance with the documentation here: https://datahubproject.io/docs/metadata-ingestion/docs/transformer/dataset_transformer#simple-add-dataset-datasetproperties
hallowed-spring-18709
10/06/2022, 1:26 PM{
transformers: [
{
type: "simple_add_dataset_properties",
config: {
properties: { prop1: "pikachu" },
},
},
{
type: "simple_add_dataset_tags",
config: {
tag_urns: [
"urnlitag:Test Tag",
],
},
},
{
type: "simple_add_dataset_terms",
config: {
term_urns: [
"urnliglossaryTerm:Test Glossary Term",
],
},
},
{
type: "simple_add_dataset_domain",
config: {
semantics: "OVERWRITE",
domains: ["urnlidomain:Test Domain"],
},
},
],
sink: {
type: "datahub-rest",
config: { server: "http://datahub-gms:8080" },
},
source: {
type: "s3",
config: {
// intentionally removed this before posting to slack
},
},
};
hallowed-spring-18709
10/06/2022, 2:56 PMIf I modify the transform, again following the guide posted above, to something like this:
Copy code
{
type: "simple_add_dataset_properties",
config: {
replace_existing: true,
properties: { ingestedDate: "2022-10-06T07:45:37.882Z" },
},
},replace_existingsimple_add_dataset_propertieshallowed-spring-18709
10/06/2022, 4:49 PMnevermind... I just found out what the problem is... which is weird... I had to move that transform to the bottom of the transformers list to make it work
d
dazzling-judge-80093
10/07/2022, 12:41 PMI’m glad it works now