This should probably go in as a bug report, but st...
# ingestiono
orange-flag-48535
11/10/2021, 11:35 AMThis should probably go in as a bug report, but still checking here once: When providing SchemaMetadata items in a MCE file, it seems Datahub generates different graphs depending on the order of the fields. When I created a file with nested fields listed before the parent fields (as discernible from its fieldPath value), it didn't nest the inner fields within the outer fields correctly. Is this known behaviour?
m
mammoth-bear-12532
11/10/2021, 3:19 PM
Hi @orange-flag-48535: when you say “generates different graphs”, are you talking about the UI rendering of the schema?
o
orange-flag-48535
11/10/2021, 4:10 PMThat's right. The UI renders the schema in different ways. If I don't list the parent fields first, the nested fields are not picked up as children of the parent, even though the fieldPaths are correct. I have attached a small MCE JSON file that demonstrates the issue.
m
mammoth-bear-12532
11/10/2021, 4:22 PM
Thanks for reporting it. There is currently an assumption both in the emitter and the rendering logic that “order of fields” matters.
mammoth-bear-12532
11/10/2021, 4:23 PM
we had previously gotten feedback that people don’t want to see their fields re-ordered (sorted) etc… so we’re basically processing the fields in the order received
b
big-carpet-38439
11/10/2021, 5:34 PMIs this something we should track, though? I feel like we should sort the fields on our side perhaps?
m
mammoth-bear-12532
11/10/2021, 6:00 PM
hmm in this case Pramod is creating a hand crafted MCE
mammoth-bear-12532
11/10/2021, 6:00 PM
so not much the client can do
mammoth-bear-12532
11/10/2021, 6:00 PM
we previously got feedback that people don’t want us changing the order of fields on them
mammoth-bear-12532
11/10/2021, 6:00 PM
which I agree with
mammoth-bear-12532
11/10/2021, 6:02 PM
so I feel inclined to say that we can either
1) say this is the expected behavior .. and just teach people what a well-formatted MCE looks like
2) detect this kind of thing in the UI and sort and display only when we detect fields are out of order, but preserve regular ordering otherwise.
b
big-carpet-38439
11/10/2021, 7:45 PMHmm I see… yeah because I think this is not intuitive for someone building a new integration, it’s an implicit contract
o
orange-flag-48535
11/11/2021, 7:25 AM@mammoth-bear-12532 small correction: I'm not hand-crafting the MCE file in my actual setup. There I'm generating it from Java code, by parsing some other JSON file provided as input.
orange-flag-48535
11/11/2021, 7:37 AMIt'd be nice to have this documented somewhere. I had started thinking of fieldPath as a nice way to create/refer-to the schema structure, irrespective of where it appears physically in an MCE file (kind of like the relational model vs network model).
m
mammoth-bear-12532
11/11/2021, 8:14 AM
Thanks Pramod this is helpful. Do you think a Java client sdk (similar to the python sdk) would be helpful here?
o
orange-flag-48535
11/11/2021, 10:18 AMYes. A Java SDK would be very useful. I have another thread going (i think in the #ingestion channel) where I've enquired about Java bindings for the MCE format.
k
kind-dawn-17532
11/14/2021, 2:28 PMOrdering of field is important in multiple scenarios. If there is an assumptionas, Let's make it explicit through code rather than being implicit. That way people can have more control over the order and they will be able to change it based on their needs. @mammoth-bear-12532