@charles could you please give an example of rather complex jsonSchema that is supposed to be used for catalog? Probably with inner fields and etc? Or at least which source can have one. Also I do not see cleanedName and dataType for fields anymore. Thats what we had

name: string;
cleanedName: string;
type: string;
dataType: string;

and I only receive now

type

and

name

. Also just in case I have selected bad source that doesn’t have

cleanedName

-

cleanedName

was renamed to

aliasName

on root level. Should I assume that we can receive such field?

I managed to semi-generate this catalog from stripe with nested jsonSchema: (you might want to edit it and remove namespace related stuffs, those are new fields in another PR)

could I assume that we can still have it later, so I will just override getter instead of removing the field totally.

That would depend if we re-introduce some configuration inside the json schema or not? how are we going to clean field names, change their types and possibly select/unselect them, right?

yes

Yes, for now we can configure only streams (not fields) with this refacto

is it okay if we just pass the json schema for now? i think this will change more in the future so i'd rather stick with something that resemebles what we already have since we don't know yet.

For now I am just passing jsonSchema as it may require additional changes on UI too 🙂

@Artem Astapenko I’ve got a more complex catalog.json now:

wait, we have anyOf?

it’s a

AnyOf

nested down in the json schema object at

customers.sources

which can be either an object or an array i think in the UI you don’t need to show the nested streams but just the top level

customers

?

ammm @charles isn’t it what we are trying to achieve? Otherwise how do we see traversing the jsonSchema if one fields we need to display and others not

i thought we had discussed this as part of fully supporting json schema. my bad, if we did not.

I am not even sure how we are going to display it in UI and not sure that it will be easy to add support for that right now.

so while we will need to support this very soon, if we have everything except oneof / anyof then i think thats an okay state.

so as for anyOf - I can display just customer.sources, right?

okay. can we get to a place where we support everything but oneof / anyof? if the ui encounters a oneof or anyof it ignores it. so it won't break and it will still send it back to the api, but it won't try to render it.

I think yes

okay. let's do that.

then it means I have no choice except having it done by that moment 🙂

thanks!