Apache Pinot

So this code is not ours, there are pinot users who have written all kinds of code/scriptware etc to generate these, and most of the time we aren't even aware of their existence.

well all the more reason for early validation :slightly_smiling_face:

I modified the schema to allow both 3 and "3" FYI

this does mean we cannot enforce additioanl constraints (for eg: min / max )

I agree, we do need validation for sure. As Subbu suggested, we can allow for level of checking (strict vs loose) and move from loose to strict over time for deployments where that is needed.

upon further reading, it doesn't look like json-schema has tunable validation levels. We might have to skip validation checks for a lot of fields (eg: inverted indices) to account for custom code generated table config.

at this point - I'm not sure if json-schema will work fo rus

then just add rules, we can still have json-schema based validation but thats just one of the rules

you mean, be able to configure json-schema as one of the validation mechanisms ?

<@U01D787RXNJ>: let's go ahead with that strategy - lets make the json schema based validation configurable. So by default - we don't enforce it. Let's finish up the remaining items in the schema and open a PR ?

I've updated the schema in the docs <https://docs.google.com/document/d/1uw0lDBKDU9tM_R1mTMgQFmcAwz9pc5QJzeNSRe1GxXE/edit#>
if it's good we can move forward with opening the PR for making it configurable

FYI: Opened a new PR for some missing validation. <@UDQU92KBK> might want to eyeball it - I dont think this will cause any issues in the LinkedIn integration tests - but doesn't hurt to verify

<https://github.com/apache/incubator-pinot/pull/6541>

lol - it just broke a bunch of integration tests :slightly_smiling_face:. Looks like some integration tests are specifying a range index on a non numeric column

ok - ready for review now: <https://github.com/apache/incubator-pinot/pull/6541>

<@UDQU92KBK> mind reviewing before we commit ?

<@UDSRQR2GP> <@UDT7GFEG6> we're using this doc to track all remaining validations: <https://docs.google.com/document/d/1uw0lDBKDU9tM_R1mTMgQFmcAwz9pc5QJzeNSRe1GxXE/edit>

As Jackie mentioned some of the recent validations are for newer features or were missed. I can use this to categorize these as MUST HAVE or SUGGESTED ?

we can also use those 2 categories for enforcing how strict the validations are

one PR in progress related to this is : <https://github.com/apache/incubator-pinot/pull/6485>

which is trying to enforce a schema on table config - this will be the most stringent check and will probably belong in the SUGGESTED cateogry