hello folks, I’ve been working with feature ‘schema evolution’ on aws EKS cluster. I set ‘pinot.server.instance.reload.consumingSegment’ as true on server part of helm chart, and tried to upsert realtime table with FULL option. Column ‘doNotFailPlease’ was added after schema edit-reloading segment part, but that column never got updated by upsert. Is schema evolution supporting upsert of realtime table? --- FYI, pic #1 : before adding a column pic #2 : schema edit pic #3 : column displays well but doesn’t update pic #4 : my kafka event sample

@Yeongju Kang The values for the newly added columns won’t be reflected within the current consuming segment(s). The next consuming segment(s) will start consuming the actual values.

@jagadesh So does it mean that rows before being sealed won't be able to have desired state? Let's say that my segment size is 2.5M and my offset is 1M. Will next 1.5M rows of the new column value be ignored?

@Jackie ^^

Schema evolution can only add default value to the new added columns for consuming segments currently. @Yeongju Kang for the example you give, yes the next 1.5M rows will be ignored and filled with default values. We have some ideas on solving this, and it is WIP

@Jackie Thanks!