Hi everyone! I am researching realtime table evolu...
# troubleshootingr
Ruslan Rodriquez
07/08/2021, 10:41 AMHi everyone! I am researching realtime table evolution. After updating pinot schema and reloading segments I see new columns in table and null values in old data. But after consuming new data with not empty newly added fields, new data are importing with null values too in new columns. Kafka messages in avro formats. When I debug the code I see that AvroRecordExtractor still uses old set of fields. Can I refresh fields set in AvroRecordExtractior and start consuming messages with new columns?
j
Jackie
07/08/2021, 4:50 PMThe current consuming segment won't be able to pick up the new values immediately because the writers for the new added columns are not setup. The next consuming segment will pick the new fields up.
Jackie
07/08/2021, 4:52 PMCan you please file an issue for the requirement? One work around we can do is to drop the current consuming segment and replace with a new one with all fields set up properly. But that also means the already consumed data within the consuming segments are dropped and will be re-consumed in the replacing segment
r
Ruslan Rodriquez
07/12/2021, 6:52 AM