Hi all, we have a usecase where • we have a table with a metric that is sourced from an ingestion aggregation • we know we will have to add columns to it relatively dynamically (user request) unfortunately up until now I had not registered this requirement from the documentation:

All metrics must have aggregation configs.

I feel like it is at the heart of what we are seeing now; in essence, you can't update the schema with a new metric, as the API says:

PUT schema response: {'code': 400, 'error': 'Invalid schema: staging_stream_st_mknaw_idle_worker_test14_sg_12. Reason: Schema is incompatible with tableConfig with name: staging_stream_st_mknaw_idle_worker_test14_sg_12_REALTIME and type: REALTIME'}

and, probably correctly, the other way around, trying to get the table update in before the schema update, also does not work

PUT table response: {'code': 400, 'error': "Invalid table config: staging_stream_st_mknaw_idle_worker_test14_sg_12 with error: The destination column 'mtr_clicks_sum' of the aggregation function must be present in the schema"}

so... is the implication that a pinot schema/table pair that has ingestion aggregation can.. never evolve? this would be unfortunate.