hi, I am struggling to update table config, I have updated schema to add new column and for same column I have written transformationFunction, in existing table it is showing the new column but not copying the value in that new column, with same table config I created new table and all is working fine. I have also reloaded the segments but not working with the existing table, record's json string is ->

{
  "header": {
    "nnTransId": "9003",
    "qid": 1,
    "timestamp": 1234567890123
  },
  "status": "N200_SUCCESS"
}

"ingestionConfig": {
      "transformConfigs": [
        {
          "columnName": "header_js",
          "transformFunction": "jsonFormat(header)"
        },
        {
          "columnName": "header_nnTransId",
          "transformFunction": "JSONPATHSTRING(header, '$.nnTransId')"
        } .....

Did you not do both at the same time? One issue I can see is that adding transform function on existing column is considered backward incompatible and is not allowed. If you created the column in schema first and later trying to add transform function you might run into this issue

both at the same time means , I first created column in schema today and just after that updated table schema. Without adding column in schema it wont allow to update column in table config.

Are you able to update the table config with transform, and confirm that it is accepted?

yes it says updated successfully

Hmm that doesn’t make sense. If table config shows transform function when you check in Pinot UI, then it should be applied

Hi, so were there any records in the table before the schema and table config were updated? If yes, can you check if the query result contains older records or new records

query is getting all new and old records and new records not getting value in this new column. I am definitely not expecting that existing records would have this value which I understood it will not get data in new column but new records also not getting this value which is really should not happen.

Yes, that's strange. Will look into it.

hi Karthik I checked again and now for new records getting data, it took time I am not sure why, I tested it with new records so many times yesterday but today I saw value is coming now.

That's good to hear! Will it be possible for you to tell after how long since making the config change did the records started getting data?

not sure about exact time because I tried yesterday and today I can see value is coming and yesterday's records still don't have value in it, after adding column I am sure about till 1 hr I have checked data was not coming

Got it. I was asking this because we have some periodic jobs that run inside Pinot. So it could be one of them picked up something and changed the output. Will have to debug a bit further though. Can you share your table config (before and after change), schema and some sample records. Do remove the secret values and brokers from it.

@Kartik Khare table is very simple , I think due to indexing this issue is happening, I have done this change in one of the table which has no indexes and it replicated instantly, but those table who has sorted and range index are the one taking time

So the table had indices even before making the change or were they added after?

they have existing indices

Thanks. I will try to reproduce the issue.