https://pinot.apache.org/ logo
Join Slack
Powered by
Hi!! I’m experiencing some issues with Schema evol...
# troubleshooting
r
Hi!! I’m experiencing some issues with Schema evolution. I have a REALTIME table consuming from a kafka topic with a defined schema. if I add a new column to the table initially the column doesn’t appear but after reloading the segments the table appears in queries but with 
null
value. The only way I have achieved to see data on the new column is restarting the server instances. Any idea about I have to do to get the data n the table after update the schema? Note: following the documentation I added the property: 
pinot.server.instance.reload.consumingSegment=true
to my 
pinot-server-config
but the result is the same.
m
What version are you using? cc @Jackie
r
I’m using the version 
chart=pinot-0.2.4
I have installed it using the helm chart
To be honest I not sure if this chart version is the same version of the pinot instance. I have tried to find the pinot version but I can’t find it in the controller.
j
This is expected. The next consuming segment will start consuming the real values. See https://docs.pinot.apache.org/users/tutorials/schema-evolution for details
r
Yes!! I followed that guide, in fact here it is my current configuration:
After update the schema I have ran a 
/segments/{tableName}/reload
but after add even if I send new events I’m always getting null on the new columns when query the table.
my doubt now is the following: this 
chart=pinot-0.2.4
is my pinot version? if it is I’m running a very old version….
s
Values from kafka will be populated when the next consuming segment starts. Until then, it will be null values for all segments including the current consuming segment.
r
I’m able to kill and starts segements using the API?
because if I have understood correctly if I simply kill the consuming segment and start a new one it should be solved
s
That part went through some changes recently. @Neha Pawar can you help to see if this works?
n
You could restart the servers, that will make the CONSUMING segments restart, and pick new values. But dont kill the consuming segments (i.e. dont DELETE those segments), that is hard to recover from
r
@Neha Pawar as I understand there is no way from the API to do it right?
For our use case we need to perform the schema update automatically and we need to be ready to ingest data after the schema update
Restart the server is not an easy option for us.
n
you can try reset API. let me confirm if this will do what you want to achieve. if you have a test environment, coould you try it out?
this is the API, you can find it under Segment section in swagger
Copy code
@POST
  @Path("segments/{tableNameWithType}/{segmentName}/reset")
r
@Neha Pawar that approach works, after calling the reset we get the column filled but we are not able to query the table during the reset which is not good. We are using a table with one segment, if we modified that it could help us?
n
you mean you will increase the replication? yes, that is a reasonable approach
r
@Neha Pawar I have tried to increase the replication but the setting seems to be ignored:
Copy code
"segmentsConfig": {
      "schemaName": "context",
      "timeColumnName": "timestamp",
      "replication": "2",
      "replicasPerPartition": "2"
    },
m
Do you have enough servers with the required tenant tag available that can be used to increase replication?
n
r
I’m running two servers and I have 2 tables, one with 4 segments and the other one with 1 segment and all of them are running on the same server
n
okay. Follow that guide to make sure you 1) tag the new server using same tenant as previous server 2) run a rebalance
r
I have created a new server and tried to rebalance getting this:
Copy code
{
  "status": "NO_OP",
  "description": "Table is already balanced",
  "instanceAssignment": {
    "CONSUMING": {
      "instancePartitionsName": "context_CONSUMING",
      "partitionToInstancesMap": {
        "0_0": [
          "Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098",
          "Server_pinot-server-2.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098",
          "Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098"
        ]
      }
    }
  },
  "segmentAssignment": {
    "context__0__0__20210810T2104Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098": "CONSUMING"
    }
  }
}
but no luck getting more segments
n
did you set the includeConsuming flag to true?
r
it seems to work but I haven’t seen any change on the number of segments on the UI
Copy code
{
  "status": "IN_PROGRESS",
  "description": "In progress, check controller logs for updates",
  "instanceAssignment": {
    "CONSUMING": {
      "instancePartitionsName": "context_CONSUMING",
      "partitionToInstancesMap": {
        "0_0": [
          "Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098",
          "Server_pinot-server-2.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098",
          "Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098"
        ]
      }
    }
  },
  "segmentAssignment": {
    "context__0__0__20210810T2104Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-server-2.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098": "CONSUMING"
    }
  }
}
n
even after a couple of minutes?
can you paster the idealState and externalView?
r
externalView:
Copy code
{
  "OFFLINE": null,
  "REALTIME": {
    "context__0__0__20210810T2236Z": {
      "Server_pinot-server-2.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098": "CONSUMING"
    }
  }
}
IdealState
Copy code
{
  "OFFLINE": null,
  "REALTIME": {
    "context__0__0__20210810T2236Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-server-2.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098": "CONSUMING"
    }
  }
}
I killed server pods and now I can see two instances for the segment
n
hmm, that should’ve been sutomatic. Any chance you saved pinot-server-1's logs before you killed it?
r
ohh I didn’t save it sorry!
it seems that even with the replication in segment, when I call reset
the queries are not available during the process
n
hmm, my bad. I shouldve realized that the resst call will disable both replicas at once. There is no one-by-one mode there. Perhaps we need one.
As for your issue, let me think if there’s another way to prevent downtime
unfortunately, right now there’s no way without that downtime during reset. Created an issue for this: https://github.com/apache/pinot/issues/7280 . Discussions/contributions welcome!
❤️ 1
r
Thank you very much for your help on this @Neha Pawar
🙂 1
Open in Slack
PreviousNext
Powered by