https://pinot.apache.org/ logo
d

Devashish Gupta

02/10/2021, 11:06 AM
Hi Team, What is the recommended way of updating table schema. I used the following job to create my table
Copy code
apiVersion: batch/v1
kind: Job
metadata:
  name: request-realtime-table-creation
  namespace: data2
spec:
  template:
    spec:
      containers:
        - name: request-realtime-table-json
          image: apachepinot/pinot:latest
          args: [ "AddTable", "-schemaFile", "/var/pinot/examples/request_schema.json", "-tableConfigFile", "/var/pinot/examples/request_realtime_table_config.json", "-controllerHost", "pinot2-controller", "-controllerPort", "9000", "-exec" ]
          env:
            - name: JAVA_OPTS
              value: "-Xms4G -Xmx4G -Dpinot.admin.system.exit=true"
          volumeMounts:
            - name: examples
              mountPath: /var/pinot/examples
      restartPolicy: OnFailure
      volumes:
        - name: examples
          configMap:
            name: pinot-table
  backoffLimit: 100
Currently I have updated the schema using the controller portal. I wanted to know if there is a way of doing this k8s natively. Also the schema change didnt reflect in the console until I did the complete segment reload. Is this a mandatory step or there is some work around to have simple schema updates like column addition as a O(1) operation.
x

Xiang Fu

02/10/2021, 5:29 PM
you can also use controller swagger api to update schema
n

Neha Pawar

02/10/2021, 5:38 PM
@Devashish Gupta adding columns to a schema necessarily needs a reload : https://docs.pinot.apache.org/operators/tutorials/schema-evolution#update-the-schema
that’s how the changes trickle to the segments
d

Devashish Gupta

02/10/2021, 7:11 PM
Alright. On a different topic, I have created a table with
Copy code
"dateTimeFieldSpecs": [
      {
        "name": "TimeStamp",
        "dataType": "LONG",
        "format": "1:SECONDS:EPOCH",
        "granularity": "1:HOURS"
      }
    ]
According to the documentation the granularity is used for bucketing. The table contains epoch values, so where is the bucketing used?
n

Neha Pawar

02/10/2021, 7:12 PM
granularity is not used anywhere atm. It will be used in the future for segment merge/rollups
d

Devashish Gupta

02/11/2021, 2:38 PM
alright. Thanks.