Hi Team We have an Upset Table in Pinot 1.2.0/1.3.0 version, which has TIMESTAMP dType as timeColumn , But after Pinot upgrade to 1.4.0 , timeColumn does not support TIMESTAMP datatype if TTL is enabled , Which is restricting us to do schema evolutions . Is there any known workaround for this , other than re-creating the table..? @Mayank, @Xiang Fu

cc: @Jackie

1. Changing the Dtype on exiting schema column does not work , since pinot schema is not backward compatible One other way we tried is • Update TTL to 0 in Table config • Update schema by adding new timeColumn. Lets say timeColumn is used to be TC1 (TIMESTAMP) , create new timeColumn TC2 with LONG type , which is mapped to TC1 using transformation Function • Update timeColumn to TC2 in table config • Update TTL back previous state But this is unnecessarily adding an extra column in our Pinot table. and also not sure if this approach is safe

Hi Badhusha, Yes, this is a known 1.4.0 validation issue. Pinot 1.4.0 validates upsert TTL columns using the logical type and rejects

TIMESTAMP

as non-numeric, even though Pinot stores

TIMESTAMP

internally as

LONG

epoch millis. This was fixed later in PR #17200: https://github.com/apache/pinot/pull/17200, included from 1.5.0 onward. Cleanest fix: upgrade/backport that PR. Then you can keep

TC1

as

TIMESTAMP

and avoid schema changes. If you must stay on stock 1.4.0, your workaround is generally valid, but I’d avoid changing

segmentsConfig.timeColumnName

. Instead: - Keep

segmentsConfig.timeColumnName = TC1

- Add a new

LONG

column, e.g.

TC1_epoch_ms

- Add transform config:

TC1_epoch_ms = TC1

- Reload/backfill all existing completed segments so the new column is materialized - Handle consuming segments/force-commit as appropriate - Verify

TC1_epoch_ms

min/max matches

TC1

epoch millis - Set

upsertConfig.comparisonColumn = TC1_epoch_ms

- Re-enable

metadataTTL

/

deletedKeysTTL

Important: do not re-enable TTL before the backfill/reload is complete. If older segments do not have the derived LONG value materialized, Pinot can see default/missing values and TTL may clean the wrong upsert metadata. So short answer: no table recreation should be required. Best path is backport the fix; otherwise use a derived LONG comparison column and fully backfill it before enabling TTL again.

If you must stay on stock 1.4.0, your workaround is generally valid, but I’d avoid changing

segmentsConfig.timeColumnName

. Instead

set

upsertConfig.comparisonColumn = TC1_epoch_ms

..? This is changing right ..?

I think instead of changing timeColumn, you can directly change the comparisonColumn

Got it , thank you