Channels
#general
Title
# general
t
Ting Chen
04/02/2021, 12:27 AM@User for column transformation, https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#column-transformation, (1) for a hybrid table, is the ingestionConfig required in both realtime and offline tables (2) can the transformation be applied to existing data? can we reload the table to do it?
n
Neha Pawar
04/02/2021, 12:28 AM
yes ingestion config is required in both tables
yes for reload. Jackie recently extended the transform configs to support derived columns
this is assuming that the arguments to the transform function are already part of the segment
t
Ting Chen
04/02/2021, 12:32 AM{"tableName": "myTable",..."ingestionConfig": {"transformConfigs": [{"columnName": "hoursSinceEpoch","transformFunction": "toEpochHours(timestamp)" // inbuilt function}]}}we have a table and I planned to apply a similar transformation function like above to it.
so what I need to do is to (1) add a new column (hoursSinceEpoch) to the table schema (2)add ingestionConfig to the table config and (3) reload the table?
n
Neha Pawar
04/02/2021, 1:24 AM
yes
t
Ting Chen
04/06/2021, 7:10 PMI applied the table config change and reload the segments. Only the new realtime table segments have the derived column with values. The derived columns on the existing segments and segments on the offline-subtable are refreshed either. Did I miss anything?
I read the ReloadSegment() code and it does not seem to add values to the derived column. @Neha Pawar
n
Neha Pawar
04/06/2021, 7:36 PM
does your deployment include this commit https://github.com/apache/incubator-pinot/pull/6494
reload invokes that V3DefaultColumnHandler