<@UDRLN8MAP> for column transformation, <https://d...
# generalt
Ting Chen
04/02/2021, 12:27 AM@User for column transformation, https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#column-transformation, (1) for a hybrid table, is the ingestionConfig required in both realtime and offline tables (2) can the transformation be applied to existing data? can we reload the table to do it?
n
Neha Pawar
04/02/2021, 12:28 AM
yes ingestion config is required in both tables
Neha Pawar
04/02/2021, 12:29 AM
yes for reload. Jackie recently extended the transform configs to support derived columns
Neha Pawar
04/02/2021, 12:30 AM
this is assuming that the arguments to the transform function are already part of the segment
t
Ting Chen
04/02/2021, 12:32 AM{"tableName": "myTable",..."ingestionConfig": {"transformConfigs": [{"columnName": "hoursSinceEpoch","transformFunction": "toEpochHours(timestamp)" // inbuilt function}]}}Ting Chen
04/02/2021, 12:33 AMwe have a table and I planned to apply a similar transformation function like above to it.
Ting Chen
04/02/2021, 12:34 AMso what I need to do is to (1) add a new column (hoursSinceEpoch) to the table schema (2)add ingestionConfig to the table config and (3) reload the table?
n
Neha Pawar
04/02/2021, 1:24 AM
yes
t
Ting Chen
04/06/2021, 7:10 PMI applied the table config change and reload the segments. Only the new realtime table segments have the derived column with values. The derived columns on the existing segments and segments on the offline-subtable are refreshed either. Did I miss anything?
Ting Chen
04/06/2021, 7:11 PMI read the ReloadSegment() code and it does not seem to add values to the derived column. @Neha Pawar
n
Neha Pawar
04/06/2021, 7:36 PM
does your deployment include this commit https://github.com/apache/incubator-pinot/pull/6494
Neha Pawar
04/06/2021, 7:38 PM
reload invokes that V3DefaultColumnHandler