Hello 👏, we are having an issue with derived columns Concretely we're trying to make derived columns -> daysTs and hoursTs, but on consuming segments on realtime table derived columns have min Long value -> -2^63 In other segments derived columns in realtime table have OK value. Also we have setup RealtimeToOffline task and when segments are transferred in offline, daysTs and hoursTs have same min Long value as on consuming segments. Ingestion config part for real time table

"ingestionConfig": {

"transformConfigs": [

{

"columnName": "id",

"transformFunction": "JSONPATHSTRING(transaction, '$.id', 'null')"

},

.

.

.,

{

"columnName": "ts",

"transformFunction": "JSONPATHLONG(transaction, '$.ts.date', 0)"

}{

"columnName": "daysTs",

"transformFunction": "toEpochDays(ts)"

},

{

"columnName": "hoursTs",

"transformFunction": "toEpochHours(ts)"

}

Derived columns are in schema (bottom two):

"dateTimeFieldSpecs": [

{

"name": "ts",

"dataType": "LONG",

"format": "1:MILLISECONDS:EPOCH",

"granularity": "1:MILLISECONDS"

},

{

"name": "cat",

"dataType": "LONG",

"format": "1:MILLISECONDS:EPOCH",

"granularity": "1:MILLISECONDS"

},

{

"name": "agn_o",

"dataType": "LONG",

"format": "1:MILLISECONDS:EPOCH",

"granularity": "1:MILLISECONDS"

},

{

"name": "_sourceTimestamp",

"dataType": "LONG",

"format": "1:MILLISECONDS:EPOCH",

"granularity": "1:MILLISECONDS"

},

{

"name": "daysTs",

"dataType": "LONG",

"format": "1:DAYS:EPOCH",

"granularity": "1:DAYS"

},

{

"name": "hoursTs",

"dataType": "LONG",

"format": "1:HOURS:EPOCH",

"granularity": "1:HOURS"

}

hoursTs is aggreate dimension in startree index as follows:

"starTreeIndexConfigs": [

{

"dimensionsSplitOrder": [

"hoursTs",

"c",

"ty",

"cu",

"dp",

"ag"

],

"skipStarNodeCreationForDimensions": [],

"functionColumnPairs": [

"SUM__a",

"COUNT__id",

"SUM__ra",

"SUM__rp",

"COUNT__*"

],

"maxLeafRecords": 10000

}

]

daysTs is blomm filter index as follows:

"bloomFilterColumns": [

"id",

"daysTs"

]

Are we missing something?

is the

ts

column working as expected?

Yes, infact I've put another pair of daysTs_n and hoursTs_n for testing, runed reload segments and min long value appeared on just consuming segments, it was ok on places where hoursTs and daysTs had min long value

can you say the last bit in another way - what's the scenario where it works?

It works on all data points on realtime segments that are committed and on new ones that are going to be created. It doesn't work on consuming segments and when offloading to offline table via RealtimeToOfflineSegmentsTask

when offloading to offline table via RealtimeToOfflineSegmentsTask

as in you start with there being correct values in the real-time table and then the values become incorrect once they're in the offline table?

Yes

I thought that task was taking the data from segments in real time tables and moving them to offline tables, I didn't think that it applied any transforms during that process. But I could be wrong, I think we need @Mayank for this one

Update -> we created a new table, set everything before consuming and all the things are working fine.

Sorry for missing the thread earlier @Vedran Krtalić, @Mark Needham. Do we know why it didn’t work earlier? Was the table config changed after consumption started? Something we should document may be @Mark Needham ?

Hello 👋, we didn't changed config, just created new table, configured everything upfront of the consumption and derived values are now fine.