Elon
02/02/2021, 10:54 PMMayank
Elon
02/02/2021, 10:55 PMMayank
Elon
02/02/2021, 10:57 PMMayank
Elon
02/02/2021, 10:59 PM"OFFLINE": {
"tableName": "enriched_orders_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"schemaName": "enriched_orders",
"segmentPushFrequency": "daily",
"segmentPushType": "APPEND",
"timeColumnName": "order_timestamp_seconds",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "365",
"replication": "3",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
},
...
"REALTIME": {
"tableName": "enriched_orders_REALTIME",
"tableType": "REALTIME",
"segmentsConfig": {
"schemaName": "enriched_orders",
"segmentPushFrequency": "daily",
"segmentPushType": "APPEND",
"timeColumnName": "order_timestamp_seconds",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "100",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
"replicasPerPartition": "3"
},
...
"dateTimeFieldSpecs": [
{
"name": "order_timestamp_seconds",
"dataType": "LONG",
"defaultNullValue": 0,
"transformFunction": "toEpochSeconds(order_timestamp_ms)",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:SECONDS"
}
order_timestamp_ms
is not included in the table.Mayank
Elon
02/02/2021, 11:01 PMMayank
Elon
02/02/2021, 11:03 PMMayank
Elon
02/02/2021, 11:03 PM--offline: min: 1601424056 max: 1612051039
--realtime: min: 1607816472 max: 1612317600
--time boundary in ms: 1611964639000
Mayank
Elon
02/02/2021, 11:04 PMNeha Pawar
"transformFunction": "toEpochSeconds(order_timestamp_ms)",
"format": "1:MILLISECONDS:EPOCH",
"granularity": "1:SECONDS"
this is incorrect ^. toEpochSeconds will convert it to millis/1000, but the format here says MILLISECONDSElon
02/02/2021, 11:17 PMNeha Pawar
Elon
02/02/2021, 11:19 PMNeha Pawar
Elon
02/02/2021, 11:20 PM1:SECONDS:EPOCH
?Neha Pawar
Elon
02/02/2021, 11:22 PMNeha Pawar
Elon
02/02/2021, 11:24 PMNeha Pawar
Elon
02/02/2021, 11:25 PMNeha Pawar
Elon
02/03/2021, 7:06 PMNeha Pawar
Elon
02/11/2021, 8:17 PM