Neha Pawar
Ranveer Singh
05/06/2020, 10:07 AMNeha Pawar
Ranveer Singh
05/06/2020, 4:17 PMNeha Pawar
Ranveer Singh
05/06/2020, 4:21 PMNeha Pawar
Ranveer Singh
05/06/2020, 4:25 PMNeha Pawar
Ranveer Singh
05/06/2020, 4:28 PMNeha Pawar
Ranveer Singh
05/06/2020, 4:32 PMNeha Pawar
Ranveer Singh
05/06/2020, 4:47 PMNeha Pawar
Ranveer Singh
05/06/2020, 5:24 PMNeha Pawar
Ranveer Singh
05/06/2020, 5:26 PMNeha Pawar
Ranveer Singh
05/06/2020, 5:32 PMNeha Pawar
Ranveer Singh
05/06/2020, 6:06 PMNeha Pawar
%a %b %d %H:%M:%S %Z %Y
?Ranveer Singh
05/06/2020, 6:20 PMNeha Pawar
The pattern of timestamp format. For strings use python datetime string pattern expression which needs to adhere to the ISO 8601 standard to ensure that the lexicographical ordering coincides with the chronological ordering. If the timestamp format does not adhere to the ISO 8601 standard you will need to define an expression and type for transforming the string into a date or timestamp. Note currently time zones are not supported. If time is stored in epoch format, put epoch_s or epoch_ms.
"needs to adhere to ISO 8601 standard".
The format you have is not ISO standard: https://en.wikipedia.org/wiki/ISO_8601{
"schemaName": "rfp",
"dimensionFieldSpecs": [
{
"name": "status",
"dataType": "STRING"
},
{
"name": "fulfilmentType",
"dataType": "STRING"
},
{
"name": "soOrderHeaderKey",
"dataType": "STRING"
},
{
"name": "SONumber",
"dataType": "STRING"
},
{
"name": "CommsResponse",
"dataType": "INT"
},
{
"name": "extnOriginalNo",
"dataType": "INT"
},
{
"name": "messageId",
"dataType": "STRING"
},
{
"name": "orderLineKey",
"dataType": "STRING"
},
{
"name": "fulfilmentSubType",
"dataType": "STRING"
},
{
"name": "storeId",
"dataType": "STRING"
},
{
"name": "soOrderLineKey",
"dataType": "STRING"
},
{
"name": "primeLineNumber",
"dataType": "STRING"
},
{
"name": "PONumber",
"dataType": "STRING"
},
{
"name": "itemId",
"dataType": "STRING"
},
{
"name": "orderHeaderKey",
"dataType": "STRING"
},
{
"name": "releaseStatusKey",
"dataType": "STRING"
},
{
"name": "RFP",
"dataType": "STRING"
},
{
"name": "EmailAck",
"dataType": "STRING"
},
{
"name": "StatusRecModifyMillis",
"dataType": "LONG",
"transformFunction": "Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', StatusRecModifyTS).getTime()}, StatusRecModifyTS)"
},
{
"name": "StatusRecCreateMillis",
"dataType": "LONG",
"transformFunction": "Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', StatusRecCreateTS).getTime()}, StatusRecCreateTS)"
},
{
"name": "EmailSendCreateMillis",
"dataType": "LONG",
"transformFunction": "Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', EmailSendCreate).getTime()}, EmailSendCreate)"
}
],
"metricFieldSpecs": [
{
"name": "TimeTaken",
"dataType": "LONG"
}
]
}
Ranveer Singh
05/07/2020, 12:30 PM@Test
public void testDateTransformFromString() {
Schema pinotSchema = new Schema();
DimensionFieldSpec dimensionFieldSpec = new DimensionFieldSpec("StatusRecCreateTSInMillis", FieldSpec.DataType.LONG, true);
dimensionFieldSpec.setTransformFunction("Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', StatusRecModifyTS).getTime()}, StatusRecModifyTS)");
pinotSchema.addField(dimensionFieldSpec);
ExpressionTransformer expressionTransformer = new ExpressionTransformer(pinotSchema);
GenericRow genericRow = new GenericRow();
genericRow.putValue("StatusRecModifyTS", "Sun Apr 26 01:38:01 UTC 2020");
// genericRow.putValue("StatusRecCreateTSInMillis", "1587865081000");
// no transformation
expressionTransformer.transform(genericRow);
Assert.assertEquals(genericRow.getValue("StatusRecCreateTSInMillis"), 1587865081000L);
/* pinotSchema = new Schema();
TimeFieldSpec timeFieldSpec = new TimeFieldSpec(new TimeGranularitySpec(FieldSpec.DataType.LONG, TimeUnit.MILLISECONDS, "incoming"), new TimeGranularitySpec(
<http://FieldSpec.DataType.INT|FieldSpec.DataType.INT>, TimeUnit.DAYS, "outgoing"));
pinotSchema.addField(timeFieldSpec);
expressionTransformer = new ExpressionTransformer(pinotSchema);
genericRow = new GenericRow();
genericRow.putValue("incoming", "123456789");
genericRow.putValue("outgoing", "123");
// no transformation
expressionTransformer.transform(genericRow);
Assert.assertEquals(genericRow.getValue("outgoing"), "123");*/
}
Neha Pawar
Ranveer Singh
05/07/2020, 2:52 PMNeha Pawar
Ranveer Singh
05/07/2020, 4:05 PMNeha Pawar
Ranveer Singh
05/07/2020, 4:47 PMNeha Pawar
Ranveer Singh
05/11/2020, 10:50 AM