Neha Pawar
Ranveer Singh
05/06/2020, 10:07 AMRanveer Singh
05/06/2020, 10:08 AMNeha Pawar
Neha Pawar
Ranveer Singh
05/06/2020, 4:17 PMRanveer Singh
05/06/2020, 4:18 PMRanveer Singh
05/06/2020, 4:19 PMRanveer Singh
05/06/2020, 4:20 PMRanveer Singh
05/06/2020, 4:20 PMNeha Pawar
Neha Pawar
Ranveer Singh
05/06/2020, 4:21 PMRanveer Singh
05/06/2020, 4:22 PMRanveer Singh
05/06/2020, 4:22 PMNeha Pawar
Neha Pawar
Neha Pawar
Ranveer Singh
05/06/2020, 4:25 PMRanveer Singh
05/06/2020, 4:25 PMRanveer Singh
05/06/2020, 4:25 PMRanveer Singh
05/06/2020, 4:26 PMNeha Pawar
Neha Pawar
Ranveer Singh
05/06/2020, 4:28 PMRanveer Singh
05/06/2020, 4:29 PMRanveer Singh
05/06/2020, 4:30 PMNeha Pawar
Ranveer Singh
05/06/2020, 4:32 PMNeha Pawar
Ranveer Singh
05/06/2020, 4:47 PMRanveer Singh
05/06/2020, 4:48 PMRanveer Singh
05/06/2020, 4:49 PMNeha Pawar
Ranveer Singh
05/06/2020, 5:24 PMNeha Pawar
Neha Pawar
Ranveer Singh
05/06/2020, 5:26 PMNeha Pawar
Ranveer Singh
05/06/2020, 5:32 PMNeha Pawar
Ranveer Singh
05/06/2020, 6:06 PMRanveer Singh
05/06/2020, 6:06 PMRanveer Singh
05/06/2020, 6:07 PMRanveer Singh
05/06/2020, 6:07 PMRanveer Singh
05/06/2020, 6:07 PMRanveer Singh
05/06/2020, 6:07 PMNeha Pawar
Neha Pawar
%a %b %d %H:%M:%S %Z %Y
?Ranveer Singh
05/06/2020, 6:20 PMRanveer Singh
05/06/2020, 6:20 PMRanveer Singh
05/06/2020, 6:20 PMRanveer Singh
05/06/2020, 6:21 PMNeha Pawar
Neha Pawar
The pattern of timestamp format. For strings use python datetime string pattern expression which needs to adhere to the ISO 8601 standard to ensure that the lexicographical ordering coincides with the chronological ordering. If the timestamp format does not adhere to the ISO 8601 standard you will need to define an expression and type for transforming the string into a date or timestamp. Note currently time zones are not supported. If time is stored in epoch format, put epoch_s or epoch_ms.
"needs to adhere to ISO 8601 standard".
The format you have is not ISO standard: https://en.wikipedia.org/wiki/ISO_8601Neha Pawar
{
"schemaName": "rfp",
"dimensionFieldSpecs": [
{
"name": "status",
"dataType": "STRING"
},
{
"name": "fulfilmentType",
"dataType": "STRING"
},
{
"name": "soOrderHeaderKey",
"dataType": "STRING"
},
{
"name": "SONumber",
"dataType": "STRING"
},
{
"name": "CommsResponse",
"dataType": "INT"
},
{
"name": "extnOriginalNo",
"dataType": "INT"
},
{
"name": "messageId",
"dataType": "STRING"
},
{
"name": "orderLineKey",
"dataType": "STRING"
},
{
"name": "fulfilmentSubType",
"dataType": "STRING"
},
{
"name": "storeId",
"dataType": "STRING"
},
{
"name": "soOrderLineKey",
"dataType": "STRING"
},
{
"name": "primeLineNumber",
"dataType": "STRING"
},
{
"name": "PONumber",
"dataType": "STRING"
},
{
"name": "itemId",
"dataType": "STRING"
},
{
"name": "orderHeaderKey",
"dataType": "STRING"
},
{
"name": "releaseStatusKey",
"dataType": "STRING"
},
{
"name": "RFP",
"dataType": "STRING"
},
{
"name": "EmailAck",
"dataType": "STRING"
},
{
"name": "StatusRecModifyMillis",
"dataType": "LONG",
"transformFunction": "Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', StatusRecModifyTS).getTime()}, StatusRecModifyTS)"
},
{
"name": "StatusRecCreateMillis",
"dataType": "LONG",
"transformFunction": "Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', StatusRecCreateTS).getTime()}, StatusRecCreateTS)"
},
{
"name": "EmailSendCreateMillis",
"dataType": "LONG",
"transformFunction": "Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', EmailSendCreate).getTime()}, EmailSendCreate)"
}
],
"metricFieldSpecs": [
{
"name": "TimeTaken",
"dataType": "LONG"
}
]
}
Neha Pawar
Ranveer Singh
05/07/2020, 12:30 PMRanveer Singh
05/07/2020, 12:31 PM@Test
public void testDateTransformFromString() {
Schema pinotSchema = new Schema();
DimensionFieldSpec dimensionFieldSpec = new DimensionFieldSpec("StatusRecCreateTSInMillis", FieldSpec.DataType.LONG, true);
dimensionFieldSpec.setTransformFunction("Groovy({new Date().parse('EEE MMM dd HH:mm:ss z yyyy', StatusRecModifyTS).getTime()}, StatusRecModifyTS)");
pinotSchema.addField(dimensionFieldSpec);
ExpressionTransformer expressionTransformer = new ExpressionTransformer(pinotSchema);
GenericRow genericRow = new GenericRow();
genericRow.putValue("StatusRecModifyTS", "Sun Apr 26 01:38:01 UTC 2020");
// genericRow.putValue("StatusRecCreateTSInMillis", "1587865081000");
// no transformation
expressionTransformer.transform(genericRow);
Assert.assertEquals(genericRow.getValue("StatusRecCreateTSInMillis"), 1587865081000L);
/* pinotSchema = new Schema();
TimeFieldSpec timeFieldSpec = new TimeFieldSpec(new TimeGranularitySpec(FieldSpec.DataType.LONG, TimeUnit.MILLISECONDS, "incoming"), new TimeGranularitySpec(
<http://FieldSpec.DataType.INT|FieldSpec.DataType.INT>, TimeUnit.DAYS, "outgoing"));
pinotSchema.addField(timeFieldSpec);
expressionTransformer = new ExpressionTransformer(pinotSchema);
genericRow = new GenericRow();
genericRow.putValue("incoming", "123456789");
genericRow.putValue("outgoing", "123");
// no transformation
expressionTransformer.transform(genericRow);
Assert.assertEquals(genericRow.getValue("outgoing"), "123");*/
}
Ranveer Singh
05/07/2020, 12:31 PMNeha Pawar
Neha Pawar
Neha Pawar
Ranveer Singh
05/07/2020, 2:52 PMRanveer Singh
05/07/2020, 2:53 PMRanveer Singh
05/07/2020, 2:53 PMNeha Pawar
Ranveer Singh
05/07/2020, 4:05 PMRanveer Singh
05/07/2020, 4:47 PMNeha Pawar
Ranveer Singh
05/07/2020, 4:47 PMRanveer Singh
05/07/2020, 4:48 PMNeha Pawar
Ranveer Singh
05/11/2020, 10:50 AM