While validating using api, dateTimeFieldSpec sect...
# generalr
Ranveer Singh
05/05/2020, 10:38 AMWhile validating using api, dateTimeFieldSpec section is getting ignored. Can someone guide me on this
k
Kishore G
05/05/2020, 3:45 PM
@Neha Pawar ^^
n
Neha Pawar
05/05/2020, 4:04 PM
what are you trying to do? Upload the schema?
Neha Pawar
05/05/2020, 4:04 PM
or create a segment using this schema?
Neha Pawar
05/05/2020, 4:05 PM
as of now atleast, Pinot doesn't automatically create these values. You have to make these data transformations before creating the segment - in some kind of preprocessing job
Neha Pawar
05/05/2020, 4:07 PM
after this gets merged, you will be able to define transform functions for the columns, in your schema" https://github.com/apache/incubator-pinot/pull/5326
r
Ranveer Singh
05/05/2020, 4:28 PMI am trying to do batch upload. I am first creating the schema
Ranveer Singh
05/05/2020, 4:28 PMWhen i am ingesting the data using csv these columns are not considered . In short i dont see these column as part of schema
n
Neha Pawar
05/05/2020, 4:31 PM
these columns
StatusRecModifyTSStatusRecCreateTSEmailSendCreatek
Kishore G
05/05/2020, 4:32 PM
lets move to #troubleshooting
r
Ranveer Singh
05/05/2020, 4:32 PMYes
Ranveer Singh
05/05/2020, 4:32 PMits part of CSV column
Ranveer Singh
05/05/2020, 4:32 PM{
"schemaName": "rfp",
"dimensionFieldSpecs": [
{
"name": "status",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "fulfilmentType",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "soOrderHeaderKey",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "SONumber",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "CommsResponse",
"dataType": "INT",
"singleValueField": true
},
{
"name": "extnOriginalNo",
"dataType": "INT",
"singleValueField": true
},
{
"name": "messageId",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "orderLineKey",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "fulfilmentSubType",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "storeId",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "soOrderLineKey",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "primeLineNumber",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "PONumber",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "itemId",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "orderHeaderKey",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "releaseStatusKey",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "RFP",
"dataType": "STRING",
"singleValueField": true
},
{
"name": "EmailAck",
"dataType": "STRING",
"singleValueField": true
}
],
"metricFieldSpecs": [
{
"name": "TimeTaken",
"dataType": "LONG",
"defaultNullValue": 0
}
],
"dateTimeFieldSpec":[
{
"name": "StatusRecModifyTS",
"dataType": "LONG",
"format" : "5MINUTESSIMPLE_DATE_FORMAT:EEE MON dd HHmmss UTC yyyy",
"granularity" : "5:MINUTES"
},
{
"name": "StatusRecCreateTS",
"dataType": "LONG",
"format" : "5MINUTESSIMPLE_DATE_FORMAT:EEE MON dd HHmmss UTC yyyy",
"granularity" : "5:MINUTES"
},
{
"name": "EmailSendCreate",
"dataType": "LONG",
"format" : "5MINUTESSIMPLE_DATE_FORMAT:EEE MON dd HHmmss UTC yyyy",
"granularity" : "5:MINUTES"
}
]
}
Ranveer Singh
05/05/2020, 4:33 PMIf i use validate Schema curl command.. As part of response, i don't see these columns