https://pinot.apache.org/ logo
#general
Title
# general
r

Ranveer Singh

05/05/2020, 10:38 AM
While validating using api, dateTimeFieldSpec section is getting ignored. Can someone guide me on this
k

Kishore G

05/05/2020, 3:45 PM
@Neha Pawar ^^
n

Neha Pawar

05/05/2020, 4:04 PM
what are you trying to do? Upload the schema?
or create a segment using this schema?
as of now atleast, Pinot doesn't automatically create these values. You have to make these data transformations before creating the segment - in some kind of preprocessing job
after this gets merged, you will be able to define transform functions for the columns, in your schema" https://github.com/apache/incubator-pinot/pull/5326
r

Ranveer Singh

05/05/2020, 4:28 PM
I am trying to do batch upload. I am first creating the schema
When i am ingesting the data using csv these columns are not considered . In short i dont see these column as part of schema
n

Neha Pawar

05/05/2020, 4:31 PM
these columns
StatusRecModifyTS
StatusRecCreateTS
EmailSendCreate
exist in your data?
k

Kishore G

05/05/2020, 4:32 PM
lets move to #troubleshooting
r

Ranveer Singh

05/05/2020, 4:32 PM
Yes
its part of CSV column
{ "schemaName": "rfp", "dimensionFieldSpecs": [ { "name": "status", "dataType": "STRING", "singleValueField": true }, { "name": "fulfilmentType", "dataType": "STRING", "singleValueField": true }, { "name": "soOrderHeaderKey", "dataType": "STRING", "singleValueField": true }, { "name": "SONumber", "dataType": "STRING", "singleValueField": true }, { "name": "CommsResponse", "dataType": "INT", "singleValueField": true }, { "name": "extnOriginalNo", "dataType": "INT", "singleValueField": true }, { "name": "messageId", "dataType": "STRING", "singleValueField": true }, { "name": "orderLineKey", "dataType": "STRING", "singleValueField": true }, { "name": "fulfilmentSubType", "dataType": "STRING", "singleValueField": true }, { "name": "storeId", "dataType": "STRING", "singleValueField": true }, { "name": "soOrderLineKey", "dataType": "STRING", "singleValueField": true }, { "name": "primeLineNumber", "dataType": "STRING", "singleValueField": true }, { "name": "PONumber", "dataType": "STRING", "singleValueField": true }, { "name": "itemId", "dataType": "STRING", "singleValueField": true }, { "name": "orderHeaderKey", "dataType": "STRING", "singleValueField": true }, { "name": "releaseStatusKey", "dataType": "STRING", "singleValueField": true }, { "name": "RFP", "dataType": "STRING", "singleValueField": true }, { "name": "EmailAck", "dataType": "STRING", "singleValueField": true } ], "metricFieldSpecs": [ { "name": "TimeTaken", "dataType": "LONG", "defaultNullValue": 0 } ], "dateTimeFieldSpec":[ { "name": "StatusRecModifyTS", "dataType": "LONG", "format" : "5MINUTESSIMPLE_DATE_FORMAT:EEE MON dd HHmmss UTC yyyy", "granularity" : "5:MINUTES" }, { "name": "StatusRecCreateTS", "dataType": "LONG", "format" : "5MINUTESSIMPLE_DATE_FORMAT:EEE MON dd HHmmss UTC yyyy", "granularity" : "5:MINUTES" }, { "name": "EmailSendCreate", "dataType": "LONG", "format" : "5MINUTESSIMPLE_DATE_FORMAT:EEE MON dd HHmmss UTC yyyy", "granularity" : "5:MINUTES" } ] }
If i use validate Schema curl command.. As part of response, i don't see these columns