https://pinot.apache.org/ logo
#general
Title
# general
m

Map

10/20/2021, 4:19 PM
Hi, if Pinot expects a field to be numeric but receives a string value, how does Pinot handle it?
k

Kishore G

10/20/2021, 5:12 PM
it will try to parse it and if it fails, it will use the default value for that data type. Default value can be overridden in the schema
m

Map

10/20/2021, 5:16 PM
Thanks @User would you mind pointing me to the code?
k

Kishore G

10/20/2021, 5:22 PM
see compositetransformer
m

Map

10/20/2021, 5:25 PM
Previously we had a case where an incorrect value type for the datetime field halted the whole ingestion. Do you have special handling for this field?
k

Kishore G

10/20/2021, 5:27 PM
I think so. yes, because retention depends on the primary time column and we avoid setting default value for that and fail fast
m

Map

10/20/2021, 9:51 PM
@User Took a look at the code and it seems it’s DataTypeTransformer that does the job which in turns relies on PinotDataType. According to PinotDataType, exceptions can be thrown if a conversion is not possible. Would you mind pointing me to the place that handles this exception and uses the default value, instead of stopping the whole ingestion?
a

Arpit

10/21/2021, 9:04 AM
I am also having the same case where a wrong datetimefield is stopping the ingestion
I provided a default value for that field but its not used I think
Does it mean primary time column should always receive a valid value otherwise it stops ingestion?
m

Map

10/23/2021, 3:02 PM
@User when you have a chance, can you take a look at questions ^? Thanks
k

Kishore G

10/23/2021, 3:51 PM
The behavior is as expected right now. We are thinking of ways to improve that and probably fail fast
m

Map

10/23/2021, 3:53 PM
@User Took a look at the code and it seems it’s DataTypeTransformer that does the job which in turns relies on PinotDataType. According to PinotDataType, exceptions can be thrown if a conversion is not possible. Would you mind pointing me to the place that handles this exception and uses the default value, instead of stopping the whole ingestion?
k

Kishore G

10/23/2021, 4:08 PM
LLRealtimeDataManager
Copy code
LLRealtimeSegmentDataManager
👍 1