Hi, if Pinot expects a field to be numeric but receives a string value, how does Pinot handle it?

it will try to parse it and if it fails, it will use the default value for that data type. Default value can be overridden in the schema

Thanks @User would you mind pointing me to the code?

see compositetransformer

Previously we had a case where an incorrect value type for the datetime field halted the whole ingestion. Do you have special handling for this field?

I think so. yes, because retention depends on the primary time column and we avoid setting default value for that and fail fast

@User Took a look at the code and it seems it’s DataTypeTransformer that does the job which in turns relies on PinotDataType. According to PinotDataType, exceptions can be thrown if a conversion is not possible. Would you mind pointing me to the place that handles this exception and uses the default value, instead of stopping the whole ingestion?

I am also having the same case where a wrong datetimefield is stopping the ingestion

@User when you have a chance, can you take a look at questions ^? Thanks

The behavior is as expected right now. We are thinking of ways to improve that and probably fail fast

@User Took a look at the code and it seems it’s DataTypeTransformer that does the job which in turns relies on PinotDataType. According to PinotDataType, exceptions can be thrown if a conversion is not possible. Would you mind pointing me to the place that handles this exception and uses the default value, instead of stopping the whole ingestion?

LLRealtimeDataManager