Hi guys, is there a safeguard when applying ingestion transformations if the input field is the default value? I.e: Given this transformation:

{
  "columnName": "dateOfBirthMs",
  "transformFunction": "fromDateTime(dateOfBirth, 'yyyy-MM-dd''T''HH:mm:ss''Z')"
}

And schema definitions:

"dimensionFieldSpecs": [
    ,...,
    {
      "name": "dateOfBirth",
      "dataType": "STRING"
    },...,
],
"dateTimeFieldSpecs": [
    ...,
    {
      "name": "dateOfBirthMs",
      "dataType": "LONG",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
  ],

I get this exception:

java.lang.IllegalStateException: Caught exception while invoking method: public static long org.apache.pinot.common.function.scalar.DateTimeFunctions.fromDateTime(java.lang.String,java.lang.String) with arguments: [null, yyyy-MM-dd'T'HH:mm:ss'Z]

I was under the impression that Pinot would not apply the transformation if the input field is null or that the transformation itself would be resilient. Is there any way around this?

Yes transform functions should be able to handle nulls. For workaround you can convert it into a groovy function and add the null check for now.

A follow-up question.... How does Pinot decide if a field in an incoming message is null to apply the defaultNullValue? Does the key of the field have to be missing?