Question, is

dataTypeTransformer

supposed to be enabled by implicitly for all tables? Is this the expected default behavior?

yes, it tries to convert read data to corresponding Pinot type

Do we expect this behavior to continue for the foreseeable future or could this behavior be dropped for efficiency concerns?

Since this is for ingestion, we don’t have any plan yet

Got it. Thanks for the straightforward answer.

Please let me know if this is a concern from your side. We can assess where time are spent during ingestion and figure out the improvement plan

Our expected ingestion rate is around 25-30GB/s and perhaps 25%+ CPU is spent on ingestion and indexing. We run our own custom transformer which parses the input json and handles value type erasure (cast to string) or optionally handle value type polymorphism by storing different value types of in different columns (i.e. field:str, field:int, etc). So the

dataTypeTransformer

is 100% redundant. We are not at the stage of optimizing for ingestion performance nor did we do any any profiling on the performance. If it does indeed become a performance issue, perhaps a simple solution would be to make

dataTypeTransformer

an opt-out pipelined transformer should be sufficient.

ack, if you have your own type inference transformer, then this is not needed. I think you can remove it. You are already an advanced user!