Is there an easy way to transform the incoming rec...
# generald
Dan Hill
03/08/2020, 5:29 PMIs there an easy way to transform the incoming records by specifying a function in the configs? Just checking.
k
Kishore G
03/08/2020, 5:57 PM
What kind of function are you looking to apply
Kishore G
03/08/2020, 5:57 PM
schema supports a field called transformFunction
d
Dan Hill
03/08/2020, 5:59 PMI currently have a logging json that I'm passing into Kinesis. It currently logs and I was going to have Pinot ingest the same thing from Kinesis. Small transformations like take an enum value and convert it to separate counts. E.g. EventType=CLICK => impressions = 0, clicks= 1.
k
Kishore G
03/08/2020, 6:01 PM
got it.
Kishore G
03/08/2020, 6:02 PM
will eventType be part of the data in Pinot?
d
Dan Hill
03/08/2020, 6:02 PMI was planning on splitting it to become separate metrics.
Dan Hill
03/08/2020, 6:02 PMThis is mostly for quick development. Longer-term, I'll probably have separate definitions for logging and pinot.
k
Kishore G
03/08/2020, 6:04 PM
we feature is not yet ready
Kishore G
03/08/2020, 6:06 PM
by the way, you can write your own custom record reader achieve this for now.
d
Dan Hill
03/08/2020, 6:06 PMTrue
Dan Hill
03/08/2020, 6:07 PMI figured doing that would make it a little more complex to deploy and probably have a similar complexity to other solutions (e.g. a lambda in Kinesis to transform the data).
k
Kishore G
03/08/2020, 6:07 PM
yes, its not generic.
Kishore G
03/08/2020, 6:08 PM
you dont want to write one custom reader for every table.