Hi, i'm streaming from Kafka and would like to hav...
# generalk
kelv
05/20/2021, 11:32 PMHi, i'm streaming from Kafka and would like to have a way to uniquely identify messages. In Kafka consumer, I can do that with offset. Is it possible to expose Kafka metadata such as offset/timestamp to Pinot clients?
m
Mayank
05/20/2021, 11:36 PM
I suppose you could write a transform function that reads that metadata and populates columns in Pinot schema.
x
Xiang Fu
05/21/2021, 12:07 AM
actually I found it’s a good ask which can be part as a hidden column for msg consumed from kafka
m
Map
05/21/2021, 1:27 AM@User is there an example for reading that metadata? Not sure what the available fields are.
k
Kishore G
05/21/2021, 1:37 AM
This is an amazing idea and easy to add as part of Kafka decoder
Kishore G
05/21/2021, 1:40 AM
@User this is not available as part of the data which means transform function cannot be do this. What we need is Kafka decoder to read this metadata and add it to generic row.. we already have access to this and use it for checkpointing.. should be easy to add this.. very good beginner task
m
Map
05/21/2021, 2:14 AM@User is $docId guranteed to be unique in a table?
Map
05/21/2021, 2:15 AMHmm doesn’t look like it
Map
05/21/2021, 2:15 AMIt’s only unique within a segment
Map
05/21/2021, 2:16 AMI am new to this community and the code base. If you can give me some pointers, I may help add this to the decoder
m
Mayank
05/21/2021, 3:18 AM
Thanks @User, I suspected as much, but wasn't sure.
Mayank
05/21/2021, 3:19 AM
@User Thanks for the offer, yes it would be great if you can help. We can provide you pointers.
n
Neha Pawar
05/21/2021, 3:20 AM
@User @User kishore suggested I point you to the code where you can make this change. Check out https://github.com/apache/incubator-pinot/blob/master/pinot-core/src/main/java/org[…]ot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java.
m
Mayank
05/21/2021, 3:20 AM
Thakns @User
n
Neha Pawar
05/21/2021, 3:20 AM
if you look at the fetchMessages method, you’ll see we have access to the message, and the offset.
Neha Pawar
05/21/2021, 3:25 AM
offsets are already being pulled up all the way to the decoder. You can fetch whatever else you need.
Neha Pawar
05/21/2021, 3:26 AM
And then here’s where you would put everything into the Pinot row: https://github.com/apache/incubator-pinot/blob/16f94a19a82807244617e7436c7ba57a1ed[…]ot/core/data/manager/realtime/LLRealtimeSegmentDataManager.java
m
Map
05/21/2021, 1:43 PM@User let me take a look