Hi, I'm trying to answer a relatively simple question: Does Pinot support extracting record metadata from Pulsar similar to Kafka (https://docs.pinot.apache.org/basics/data-import/pinot-stream-ingestion/import-from-apache-kafka#extract-record-headers-as-pinot-table-columns). The Design doc for the feature does mention Pulsar and Kinesis (https://docs.google.com/document/d/1kTUfBud1SBSh_703mvu6ybkbIwiKKH9eXpdcpEmhC2E/edit) but based on the docs I wasn't sure if it was actually implemented for more than just Kafka. Thank you!

@Jeff Bolle Yes. we introduced this capability for kafka. For kinesis, the PR is pending merge https://github.com/apache/pinot/pull/9713 (for some reason, I keep forgetting about wrapping this up). For pulsar, I haven't invested time in it.

I can also take it on. I think you already did the hard work

amazing! would love to have your contribution 🙏 Happy to review and shepherd your changes when they are ready. feel free to ping me!

That would be awesome @Jeff Bolle

I'll start work on it tomorrow hopefully...should I use the Kinesis PR as a model?

should I use the Kinesis PR as a model?

yes you can

I have a question on the key value stitch in the Pulsar code....do you know why this was done? Wish a PulsarStreamMessage that keeps the key and value separate, I feel like we can replace all of that, but I'm not sure what purpose it served.

@Jeff Bolle that's an apt question 🙂 the stitching approach was done since at that time we didn't have a way to parse the headers . we should be replacing it with the right approach for pulsar too. unfortunately, now it has to be done in a backward compatible manner 😛

I'll see what I can do. I assume there are some existing tests in place that will let me know if I did it wrong 😉

@Jeff Bolle please use

PulsarPartitionLevelConsumer

. stream level consumer would be the high level consumer, which we determined last time doesn't work.

Got it. Should we mark as deprecated or include a warning?

I will start a one-pager on deprecating the high level consumer model and share it with pinot dev in oss. I don't know how much push back will be there. Imo, high level consumption model should no longer be used and its time to sunset. @Mayank wdyt?

We can create a GH issue and discussing there? (I might be very wrong, but I think there’s some team using HLC).

Ok. We can perhaps limit it to Kafka and deprecate the remaining. I don't think we even test those implementations.

I would say, from the position of a relatively new Pinot user, at least putting a note in the documentation would be an important step.

@Jeff Bolle yeah I agree. for pulsar, can you highlight the things that don't work?

I don't remember off the top of my head. I'll have to review. I believe the last issue was protobuf deserialization.

yes. protobuf deserialization and metadata extraction are 2 sticking points.

yeah, that was the big issue for me, once we knew to use the lowlevel consumer, our protobuf serialization from pulsar worked, and we didn't look back.

I will take a look at the PR today @Jeff Bolle. Meanwhile, looks like we might have a path forward to sunset the stream level apis. See https://github.com/apache/pinot/issues/10996

Wow. Amazing job.

@Jeff Bolle Looks like your PR was merged last week. Can you please contribute to the docs as well? I think you can edit in gitbook here . TIA !

I was just going to ask about docs. I definitely think we need some explanations to help people understand what to configure!

no worries. please tag me and if you don't hear back from me, you can DM. Sometimes I miss the notifications.