Hi all, I want to do insert of a single record to Pinot from an application. Setting up Kafka for real time ingestion seems too complicated for very small volume insert calls. Is there some other way?

Pinot does not support row level insert api as of now. You can use the batch api

You can try out the SegmentWriter interface. There’s no document yet, but this test demonstrates how to use https://github.com/apache/pinot/pull/6718/files#diff-7f887860478edd25adc4277de42a38b84ee6e6571c89d0eb05172fa1b6073d4d But in any case, creating segments with single row isn’t the best idea

Thanks I will check it out. But yes the single row thing is what’s troubling me. I use the table for searchbox so I don’t want to batch process

Are you trying to use Pinot for near real time search?

Its a normal search bar. It will be a regex query on one of the columns. The reason to use Pinot is I have some dashboarding needs which require fast aggregations. Having a different database like Mysql for just one table (the search query one) seemed like an unnecessary layer. So I am thinking to use Pinot for the search

OK, but normally adding row-by-row (not batch) means you want near-real time (NRT) search. As in, soon after data is available you want it to be searchable. Otherwise you could just use batch to generate segments every day (as an example).

oh yes, I want it near real time. Data should available for all as soon as insert is done

If you are planning to use this in production and expect strong guarantee.. it’s better to use Kafka

I decided to go with SQL sever for this

Have you looked into Apache pulsar instead?

Have you looked into Apache pulsar instead?- No let me check

@User how would introducing Pulsar be operationally very different than setting up Kafka for this?

With Pulsar, the amount of operational steps you have to take to set up your clusters isn’t as extensive as Kafka.