Hi team, can Pinot ingest one partition of one Kafka topic which has many partition?

If you mean you want to ignore all other partitions, you might be able to achieve it by having filter functions in ingestion. However, this is not a setup that I would recommend. What’s the use case?

yes, like I can specify a partition number and Pinot can ingest just that partition and save the data to one table.

you can also filter on

$segmentName

which has some convention to put the partition number in the segment name, but i doubt it’s fast enough.

Right, but what’s the business case where you want to do this? Note, Pinot scales ingestion by having parallel consumers (one per partition). If you just have one partition to consume from you will lose out on that.

Multi Business data is merged and written to one Kafka topic and We depend on Pinot to separate data into different tables. Hence this topic is consumed repeatedly and it’s resource consuming.

What is the total data size across all partitions (per day)?

That’s a key point. Maybe 0.15 billion per day.

0.15 B rows?

yes

You can just have a single table partitioned by org. It will be much simpler.

I’m afraid one table is not ok. Different business data has different fields.

You can create multiple tables and have filter function in ingestion that each table only ingests one transform, that is definitely possible. But like I mentioned, it is not the most ideal/cost-effective solution. Is there an option to repartition upstream?

I see. You can create multiple tables and have filter function in ingestion that each table only ingests one transform, that is definitely possible. Like you mentioned here, that’s what we want to do and considering the cost I had the question here.

You could have a stream processing upstream that takes your kafka stream, and splits it into multiple topics (one per org) with partitions within each partition

Thanks, Mayank. I like your idea and it makes things much easier.

Btw, we did some changes when we added the support for kinesis.. those changes will actually make it easier to add support to consume a subset of partitions..

Sure, I’d like to give it a try.