Hi team, does anyone know if one real time table can ingest data from two kafka topics?

nope 🙂 Question asked many times here 😉

ahh… so if I really want to consume data from two topics, do I have to merge the two streams before ingestion like using Flink to combine the two streams first?

ingest in seperate table then join with trino 🙂 Or even using now multistage query directly in pinot

@Eric Liu currently, a single pinot cannot ingest from more than one input topic. Iiuc, you want to achieve a stream-stream join during ingestion. This can be done via stream processing frameworks prior to ingestion or as @francoisa suggested, ingest into separate tables and join at query time. for the data in your lookup stream (low volume

B

), is the data mostly static, meaning are there going to be multiple updates for the same primary key ? I am trying to understand if this is an actual stream-stream join or more like a data enrichment usecase.

Thanks Francoisa and Navina for the answers. The low volume lookup stream

B

is also

id

based event stream, not static. It can have multiple updates (at most 2 times for the same primary key, e.g. first time an event from

B

comes in w/

d

as false, and later another event comes in as

true

. The combinations of the possibility are true + false, true + true, or just true). The high volume stream A can be hundreds of GBs or TBs, and the low volume B might be ten to hundreds of GBs, but potentially to grow larger. Some questions: • How is the performance look like in general for joins at query time at this scale using either Trino or V2 query engine? • How far is the V2 query engine from production ready?

@Rong R can you help answer the above questions from Eric ? ☝️

• If both table are partition by the primary join key it should be fairly fast. If you join with not the partition key it would be much slower and would probably perform similar than trino. • We are targeting the end of July as our production "beta" ready. The copartition optimizer is still under development. But you can try it out now and see

Thanks all! I will try out the V2 query first and see.

Thank you please share your feedback so we can improve upon the use case :-)

Will do.

Good question. Our goal is to tackle queries that could potentially push down to leaf stage to filter out and exclusively send the necessary data over the wire. If we were to ship giant amount of data across the wire it is not going to be efficient anyway