Apache Pinot

Hi All, Very impressed with what Apache Pinot can do, and I am considering Pinot for a critical use case, and we are not Java experts in our team - Would Java be a key skill to adopt Pinot successfully? An additional question, will join between two realtime tables work? Information online seem to suggest, that join between two realtime tables are not recommended for Production, just checking if anyone here has experiences around this - thanks.

Java knowledge not necessary unless you want to contribute (also we are in the world where cursor and copilot rule, apparently :wink:).

What kind of joins are you thinking for realtime table? It is always a function of things like the join you want to do (hence the data shuffle), your SLA and the cost you are willing to pay.

thanks a lot Mayank, See below query for example where all three tables are realtime tables streaming from Kafka

`select portfolio, sum(move + impact_price) as pnl_approx`
`from pnl pnl`
`join (`
    `select id,`
      `version_id,`
      `subversion_id,`
      `leg_id,`
      `leg_profile_id,`
      `sum((p.value - pe.price) *native_unit) impact_price`
    `from pe_deal pe`
      `join price p on p.price_idx = pe.price_idx`
      `and p.price_point_id = pe.price_point_id`
    `group by id,`
      `version_id,`
      `subversion_id,`
      `leg_id,`
      `leg_profile_id`
  `) exp`
 `on exp.id = pnl.id`
 `and exp.version_id = pnl.version_id`
 `and exp.subversion_id = pnl.subversion_id`
 `and exp.leg_id = pnl.leg_id`
 `and exp.leg_profile_id = pnl.leg_profile_id`
 `group by pnl.portfolio`

1. to add additional context, I am expecting 3 Million records in pe_deal &amp; pnl table, and a few thousand records in price. 

• If the queries return result in &lt; 3 secs, that is good enough (as long as the results are accurate)

• cost is not a major concern, but accuracy and latency is

your data size doesn't seem large, so guessing that realtime ingestion rate would be small too

with the right optimizations (eg colocation, partition on join key), it should be fine

makes sense, ingestion rate shouldn't be huge, may be 1000 records/sec, but a bit infrequently.

<https://docs.pinot.apache.org/users/user-guide-query/multi-stage-query/optimizing-joins>