When bootstraping the table

Actually the bootstrap could also be done via kafka

The reason we have to do segment replacement is because Pinot is append only, but that is not the case with upsert

So actually it might make sense to have everything through kafka for upsert table

There is usually 10-20x throughout difference between direct push vs going through kafka

In our past observation, backfill via kafka may take days whereas going with direct push may just take a couple of hours

The initial bootstrap can be done via direct push, then the updates come through kafka

The problem here is that we should not change the history for upsert table. Also it is better to pay the extra cost at write time instead of query time

Also, if we can make the assumption that a primary key won't get any update after a period of time (say 3 days for an order), we might be able to flush it to an offline table

Bootstrap makes sense

If we cannot easily change history, then we shall consider a feature to replace an existing table

Think from user perspective how the data correction flow shall be like

To correct the record for a primary key, simply put the desired record with the current (latest) timestamp

Hmm, for different scenario, user might want to put different timestamp for the update message

But the update logic should be general enough to handle all these custom logic

hello @User i used the lastest master branch and run quick-start-streaming.sh demo, and observed the following error. is it possible there is a potential bug in the RealtimeQuickStart producer?

➜  apache-pinot-incubating-0.8.0-SNAPSHOT-bin git:(master) bin/quick-start-streaming.sh
***** Starting Kafka *****
***** Starting meetup data stream and publishing to Kafka *****
***** Starting Zookeeper, controller, server and broker *****
May 21, 2021 7:06:37 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:9000]
May 21, 2021 7:06:37 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
May 21, 2021 7:06:46 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:8000]
May 21, 2021 7:06:46 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer-1] Started.
May 21, 2021 7:06:53 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:7500]
May 21, 2021 7:06:53 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer-2] Started.
***** Bootstrap meetupRSVP table *****
***** Waiting for 5 seconds for a few events to get populated *****
***** Realtime quickstart setup complete *****
Total number of documents in the table
Query : select count(*) from meetupRsvp limit 1
Exception caught: 
java.lang.NullPointerException: null
	at org.apache.pinot.tools.Quickstart.prettyPrintResponse(Quickstart.java:113) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc]
	at org.apache.pinot.tools.RealtimeQuickStart.execute(RealtimeQuickStart.java:111) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc]
	at org.apache.pinot.tools.admin.command.QuickStartCommand.execute(QuickStartCommand.java:147) ~[pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc]
	at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:166) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc]
	at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:186) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc]
	at org.apache.pinot.tools.RealtimeQuickStart.main(RealtimeQuickStart.java:50) [pinot-all-0.8.0-SNAPSHOT-jar-with-dependencies.jar:0.8.0-SNAPSHOT-08b909c45e85f9bf8d8659561a2d13b4cc443ebc]

http://localhost:9000/#/query loaded successfully, but no data shown up in the table meetupRsvp

Is this from a fresh master checkout?

@User Can you try rebuild the project and see if the problem still exist?

It should not have problem because it is part of the CI test on github

@User @User let me try reproduce this issue and see if still exist

@User @User @User @User @User fyi, I added some context on the need of segment reader for partial upsert table https://github.com/apache/incubator-pinot/issues/7036

So basically leveraging partial upsert to merge the incomplete records?

By directly reading the pinot segment, all the records will be returned, instead of the ones with the latest timestamp. @User Do you think it suits your use case?

can the segment reader support upsert?

No, the upsert is not handled in the reader level

In order to use Pinot features, or push down filters, the table dump should be modeled as a query IMO

We can potentially use the gRPC streaming server to reduce the memory footprint

true, you need to read from server to use the upsert metadata

i wonder if it’s easy to write a Hive ETL to do the compaction: read the latest record of the pk