This message was deleted.

runs “--new-store”/“--update-store <quotas>“/”--empty-push” using admin tool. This should be enough to start submitting the data.

Maybe not. Can you share mode details of how you create the store, e.g. configs or scripts? For store to ingest real-time writes, it has to have real-time kafka topic which should be created when the store is converted to hybrid and requires a push job to take it effective (my memory could be wrong). Since you can eventually get the records, I’d assume you have done something to enable it but is it after the empty push?

For 1) yeah that's a Kafka client behavior. Flush drains, I 'think' there's a timeout API? I'd need to check (on we phone atm). For 2), nearline writes apply across versions and are sent to Kafka directly. If that topic exists it presumes that there was a server somewhere with a version ready to create that topic at some time (though that topic persists through versions). Even if things are down, the server will ingest writes when it's ready. Though I realize now, what is it you mean by 'ready'?

"ready" as in: if I sent the data there it will actually appear there. The scary part of the behavior is that the producer does not fail / does not throw, the future it returned completes normally but the data does not show up in the store ("get" does not return it). If the sleep() enough to fix it I'd love to have a way to wait on something else. Or/and for the producer to fail.

Ok. It should only take about a second at most if everything is running correctly. As for guarantees things have drained @Nisarg Thakkar might have some thoughts as he's working on the online producer. If you want positive acknowledgement that a write is processed server side. there is maybe a few options. One would be to use the change capture client (which is relatively new).

I want to be sure that data is not lost. E.g. in this scenario it is not the case of data appearing with the delay; the data simply never makes it to venice. producer.put() succeeds, get return null. I can wait a few minutes and try get again it it consistently returns null. I'd be less concerned if put succeeded and get returned data just a few sec later.

Right. So it should be exactly as you've described. If you produce Venice will get it and even takes steps to verify that all parts are received. So I think we need to look at your set up

producer.put part: https://github.com/dlg99/venice/blob/0a6ea9e0854faeea3b336304f8a8f96fd8e34274/clie[…]r/src/main/java/com/linkedin/venice/pulsar/sink/VeniceSink.java

Sounds good. The Venice team is actually driving go-karts at the moment 😅 so Monday traction should be better

Have fun 😉