it seems like pinot does not have google pub/sub a...
# generalb
Ben San Nicolas
08/10/2022, 6:44 PMit seems like pinot does not have google pub/sub as a streaming source because it doesn't fit into the stream model pinot uses with offsets. however, google also offers pubsub lite that is compatible with the kafka api. i think it would require some refactoring since the kafkaconsumerfactory doesn't construct the kafka consumer directly, but would something like this be possible?
m
Mayank
08/10/2022, 6:56 PM
There’s a pub-sub consumer implementation in the closed source for Pinot. However, we have experienced issues on the pub-sub side, esp related to message ordering and duplicates.
Mayank
08/10/2022, 6:57 PM
Purely my personal opinion based on personal experience, it seems a better fit for messaging between micro-services, and not so much on streaming side.
b
Ben San Nicolas
08/10/2022, 6:59 PMis this for regular pubsub or pubsub lite?
m
Mayank
08/10/2022, 7:01 PM
Former.
b
Ben San Nicolas
08/10/2022, 7:03 PMpubsub lite has partitions and is very close to the kafka model (not sure if/how it's different). it even has a shim for kafka https://cloud.google.com/pubsub/lite/docs/publish-receive-messages-apache-kafka which is why i'm thinking it could be used relatively easily
m
Mayank
08/10/2022, 7:04 PM
@Navina ^^
n
Navina
08/11/2022, 6:04 AM@Ben San Nicolas yeah we initially didn't consider the pubsub lite option as it required the user to manage their own topic. however, in hindsight, having seen the problems ( in terms of ordering guarantees, need to handle msg re-deliveries etc) with interacting with pubsub, perhaps pubsub lite might have been a better choice.
Navina
08/11/2022, 6:10 AMjust fyi: Pinot expects the following properties from its connector implementation:
1. ordered delivery of messages
2. monotonically increasing message offset (or some notion of it). (#1 and #2 become critical when used with features like upsert)
Iiuc, pubsub lite also doesn't expose any "offset" kind of pointers into the stream, which means we will still have the overhead of using the messageId and timestamp as a form of watermark to seek across the streams. This has been quite challenging to get it right as well.
Navina
08/11/2022, 6:21 AMmy bad. looks like pubsub-lite does provide
MessageMetadatam
mod
02/24/2023, 8:31 AM@Navina Is pinot supporting pubsub lite as a streaming source currently?