It seems like when I'm reading data the data comes...
# troubleshooting
t
It seems like when I'm reading data the data comes in a different order each time, and that makes the results very wrong, or it's just failing altogether. Pulsar says it is setting event time to the time the message arrived on the broker each by default, and I assumed that connected streams just kind of automatically handled keeping things in line somewhat by event time. I see there is some notion of that with
BoundedOutOfOrdernessWatermarks
, I'm just not sure if or how that applies to connected streams, or if that just applies to the data in that particular stream.
d
Hey Travis, I have a pretty similar problem i'm tackling. My dataset is such that I may need to reprocess the entire history of data, in addition to processing incoming data. The historical data could take some time to read and process(over 1 hr) mean while I get new data every 20 minutes. And the new data is completely useless without the historical state being processed before new state. My current thought is this: • reprocessing section of stream broadcasts state of most recent record processed(reprocess_ts) • end of current incoming data broadcasts state of most recent record processed(current_ts) • early in the current incoming data stream, a check is made between reprocess_ts and current_ts, if reprocess is behind current, then the incoming data is queued. When current and reprocess are equal, the queued data is released This really is a rough sketch, and I think there is probably details i'm missing.