Hi all! Just a query about seeking to prior messages in pulsar - is this supposed to be a quick operation? Sometimes it can take upwards of 30s for us to seek an hour back in time (ie about 1GB of data / 100k messages) and Id be curious to know if this is expected or the time indicates something is wrong

It isn't optimised. There's a workaround mentioned in this issue: https://github.com/apache/pulsar/issues/10488

Right now the primary use case is for a developer only UI which we use to inspect trading decisions made at time X, which are are all contained in a specific pulsar topic. So if client A says "something odd happened at 10am yesterday" we'd open the UI, put in that time, and behind the scenes a reader seeks to to 9:50am, consumes messages up to 10:10am, and then closes. The issue is that that initial seek, if its more than a few hours in the past, tends to time out even with 120s timeout config set. EDIT: For more context, theres normally about 10kB message every few seconds or so into the topic, for about 200MB per day of storage. So scrolling back a few hours I was thinking should be blazing fast. Are there any plans to add in-built time-based indexing (like a btree linking time to message id, if seeking to a message id is faster?)

@Samuel Hinton thanks I added some comments about the missing optimization to the issue https://github.com/apache/pulsar/issues/22129

Thanks mate, appreciate the directions. Its been well over a decade since I dived into java so I cant promise itll make sense, but I'll spent some time reading over it and hoping it clicks

here are some guides to work with Pulsar source code: • https://pulsar.apache.org/contribute/setup-building/ • https://pulsar.apache.org/contribute/setup-ide/ • https://pulsar.apache.org/contribute/setup-debugging/ also: https://github.com/apache/pulsar?tab=readme-ov-file#build