This message was deleted.

@David K I do rememeber seeing Server Side Entry filters - maybe they can help

Well IIUC, the StreamNative Pulsar Functions SQL is not a "server-side" filtering. It's a Function that will consume from a topic, apply the filtering and write to another topic. This will involve more resource than a server-side filtering or even a client-side filtering. To do server-side filtering, the solution would be to use an EntryFilter: https://pulsar.apache.org/docs/next/develop-plugin/#entry-filter

The StreamNative Pulsar Functions SQL runtime is on the broker side, so the data is filtered before it is transmitted over the network to the client. The EntryFilter is another option, but if IIUC that approach requires a re-compilation of the code if you want to change your filtering criteria.

The queries are

INSERT INTO public/default/output SELECT FROM public/default/input

so I'm pretty sure a Pulsar consumer and a producer are involved. Just like for Flink-Pulsar. So there is an overhead. Maybe it's not visible for the user in the SN platform but it is there and someone pays for it.

Yes, there is a consumer and producer involved, but that is true of all solutions. 😃 The difference is where they both run. If they both run on the broker side, then the networking cost is minimal vs. the alternative of having the application decide to skip the message after it was delivered. I think to original question was focused on manipulating a subscription cursor manually using the pulsar-admin tool, so I assumed that the subscription was being used by an application running outside of Pulsar. In that case, having the pfSQL pre-filter the messages into a topic and having the application consume from the filtered topic rather than the original topic eliminates the sending of messages that will be ignored. For some filters, this could be a significant savings in throughput.

The EntryFilter will be the most efficient solution by far (and I’m saying this as the Pulsar transformation author…)

It’s ok to take pride in your work @Christophe Bornet 😃. It appears to be an efficient approach, and my only critique is the lack of flexibility in changing the filtering logic. A code change and broker restart does seem a bit heavyweight unless I am missing something?

I'm not an expert in EntryFilter. I think subscriptionProperties can be used to provide some configuration flexibility but I wouldn't bet my life on it. (Pulsar-transformations != EntryFilter)

Based on the PIP, an entry filter can read properties of the subscription to set the filter logic.