Hi. Why does Pulsar Sink crash if it is created with

EFFECTIVELY_ONCE

guarantee and the sink calls the

record.fail()

?

I don't have a direct answer to that. Slightly related, there's "PIP-439: Adding Transaction Support to Pulsar Functions Through Managed Transaction Wrapping" in progress to add transactions support, more details in https://lists.apache.org/thread/8g43xgv8d0p3m918c9zrpwn81p4lh9d4

Thanks, Lari. Yeah it might be useful going forward but I was looking for some backward compatible way. Knowing the reason can help me shape up our app or raise a PR if some fix is required. Can you please tag someone else who might be knowing the reason?

Unlike other processing guarantees, when

EFFECTIVELY_ONCE

is set, the message is NOT negatively acknowledged (

consumer.negativeAcknowledge(message)

is skipped). Instead, a

RuntimeException

is thrown. Why this design?: This behavior aligns with exactly-once semantics: • Negative acknowledgment would typically cause message redelivery • With exactly-once processing, you want to avoid automatic redelivery that could lead to duplicate processing • The exception bubbles up to higher-level error handling, allowing the system to decide how to handle the failure (restart, dead letter queue, etc.)

Thanks for the clarification @David K. Let's assume there is some temporary error in the sink, say the timeout happened on one of the records due to some network issue between sink and the destination system. What should we do in this case? Sending a positive ack is incorrect despite of the error, calling fail kills the sink which also is undesirable. Please advice.

In this scenario, the thrown exception stops the processing, which is what you want in the case of a transient error. You should add logic to catch the exception and retry the processing after a delay, e.g., exponential backoff.

Okay. Thanks for the suggestion.