@RunLLM How to expose the metric for last read position and mark delete position of the cursor ?

To expose metrics for the last read position and mark delete position of a cursor in Pulsar, enable

exposeTopicLevelMetricsInPrometheus

in your broker configuration. This will make cursor-related metrics available in Prometheus, including

pulsar_subscription_last_acked_timestamp

(last mark delete/ack position) and

pulsar_subscription_last_consumed_timestamp

(last read/consumed position), labeled by cluster, namespace, topic, and subscription name. These metrics provide timestamps for the last acknowledged and last consumed messages per subscription. For more granular cursor stats (including positions), you can also enable

exposeManagedCursorMetricsInPrometheus

for managed ledger metrics, but the main subscription-level metrics above are sufficient for most monitoring needs. Would you like more detail on how to interpret or use these metrics? Sources: - Pulsar subscription metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM

pulsar_subscription_last_consumed_timestamp is always 0 for my consumer

If

pulsar_subscription_last_consumed_timestamp

is always 0, it means the broker has not recorded any message as consumed for that subscription. This can happen if the consumer has not yet received any messages, or if the consumer is not actively consuming (e.g., it's connected but idle, or not requesting messages due to available permits or flow control). Check that your consumer is actually receiving and processing messages. Also, ensure your Pulsar version and client are up to date, as some older versions may not update this metric correctly. Would you like more detail on troubleshooting this scenario? Sources: - Pulsar subscription metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM

pulsar_subscription_last_consumed_timestamp is o form the subscription

If

pulsar_subscription_last_consumed_timestamp

is 0 for the subscription, it indicates that no messages have been consumed (read) by any consumer on that subscription since its creation. This is expected if the consumer has not yet received any messages, or if the subscription is new and idle. To resolve, ensure your consumer is actively receiving messages and that there is data available on the topic. If the consumer is running and messages exist, check for issues like flow control, available permits, or misconfiguration. Would you like more detail on how to debug this further? Sources: - Pulsar subscription metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM if reset cursor is executed on the subscription will the last read and last mark delete position will be 0?

No, executing a reset cursor operation on a subscription will not set the last read position or last mark delete position to 0. Instead, both positions are updated to the reset target (the message ID or timestamp specified). The positions reflect the new start point for consumption, not a zero value, after the reset completes. You can verify this by checking the

markDeletePosition

and

readPosition

fields in the topic's internal stats after a reset; they will show the new position, not 0, as per ManagedCursorImpl.java and PersistentSubscription.java. Would you like more detail on how these positions are managed during a reset? Sources: - ManagedCursorImpl.java - PersistentSubscription.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM topic internal stats shows the mark delete postion and read position for cursor but the prometheus metrics is always 0 ?

Prometheus metrics like

pulsar_subscription_last_acked_timestamp

and

pulsar_subscription_last_consumed_timestamp

expose timestamps (not ledger positions) for the last acknowledged and last consumed messages, not the actual mark delete or read positions. If these metrics are always 0, but the internal stats show valid positions, it means messages have been acknowledged/read, but the timestamp fields were not updated—often due to no recent message activity or a client/broker version issue. Prometheus does not expose the raw ledger positions; it only provides timestamps. For actual positions, you must use the internal stats API or CLI, not Prometheus metrics. Would you like more detail on how to correlate these values or troubleshoot further? Sources: - Pulsar subscription metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yes

Prometheus metrics such as

pulsar_subscription_last_acked_timestamp

and

pulsar_subscription_last_consumed_timestamp

are designed to show the timestamp (in milliseconds since epoch) of the last acknowledged and last consumed messages for a subscription, not the ledger positions themselves. If these metrics remain at 0 while the internal stats API shows non-zero

markDeletePosition

and

readPosition

, here’s what’s happening: 1. Metrics vs. Internal Stats: - Prometheus metrics only expose timestamps, not ledger positions. - The internal stats API (e.g.,

pulsar-admin topics stats-internal

) shows the actual ledger positions (

markDeletePosition

,

readPosition

) for each cursor/subscription. 2. Why Timestamps Might Be 0: - If no messages have been consumed or acknowledged since the subscription was created, the timestamps will remain 0. - Some Pulsar versions or client libraries may not update these timestamp fields correctly, especially if the consumer is idle or not actively acknowledging messages. - The metrics are only updated when a message is actually consumed or acknowledged. 3. How to Correlate: - To see the actual positions, always use the internal stats API. - To monitor activity (i.e., when the last message was consumed/acked), use the Prometheus timestamp metrics. - If you see positions advancing in internal stats but timestamps remain 0, check your Pulsar version and client library for known issues or consider updating. 4. Troubleshooting Steps: - Ensure your consumer is actively consuming and acknowledging messages. - Check for flow control or permit issues that might prevent message delivery. - Review your broker and client versions for bugs related to metrics reporting. - If you need to monitor positions, rely on the internal stats API, as Prometheus will not provide this detail. Let me know if you want a step-by-step example or further troubleshooting guidance. Sources: - Pulsar subscription metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM step by step for providing the ledger position in prometheus metrics

Prometheus metrics in Pulsar do not natively expose the actual ledger positions (

markDeletePosition

,

readPosition

)—they only expose timestamps for last acked and consumed messages. To provide ledger positions in Prometheus, you would need to extend Pulsar’s metrics system by modifying the broker code to export these fields as custom metrics. Step-by-step summary: 1. Modify the Pulsar broker code (e.g., in

PrometheusMetricsGenerator.java

) to add new metrics for

markDeletePosition

and

readPosition

per subscription. 2. Register these new metrics with Prometheus, labeling them by cluster, namespace, topic, and subscription. 3. Rebuild and redeploy the broker with your changes. 4. Scrape the new metrics from the Prometheus endpoint. This is not supported out-of-the-box and requires custom development. Would you like more detail on which files to modify or how to implement this? Sources: - Pulsar subscription metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yes

It seems likely that to expose ledger positions (

markDeletePosition

,

readPosition

) in Prometheus metrics, you would need to: 1. Locate Metric Generation Code: Find where subscription metrics are generated, such as in `PrometheusMetricsGenerator.java` and `NamespaceStatsAggregator.java`. 2. Extract Positions: In the metric generation loop, extract

markDeletePosition

and

readPosition

for each subscription from the topic’s internal stats. 3. Add New Metrics: Define new Prometheus metrics (e.g.,

pulsar_subscription_mark_delete_position

,

pulsar_subscription_read_position

) and output their values, labeled by cluster, namespace, topic, and subscription. 4. Update Metric Documentation: Document the new metrics for clarity and future maintenance. 5. Rebuild and Deploy: Rebuild the broker and deploy the updated version. Prometheus will then scrape these new metrics. Example (pseudo-Java, for illustration):

// Inside PrometheusMetricsGenerator or NamespaceStatsAggregator
for (Subscription sub : topic.getSubscriptions()) {
    String markDeletePos = sub.getCursor().getMarkDeletePosition().toString();
    String readPos = sub.getCursor().getReadPosition().toString();
    // Output as Prometheus gauge metrics (may need to encode position as string label or split into ledger/entry)
    writeMetric(stream, "pulsar_subscription_mark_delete_position", markDeletePos, cluster, namespace, topic, subName);
    writeMetric(stream, "pulsar_subscription_read_position", readPos, cluster, namespace, topic, subName);
}

Note: Prometheus metrics are numeric, so you may need to split the position into ledgerId and entryId and expose them as separate metrics or labels. Would you like a more detailed code example or guidance on handling position encoding? Sources: - `PrometheusMetricsGenerator.java` - `NamespaceStatsAggregator.java` 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.