<@U078Y59KC3B> what are the main tunables for memo...
# ask-ai
v
@RunLLM what are the main tunables for memory limits with Pulsar ?
r
In Apache Pulsar, memory limits can be controlled using several tunable parameters. The main tunables include setting memory limits for the Pulsar client, which can be configured to control the total client memory usage. This is particularly useful for managing the memory consumption of producers and consumers under a client instance. According to the Pulsar documentation, you can set memory limits using the
memoryLimit
parameter in the client configuration. This parameter allows you to specify the maximum memory the client can use, either in absolute terms or as a percentage of the JVM's direct memory. Additionally, the Pulsar client configuration provides options to set a memory limit for the client instance, which can be disabled by setting it to 0. This feature is part of the enhancements introduced in PIP-74, which aims to provide better control over client memory usage. For more detailed configuration, you can refer to the StreamNative documentation for advanced configuration options, including JVM options that can impact memory usage. Would you like more detail on any specific aspect of memory tuning in Pulsar? Sources: - Pulsar documentation - Pulsar client configuration - StreamNative documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
v
@RunLLM i'm talking about the pulsar broker
r
For tuning memory limits in the Pulsar broker, several configuration parameters can be adjusted to optimize performance and manage memory usage effectively. Key tunables include: 1. Managed Ledger Cache Size: The
managedLedgerCacheSizeMB
parameter controls the amount of memory allocated for caching data payloads in the managed ledger. By default, it uses 1/5th of the available direct memory. Adjusting this can help manage memory usage for topics with high read/write loads. 2. Direct Memory Settings: The JVM options for the broker can be configured to set the maximum direct memory size using
-XX:MaxDirectMemorySize
. This is crucial for managing the memory used by the broker for off-heap storage, which is often used for caching and buffering. 3. Broker Cache Settings: Parameters such as
maxMessagePublishBufferSizeInMB
and
managedLedgerMaxReadsInFlightSizeInMB
can be tuned to control the memory used for message buffering and read operations. These settings help prevent out-of-memory errors by limiting the amount of data held in memory at any given time. For more detailed configuration options, you can refer to the Pulsar broker configuration file, which provides a comprehensive list of tunable parameters. Would you like more detail on any specific configuration? Sources: - Pulsar broker configuration - StreamNative documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.