Vincent Rischmann
04/16/2025, 12:05 PMRunLLM
04/16/2025, 12:05 PMmemoryLimit parameter in the client configuration. This parameter allows you to specify the maximum memory the client can use, either in absolute terms or as a percentage of the JVM's direct memory.
Additionally, the Pulsar client configuration provides options to set a memory limit for the client instance, which can be disabled by setting it to 0. This feature is part of the enhancements introduced in PIP-74, which aims to provide better control over client memory usage. For more detailed configuration, you can refer to the StreamNative documentation for advanced configuration options, including JVM options that can impact memory usage.
Would you like more detail on any specific aspect of memory tuning in Pulsar?
Sources:
- Pulsar documentation
- Pulsar client configuration
- StreamNative documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.Vincent Rischmann
04/16/2025, 12:08 PMRunLLM
04/16/2025, 12:08 PMmanagedLedgerCacheSizeMB parameter controls the amount of memory allocated for caching data payloads in the managed ledger. By default, it uses 1/5th of the available direct memory. Adjusting this can help manage memory usage for topics with high read/write loads.
2. Direct Memory Settings: The JVM options for the broker can be configured to set the maximum direct memory size using -XX:MaxDirectMemorySize. This is crucial for managing the memory used by the broker for off-heap storage, which is often used for caching and buffering.
3. Broker Cache Settings: Parameters such as maxMessagePublishBufferSizeInMB and managedLedgerMaxReadsInFlightSizeInMB can be tuned to control the memory used for message buffering and read operations. These settings help prevent out-of-memory errors by limiting the amount of data held in memory at any given time.
For more detailed configuration options, you can refer to the Pulsar broker configuration file, which provides a comprehensive list of tunable parameters.
Would you like more detail on any specific configuration?
Sources:
- Pulsar broker configuration
- StreamNative documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.