@RunLLM give me an example how to configure

storageOffload

aws using service account

To configure Pulsar or StreamNative tiered storage offload to AWS S3 using a service account, you must set the AWS credentials (access key and secret key) as environment variables before starting the broker, and specify the S3 bucket and region in your configuration. Example: 1. Set environment variables in `conf/pulsar_env.sh`:

export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY

2. In

broker.conf

, configure:

managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=your-bucket-name
s3ManagedLedgerOffloadRegion=your-region

This uses the AWS IAM user/service account credentials for authentication. For more, see Pulsar AWS S3 offload docs. Would you like more detail? Sources: - Pulsar AWS S3 offload docs - Pulsar Tiered Storage Cookbook 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM using helm and values.yaml

To configure AWS S3 offload in Pulsar/StreamNative using Helm and values.yaml, set the required S3 fields under the broker section in your values.yaml. Also, create a Kubernetes secret with your AWS credentials and reference it in the config. Example values.yaml:

broker:
  offload:
    enabled: true
    managedLedgerOffloadDriver: aws-s3
    s3:
      enabled: true
      s3ManagedLedgerOffloadBucket: your-bucket
      s3ManagedLedgerOffloadRegion: your-region
      secret: your-aws-secret

Create the secret:

kubectl -n <k8s_namespace> create secret generic your-aws-secret \
  --from-literal=AWS_ACCESS_KEY_ID=<access_key> \
  --from-literal=AWS_SECRET_ACCESS_KEY=<secret_key>

For details, see StreamNative AWS S3 Tiered Storage. Would you like more detail? Sources: - StreamNative AWS S3 Tiered Storage 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

use service account @RunLLM

This error means your AWS credentials or S3 endpoint region do not match the actual region of your S3 bucket. You must set

s3ManagedLedgerOffloadRegion

in your values.yaml to the correct region, and if using a custom endpoint, set

s3ManagedLedgerOffloadServiceEndpoint

to match the region (e.g.,

https://s3.YOUR_REGION.amazonaws.com

). See StreamNative AWS S3 Tiered Storage. Would you like more detail? Sources: - StreamNative AWS S3 Tiered Storage 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM i need setings for official helm chart

For the official Apache Pulsar Helm chart, set AWS S3 offload in your values.yaml under the broker config. Example:

broker:
  configData:
    managedLedgerOffloadDriver: aws-s3
    s3ManagedLedgerOffloadBucket: your-bucket
    s3ManagedLedgerOffloadRegion: your-region
    # Optional: s3ManagedLedgerOffloadServiceEndpoint: https://s3.your-region.amazonaws.com

Set AWS credentials using a Kubernetes secret and mount it as environment variables, or use an IAM role if running on AWS. For details, see Pulsar AWS S3 offload docs. Would you like more detail? Sources: - Pulsar AWS S3 offload docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM with such config I have an error with regions

service_account:
    annotations:
      <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>:  
storageOffload:
    driver: aws-s3
    bucket: pulsar-bucket
    region: us-east-2

The error occurs if the S3 bucket is not actually in the us-east-2 region, or if the IAM role is not assumed correctly. Ensure your S3 bucket is created in us-east-2, and that your EKS service account is annotated with the correct IAM role ARN that has S3 access. Also, make sure you do not have conflicting region settings elsewhere (e.g., defaulting to us-east-1). For region errors, double-check: - The bucket's actual region in AWS S3 console. - The value of

region

in your config matches the bucket's region. - No other config or environment variable is forcing a different region. See Pulsar AWS S3 offload docs. Would you like more detail? Sources: - Pulsar AWS S3 offload docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.