eric.ng
06/28/2024, 2:35 AMRunLLM
06/28/2024, 2:35 AMbroker.conf or standalone.conf).
3. Configure Pulsar Proxy:
- Create a PulsarProxy Custom Resource (CR) with the necessary details such as image version, replica count, and broker connection address.
Example configuration for `PulsarProxy`:
apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarProxy
metadata:
name: <PulsarProxy name>
namespace: <namespace>
labels:
k8s.streamnative.io/coordinator-name: <Your PulsarCoordinator name>
spec:
image: <PulsarProxy image version>
replicas: <PulsarProxy replica count>
brokerAddress: <PulsarBroker connection address>
pod:
resources:
requests:
cpu: <PulsarProxy cpu requests size>
memory: <PulsarProxy memory requests size>
securityContext:
runAsNonRoot: true
4. Health Check:
- Use pulsarctl to run a health check against the broker.
Example usage of pulsarctl for health check:
pulsarctl brokers healthcheck --admin-service-url http://localhost:8080
Would you like more details on any specific step?
Sources:
- https://pulsar.apache.org/docs/3.2.x/concepts-proxy-sni-routing#docusaurus_skipToContent_fallback
- https://github.com/apache/pulsar/pull/22133 π 0 button π 1 button π‘ Hint: Mention @RunLLM in the thread for followups.eric.ng
06/28/2024, 2:37 AMRunLLM
06/28/2024, 2:37 AMapiVersion: apps/v1
kind: Deployment
metadata:
name: pulsar-proxy
namespace: pulsar
spec:
replicas: 3
selector:
matchLabels:
app: pulsar-proxy
template:
metadata:
labels:
app: pulsar-proxy
spec:
containers:
- name: pulsar-proxy
image: apachepulsar/pulsar:latest
command: ["/bin/bash", "-c", "bin/pulsar proxy"]
env:
- name: PULSAR_MEM
value: "-Xms512m -Xmx512m -XX:MaxDirectMemorySize=512m"
- name: PULSAR_LOG_ROOT_LEVEL
value: "info"
ports:
- containerPort: 6650
- containerPort: 8080
readinessProbe:
httpGet:
path: /admin/v2/brokers/health
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
livenessProbe:
httpGet:
path: /admin/v2/brokers/health
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
volumeMounts:
- name: config-volume
mountPath: /pulsar/conf
volumes:
- name: config-volume
configMap:
name: pulsar-proxy-config
Step 3: Create a ConfigMap for Pulsar Proxy Configuration
Create a ConfigMap to store the Pulsar Proxy configuration.
apiVersion: v1
kind: ConfigMap
metadata:
name: pulsar-proxy-config
namespace: pulsar
data:
proxy.conf: |
zookeeperServers=zookeeper_servers
configurationStoreServers=configuration_store_servers
brokerServiceURL=pulsar://broker_service_url:6650
brokerWebServiceURL=http://broker_web_service_url:8080
Step 4: Create a Service for Pulsar Proxy
Create a Kubernetes Service to expose the Pulsar Proxy.
apiVersion: v1
kind: Service
metadata:
name: pulsar-proxy
namespace: pulsar
spec:
selector:
app: pulsar-proxy
ports:
- protocol: TCP
port: 6650
targetPort: 6650
- protocol: TCP
port: 8080
targetPort: 8080
Step 5: Apply the Kubernetes Resources
Apply the created resources using kubectl.
kubectl apply -f pulsar-proxy-config.yaml
kubectl apply -f pulsar-proxy-deployment.yaml
kubectl apply -f pulsar-proxy-service.yaml
Step 6: Verify the Setup
Check the status of the Pulsar Proxy pods and services.
kubectl get pods -n pulsar
kubectl get services -n pulsar
Step 7: Health CheckRunLLM
06/28/2024, 2:38 AMproxy.conf) is correctly set up with the appropriate ZooKeeper servers, configuration store servers, and broker service URLs.
If you need more details or specific configurations, please let me know!
If this doesn't help, please reach out to the StreamNative support team for further assistance. π 0 button π 1 button π‘ Hint: Mention @RunLLM in the thread for followups.eric.ng
06/28/2024, 2:38 AMRunLLM
06/28/2024, 2:38 AMapiVersion: apps/v1
kind: Deployment
metadata:
name: pulsar-proxy
namespace: pulsar
spec:
replicas: 3
selector:
matchLabels:
app: pulsar-proxy
template:
metadata:
labels:
app: pulsar-proxy
spec:
containers:
- name: pulsar-proxy
image: apachepulsar/pulsar:latest
command: ["/bin/bash", "-c", "bin/pulsar proxy"]
env:
- name: PULSAR_MEM
value: "-Xms512m -Xmx512m -XX:MaxDirectMemorySize=512m"
- name: PULSAR_LOG_ROOT_LEVEL
value: "info"
ports:
- containerPort: 6650
- containerPort: 8080
readinessProbe:
httpGet:
path: /admin/v2/brokers/health
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /admin/v2/brokers/health
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: config-volume
mountPath: /pulsar/conf
volumes:
- name: config-volume
configMap:
name: pulsar-proxy-config
Explanation
- Readiness Probe:
- httpGet: Specifies that the probe will perform an HTTP GET request.
- path: The endpoint to check. For Pulsar Proxy, /admin/v2/brokers/health is used.
- port: The port on which the HTTP request is made (8080 in this case).
- initialDelaySeconds: The delay before the probe starts checking (10 seconds).
- periodSeconds: The interval between probe checks (10 seconds).
- timeoutSeconds: The time to wait for a response before marking the probe as failed (5 seconds).
- failureThreshold: The number of consecutive failures before the container is considered unhealthy (3).
- Liveness Probe:
- Configured similarly to the readiness probe, it ensures the container is still running and responsive.
Applying the Configuration
Apply the updated deployment configuration using `kubectl`:
kubectl apply -f pulsar-proxy-deployment.yaml
Would you like more details on any specific configuration?
Sources:
- Pulsar GitHub Issue π 0 button π 1 button π‘ Hint: Mention @RunLLM in the thread for followups.eric.ng
06/28/2024, 2:40 AM/admin/v2/brokers/health is require authRunLLM
06/28/2024, 2:40 AM/admin/v2/brokers/health endpoint requires authentication. To configure readiness and liveness probes for Pulsar Proxy when authentication is enabled, you need to include the necessary authentication headers in the probe configuration.
Here's an example of how to set up the probes with basic authentication:
Example Deployment with Authenticated Probes
apiVersion: apps/v1
kind: Deployment
metadata:
name: pulsar-proxy
namespace: pulsar
spec:
replicas: 3
selector:
matchLabels:
app: pulsar-proxy
template:
metadata:
labels:
app: pulsar-proxy
spec:
containers:
- name: pulsar-proxy
image: apachepulsar/pulsar:latest
command: ["/bin/bash", "-c", "bin/pulsar proxy"]
env:
- name: PULSAR_MEM
value: "-Xms512m -Xmx512m -XX:MaxDirectMemorySize=512m"
- name: PULSAR_LOG_ROOT_LEVEL
value: "info"
ports:
- containerPort: 6650
- containerPort: 8080
readinessProbe:
httpGet:
path: /admin/v2/brokers/health
port: 8080
httpHeaders:
- name: Authorization
value: "Basic base64-encoded-credentials"
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /admin/v2/brokers/health
port: 8080
httpHeaders:
- name: Authorization
value: "Basic base64-encoded-credentials"
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
volumeMounts:
- name: config-volume
mountPath: /pulsar/conf
volumes:
- name: config-volume
configMap:
name: pulsar-proxy-config
Explanation
- httpHeaders: This field is used to add HTTP headers to the probe requests.
- name: The name of the HTTP header (Authorization).
- value: The value of the HTTP header, which should be the base64-encoded credentials for basic authentication.
Generating Base64-Encoded Credentials
You can generate the base64-encoded credentials using the following command:
echo -n "username:password" | base64
Replace username and password with your actual Pulsar credentials. Use the resulting base64 string in the Authorization header.
Applying the Configuration
Apply the updated deployment configuration using `kubectl`:
kubectl apply -f pulsar-proxy-deployment.yaml
Would you like more details on any specific configuration?
Sources:
- Pulsar Security Basic Auth
- Pulsar Proxy Configuration π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.