https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
a
Afaik there is no way to currently do this. However, you could look into extending the keda triggers for Pulsar if you really need this functionality
m
The pulsar function worker does not support adding an HPA to a function statefulset.
t
ah - that's awkward Michael, how do we do it then?
It's a normal situation to have peaky loads i.e maybe 10 events a second spiking to 500 events a second etc
can pulsar not handle such a trivial normal situation?
appreciate any help btw - so far it's been great to use but this issue is a major blocker
m
The Pulsar function work just hasn't added support for HPA. One option is to add that feature. Another option is to deploy your app on your own and add an HPA yourself
t
ok cool - so, like, deploy a HPA manifest pointing to a statefulset that has already been deployed (just spitballing)
i don't have much experience with this to be honest
m
Have you verified that you actually need auto scaling? I would guess that a pulsar function can normally handle 500 events a second. However, it obviously depends on the cost to handle each message.
t
yes, we have a specific situation where we can have up to 500 events a second - with a failure rate of around 20% (enrichment logic, but async so sometimes events arrive late)
m
I think you could add an HPA to an existing function sts, but the risk is that the function worker state will diverge from the actual state
t
i want to mitigate pod error retry backoff
fair enough Michael appreciate your support, will look into HPA with existing pods
s
@Tom (Xiatech) You can use Function Mesh to run functions. It supports native integration with HPA and VPA: https://functionmesh.io/docs/scaling
t
Hi Sijeg, we've tried using Function Mesh - but didn't have much luck? Would be good to talk about it more with you, or a member of your team, as we encountered a blocker.
n
if you are facing the following error when running operator locally:
Copy code
Error from server (InternalError): error when creating "config/samples/compute_v1alpha1_function.yaml": Internal error occurred: failed calling webhook "<http://mfunction.kb.io|mfunction.kb.io>": failed to call webhook: Post "<https://function-mesh-admission-webhook-service.function-mesh.svc:443/mutate-compute-functionmesh-io-v1alpha1-function?timeout=10s>": context deadline exceeded
please try to set the environment variable ENABLE_WEBHOOKS=false to disable the webhooks
s
@Tom (Xiatech) - btw @Neng is the creator of function mesh.
t
Thanks Neng, Sijeg. Appreciate the support - will give that suggestion a go and will respond here
👍 1
morning - quick question, where to set this env variable? Locally on our machine, or on one of the pods?
We have been following the docs here: https://functionmesh.io/docs/install-function-mesh/ and have: • packaged function • deployed the function mesh operator helm chart we are at the point where we are trying to deploy a function using the manifests described here: https://functionmesh.io/docs/functions/run-function/run-go-function and this is where we encounter the error - when we try to apply the manifest
i suspect we're doing something wrong @Neng - when you have a moment, would appreciate your guidance regarding using function mesh
@sijieg one for you - could you help facilitate conversation between Xiatech and your team regarding the 'Bring your own cloud' proposition? We have followed instructions online, but have had no contact from your team. I suspect there's some blocker somewhere, but can't tell what it is.
👌 1
n
@Tom (Xiatech) just set that locally on your machine before you run the operator
one thing worth to know is the webhook will do validation and admission to the submitted CRD. Since we turn if off when running locally, we may need to fill in some fields manually. But please let me know if you met some further issues, and I would like to help
Open in Slack
PreviousNext
Powered by