https://promptfoo.dev/ logo
High CPU usage
# ❓|questions
m
I noticed that when using promptfoo. I always faced CPU throttle (>500%). I dont think it should be that high even with concurrency
m
Hi @mrwildler I'm sorry to hear that you're experiencing issues. That is not normal. Can you share more information about your system and your promptfoo configuration?
m
my use case is I'm building an evaluation pipeline that uses promptfoo as a backend. The pipeline is written in Python and will trigger a subprocess to run promptfoo. I observed that with concurrency set to 4, we will need at least 4Gb in memory and 2.5 CPU.
i
what types of assertions are you using? any chance you have a shareable example that reproduces the issue?
m
Hi @ian , the assertions run langchain postgre to retrieve documents and calculate base metrics like f1 score. I ran it on pure python and it cause very minimal CPU + Memory usage while with promptfoo as backend. The CPU always spike above 500%
3 Views