I noticed that when using promptfoo. I always faced CPU throttle (>500%). I dont think it should be that high even with concurrency
m
michaelmichaelmichael
07/02/2024, 9:08 PM
Hi @mrwildler I'm sorry to hear that you're experiencing issues. That is not normal. Can you share more information about your system and your promptfoo configuration?
m
mrwildler
07/03/2024, 6:27 AM
my use case is I'm building an evaluation pipeline that uses promptfoo as a backend. The pipeline is written in Python and will trigger a subprocess to run promptfoo.
I observed that with concurrency set to 4, we will need at least 4Gb in memory and 2.5 CPU.
what types of assertions are you using? any chance you have a shareable example that reproduces the issue?
m
mrwildler
07/05/2024, 6:17 AM
Hi @ian , the assertions run langchain postgre to retrieve documents and calculate base metrics like f1 score. I ran it on pure python and it cause very minimal CPU + Memory usage while with promptfoo as backend. The CPU always spike above 500%