https://promptfoo.dev/ logo
Monitoring feature
# ❓|questions
g
Dear Team, Wondering here if you could provide more context into the monitoring functionality. I guess is more of a broader question but when it comes to LLMs, what is the point of monitorin real time or on a schedule with the same prompts? Unless the model has been retrained ofc. And if it is with different promts and tests, I would not call it monitoring as is just a different scan. How is the current risk level calculated? Or maybe is it part of the plan that as soon as a new vulnerability is disclosed you are in charge or testing it into your costumer system? Thanks for your clarification! 🙂
i
Hi @gabhoo_83877, monitoring is either done on a schedule, or triggered by changes in CI/CD (i.e. when the model or relevant application code has changed) The system will run the same tests repeatedly by default, unless the configuration of the scan itself has changed. Right now each risk area is mapped to a severity level (e.g. child exploitation is always "Critical"). This can be customized Hope this helps! Let me know if you have any more questions