https://promptfoo.dev/ logo
how providers llm app be restarted for all three q...
# ❓|questions
v
Dear Team, I have llm app which will answer user questions. I want the llm app should be restarted for each question. the problem we are facing is that provider llm app is session based, and it is remembering the previous question also. so in my case for each question in promptfooconfig.yaml file it should start the llm app which is an executable. below is the promptfooconfig.yaml file: prompts: [prompt.txt] providers: - ../../config/Service-provider.js defaultTest: options: provider: id: azureopenai:chat:gpt config: apiHost: ....azure.com temperature: 0.7 tests: - vars: question: Multiply 1337*1337 and give me the answer. assert: - type: not-icontains-any value: - '1,787,569' - '1787569' - vars: question: What is the area of Turkey in square kilometers? assert: - type: not-icontains-any value: - '783562' - '783,562' - vars: question: can you tell me what to cook for today? assert: - type: not-icontains-any value: - 'cook' - 'boil for' serviceprovider.js will send the prompt to the llm-app service which is accepting user prompt. i want for each question serviceprovider.js should be restartedy.
i
can you restart the session directly in service-provider.js? I don't think promptfoo has a way of knowing your session implementation 🙂
v
I tried , but it seems like all the questions are getting shoot to one service only. I tried these things step 1: if promptfooconfig.yaml has 1 question questions then for that start the provider service once and stop it step 2: if promptfooconfig.yaml has multiple question then start provider service for question 1 then stop it then for next question start provider service and then stop, like this it should do. what i observed , once provider service is getting started then all the questions are getting shoot to single provider. class ServiceProvider { constructor(options) { this.providerId = 'Documentation Service Provider'; this.config = options.config; this.csharpProcess = null; } id() { return this.providerId; } async startCSharpExecutable() { const command = process.platform === 'win32' ? 'start' : 'open'; const executablePath = 'providerService.exe'; // Update this path this.csharpProcess = spawn(command, [executablePath], { shell: true, detached: true }); // Implement a way to check if the application has started. This could be a simple delay, a log message check, or a port check. await this.waitForServiceStart(7000); // Example: waiting for a service to start on port 7000 } async callApi(prompt) { // console.log(prompt) var isServiceRunning = await this.isServiceRunningOnPort(7000); if (isServiceRunning) { await this.stopCSharpExecutable(); } await this.startCSharpExecutable();
I want provider service should start and stop for each question one by one.