Hi! I'm trying to use Ollama to test llama3 models and also use llama3 model for the evaluation part but I have the "ECONNREFUSED" error message. I know it is already covered in the Ollama part of Promptfoo documentation but it's not working for me.
I have a case quite different since I'm running my evals on google colab and not directly on my machine.
It seems like the problem is coming from the endpoint "/api/generate" because without it my Ollama is running and when i add it I have a "404 page not found" error...
Would you guys have any idea on how to solve this issue ?
Thanks a lot!
m
michaelmichaelmichael
09/11/2024, 6:11 PM
I think issue you're facing is due to Colab running in a separate environment from your local Ollama instance. I’d recommend two possible solutions:
1. Tunneling with ngrok: see https://pyngrok.readthedocs.io/en/latest/integrations.html#google-colaboratory
2. Using a Cloud-hosted LLM API: would allow you to avoid local connection issues altogether. For example - you could use https://openrouter.ai/ or https://replicate.com/ with llama
Let me know if you get stuck. I am happy to discuss with you on a quick call.
f
Fav
09/13/2024, 1:02 PM
Thanks a lot for your help, I managed to find a solution with ngrok !