https://promptfoo.dev/ logo
Using Huggingface inference endpoints
# ❓|questions
r
I would like to load a big LLM and evaluate it, but there's something wrong with the code below # This configuration compares LLM output of 2 prompts x 2 GPT models across 3 test cases. # Learn more: https://promptfoo.dev/docs/configuration/guide description: 'My first eval' prompts: - "Translate to English: اهلا" #providers: [huggingface:text-generation:FreedomIntelligence/AceGPT-7B] providers: - id: huggingface:text-generation:FreedomIntelligence/AceGPT-7B config: apiEndpoint: https://api-inference.huggingface.co/models/FreedomIntelligence/AceGPT-7B tests: assert: - type: similar value: 'Welcome' threshold: 0.8 HF_API_TOKEN : ''
i
The
apiEndpoint
you have doesn't look like a huggingface hosted inference endpoint. The endpoint URL should look something like this:
Copy code
https://v9igsezez4ei3cq4.us-east-1.aws.endpoints.huggingface.cloud
And you need to create an endpoint using the "Create a new Endpoint" functionality
Here's an example config that uses a private endpoint:
Copy code
yaml
description: 'HF private inference endpoint'

prompts:
  - "Write a tweet about {{topic}}:"

providers:
  - id: huggingface:text-generation:gemma-7b-it
    config:
      apiEndpoint: https://v9igsezez4ei3cq4.us-east-1.aws.endpoints.huggingface.cloud
      # apiKey: abc123   # Or set HF_API_TOKEN environment variable

tests:
  - vars:
      topic: bananas
  - vars:
      topic: potatoes
r
Thank you so much