https://promptfoo.dev/ logo
Translation evaluation
# ❓|questions
r
Hi, I am attempting to evaluate LLMs on translation task. I have a benchmarked translated content along with the original text. I need help with setting up the YAML file, in where the assertions include the translated text and I get to score the models output. I have found an example of loading assertions from a CSV, where i can add the 'expected' translated outcome. Is this right? and how do i add the csv file as an assertion in the YAML file? Also how can I add a score metric Your help is appreciated!
Is ROUGE-N a good metric for this? Is there a BLEU score metric available
i
Hi @raghadn3, You can set it up via CSV, or you can have the tests in the YAML file itself. To add the CSV, see [loading tests from CSV](https://promptfoo.dev/docs/configuration/guide#loading-tests-from-csv). Basically it looks like this:
Copy code
yaml
prompts: [prompt1.txt, prompt2.txt]
providers: [openai:gpt-3.5-turbo, ...]
tests: tests.csv
Your prompt might look like:
Copy code
Translate this text to {{language}}:

{{document}}
and the CSV format could looks like:
Copy code
language,document,__expected1,__expected2
French,"Greetings",contains:bonjour,similar(0.5):Hello world
Personally I find it a bit easier to just include it all in the YAML:
Copy code
yaml
prompts: [prompt1.txt, prompt2.txt]
providers: [openai:gpt-3.5-turbo, ...]
tests:
  - vars:
      language: French
      document: "..."
    assert:
      # Semantic similarity
      - type: similar
        threshold: 0.5
        value: "Hello world"
      # String contains
      - type: contains
        value: "Bonjour..."
2 Views