abhagsain
04/03/2024, 6:51 PMyaml
prompts: [prompt1.txt, prompt2.txt]
providers: [openai:gpt-3.5-turbo, vertex:gemini-pro]
tests: https://docs.google.com/spreadsheets/d/1eqFnv1vzkPvS7zG-mYsqNDwOzvSaiIAsKB3zKg9H18c/edit?usp=sharing
If we're add the tests object with link to sheets, where do I specify the llm rubric or other assert statements?ian
04/04/2024, 2:05 AM__expected to include assertions. For example:
__expected1,__expected2
llm-rubric: does not apologize,contains:foobarabhagsain
04/04/2024, 7:34 AMapiKeyEnvvar but it's not working. The same azure variable won't work with both.
yaml
defaultTest:
options:
provider:
id: azureopenai:chat:evaluation-us-south-central
config:
apiHost: evaluation-south-central-us.openai.azure.com
temperature: 0.2
apiVersion: 2024-02-15-preview
apiKey: 'HARDCODING KEY HERE WORKS>'
apiKeyEnvar: AZURE_EVALUATION_API_KEY <❌ THIS DOESNT>ian
04/04/2024, 9:57 AM