https://promptfoo.dev/ logo
SyntaxError: "undefined" is not valid JSON
# ❓|questions
g
Hi, I am testing the dataset generation functionality but when I run
npx promptfoo@latest generate dataset --config generate_dataset.yaml
I get the following error:
Copy code
SyntaxError: "undefined" is not valid JSON
    at JSON.parse (<anonymous>)
    at synthesize (/Users/username/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/testCases.js:253:27)
    at async Command.<anonymous> (/Users/username/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/commands/generate.js:157:25)
generate_dataset.yaml contains the following:
Copy code
prompts:
  - 'Act as a travel guide for {{location}}'
  - 'I want you to act as a travel guide. I will write you my location and you will suggest a place to visit near my location. In some cases, I will also give you the type of places I will visit. You will also suggest me places of similar type that are close to my first location. My current location is {{location}}'

providers:
  - id: azureopenai:chat:deploymentNameHere
    config:
      apiHost: 'xxxxxxxx.openai.azure.com'
  
tests:
  - vars:
      location: 'San Francisco'
  - vars:
      location: 'Wyoming'
  - vars:
      location: 'Kyoto'
  - vars:
      location: 'Great Barrier Reef'
Normal evaluation using a
promptfooconfig.yaml
file with same provider works perfectly. Any ideas on what could be wrong? Thanks!
m
Hi @gianfree it's not working is because dataset generation relies on gpt-4 from openai: https://github.com/promptfoo/promptfoo/blob/main/src/testCases.ts#L22 I will work on using the default provider for you.
g
Hi @User , thanks for answering! I see, is there any plan to include other providers for this feature?
m
there is an open pull request right now that adds support for this: https://github.com/promptfoo/promptfoo/pull/1158
Please feel free to try it / leave comments / etc. If you share your github username I can tag you in it.
I am going to try to get it merged and deployed within the next ~20 hours.
g
Thanks a lot! I will have a look!
On a similar note, I get the following error when I try to run `npx promptfoo@latest generate redteam -w`:
Copy code
Generating...
 █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 2% | ETA: 1s | 2/100/Users/username/.npm/_npx/81bbc6515d992ace/node_modules/tiny-invariant/dist/tiny-invariant.cjs.js:14
    throw new Error(value);
          ^

Error: Invariant failed: Expected purpose to be a string, got: undefined
    at invariant (/Users/username/.npm/_npx/81bbc6515d992ace/node_modules/tiny-invariant/dist/tiny-invariant.cjs.js:14:11)
    at getPurpose (/Users/username/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/redteam/index.js:49:34)
    at async synthesize (/Users/username/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/redteam/index.js:73:41)
    at async doGenerateRedteam (/Users/username/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/commands/generate.js:76:26)

Node.js v20.9.0
Again I am using Azure as a provider (the config file is the same as above, but no tests). Do you have suggestions also for this issue?
m
You should be able to pass a `--provider azureopenai:chat:gpt-4o-deployment-name `flag to override the default. I haven't tested this myself with Azure and will today
g
Hi! Thanks again for the answer. I tested the features with
--provider
but I got a few errors (reported below in order of appearance): -
You must specify at least 1 provider (for example, openai:gpt-4o)
-> so I added the provider also in the yaml file (so provider is present in the yaml and passed when lunching the red-teaming feature) -
Error: Invariant failed: Expected purpose to be a string, got: undefined
-> so I passed
--purpose 'testing'
-
Error: Invariant failed: Expected generatedPrompts to be a string
-> at the moment I am stuck here Any ideas about the last error?
m
I am sorry about your experience. I am working on this now for you. It seems like these are all provider errors relating to your azure integration. Re. the purpose Your error was coming from https://github.com/promptfoo/promptfoo/blob/main/src/redteam/purpose.ts#L26 When you pass in a purpose that function is not called https://github.com/promptfoo/promptfoo/blob/main/src/redteam/index.ts#L177 I just signed up for an azure account and will get back to you.
g
Thanks!!
Hi @michaelmichaelmichael , did you have a chance to check the azure integration?
m
any news?
i got a promptfooconfig.yaml with defaultTest: options: provider: id: openai:chat:openai/gpt-oss-120b config: apiBaseUrl: http://172.18.21.143:8000/v1 apiKey: empty config: reasoning_effort: high providers: - id: openai:chat:openai/gpt-oss-120b config: apiBaseUrl: http://172.18.21.143:8000/v1 apiKey: empty when generating a dataset OPENAI_API_KEY=empty promptfoo generate dataset --config promptfooconfig.yaml --output path_to_output.yaml --instructions "You are an AI assistant" --numPersonas 2 --numTestCasesPerPersona 5 --provider openai:chat:openai/gpt-oss-120b --env-file .env async doGenerateDataset (/node-versions/v22.14.0/installation/lib/node_modules/promptfoo/dis{ Error: Invariant failed: resp.output must be defined any tips?
4 Views