https://promptfoo.dev/ logo
Function callbacks not triggered
# ❓|questions
r
Hey! I was trying to use the function callbacks feature on an openai model. I have used this config from the docs:
Copy code
module.exports = /** @type {import('promptfoo').TestSuiteConfig} */ ({
  prompts: "Please add the following numbers together: {{a}} and {{b}}",
  providers: [
    {
      id: "openai:gpt-4o-mini",
      config:
        /** @type {InstanceType<import('promptfoo')["providers"]["OpenAiAssistantProvider"]>["config"]} */ ({
          model: "openai:gpt-4o-mini",
          instructions:
            "You can add two numbers together using the addNumbers tool",
          tools: [
            {
              type: "function",
              function: {
                name: "addNumbers",
                description: "Add two numbers together",
                parameters: {
                  type: "object",
                  properties: {
                    a: { type: "number" },
                    b: { type: "number" },
                  },
                  required: ["a", "b"],
                },
              },
            },
          ],
          functionToolCallbacks: {
            addNumbers: (parametersJsonString) => {
              const { a, b } = JSON.parse(parametersJsonString);
              return JSON.stringify(a + b);
            },
          },
        }),
    },
  ],
  tests: [
    {
      vars: { a: 5, b: 6 },
    },
  ],
});
I was expecting that: - my prompt gets passed as a user role message - then i receive an assistant role message with the tool calls array - then i receive a tool role message with the result of the function and possibly the next assistant message However, it looks like after running eval I get as output just the confirmation that my function would be called:
[{"id":"call_HVJQjd2cVSPdSsxwsepxicSA","type":"function","function":{"name":"addNumbers","arguments":"{\"a\":5,\"b\":6}"}}]
Did i get this function callbacks feature wrong or am i missing something?
i
I believe @michaelmichaelmichael is looking into this
m
I have a fix with an example in https://github.com/promptfoo/promptfoo/pull/1648 I will get it released today