I'm running a series of 'model-graded-closedqa' assertion tests on a prompt. Generally it seems to work pretty decently, however I often get something along these lines back:
Model grader produced a malformed response:
The criterion requires the submission to mention something similar to "X". In the submission, it is stated that "X" This statement clearly indicates Y, which aligns with the criterion. Therefore, the submission meets the criterion.
The response is saying "the submission meets the criterion", but it ends up failing the assertion. Any idea?