A poor man's concept of evals

did you make this?

yes

This is really cool. Does it do side effect evaluation?

Side effects as in passing the right parameter/s to tool calls? No; not yet. But that is a great idea!

Side effects when you want to evaluate external interactions of an agent. For example, the agent makes file changes or API calls.

Ah. That's a little bit beyond the scope of this thing. I am focused in on the evals of the agents; not the correctness of the action. for example, if a tool intends to make a append entry into the file, this scope does not check whether the tool actually made that append entry into specififc file. But what the eval can do is say whether the correct parameters/s were passed to the tool.

That makes sense. I just find that sometimes I have multi-layer LLM calls where one LLM output + some code determines the next LLM call. In those cases adjusting one of components (LLM1, Algorithm, or LLM2) can result in cascading failures that are impossible to find without running them fully integrated. So I vibecoded my own evaluation tooling that runs code and can evaluate both LLM outputs and make sure that those outputs match actual file changes.