cc @millions-afternoon-47582

not sure why llm e2e tests are failing, separate from this pr will take a look in a bit