https://docs.all-hands.dev/ logo
Join Slack
Powered by
Yes mutation testing tools like stryker make small...
# proj-agent-canvas
d
Yes mutation testing tools like stryker make small modifications to your code and then run your unit tests. If your unit tests would not have caught that small modification, it means your tests are not effective at catching changes aka "real" coverage.
c
Are these small modifications to the code deterministic, or random, applied on potentially every line of covered code?
I mean, then if I understand correctly: • unit tests seem to cover part A of the code • Stryker messes up lines in part A • and expects the unit tests to catch that
So it's basically verifying that the unit tests are sound
Ah if only we had something like that back in the days of Sonnet 3.7! It was extraordinarily reward-hacking tests!
Copy code
// placeholder to pass the tests
assert True is True
This is not a joke 🤣
🤣 1
d
yep mutation testing is a way to verify that the unit tests are sound!
yeah that test adds coverage but provides no real value 😬
c
About the PR: it doesn’t add it to CI, but it adds over 1000 lines to package.json 🫠 Does it need to… How do we want to run it?
d
it doesn't need to run in CI
hmmm
let me see if it needs all those dependencies
I was thinking of just making the setup available for anyone can run it if they want to
to improve their tests
c
Agree, but how about … a weekly workflow/automation, installing everything, give us a report on Monday?
d
a weekly automation sounds awesome!!!
so stryker just has a lot of transitive dependencies
they are needed unfortunately 😦
at least stryker was only added with dev dependencies
aha 1
c
Technically, I guess we could install all deps only in the automation, unless that sounds too terrible 🤔
d
how about we start with that? thinky spin if there is a pain point we can look into optimizing further
c
With an automation that sets it all up itself, as opposed to expecting users to?
Quick thoughts: • setting up for people seems like a good idea • except humans kinda don’t actually run tests manually anymore ◦ the agent will ◦ so we’d have a higher surface for deps/risk for ~no humans • setting up on the fly in an automation should be doable • need to diff from a week to another, that is, a stable place to save what it already went through last week (I think?) ◦ but this is true anyway I guess ◦ (in a cloud/remote automation)
• on the other hand, not setting it up for agents in normal course of PR work means the agent won’t catch its own sillyness
• we'd catch it weekly…
d
do you think mutation testing is a good use case for github.com/OpenHands/…/plugins as a plugin? the Paul Hammond skill is Stryker specific. For python backend I like mutmut. We could have a more general skill that is language framework agnostic?
🤔 1
c
My thinking is that running it in an automation may still be simpler, because of the ephemeral nature of a cloud sandbox: • we could make a skill, sure • if the user is running it on their repo, though, it needs to install all deps • then it gives some page/report/results • the user needs the report; but they may end up with a bunch of stuff installed While in an automation, it can do all those things, put the report somewhere, and then tear down. But maybe I’m overthinking it!
d
ohh neat
I saw we transitioned the pr review from a plugin to an automation as well
c
What if we start by running it in GH actions? Deterministic, install, run, save report, publish a comment with report… it also needs to save state, doesn’t it? How does it save, e.g. “ran on all tests at commit hash x”?
Maybe not. What if you tell OpenHands on cloud to create an automation?
d
for a github action we could just view the report in the github action job thinky spin
can certainly try the automation
i responded to your comments on github.com/OpenHands/agent-canvas/pull/1688!
Open in Slack
PreviousNext
Powered by