@Adam Cameron et al, I'm totally new to TestBox, haven't even downloaded it, just thinking about how it would fit in our dev workflow, so excuse the Fred Flintstone questions here. I get the general pattern for spec'ing out tests, but what actually runs them? We don't have a build process per se as of now. Code gets committed to source control, pushed up the hierarchy to the production stream and workspace, then that workspace gets updated, so it gets the current state of that code. On the way, it passes through the stage stream and workspace, where user testing happens. TestBox gets talked about as automated testing, so I'm thinking some sort of scheduled process would run our suite of tests, and complain somehow if any tests failed. How is that usually orchestrated?

A generalised concept for this is the "pipeline". A commit can trigger a pipeline and steps in the pipeline can run to perform various tasks. Back in the day, Jenkins was a champion for running this automated tasks from triggers. Today, there are a lot of options. Gitlab, Bitbucket, Github all support pipelines. At Pixl8, we use and adore Gitlab. So it is our source control and also what we use to have pipelines for building, deploying, running tests, etc. I'd say that moving to a flow where commits in certain branches launch workflows can be really transformative for your devops workflow so something 100% worth the time and effort in exploring.

Yeah, we use BitBucket for our

git

repository and it can run pipelines on commits to branches (including the main branch). That's how we run all our automated tests. We used Jenkins before that. Plus, with TestBox, developers can run tests any time they want via CommandBox or via a URL in a browser.

> but what actually runs them? All your devs. Whenever appropriate. Plus all the pipeline carry-on that @domwatson has given you a good steer on. Perhaps one point of clarification might help: "test automation" is not intrinsically and only "running the tests as part of a delivery pipeline", the "automation" part of the concept is the act of writing tests that can be run [whenever], instead of relying on a human to click around and hope for the best. I'm not gonna regurgitate the TDD paradigm here, but it's a useful approach to start with writing an empty test case that is along the lines of:

it("does x when y happens", ()=>{})

Write down in the description what part of the work you're doing. Like

"it adds tax when the order is over $$$"

or something. Even doing this keeps you focused on what yer doing. Ideally write the test that will verify that the tax has been added (or whatevs), then write the code. But even if you write the test afterwards, writing the description first and the the code and then the test is still better than not even thinking about the tests until afterwards. I can tell when my devs aren't doing TDD cos their tests will be "it returns true when the inputs is [blah]" or something like that. That's testing implementation detail. No-one gives a shit about that. One only gives a shit that the feature that the client needs has been fulfilled (they do not care about booleans being returned from functions ;-)). And then the various variants of the feature have been fulfilled. Rinse and repeat. Small steps. Run yer tests often. Update yer main branch on local... run the tests to make sure no other oik has broken them Work on yer branch (or whatever)... do small steps and run yer tests whenever you pause to think "OK what's next?" (cos "what's next?" might be "ah bugger... I broke a test" 😉 You'll adopt a cadence and it will become muscle memory.

Report from the trenches of doing the changes I planned, and thinking about writing tests for them... The little micro-section of code I built and tested and was considering writing actual tests for is fine, but how and where it integrates into the overall app is, surprise, much messier. Basically, I'm replacing an [Inactive] bit field in a lookup table with an N-to-N table, allowing these items to be differently active based on a value in another table. That's simple enough, and easy to test in isolation, but all I'm really testing is that I get back these few ID pairs from the table my sql update script put them in. Is that even worth testing? The real application uses that data in various queries that join roughly 25 tables, some of which are lookups like the one I'm fiddling with, and some that are user entered data. Yes I could mock the dynamic ones so they're stable enough to test, but that's still going to be hella verbose, and again really only tests that my lookup data and mocks all match up. I feel like I'm missing the point.

It's probs more a case that this is one of the challenges when one starts trying to test an app which hasn't been designed in a tiered / modular / mindful-of-testing way. TBH I seldom test the integration of new testable code into legacy untestable code, as it's not a good use of anyone's time. As you are facing... a lot of work for little improved "safety" I'd rely on end-to-end / acceptance testing for this (like with Cypress / Playwright / whatever-is-this-month's-flavour).

Mmmm. How about the "Is that even worth testing" part? What exactly am I trying to verify? That the test script and the sql update script that populated that n2n table are (still) in sync?

Testing basic CRUD is rarely worth the effort -- databases "Just. Work." -- but testing "if I run function X with arguments Y, I expect SQL query Z to produce <expected results>" or "if I insert/update <test data> in the database, then run function X with arguments Y, I expect to get back <expected results>" are both valid tests. So "what exactly am I trying to verify" will depend on functions that you've written (or are about to write!) behave in the "expected" way. If your code isn't really structured as a set of collaborating functions with specific inputs/outputs (and maybe side effects), then coming up with tests is likely to be much harder. This plays into what Adam said about code being written without testing in mind. If you can describe the input state and the expected computed output state, for a specific operation, you can write a test for it. That transformation of (part of) the world is what you're trying to verify. The key being "a specific operation", and figuring out the minimal input state setup and the relevant parts of the output state. Testing code that "does too much" is always hard -- and sometimes not worth it at any detailed level since that will almost certainly lead to brittle tests that will fail as soon as you actually start to refactor the big bowl of spaghetti into something more maintainable. It is worth having some automated tests at some level for spaghetti code before you start refactoring -- and e2e testing is probably about as "low-level" as you can reasonably get for an app not written using tests 😐