Of course not! Our automation should always run in seconds, or maybe “ones of minutes”. Right? Right? Insert that meme of Padme and Anakin in the meadow. A duration of 45 minutes is clearly an anti-pattern and always a bad idea, right?

Y’all know what’s going to happen when I ask absolute questions like that, don’t y’all?

Yes, of course, in an ideal environment, we would be able to load data into our test environment and start an automated test script at an arbitrary state in our product and begin execution there. It would be similar to a unit test but against deployed and running software. The test script would be short, sweet, and speedy. The ability to perform automation in this manner has many benefits including

• “Fast” feedback for what that script is checking.
• The ability to run smaller groups of test scripts to directly target specific features or fixes.
• It can be easier to parallelize smaller more independent test scripts.
• And many other things, I’m sure.

When possible, this is great! I would love to live in this kind of development environment, and I suspect most others would as well.

Unfortunately, not everyone has that privilege. Many teams are trying to evolve in an ecosystem that was created without testability and, certainly, without automatability. The teams in many of these ecosystems are reluctant to evolve to a situation where automatability becomes valuable. They think that changing app code is a risk, and they’re right! Adding testability and automatability, which is changing app code, is a risk… and a cost! So, if we can’t do “bite-sized” automation, whatever shall we do?

We provide value as best we can!

I once had a client that is a manufacturer of hardware components. Their physical component delivery workflow progressed through multiple “workstations”; each workstation performed a specific set of manufacturing tasks, but those tasks were dependent upon the successful completion of the previous workstations’ tasks. Task N+1 could not, and should not, be attempted if Task N failed to occur successfully.

Their automation lead was really smart. She was self-taught in test automation, but she built a great set of automated test scripts. She used an open source framework to build their suite and her organization found that the suite provided value to them. The automation scripts progressed the software “ticket” through the various workstations to see if things could work in the “happy path”; their other tester worked on the more “devious” testing.

I was brought in to help the team upgrade their automation framework to the most current version and to help evolve their approach where appropriate. When I work with a new client, I like to get a walkthrough of what they’ve done, what seems to be working well, and what seems to be a struggle. As part of her overview of their automation endeavor; she told me about how some of the test scripts took 45 minutes or more to run, each! Holy schnikes! How could someone think a 45-minute test script would be the appropriate way to proceed?

I suggested that they should take a different approach: work with some of their app’s APIs to accommodate at least some reduction in execution time. Perhaps we could inject the data for Workstation 3 so we can jump directly to testing Workstation 4. Nope. The application was not implemented to allow that. I suggested that we change the application for testability and automatability. Nope. Because of, uh, reasons (some appropriate, some maybe not so much), that would not happen (remember when I said above, “reluctant to evolve”?).

So, their lead did her best; she found a way to add value.

The manual effort to perform this specific test was over 8 hours on average for each test flow due to the number of workstations and the number of details that needed to be checked. Bringing that duration down to under an hour was an incredible boost to their productivity.

So, was an automated test script duration of 45 minutes unacceptable? To answer questions like that we must check the context. In this case, I could not argue that she should have done something different, at least not in her current context. Of course, in tandem with this approach, she, her boss, and I were working with the development team to convince them to build interfaces that would allow automation to start at Workstation X and then proceed to X+1, but that would be months in coming. In the meantime, a subset of their tests now ran in tens-of-minutes instead of tens-of-hours.

We often get caught up in thinking things like “automation is going to help us release faster so the automation also needs to be fast”. We must be mindful that automation is about getting machines to do things that machines can do valuably so that we, the humans, the testers, can do things that are not well suited to mechanization. This “sharing of work” between humans and machines can help us to perform tasks faster than we did before. Sometimes that’s due to the speed of the automation itself, but sometimes that “speed” is due to the fact that the testers can perform some tasks sooner because of a reduced opportunity cost provided by the automation. In some contexts, “slow” may actually be “fast enough”, at least for now.

Like this? Catch me at or book me for an upcoming event!