The first record I won from our local radio station was Paradise Theatre by Styx. It was awesome; Side B of the album was laser-etched with the Styx logo and other graphics. It was a very cool piece and I still have it.

But that’s not the record we’re talking about today. Styx’s 1977 album The Grand Illusion is much more salient for this writing. Though that album’s more famous tune is Come Sail Away, the title track is what resonates with me when talking about testing and automation. Let’s take it from the top…

Welcome to the grand illusion
Come on in and see what’s happening
Pay the price, get your tickets for the show

This is how automation is often sold: as a grand illusion. Just a couple of weeks ago, I received an email saying, and I quote, “automate your entire testing process”. Don’t get me wrong, I’ve spent my career in automation and automation-adjacent activities. There can be value in automation. In fact, when applied judiciously, there’s almost always some value in automating some aspects of our testing activities. But, spoiler alert, that panacea of “100% automated testing” is almost always a fantasy (oops, spoiler for the next set of lyrics)

But don’t be fooled by the radio
The TV or the magazines
They show you photographs of how your life should be
But they’re just someone else’s fantasy

Here is one of our biggest automation stumbling blocks: Company X is doing totally awesome Automation Y with Tool Z; we need to do that too! We read about the big-name companies and their thought leaders promoting how they “automated everything” and “they don’t have testers anymore”. It sounds great! Use the computers! Mechanize the things! Eliminate the testers! Save all the costs! But you can only attain these successes if we do what Company X did using Tool Z!

There are several flights of “fantasy” to mention here. Not to put too fine a point on it, but unless you work at one of those big-name companies, you, well, don’t work at one of those big-name companies. Your company’s pockets probably aren’t as deep, its business profile is probably much more conservative, and it probably can’t tolerate being as aggressive as the big-name companies when it comes to risk. If a company has “automated everything”, there are few, if any, human eyes on the product before it’s released to the users; this is a risk. If Facebook or Twitter goes down for a bit, big deal. They get some bad press in the media but in a few days, it dies down. Did they lose users or revenue? Probably not in appreciable numbers.

But what about your company’s product? What happens if your app goes down for a few hours? Do you owe penalties? Might you be sued? Did this outage cause a life-affecting situation? These are questions that your company needs to consider and answer before living the fantasy of “how your life should be”.

Looking to companies like Google, Microsoft, or Amazon for ideas and concepts is an awesome idea; doing something they do simply because they do it is a bad idea.

America spells competition, join us in our blind ambition
Get yourself a brand new motor car

Based on my experience, these lyrics could be rewritten as “automation spells competition” and “join us in our blind automation”. Yes, indeed! We can go ahead and get ourselves that “brand new motor car”; get that newest, shiniest automation tool that lets “anyone” create automation. But is that really our goal? To allow “anyone” to create test automation?

It’s one thing to enable people with a certain skill set to exploit an additional, related skill set. It’s a totally different thing, and usually, folly, to let people with no context wield a powerful tool. For example, I know that tree branches occasionally need to be pruned. It’s a bad idea to hand me a chainsaw and say, “Cut off the branches hanging over the house”. A really, really bad idea.

If, however, we put that kind of power into the hands of experienced testing and automation professionals, we have the potential to obtain great value.

There will always be a “brand new motorcar”, I mean, a new automation tool. Always. It will promise to do things better, stronger, and faster. And cheaper. And we’ll always be asked why we aren’t using it. And it will be tempting. But what of our current automation endeavor? Organizations spend a lot of money creating their test automation, most of which is married to the test tool or framework they are using. When we find the new, best tool, shouldn’t we switch to it, regardless of the cost and effort?

This is a hard question to answer, even within the context of a specific organization. Humans often get caught in the sunk-cost fallacy, meaning they hope that if they keep spending money on an endeavor, it will eventually be valuable. If we identify that our current tool or approach isn’t producing value, we need to evaluate whether the better business value is to keep investing in (i.e., keep “sinking cost”) or abandon the endeavor.

On the surface, the answer seems easy, “this isn’t working so let’s start over”, but starting over has impacts and consequences. Do we abandon our existing automation and completely reimplement it in the new tool or framework? How long will that take? Instead, perhaps we should stop creating new automation in the “old tool” and create all new automation in the “new tool”. Unfortunately, that means we have two tool sets to maintain. We probably will have automation code that’s duplicated between the two tools, e.g., code for logging in, logging out, and checking various statuses. When a product change requires a change in workflow, that change needs to be reflected in both toolchains. We also need to support both toolchains in our CI/CD/unattended execution environments.

None of these things are intrinsically bad or wrong, but they incur costs. The most obvious costs are the outlay of money for licenses, if any, and the effort required to create the additional automation. Less obvious is the maintenance effort mentioned in the previous paragraph such as maintaining parity with the product across two toolchains.

An even less obvious cost is the opportunity cost. Briefly, opportunity cost is the cost the organization incurs when performing Activity A at the expense of Activity B. Presumedly, Activity B would provide additional revenue or cost savings to the organization; those benefits will be delayed if Activity A delays the completion of Activity B. In most organizations, we can’t do all the activities at once; some activities must wait, and those activities incur the opportunity cost.

Changing our automation tool almost always causes an opportunity cost; we have to learn the new tool and possibly reimplement existing automation in that new tool. Again, this isn’t necessarily bad or wrong, but this fact must be taken into account when deciding on that new “motor car”.

Someday soon we’ll stop to ponder what on earth’s this spell we’re under
We made the grade and still we wonder who the hell we are

Styx was so prescient here. Certainly, they didn’t have test automation in mind when they wrote this song. If, however, we look at these lyrics and think, “Hey, our test automation endeavor is delivering great value to our team, but we need to be doing so much more”, that may be the spell we’re under: we need more!

Can we always improve? Probably. Can we improve in a way that provides appropriate business value? Well, that’s a different question and the answer is context specific. Some organizations will say yes, and some will say no. There’s no wrong answer to that question, just wrong ways to determine the answer. The number one wrong way of determining that answer is ignoring our context and striving for that perception we have, that illusion, that we simply must do more. We need to make our own decisions based on our own contexts.

So if you think your life is complete confusion
Because your neighbors got it made
Just remember that it’s a grand illusion

Wow, that all sounded like a rant. I guess some of it was, but there is a theme here. The lyrics above talk about looking at our neighbors’ accomplishments or our organizations’ longing to be the shining star of the company or the internet. Often those desires are due to us trying to be like other organizations because it seems like they “got it made”. Inspiration from our peers is awesome and valuable; chasing what we perceive as their panacea is not. So, as long as we’re not “fooled by the radio, the TV, or the magazines” and we make responsible decisions about how, when, and what to automate, it’ll be hard to say we succumbed to The Grand Illusion.

Like this? Catch me at an upcoming event!