Yeah, this post is much later than I’d planned, and then it got later than I replanned it. Life and other ideas intervened. I think, however, this is still something interesting to consider when designing, testing, and automating… or not automating.

It was Christmas Day, 2020. My 7-year-old twins got a Nintendo Switch as a present. Whoa were they excited! They also got three games, including Mario Kart.  As expected, racing was the preferred first choice of activity, so the morning was filled with karts and relearning how to lose, and win, gracefully.

After lunch, the boys agreed with each other to try one of the other games, one which is advertised as a multi-player game. To be fair, it really is a multi-player game, both on-console and across the internet. There are… “slownesses”… when starting to play the game; we’ll get to the multiplayer aspect in a minute. Recall, we are talking about 7-year-olds; in my case, highly active and highly engaged 7-year-olds.

It’s been a while now, but as I recall, the setup for the initial character took about 30 – 40 minutes. During this time, Kid #1 chose his character, the character attributes, stuff about his environment, and various other non-skippable configuration items. During this time, Kid #2, to his credit, was only mildly impatient with not being able to have a turn. Finally, it was Kid #2’s turn to set up his character which took 20 – 30 minutes. Altogether it was about an hour before they could start playing together. That’s an eternity to a 7-year-old (and to some 50-somethings, but I digress).

During this time, I didn’t notice anything that would be considered a “classic” software issue. No bug/glitch/error/hiccup/crash/restart; not a one. What I was seeing, however, was an issue, at least to the boys and me. To us, this inability to concurrently configure our characters, ala Mario Kart, was an issue. This software was not meeting our expectations and causing unhappiness. But why?

Most software is intended to be used by many different types of people; game software, in particular, is meant to be enjoyed by people of many ages and backgrounds. This specific game is ostensibly suitable for ages 3 and above and has a rating of “E for everyone”.

Let’s examine “everyone”. In my experience, “everyone” is primarily made up of people who are sufficiently patient

  • For a “long” (albeit gamified) configuration sequence.
  • For a sequential configuration of individual characters (what if I had more kids?!).
  • To Google “how to add an additional player”.

In my experience of this game, my kids barely fit under that “everyone” umbrella. If they had been 5 years old instead of 7, I think we would all have struggled to be patient. Though the content is certainly appropriate for many 3-year-olds, I can’t imagine a 3-year-old spending that amount of time just getting ready to play a game.

So, why am I, an automation guy, bringing all this up? Because it has to do with testing. Why was the initial game configuration so time-consuming? Was it out of necessity? Was it thought to be fun? Perhaps all the testers and business leaders thought it was cool and appropriate. Perhaps, however, some personas were missed during testing.

When I say personas, I mean the kinds of people who will interact with our software; other people have different terms for personas such as roles or actors. A very small set of example personas might include

  • A white man between the ages of 25 and 40.
  • A black woman between the ages of 55 and 70.
  • A color-blind woman in her 30s.
  • White, 7-year-old twin boys on Christmas morning (though that may be oddly specific).

Each of you can likely come up with at least 10 additional personas and certainly each of the above can be carved into many subsets. That’s the point! There are a ton of personas that we should consider when testing software. We need to try to account for as many of them as feasible we test our software. Due to the sheer number of potential personas, we probably can’t test for all of them so we must prioritize for those for whom we expect the highest usage of our software. It’s not that we don’t care about the other users; we want them to enjoy our software too, but we likely won’t have time to test every persona we identify. Heck, we can’t even identify all the personas in the first place and of the ones we do identify, we can’t always be sufficient proxies for them. As humans, we’re limited in that capacity.

So, again, why am I, an automation guy, bringing all this up? Because it also has to do with automation. For the most part, we can configure different kinds of users in our test systems so that those test users have appropriate privileges and don’t have inappropriate privileges based on their role. That helps the automation kind of behave like the actual personas they represent, but they are not really those personas. Only those personas can accurately behave like those personas, and really only a subset of those personas! As I wrote, as humans, we’re limited in our capacity to proxy personas, but machines are even more limited in that respect.

So, whither is automation? It’s where it belongs, helping testers do their jobs. That’s certainly a context-sensitive answer, but it’s for a context-specific world. In some contexts, we can highly automate based on configuration such as the aforementioned privileges. In other contexts, we can’t highly automate because we absolutely need humans (i.e., testers) sympathizing and empathizing with our users. Automating in those situations can be helpful, but if we rely solely on automation in those situations, we run the risk of missing the “human aspect” of the testing, e.g., pretending to be an impatient 7-year-old.

I once interviewed with a gaming company for a “director of automation” position. When I told the recruiter that not all testing can or even should be automated, he responded with, “you can’t automate fun, right?”. He was spot on. You can’t (yet, at least) automate fun, impatience, frustration, or any other emotional response. As long as there is an emotional aspect, a human aspect, to software usage, there will be things that are inappropriate to automate. But don’t let that prevent you from using automation to help do your job; just use it responsibly.

Like this? Catch me at an upcoming event!