As you may know from my writing and my conference speaking, I’m a kid of the 70s and a teenager of the 80s. The pop culture I digested in those decades colors many aspects of my life. From a lesser-known sitcom from the late 70s, there came a catchphrase, “Handle it, Roy. Handle it, handle it”.

Handling it. That’s something we too frequently program our automation to do. Here are some things I’ve seen teams handle:

  • Sometimes we get this popup so we check to see if it’s there. If it is, we click “Cancel” and then continue.
  • Sometimes we get a web page that says “System is unavailable” so we just click F5 and then continue because it only happens in the QA environment.
  • Sometimes the app makes us log in a second time during payment.
  • These OS popups come in a non-deterministic order so we wrote code to handle them in an arbitrary order.

Why do we do this? For various reasons, automators sometimes feel compelled to “make the script pass” which leads to creating automation that handles differences in the workflow of the feature we are testing. The problem with “just handling” these differences is that we may be masking legitimate issues that should be evaluated and possibly addressed. That said, and though “green checkmarks” are not the goal of automation, there are cases where it might be appropriate to write scripts that “get past” specific non-deterministic scenarios; this approach may be necessary in order to get to the part of the application the script is intended to target.

We can separate the situations where this might happen into categories: those that entail appreciable differences and those that entail non-appreciable differences. Appreciable differences are those differences that present intolerable risks if our automated test scripts treat the differences as equivalent. In contrast, non-appreciable differences are those differences where “just handling them” presents a tolerable risk. Appreciable differences are generally related to higher risks while non-appreciable differences are generally related to lower risks.

To help explain, here’s an example that I encountered recently. My team was creating automated test scripts for a mobile app on iOS. After logging into the app, the user was prompted to accept or reject two capabilities via iOS popups, namely location services and push notifications. Due to various implementation factors, the order in which the user received the popups was non-deterministic: sometimes the location services popup appeared first, and sometimes it appeared second.

For the testing that we were doing, the automation that we were creating, and the usage pattern of the app’s users, was programming the automation to “just handle” that non-determinism a tolerable risk? You, the reader, can’t answer that because you lack the context. My team suspected handling the non-determinism was a tolerable risk, but we could not answer that question either because we needed our client to provide the full context. We met with that client and they agreed that it was a non-appreciable difference and therefore a tolerable risk. As such, we created the automation to handle the non-determinism in a way that all test scripts could continue past those popups regardless of the order of their appearance.  We treated that difference as equivalent.

We can contrast that scenario, with the “missing signature” scenario. In this same app, depending on how the test data was configured, a user would be prompted to provide a signature or, if the signature had already been supplied, the user would not be prompted. For this case, we all agreed that these two scenarios constituted an appreciable difference. The business impact of not receiving the “missing signature” prompt when required was too large for the client; handling a missing vs a non-missing signature was deemed to require multiple test scripts because the difference in those scenarios was appreciable. We did not treat that difference as equivalent.

Many of you, myself included, may be thinking, we should start our automation test script’s execution as close as possible to the scenario we are testing. The ability to do this would eliminate the need to handle the popup non-determinism described above. In some cases, however, this is not feasible or not possible. Consider testing a GUI application where all of the business-critical features are inaccessible unless a user has logged into the application. It can be difficult or impossible to get to a specific feature that you want to test without going through the login screens or steps. If there are scenarios that the script needs to just get past during those login steps, adding code to address those steps will usually be required, such as in the “iOS pop-up” example above. As long as the differences in those steps are deemed non-appreciable, causing the automation to handle those differences can be the appropriate choice.

Slightly different, but related, is something that I often call “the trust approach”. There are scenarios where I have treated appreciable differences as non-appreciable. Why the heck would I do that? I would only do that if I knew of other tests that sufficiently explored the appreciable differences such that subsequent testing could “trust” that the previous testing allowed the differences to become non-appreciable for those subsequent tests.

As I wrote above, script success is not the goal for automation; obtaining useful information is the goal. But, if we can’t reliably access the feature we are trying to test, we will be limited in the information we can obtain and be limited in the value our automation can provide. We need to be mindful of and deliberate in the way we handle appreciable differences in our automation.

Like this? Catch me at an upcoming event!