Following the release of their immensely popular, considered-a-classic British Steel album, Judas Priest returned with 1981’s more commercial-sounding album Point of Entry. Oft overlooked, this album contains one of my favorite Priest songs: Solar Angels. Coincidentally, some of the lyrics from Solar Angels, “Diamond visions softly breathing fire”, come to mind when I think about this topic. Specifically, the word “softly”.

Recently, I was working with a couple of people who were unfamiliar with the concept of a “soft assert”. My thoughts on the necessity of assertions notwithstanding, I thought I’d write this up in case other people were also unfamiliar with the concept.

“Traditional” or hard asserts evaluate the specified condition and if that condition evaluates to false, the test script’s execution is halted at that point. Soft asserts behave differently. A soft assert is an assert that the result of which is recorded but does not halt the test script’s execution at that point. The results of all soft asserts are evaluated at an author-specified point in the test script, usually at the end; if any soft assert condition evaluates to false, the soft assert reports false as its result and the test script is typically reported as a failure.

Below is an example of using a soft assert, specifically, the SoftAssert Java class provided by TestNG.

SoftAssert softAssert = new SoftAssert();

softAssert.assertEquals("x", "y", "1st soft assert failed");

softAssert.assertTrue("x".equals("x"), "2nd assert failed");

softAssert.assertTrue("xxx".equals("yyy"), "3rd assert failed");

softAssert.assertAll();

In this code snippet, none of the assertEquals or assertTrue cause the script to stop at the call sites; the script will stop once assertAll is called because two of the assertions fail: “x” does not equal “y” and “xxx” does not equal “yyy”.

Other test frameworks may also have soft asserts built-in or that you can get from 3rd parties. If there’s nothing available for your programming language or for your specific needs, creating one for your own use may not be too hard. The essence of the implementation is that a soft assert method catches a hard assertion failure and stores the failure information until assertAll, or the analogous method is called.

If you find yourself in the situation where you need to implement your own soft assert, below is a sample implementation C#; certainly, tweaks and improvements can and should be made, but this can get you started.

class MySoftAssert
{
  private List<AssertFailedException> failedAsserts =
                             new List<AssertFailedException>();

  public void AreEqual(object o1, object o2)
  {
      try
      {
          Assert.AreEqual(o1, o2);
      }
      catch(AssertFailedException e)
      {
          failedAsserts.Add(e);
      }
  }

  public void AssertAll()
  {
      if(failedAsserts.Count != 0)
      {
          throw new AssertFailedException(
                    "One or more soft assertions failed\n" +
                    string.Join("\n", failedAsserts));
      }
  }
}

Now, why in the world (to quote my kids) would we want something like this?

Sometimes, checking “one result” of a set of automation steps, in reality, requires us to check many sub-parts of that result in order to determine whether or not an automated test script should be marked as a success or a failure. As an example, consider the response to an HTTP GET message. We probably should check the HTTP response code, but we probably want to check multiple, or even all, of the fields in the response payload.

Certainly, we could create multiple test scripts, one to check each of the payload values. The challenge here is that this approach can easily inflate the number of automation scripts that we have to execute, review the results of, and maintain. This approach could also lead to appreciably longer execution durations for a set of scripts, because there are now many more scripts, or needing additional computing power to parallelize execution to retain a tolerable execution duration. Either of these possible outcomes could result in a material impact on the budget. Utilizing soft asserts can prevent these results. When using soft asserts, we can assert on several, related values in one test script; if any of the assertions fails, the script fails but we get the failure information for all the failures.

Be careful, though. We must keep in mind that soft asserts usually only report the assertion failures if you call the assertAll method (or its equivalent). If that method is not called, we won’t know if any of the assertion conditions failed. This makes the calling of the method a critical part of the test script. Since it can be easy to forget the call to assertAll, testing and code review of our automation is essential when using soft asserts.

Like this? Catch me at an upcoming event!

Advertisement