The Power of Failure in Testing

Amsterdam, The Netherlands | Photography by Raphael Awoseyin

DON’T LOOK DIRECTLY INTO IT

I remember walking through central Amsterdam one mild winter afternoon with a buddy of mine. Crossing the Torensluis, and right in front of the bust of famous Dutch author, Multalui, lay one of those elegantly-designed red street lamps, collapsed over one of the many single-speed bicycles that make up a good size of the local population. The street lighting system was first introduced in the 17th century, following the design of painter Jan van der Heyden, and is part of the charm of this vibrant city. When night falls, the illumination of these artistically crafted ornaments, tell a story of pride and character. Perhaps it was this fondness that made the scene particularly dramatic. Funny, I never paid much attention to those street lamps, as striking as they may be. I guess I just expected the streets to be lit like any other European city. Who would have imagined one falling over, seemingly unprovoked? The beauty of the accident was not lost on the passerby, who curiously approached to take, hopefully, instagramable pictures. I was one of them.

FEAR OF A RED PLANET

Failures are so visible, we instinctively want to hide them. What good is it to our image? Books and articles are littered with success stories. We’ve been taught for so long about the shame in failing. It damages our self-confidence and self-esteem. However, failure is a natural part of life. The ability to fail and learn from failures has been seen as a trait in many of the great men and women of our time. Such greats include Albert Einstein, Charles Darwin, Steve Jobs, Oprah Winfrey, Walt Disney, JK Rowling, and many more. Even without going too deep into this topic and research, we could easily go so far as to say that we learn more from failure than we do from success. Why?

The thing about success is, it’s easy to attribute to our own efforts and abilities while disregarding circumstantial elements, timing, and quite honestly, pure luck. Failure gives concrete feedback on our lack of perfection and should trigger self-reflection on some level. Not to say we can not fail out of “bad luck”, but at least it unveils the inherent uncertainty which lurks in the world we live. Success is a false sense of security that can impede us from asking the tough questions. Is this good enough? Would I have made it otherwise?

I like to see quality as it pertains to software, on one hand as materialising user expectations, and on the other as minimising bad experiences. In software testing, success is what gives us confidence in the stability and quality of the application under test. It is what we want before launching any new product. But how can we truly have confidence in a test that never fails?

“How can you know light, if you’ve never known darkness?”

—Lao Tzu

Tests are generally about success and failure. How good is good enough to pass? We set this threshold, and we dispute any result that doesn’t meet our criteria. Failure is a sign of problems, while success is a sign of expectations being met. How often do we reexamine our expectations after a winning delivery versus after a misfire? The proof is in the pudding, as they say.

THE PARTIAL TRUTH ABOUT GREEN

We all love that new all-green smell of a freshly run test suite. It confirms what we hope to be the case, “Nothing to see here, let’s keep it moving. Everything’s in order, and we’re so proud of our passing tests and the efficiency they bring. No manual checking for us, the automation gods say it’s all good.” So much so that we forget that these are actually…well, tests! More specifically, automated tests. An automated test that always passes tends to draw little attention from the team. In fact, it is often praised as being the gold standard for stable automation. But once we see a major production bug slip by the fortified walls of what we thought we were testing, we begin to question the honesty in those positive responses. In other words, one failure in production is enough to tarnish all the green festivities of the past. Are our tests just “yes-men“?

Test automation for one thing, only verifies precisely what we ask it with zero deviation. A green test only tells you that the exact conditions you set were fulfilled. It gives you a definite answer, so we don't question it but move on. However, if we just modified our software and all our tests still pass, shouldn’t we at the very least reexamine what we are actually testing? But unfortunately, a passing test is often ignored. You see, when we run a test, we secretly hope that it is the time-saving elixir that’ll keep our manual testing fingers ever-young. An all-green report card is the sigh of relief our automation script debugging hearts crave. In some ways, this is the issue with automation being programming. We put on the programmers’ hat and become “creators” aka developers, rather than “destroyers” aka analysts. We’re in the mindset of putting things together to make them work, as opposed to picking things apart to show why they may not be perfect.

We call it an expected result, and it would do some good to challenge our expectations every once in a while. Driving on a street, we don’t question the malfunctioning of traffic lights if they are all green as we speed away to our destination. However, an unchanging red light, just lasting a little longer than we would like, may start to raise suspicion. The green light shouldn’t mean ‘go’ but rather, like a flashing yellow, we should proceed with caution.

A red test tells us our conditions failed. But in addition, it invokes reflection on the structure and implementation of your test. Are the conditions right? Why did it fail? You are pushed to think more critically about your test. Like a no RSVP, we wonder why. Too busy? Not interested? Even a maybe would invoke introspection.

BUG SEVERITY LEVEL: APOCALYPSE

Have you ever had that major bug? You know, the one that shuts down the company’s main source of revenue, leading to a panic room-like meeting? Until one of these occurs, people are generally not aware of the seriousness of QA. We learn best from our mistakes. Some major mistakes bring about pain and hardship, others bring about revolutions and new republics, but very few bring about a state of complacency and nonchalance. Automated tests should be routinely evaluated, regardless of its pass rate. We can not afford false positives, and neither can we waste time with false negatives. Ignoring failed tests is a dangerous path for all involved. Knowing our software may not be perfect, is in itself, motivation for continuous improvement.