Open eyes Working brain

If i had to award the best talk at GTAC 2009, the no-brainer choice would be Prof. Niklaus Wirth opening talk. That’s not surprising if you consider who the speaker is, one of the great pioneers of computer science in the field of programming languages. What’s most surprising to me is that he presented a (pre)historical review of problems which turned out to be incredibly relevant today and, somehow, forced me to reframe my understanding of testing.

Building on 1972 Dijkstra dismissal of software testing

program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.

Wirth explained that testing is treating symptoms instead of the disease, with the disease being our failure to prove correctness of programs by analytical means. This failure has its roots in distant past but it still holds today with languages and tools too complex and unreliable. Languages and tools providing proper abstraction, really hiding the system beneath, would give us the simple and rigorous ground to make programs easy to prove correct, so that no testing would be needed. Unfortunately, this looks far from happening

Programming languages are further from being mathematically nice than they were 50 years ago! They’re huge and complicated. They contain big libraries, and most of a programmer’s time is spent finding and learning the right libraries.

and again

What progress has this field actually made? We still struggle with the same problems as 50 years ago: iteration times, debugging, scratching our heads trying to figure out what went wrong.

How insightful! Empirical evidence that computer science made no sizable improvement in software construction is everywhere. It’s like a hamster in a wheel, running nowhere. Why is that? Maybe, we’ve been piling leaky abstractions on leaky abstractions, apparently hiding information without really simplifying, to the point where progress is drowning in complexity.

Testing is a nice way to easily lay down executable specifications, yet it requires maintenance and quickly degrades as we try to cover more cases. But when we code we have the chance to write self-describing programs which are executable specifications, reducing the coverage of tests needed, ideally to zero.

What does it mean in practice? Whenever it’s possible one should aim at declarative code. Domain specific languages and functional programming come to mind. The point being, if a program matches closely its specification, what’s left to test?

def factorial(n)
    if n == 0
        1
    else
        n * factorial(n-1)
    end
end

I’ve finished reading “Perfect Software and other illusions about testing” by Gerald Weinberg. A short, easy to read, low technical book, full of anecdotes which drive you through illusions and challenges of software testing. The nice thing is when the author of a small well written book about testing happens to be the father of software testing himself, epiphany becomes possible.

But beware developers, that’s not about automated software testing, it’s from perspective of the person who leads the testing process. One who knows the true nature of testing and answers to the big questions:

What’s testing? Why do we test?

These naive answers particularly stuck to me:

1. Humans are irrational, emotional, imperfect thinkers
2. Humans have to make decisions (ie. about software)
3. Those decisions are risky (because we make mistakes)
4. Testing is gathering information to reduce risk

So testing is about taking less risky, more informed decisions.

Also, by this definition, testing is much bigger than what you do in front of your computer:

…let me tell you about a job i turned down at a large insurance company before taking the job here. It paid more money, but it failed The White Glove Test. I asked my prospective manager whether the company used a defined testing process for its testers. ‘Oh, yes,’ he said. ‘All one-hundred-fifty use the defined test process.’ I then asked him what that process was, and he said, ‘It’s defined in our testing process manual, which is kept with the CMM library.’ I next asked him if it was the only copy. He said yes, so I asked him to show me the library. In the library, i put on my white glove and dragged my finger along the top ot the Testing Process Manual. I came away with a quarter-inch of dust. Was i testing? It gave me an estimate of how long it had been since any of the company’s one-hundred-fifty testers touched the manual. I performed a meta-test of the testing organization. I decided i didn’t want to work there.

In other words

The number one testing tool is not the computer, but the human brain in conjunction with eyes, ears, and other sense organs.

Another interesting point is that information you get by testing is neutral, which means it’s not implicitly good or bad until it impacts on us, emotional thinkers, bringing new reactions. If you want to properly assess and report information from and to others, you have to understand how humans think and behave…ehm… let me search wikipedia:

Psychology is an academic and applied discipline involving the scientific study of human mental functions and behavior.

Ouch, so as weird as it can seem, testing is also about psychology. It’s about understanding what others really mean and make them understand what you mean, because information without proper communication can turn out useless…

People listen selectively. A tester may say, “At the current rate we’re finding and fixing bugs, there’s very little chance that we can ship by the first of September.” The project manager whose job is on the line is likely to hear, “Blah blah blah blah we can ship by the first of September.” Only give and accept dates in written form.

I definitely recommend this reading for anyone who wants to really understand testing. I’m leaving with some more gems.

Major Testing Fallacies

• The Blaming Fallacy: The more time and effort someone spends looking for someone else to blame for a problem, the less the chance of solving the problem.
• The Exhaustive Testing Fallacy: The only real kind of exhaustive testing is when the tester is too exhausted to continue.
• The Testing-Produces-Quality Fallacy: Quality is a product of the entire development process. Poor testing can lead to poor quality, but good testing won’t lead to good quality unless all other parts of the process are performed properly.
• The Decomposition Fallacy: Test the parts and you’ve tested the whole.
• The Composition Fallacy: Test the whole and you’ve tested the parts.