Google Test Automation Conference: Testing is not enough
Monday, October 26th, 2009
If i had to award the best talk at GTAC 2009, the no-brainer choice would be Prof. Niklaus Wirth opening talk. That’s not surprising if you consider who the speaker is, one of the great pioneers of computer science in the field of programming languages. What’s most surprising to me is that he presented a (pre)historical review of problems which turned out to be incredibly relevant today and, somehow, forced me to reframe my understanding of testing.
Building on 1972 Dijkstra dismissal of software testing
program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.
Wirth explained that testing is treating symptoms instead of the disease, with the disease being our failure to prove correctness of programs by analytical means. This failure has its roots in distant past but it still holds today with languages and tools too complex and unreliable. Languages and tools providing proper abstraction, really hiding the system beneath, would give us the simple and rigorous ground to make programs easy to prove correct, so that no testing would be needed. Unfortunately, this looks far from happening
Programming languages are further from being mathematically nice than they were 50 years ago! They’re huge and complicated. They contain big libraries, and most of a programmer’s time is spent finding and learning the right libraries.
and again
What progress has this field actually made? We still struggle with the same problems as 50 years ago: iteration times, debugging, scratching our heads trying to figure out what went wrong.
How insightful! Empirical evidence that computer science made no sizable improvement in software construction is everywhere. It’s like a hamster in a wheel, running nowhere. Why is that? Maybe, we’ve been piling leaky abstractions on leaky abstractions, apparently hiding information without really simplifying, to the point where progress is drowning in complexity.
Testing is a nice way to easily lay down executable specifications, yet it requires maintenance and quickly degrades as we try to cover more cases. But when we code we have the chance to write self-describing programs which are executable specifications, reducing the coverage of tests needed, ideally to zero.
What does it mean in practice? Whenever it’s possible one should aim at declarative code. Domain specific languages and functional programming come to mind. The point being, if a program matches closely its specification, what’s left to test?
def factorial(n)
if n == 0
1
else
n * factorial(n-1)
end
end