Making Choices about What and How Much

How far do we break things up?

Do we test each individual thing exhaustively?

When to implement tests?

Exercise - Think about what to do with testing and the scenarios in the room

Get feedback from the room and come up with what people think is a reasonable and feasible approach to testing

Automated tests can be run:

A simple automatic triggering of automated tests is via a Unix cron job.

A more advanced approach is via a continuous integration server. These trigger automated test runs and publish the results.

Muon Ion Cooling Experiment (MICE) have a large number of tests written in Python. They use Jenkins to build their code and trigger the running of the tests which are then published online.
Apache Hadoop Common Jenkins dashboard

Show an example of a CI test server.

Review tests and avoid tests that:

def test_critical_correctness():
  # TODO - will complete this tomorrow!
  pass

Tests:

Set-up expected outputs given known inputs.
Run component on known inputs.
Check if actual outputs match expected outputs - values if possible, form otherwise.

When to test:

Always!
Early. Don't wait till after the code's been used to generate data for an important paper, or been given to someone else.
Often. So any bugs can be identified a.s.a.p. Bugs are easier to fix if they're identified at the time the relevant code is being actively developed.
Turn bugs into assertions or tests. Check that bugs do not reappear.

When to finish:

"It is nearly impossible to test software at the level of 100 percent of its logic paths", fact 32 in R. L. Glass (2002) Facts and Fallacies of Software Engineering (PDF).
But, no excuse for not testing anything.
When do you finish proof reading a paper? Learn from experience.

"If it's not tested, it's broken" - Bruce Eckel, in Thinking in Java, 3rd Edition.