One thing I am constantly fighting with while developing software systems, is the issue of when to do automated testing, and when not to. Some evangelists of automated testing procedures will try and convince you to always automate your testing.
I believe differently. Like everything in life, there should be a balance between using automated procedures and manual testing. Web based systems are growing at a very rapid pace, and this is a prime example of where a delicate balance is needed. Automating back end logic is much easier than automating front end logic (i.e. the GUI). This is simply because the user interface is not easily accessible using programmatic methods such as API's, services etc.
Most (clever) people adopted the coding style of separating business logic out of the UI, into a separate business layer. This eases automated testing dramatically, since now most of the functionality can be automated and only the thin UI layer needs to be tested manually.
However, even with the business logic separated from the GUI, it sometimes is still a huge pain to write automated tests. A simple example is a test to ensure the side effect a certain method has on the database is correct. This implies the test needs to go through these steps:
- Run the test to generate the DB entries
- Compare the DB entries to what is expected
- Delete the entries generated to prepare for the next run (the test needs to be idempotent)
If the side effects are huge, such as an entity consisting of several linked relationships, then this can become quite cumbersome to automate - to the point where writing the automated test might take longer than all the cumulative time spent if tested manually.
I believe one should consider the following aspects in order to make the best decision:
- How critical is the functionality that you need to test for the system's success?
- How complex is that functionality?
- Does that functionality lay in the critical path of the system, i.e. how often is it executed?
- How long would it take to automate that test?
- How long would it take to manually test the functionality
- How regularly does that code changes or gets worked on?
You should automate your tests if and only if:
- (1) is important to critical
- (2) is moderate to high and (1) is important to critical
- (3) is often to regularly
- (4) is less than (5) * (6)
I would say for the other times, one should be able to get away with more efficiency using manual testing. I would love to hear what you think? Please comment away...
If you are a single developer working on a single project, the above observations make sense. However, the moment the project grows beyond say 2 or 3 programmers, test code becomes essential. See the test code as your (very thin) thread which ensures that things still work as you intended. Write test code, not to *test* everything, but to show your intentions. From the erlang docs:
"There has been much debate over how much test code, compared to production code, should be written in a project. More test code finds more bugs, but test code needs to be maintained just like the production code, and it's expensive to write it in the first place. In several articles from relatively mature software organizations that I have read, the amount of test code has been about the same as the production code. "
This seems to be a bit extreme. In my view, if you can test many things with a (relatively) simple piece of code, do so.