One thing I am constantly fighting with while developing software systems, is the issue of when to do automated testing, and when not to. Some evangelists of automated testing procedures will try and convince you to always automate your testing.
I believe differently. Like everything in life, there should be a balance between using automated procedures and manual testing. Web based systems are growing at a very rapid pace, and this is a prime example of where a delicate balance is needed. Automating back end logic is much easier than automating front end logic (i.e. the GUI). This is simply because the user interface is not easily accessible using programmatic methods such as API's, services etc.
Most (clever) people adopted the coding style of separating business logic out of the UI, into a separate business layer. This eases automated testing dramatically, since now most of the functionality can be automated and only the thin UI layer needs to be tested manually.
However, even with the business logic separated from the GUI, it sometimes is still a huge pain to write automated tests. A simple example is a test to ensure the side effect a certain method has on the database is correct. This implies the test needs to go through these steps:
- Run the test to generate the DB entries
- Compare the DB entries to what is expected
- Delete the entries generated to prepare for the next run (the test needs to be idempotent)
If the side effects are huge, such as an entity consisting of several linked relationships, then this can become quite cumbersome to automate - to the point where writing the automated test might take longer than all the cumulative time spent if tested manually.
I believe one should consider the following aspects in order to make the best decision:
- How critical is the functionality that you need to test for the system's success?
- How complex is that functionality?
- Does that functionality lay in the critical path of the system, i.e. how often is it executed?
- How long would it take to automate that test?
- How long would it take to manually test the functionality
- How regularly does that code changes or gets worked on?
You should automate your tests if and only if:
- (1) is important to critical
- (2) is moderate to high and (1) is important to critical
- (3) is often to regularly
- (4) is less than (5) * (6)
I would say for the other times, one should be able to get away with more efficiency using manual testing. I would love to hear what you think? Please comment away...
