Research conducted in the field shows that about 80 % of all errors made while developing software are bugs in code of particular components (the main source of bugs in software is project documentation - usually because of its imprecision, incompleteness, inconsistency, frequently-made changes etc.; here on the other hand we're talking about mistakes made directly in the software). As a consequence, their detection and correction should greatly affect the quality of the system being developed. As the same time, the cost of fixing a bug strongly depends on the moment the bug has been found. According to Barry Boehm, removal of a bug found during unit testing can cost up to several hundred times less than removal of the same bug after the software has already been shipped to the customer.

Therefore, conduction of unit (component) tests greatly influences the quality of developed software. The main goal of conducting such tests is to dynamic, now-level verification of systems under tests (from hereafter referred to as SUT) at an early stage of course of a project. Having defined components of SUT such as modules, classes and their methods, one should verify correct operation of each of those components separately (in isolation) from the rest of the application.

 

Figure 1. The V model

 

What unit tests give us:

• they enable us to unambiguously pinpoint the occurrence of a bug, as well as to point out inefficient lines of program code at the level of separate modules,

• they prevent shadowing of bugs in one component with operation of another component,

• by operating directly on the code of systems, they grant access to vital information about the quality and coverage of tests at the lowest possible cost of all kinds of tests.

What is more, the classic V model (see Figure 1.) implies unit testing is the first stage of dynamic testing for developed software. In many organisations this leads to a conflict between the development and the testing teams regarding who should be responsible for preparation and execution of unit tests. On one hand, arguments exist for handing these tasks over to a programmer, as:

• he/she knows best the code of a certain system component,

• changes to code can lead to side effects which are easy to spot for the author of that code but much harder for a third party (who naturally doesn't know the code as well as its author).

On the other hand, programmers do not want to test their own code because they think that:

• they have got no time for testing - the schedule gives no room for anything beyond development of new code,

• tests are boring and pointless,

• programmers do not make mistakes,

• they lack competence with respect to testing,

• the SUT will be tested by the testing team anyway.

Advocates of soft methodologies know a solution to this problem - namely, the methodology called Test Driven Development (TDD in short). Its application facilitates solving problems between the developers and the testers that concern uni testing. Nevertheless, it causes reluctance in many environments, including project and development team leaders, as there are always misgivings that its application would generate certain - high, possibly - additional cost and increase the time required to create code, with mediocre or no final effect. In this article we will show that code development following the TDD methodology is neither expensive nor time-consuming and greatly improves quality of the final product. In order to be well-understood we will begin by presenting the basic concepts of TDD, after which we will describe a simple experiment demonstrating the achieved effects. We will also present an example collection of tools facilitating preparation of modules adhering to this methodology.

Test Driven Development

Test Driven Development (TDD) is a methodology of module development, the name of which comes from the title of a book by Kent Beck published in 2002: Test Driven Development By Example. In presents a modification of a concept from eXtreme Programming (XP), in XP jargon known as test-first programming or test-first development. The main idea of this methodology can be summed up with one slogan:

WRITE TESTS FIRST AND CODE AFTERWARDS!

This approach strives to achieve two goals:

• it is enforced on a programmer to think a problem over before beginning to write code (i.e. more at specification than at verification),

• it increases quality of developed code.

The diagram of actions taken while following the TDD methodology can be described by four simple steps (see Figure 2.).

The first step:

• Think the problem over.

• Think how to test this.

• Write a simple test with a good interface.

The second step:

• Write only as much code as required to run the test.

• Observe why the test failed - that will give you knowledge about what it is that you really are to program.

The third step:

• Write only as much code as required for it to pass the test.

• Run the test again, check if the module in question passes it (along with all the tests written before).

The fourth step:

• Remove all repetitions, increase clarity of code.

• Run the tests again.

• If everything works fine, store the module.

 

 

Figure 2. The diagram of actions in TDD

 

Unfortunately this is not as simple in practice as it might seem at the first glance. First of all, the approach requires devoting more care to work - it is not enough to program well, one has to crack the nature of the issue so that the tests are prepared well. Then again, it seems this problem is not particularly tedious, furthermore with a bit of practice one can do this efficiently and quite quickly. What is more, many free tools exist which support development of programs in the spirit of TDD.

In case of development of Java applications - which is that we're describing in the article - the best and the best-known of them is JUnit - a tool from the XUnit group, devoted to unit testing, along with its extension called JUnitPerf, the purpose of which is to conduct performance unit tests (XUnit is a whole family of unit-test tools - one can find a link to JUnit, as well as to other tools from the XUnit meant for other programming languages, at the end of the article). Unit tests written with the JUnit library can easily be supplemented with coverage tests conducted with the aid of the application Emma. Also, in many situations, especially in case of a freshly-assembled development team, it won't hurt to perform static analysis of code (for its compliance to programming standards) - here performed using the tool called Hammurapi. All the aforementioned tools will be described in greater detail in the last section, also featuring a fuller description of our toolkit.

In order to determine the cost and the overhead of applying such tools, two comparable groups have been created which were to develop identical simple applications (about 6 hours of work for a three-person team):

• a test group, following the TDD methodology,

• a control group, working in a traditional way, i.e. not designing unit tests.

Both teams worked in the same environment: Eclipse SDK 3.0.2 (built-in JUnit, built-in Hammurapi) and Emma.

 

 

Figure 3. Time of completing tasks

 

Figure 3. shows the time each of the teams took to complete the tasks. In the initial phase the team which followed the TDD methodology required significantly (over 1/3) more time - the extra time was devoted mainly to learning proper use of provided tools (JUnit and Hammurapi - in our experiment Emma was used mainly to achieve a predefined level of test coverage). After the TDD team has gained experience (tasks 4 and 5) this time difference ceased to be as significant - the team required only slightly (less than 10 %) more time to complete its tasks.

As shown by the above, making use of tools that control quality of developed code is not that time-consuming. And how does this affect the quality of products? This can be seen in Figure 4.

 

 

Figure 4. Number of mistakes in code

 

In the initial phase the control group achieved better results; in our opinion this has been caused by additional stress the TDD-following group had been subject to. However, it is clearly visible that no later that from the third tasks have those proportion been changing in definite favour of the group following the TDD methodology, the results of which were better by 50 %. In order to cast away any doubts we emphasise that the modules were being tested both with black-box methods (for their conformance with original concepts) and through manual - static - code analysis.

Another chart (Figure 5.) shows that the TDD-following teams produced code which was definitely more compliant with standards.

 

 

Figure 5. Quality of code (according to Hammurapi)

Summary

Summing up this simple experiment we can state that application of the TDD methodology for development of modules implies:

• minor additional cost - after the methodology has been learned, the increase of time required to complete a module is small (oscillating between 7 % and 10 %),

• a significant increase of quality of code (50 % fewer bugs with TDD).

What is more, the code developed using TDD is more prone to refactoring - analyses of results produced by Hammurapi leads to discovery of good programming practices.

Despite its numerous advantages, the TDD methodology is not devoid of drawbacks. Here we would like to mention the three primary ones:

• while using the TDD methodology it is either all or nothing, it is not possible to finish the work without completing the module in its entirety; one could of course use a stub and leave a part for later, but such a module would have to be returned to very soon;

• TDD is not a substitution for unit tests and if the contract specifies they are required, we will have to conduct them as well;

• it is a soft methodology, thus making many organisations reject it as unusable.

While the final allegation is arguable at best, the former two could be a severe limitation. Still, we believe that the achievable gain (over 50 % fewer bugs in modules with less than 10 % more required time) is a strong arguments for its application.

Description of the Toolkit (for Java Applications)

In this section we would like to present an integrated testing environment for applications written in Java, based on the IDE Eclipse and a built-in library Ant facilitating execution of tests. The tools mentioned here are free, even for commercial use. Configuration of all the tools is put into build.xml, thus making it possible for us to run full tests (making use of all the tools described in this section) by pressing a single button!

The JUnit Library

JUnit is a tool from the XUnit group the purpose of which is to perform unit tests on software developed in Java. The main components of the library are:

• an abstract class TestCase, using which one creates a test case (the class contains the methods: setUp() and tearDown() - this makes it possible to load input data required by a test case or to perform clean-ups after the test has been completed);

• a class Assert, containing a set of assertion comparing expected results with the real ones;

• a class TestRunner (junit.textui.TestRunner and junit.swingui.TestRunner), which makes it possible to execute a test case;

• a class TestSuite, enabling grouping of multiple test cases into sets or scenarios.

By default the results of tests are displayed on the console; however, running tests as Ant tasks we can generate a XML/HTML report from test cases. An example report has been shown in Figure 6.

 

 

Figure 6. An example report from JUnit

 

JUnit intentionally doesn't support performance tests. In order to conduct such tests one should obtain the tool called JUnitPerf.

The JUnitPerf Library

JUnitPerf is a library making it possible to run JUnit tests with performance parameters. The main components of the library are:

• a class TimedTest, using which one can develop tests checking execution time of a test case;

• a class LoadTest, enabling simulation of actions of multiple users in a given number of iterations, for a test case.

Configuration of JUnitPerf in Ant is similar to that of JUnit. Unfortunately, execution time of individual tests includes the time required to prepare the fixture (that is, including e.g. the time of executing the methods: setUp() and tearDown() - which can greatly affect the test result, especially whenever much data must be loaded into a database before the test is executed); as a result, one should give much thought to design of test scenarios while developing tests.

The JUnitPerf has not been made use of in the experiment presented in the article; however, it is absolutely necessary if we want to verify the efficiency of our code while running unit tests prepared with JUnit.

Hammurapi

Hammurapi is a tool that checks code for compliance with programming standards. The complete application includes:

• Hammurapi - a tool for automatic analyses of Java code. It loads the whole code into a repository and generates a detailed report. Aimed mostly at designers.

• Quickrapi - a fast version of Hammurapi. It performs analyses in memory and generates simplified reports. Aimed mostly at developers.

• Archiver.

Hammurapi contains a list of 120 inspectors - rules verifying correctness of the language. One can add new or disable rules, as well as group them into sets (and use individual sets e.g. in certain classes of tests or technologies). The full list of inspectors, along with descriptions and code, is available at: http://www.hammurapi.org/content/inspectors.html

Hammurapi's Output Listener makes it possible to generate reports in XML (HTML) format, assigning priorities to listed flaws and at the same time providing information about used benchmarks (e.g. NCSS, class complexity etc.). An example report window has been shown in Figure 7.

 

 

Figure 7. An example report window from Hammurapi

Emma

While developing unit tests it might be very important to have information about test coverage of code. Such information is provided by another tool - Emma.

Emma is a tool written in Java, independent from external libraries and capable of running either in standalone mode (from the command line), and as an Ant task. Emma measures test coverage in classes, methods, basic blocks (not to be mistaken with statements) and code lines, against *.class or *.jar files (it's a significant difference from the more popular, commercial program Clover, which measures coverage against *.java files). The tool provides two methods of instrumentalisation: the on-the-fly mode for small applications and the offline mode for more complex systems (J2EE etc.). Statistics, generated as XML, HTML or TXT, are aggregated at several levels (e.g. packages, classes etc.), with the covered lines of code tagged with colours. Additionally, Emma enables filtering of data submitted to the report (e.g. to measure coverage only of certain packages).

One should be aware of the fact Emma reports are not in 100 % agreement with a typical recommended programming technique (mainly with respect to handling exceptions); still, it is a very useful tool. An example report from Emma is shown in Figure 8.

 

Figure 8. A report from Emma

On the Net

The Xunit tool family

• Junit: http://www.junit.org/

• VBJunit: http://www.vbunit.org/

• CPPUnit: http://cppunit.sourceforge.net/

• Nunit: http://nunit.nunit.org/

• PyUnit: http://pyunit.sourceforge.net/

• XMLUnit: http://xmlunit.sourceforge.net/

Other tools presented in the article

• JunitPerf: http://www.clarkware.com/software/JUnitPerf.html

• Hammurapi: http://www.hammurapi.org/

• Emma: http://emma.sourceforge.net

Resources

This article is an extended version of the article published in Tester.pl.

 

• K. Beck, Test Driven Development By Example, Addison-Wesley, Boston 2002,

• T. Gilb, D. Graham, Software Inspection, Addison-Wesley , Harlow 1993,

• James W. Newkirk,. Alexei A. Vorontsov, Test Driven Development in Microsoft .NET, Microsoft Professional, 2005,

• Andy Hunt, Dave Thomas, Pragmatic Unit Testing in C# with NUnit, Pragmatic Programmers, 2005,

• J. Nowakowska, L. Stapp, Test Driven Development,, Tester.pl, issue 4 pages 27-37 http://www.sjsi.org/bin/doc/kwartalnik_nr_4.pd