iValidator is an open source test automation framework, designed especially for integration tests. Guenter, in this article, motivates why there is a need for a new framework and how it is used.

iValidator is an open source test automation framework, designed especially for integration tests. The article motivates why there is a need for a new framework and how it is used.

We started to build the iValidator framework for integration tests when we were asked to automate the test of a complex application (running a fully automated container terminal) since it tooks the test team several days to test the system manually for each release.

The system was composed of two main components that were developed and released independently, and several other systems that were connected to it. Each component included hundreds of classes and communicated via different interfaces (e. g., RMI, JMS). The customer had about 230 test scenarios in mind, with about 15 steps each. Many steps occurred several times in different scenarios. Some of the systems connected had to be simulated (since they moved containers physically, for example) and these simulators had to be controlled by the automatic tests since they receive and trigger events. Even to build a valid starting point for a scenario was a difficult task that included the initialization of several systems. In the end the test scenarios and test data were defined in several XML files (consisting of 30000 lines). They produced a report in an XML file on each run that took about 10 hours.

Test process and test scenarios: An example

The test process starts in the analysis phase of the project with the use cases of the system under development - becoming the system under test (SUT) soon. From these use cases the test scenarios for component integration tests are derived. Each test scenario shows that a business requirement is fulfilled.

Let's consider a simple order management system. It includes these components:

• Customer Management

• Order Management

Each component consists of several classes that store the data and enable the services of the component, e. g., creating a new customer, finding a customer, changing the attributes of a customer, selecting customers that match a search expression.

The components are developed by different teams. Each class of the different components is thoroughly tested by JUnit tests. Even the components are tested by JUnit. But who is responsible for the integration test? A test team that should test manually and cannot cope with the flow of major releases, minor releases, bug-fix releases and quick-fixes delivered by the development team. So they test only the parts changed. Let´s see what happens.

Some poor guy John sits in the call center using our system, finding that the main use case he requests is not fulfilled:

• A new customer calls, John creates the customer using the customer management. Great, it worked! Some minor problem: The address of the customer could not be checked since the post code checking service is down (the SOAP services not working reliable). But no problem: The status of the customer address simply remains ´unchecked´ until the service is up again and will be checked then automatically (smart system, uh!).

• John enters the orders of the customer. No problem either. But then John has to tell where to deliver the goods to - hoops, ´unchecked´ addresses cannot be selected (this did work until some guy in controlling found out that 5% of the deliveries were returned due to wrong addresses and requested the change of the customer component - which was tested, sure). We are stuck! Saving the order is not possible. The customer is lost.

Each component works for itself fulfilling its requirements - but the process that uses both components does not work any more since one component was changed and nobody checked the overall process - or he checked the process and it worked - but he did not consider the failure of the post code checking service since this failure is difficult to simulate.

Figure 1. Context

We will find many variations of this scenario that do differ slightly. e. g., the good one - all works fine; the address is not correct; we select a customer instead of creating him; the customer did order yesterday so we may combine the deliveries; the good ordered is not in stock...

For each release of the system each of these scenarios has to be checked - by some poor guy or by automatic tests. Lets choose the automatic tests and let the poor guy derive the test scenarios from the use cases and automate them.

Requirements for an integration test framework

What are our requirements for an automatic integration test framework?

• Automation: The tests should run automatically (obviously) - we will check the results the next morning.

• Fail-safe: A failing test scenario should not abort the whole process - we want to see how many scenarios still work.

• Error-tolerant: Each error has to be reported, but it may be feasible not to abort the whole scenario, e. g., solely because the age of the customer is not correctly calculated from his birthday - since other test steps may not refer to this data.

• Flexible: A new test scenario that it is only a variation of existing scenarios or uses another set of test data may be added without programming - our test team does not like to wake up the developers for each new scenario they are inventing.

• Configurable: A test scenario should be runnable easily on itself - so that we may show our developers the errors with full logging and debugging switched on if they do not believe that their system fails.

• Adaptable: Technical changes should not require our whole testing system to be re-implemented - so when our lead architect decides that the components do not implement CORBA interfaces any more but deliver SOAP web services we will run our test scenarios against them the next day without complaining.

• Extensible: New components or simulators may be integrated easily - with our evolving test scenarios trying to cover each hole left in the use cases.

• Integration: The framework should integrate into different environments, i. e., the test scenarios and test data may be defined in different ways, e. g., in XML files or in a database.

Building blocks and interfaces of the test framework

Simply spoken, iValidator implements a test runner like JUnit. But since integration testing is a little bit more complex than class testing, it defines some more building blocks and interfaces.

The building blocks of test scenarios are:

• Setups and Teardowns: In contrast to the simple setUp and tearDown methods in JUnit we have classes that implement setups and teardowns since we want to reuse in different test scenarios, maybe even parameterizing them.

• Test steps: Each test step realizes a reusable business task, e. g., creating a customer. Since this step may be part of different test scenarios it has to be parameterized.

• Checkups: A checkup checks the results of a test or several test steps for the expected outcome. If it finds an error it may decide how severe the failure is - and in the test scenario description it has to be decided if the scenario has to be stopped or may continue on this error.

• Flows: A flow combines several Setups, test steps, checkups and/or tear downs to create a reusable higher abstraction. Flows may be nested. A test scenario in therefore a flow - maybe consisting of several sub-flows.

• Adapters: An adapter abstracts from the technical and business interface of the SUT or a simulator. It is used to call the SUT or a simulator from the setups, test steps, checkups, and teardowns. When the technical realization of the interface changes, e. g., from CORBA to SOAP, only the adapter has to be changed - the test scenarios using the adapters are left unchanged. If the business interface changes, e. g., taking one more parameter, the adapter may provide a default parameter while another adapter exposes the new parameter for usage in new scenarios.

• Test parameters and test data: Test parameters and test data are fed into the scenario and flow through the scenario, being changed or enhanced (consider generated technical ids, for example) by the setups and test steps. The teardowns may use them to clean up.

The interfaces of iValidator are used to integrate the framework into different environments:

• Repository: The repository contains the definition of test scenarios, test parameters and test data. The repository may be, e. g., a set of XML files, some database tables. iValidator simply requires the environment to supply these resources. iValidator contains an XML reader as standard adapter for the adapter.

• Reporting: The result of the test run is reported via the reporting interface. The standard implementation creates XML files but other reporting sinks as, e. g., a database, are supported.

• Logging: iValidator may use different logging systems, e. g., log4j or Java 1.4 logging, to log its execution. This logging is separate from the SUT logging.

Developing tests with iValidator

The major design principle behind iValidator was to enable reuse as much as possible and such reduce costs for testing (btw cost is the main reason for not testing). The main reusable components are:

• units

• test data

• adapters

Units -divide test cases in small, reusable parts. The developer has to write small parts that could be used in different scenarios. Therefore the unit's parameters are separated from the unit´s implementation. The parameters are provided to the unit on test execution. The assembling of units to test cases is done declaratively using XML meta data. (This was true in version 1, now with iValidator version 2 XML is the default repository but it can be replaced by, e. g., a database.)

Most integration testing needs a lot of test data as input for the SUT and for control purposes. Data can be defined and grouped within the test description once and used in different test scenarios. For instance, you can reference a whole bunch of data and pull it into the scenario at run time.

Figure 2. Plugin

Adapters encapsulate the connection to the components of the SUT (interfaces, database, GUI ...). An adapter has to be developed only once and than can be used in any unit. (Btw, the adapter concept allows you to test any system that could be reached via Java, not only J2EE software.)

Roughly speaking, developing automated integration tests with iValidator means to develop units, test data and adapters, and than compose these parts to tests and test suites.

It is by no means trivial to build integration tests.It needs a very careful planning and a deep understanding of the problem domain. We strongly recommend to start planning test scenarios when describing the use cases for the system. When you are able to describe what the system should do you are also able to describe how you would verify that the system does what it should.

Implementing Units

On running tests with iValidator a unit is the smallest item. It is nothing more than a method of a Java class. There are only two requirements, the framework has. First there must be a default public constructor and second the method must not have parameters.

Listing 1. AnyClass example

1: public class AnyClass()
2: {
3:     public AnyClass(){}
4:     public void anyMethod()
5:     {
6:         System.out.println("Test");
7:     }
8: }

Here you can see that it is easy to use existing code for instance JUnit test classes. But if you want to use the frameworks enhanced features the class in Listing 1 isn't enough.

Listing 2. UnitClass.java

 1: import org.ivalidator.framework.test.Unit;
 2: import org.ivalidator.framework.test.UnitContext;
 3:  
 4: public class UnitClass implements Unit 
 5: {
 6:     private UnitContext context;
 7:  
 8:     public void setContext ( UnitContext context )
 9:     {
10:         this.context = context;
11:     }
12:  
13:     public void unitMethod()
14:     {
15:         context.getLogger().info("test");
16:     }
17: }

Listing 2 shows an enhanced version of the class of Listing 1. This time the class implements the interface Unit. This interface has just one method to give the class access to the framework. A unit is used in the following order:

• the framework creates an instance of the class

• the method setContext ( UnitContext ) is called

• the unit's method is called

The interface UnitContext is the central interface between the unit and the framework. In listing 2 there is a simple access to the frameworks logging facilities shown. Using this interface the unit has access to all of the frameworks features. But before we show more of these features we should have a closer look at the XML part and see what is necessary to execute units.

The default way to declare the test flow is XML.

Listing 3. input.xml file

1: <?xml version="1.0" encoding ="UTF-8"?>
2:  
3: <descriptor-repository xmlns="http://www.ivalidator.org/schemas/xml-repository">
4:     <test name="mytest">
5:         <instance class="UnitClass" method="unitMethod">
6:     </test>
7: </descriptor-repository>

Listing 3 shows the simplest case of a test description. There is exactly one test with the name test who runs the method unitMethod(). The connection comes through the tag .

Now we need one more XML file that tells the framework which test description should be executed. Listing 4 shows this.

Listing 4. configuration.xml

 
 1: <?xml version="1.0" encoding="UTF-8"?>
 2:  
 3: <config xmlns="http://www.ivalidator.org/schemas/ivalidator" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
 4:     <descriptor repository="xml">
 5:         <property name="inputpath" value="."/>
 6:         <property name="input" value="input.xml"/>
 7:         <property name="classpath" value="."/>
 8:     </descriptor>
 9:  
10:     <report repository="xml">
11:         <property name="outdir" value="."/>
12:     </report>
13: </config>

For this test the files: configuration.xml, input.xml and UnitClass.class have to be stored in the same directory. The iValidator's TestRunner can now be started in this directory issuing the following command:

java -jar [install dir]/ivalidator.jar

The last dot means, that the configuration.xml in this directory should be used. The output of this test execution is an XML file that will be stored in the default directory. This file looks like Listing 5.

Listing 5. Output file

 1: <?xml version="1.0" encoding="UTF-8"?>
 2: <report xmlns="http://www.ivalidator.org/schemas/xml-repository">
 3:     <type>test</type>
 4:     <name>mytest</name>
 5:     <result>success</result>
 6:     <note></note>
 7:     <start-time>2005-01-24T15:03:06.015+01:00</start-time>
 8:     <end-time>2005-01-24T15:03:06.035+01:00</end-time>
 9:     <duration>00:00:00.020</duration>
10: </report>

If you now compare iValidator and JUnit you can't see any advantage for iValidator, but keep in mind, iValidator is designed for scenario testing. You will see the power of iValidator soon.

From unit to suite

Now imagine you want to reuse units with different sets of parameters. We will show how to build complex scenarios. First we will see how to compose units to suites.

Listing 6. Test method

 1: <?xml version="1.0" encoding="UTF-8"?>
 2:  
 3: <descriptor-repository xmlns="http://www.ivalidator.org/schemas/xml-repository">
 4:     <test name="suite">
 5:         <flow>
 6:             <test name="test1" alias="unitMethod"/>
 7:             <test name="test2" alias="unitMethod"/>
 8:         </flow>
 9:     </test>
10:  
11:     <unitlib>
12:         <unit name="unitMethod">
13:             <instance class="UnitClass" method="unitMethod"/>
14:         </unit>
15:     </unitlib>
16: </descriptor-repository>

Listing 6 shows two important points. First there is a tag . This allows the developer to define references to units once and refer to them via the attribute alias. Second we've combined tests. A test with a flow is called a suite. Such a test can't reference units using the tag.

In a flow there can be any number of tests. Each test can reference a unit or contain a suite.

Besides the flow a suite can contain setups, checkups and teardowns. A setup is executed before the flow, checkups and teardowns after the flow.

Figure 3. Run

Typically setups and teardowns are responsible for providing defined system states before and after test execution. Checkups prove conditions after test execution. The framework executes a suite from top to bottom. Later we will see how it handles errors while testing.

Listing 7. Complete suite

 1: <?xml version="1.0" encoding="UTF-8"?>
 2: <descriptor-repository xmlns="http://www.ivalidator.org/schemas/xml-repository">
 3:     <test name="suite">
 4:         <setups>
 5:             <setup name="setup">
 6:                 <instance class="UnitClass" method="unitMethod"/>
 7:             </setup>
 8:         </setups>
 9:         <flow>
10:             <test name="test" alias="unitMethod"/>
11:         </flow>
12:         <checkups>
13:             <checkup name="checkup" alias="unitMethod"/>
14:         </checkups>
15:         <teardowns>
16:             <teardown name="teardown" alias="unitMethod"/>
17:         </teardowns>
18:     </test>
19:  
20:     <unitlib>
21:         <unit name="unitMethod">
22:             <instance class="UnitClass" method="unitMethod"/>
23:         </unit>
24:     </unitlib>
25: </descriptor-repository>

As you can see in Listing 7, a unit can be used as setup, test, checkup or teardown. They can be referenced directly via the tag or indirectly using the UnitLib.

Parameters

The iValidator framework gives the user the ability to use parameters in a flexible way. Parameters can be inherited, overwritten or put into libraries.

A parameter always consists of a name - value pair and is defined in the test description (Listing 8).

Listing 8. Test parameters

 1: <?xml version="1.0" encoding="UTF-8"?>
 2:  
 3: <descriptor-repository xmlns="http://www.ivalidator.org/schemas/xml-repository">
 4:     ...
 5:  
 6:     <parameterlib>
 7:         <parametergroup name="params">
 8:             <parameter name="param1" value="value1"/>
 9:             <parameter name="param2" value="value2"/>
10:         </parametergroup>
11:     </parameterlib>
12:  
13:     ...
14:  
15: </descriptor-repository>

At last we show how a unit can access parameters and parameter lists. Therefore it uses the context (Listing 9).

Listing 9. Access to parameters

 1: import org.ivalidator.framework.test.Unit;
 2: import org.ivalidator.framework.test.UnitContext;
 3: import org.ivalidator.framework.test.Parameter;
 4: import org.ivalidator.framework.test.Parameterlist;
 5:  
 6: public class UnitClass implements Unit 
 7: {
 8:     private UnitContext context;
 9:  
10:     public void setContext ( UnitContext context )
11:     {
12:         this.context = context;
13:     }
14:  
15:     public void unitMethod ()
16:     {
17:         Parameter param1 = context.getParameter("param1");
18:         Parameter param2 = context.getParameter("param2");
19:         Parameter param3 = context.getParameter("param3");
20:         Parameterlist list = context.getParameterlist("fruits");
21:  
22:         boolean b = param2.asBoolean();
23:         double d = param3.asDouble();
24:         list.get(0).getParameter("color").asString();
25:     }

Parameters increase the reuse of units. The different possibilities of defining parameters in the test description allow for a minimal effort on their maintenance.

Data

There are limitations on parameters, they are static and they have to be defined prior to test execution. Therefore, iValidator provides the additional concept of test data to handle dynamically created values and to forward data from unit to unit.

Data is like parameters built of name - value pairs. Any Java object could be used as value. Units can register, recall and delete data.

Listing 10. Unit methods

 1: import org.ivalidator.framework.test.Unit;
 2: import org.ivalidator.framework.test.UnitContext;
 3: import java.util.Properties;
 4:  
 5: public class UnitClass implements Unit 
 6: {
 7:     ...
 8:     public void unitMethod1()
 9:     {
10:         String data1 = "Hello";
11:         Properties data2 = System.getProperties();
12:  
13:         context.setData("data1", data1);
14:         context.setData("data2", data2);
15:     }
16:  
17:     public void unitMethod2()
18:     {
19:         String data1 = (String)context.getData("data1");
20:         Properties data2 = (Properties)context.removeData("data2");
21:  
22:         context.resetData();
23:  
24:         context.getLogger().info(data1);
25:         context.getLogger().info(data2);
26:     }
27: }

Listing 10 assumes that unitMethod1 is called before unitMethod2. The method removeData() delivers the data and removes it from the frameworks cache, resetData() deletes all registered data.

Errors and flow control

It is a test framework`s task to find errors but iValidator can not know what is and what isn't an error in the system under test (except RuntimeExceptions). So the unit's developer has to inform the framework about errors. This is done via the context. Currently we distinguish four different error types:

• failure

• severe failure

• error

• severe error

RuntimeExceptions are handled by the framework like severe errors.

The frameworks behavior on errors can be configured in every flow. Even so on succesful executed tests. Let's have a look on Listing 11 how the unit tells the framework about success or error.

Listing 11. Success or error

 1: import org.ivalidator.framework.test.Unit;
 2: import org.ivalidator.framework.test.UnitContext;
 3:  
 4: public class UnitClass implements Unit 
 5: {
 6:     ...
 7:     public void testmethod()
 8:     {
 9:         int a = 1;
10:         int b = 2;
11:  
12:         context.failure( a == b, "1 != 2");
13:  
14:         try
15:         {
16:             new Integer("4E");
17:         }
18:         catch (Exception e)
19:         {
20:             context.failure(e);
21:         }
22:     }
23: }

The framework has different methods for all four error types. In principle there two different ways to handle errors. The first leads immediately to abortion and needs an exception as parameter. This a feasible method to propagate caught exceptions to the framework giving a hint about the severeness of the exception. The second needs a boolean expression and optionally a description of the error. When the boolean expression evaluates to false the error is reported to the framework.

Internally all these methods throw RuntimeExceptions. For instance if the call of context().failure(false) is inside a try/catch block, the error would not propagate to the framework.

The developer can control the reaction on errors inside the test description.

Listing 12. Error type "failure"

1: <flow>
2:     <control type="result" value="failure" action="next_test_noinits"/>
3:     ...
4:     <test>
5:         ...
6:     </test>
7:     ...
8: </flow>

Listing 12 shows an example for the error type „failure“. It means if this error type is raised inside a test in this flow, then the next test should be executed.. The framework knows four different follow up actions:

• next_test_noinits

• next_test_inits

• next_flow

• stop

Next_test_noinits means no setups and no teardowns are executed; next_test_inits would initiate execution of first the teardowns and then the setups of the suite before the next test starts next_flow stops only the actual flow and runs the suite's teardowns, stop leads to a stop of the whole test execution. This kind of control can only be done inside a flow. If errors happen in a setup the whole suite will be stopped. It doesn't make sense to execute the test flow if the setup fails. Only teardowns will be executed to reduce effects on following suites. Errors in checkup and teardown have no influence on the flow's control but on test results.

Adapters

Tests have to use the interfaces of the system under test and of other systems or simulators. To reduce maintenance costs iValidator uses adapters for encapsulation of the interfaces. When the technical representation of the interface changes, e. g., from RMI to SOAP, only the adapter has to be changed, not the test unit. If the business interface changes, e. g., a new parameter is added, the adapter may provide a default parameter leaving all test units unchanged. Another adapter is written for new test units that may use the new parameter.

An adapter can encapsulate whatever the developer wants. Maybe a database access or an interface to some subsystem of the sut. An adapter has to be instanciated once and can be used by any unit (Listing 13).

Listing 13. Adapter implementation

 1: import org.ivalidator.framework.test.Adapter;
 2: import org.ivalidator.framework.test.AdapterContext;
 3: import org.ivalidator.framework.test.AdapterFactory;
 4:  
 5: public class JndiAdapter implements Adapter 
 6: {
 7:     private AdapterContext context;
 8:  
 9:     public void setContext(AdapterContext context)
10:     {
11:         this.context = context;
12:     }
13:  
14:     public void close()
15:     {
16:         ...
17:     }
18:  
19:     public static class Factory implements AdapterFactory
20:     {
21:         public Adapter createAdapter(String name, Parameters parameters, Object initObject)
22:         {
23:             ...
24:         }
25:     }
26: }

The class JndiAdapter shows what has to be implemented to provide a adapter and its factory. You can create an adapter inside a unit like the code in Listing 14:

Listing 14. Create an adapter inside a unit

1: public void testMethod()
2: {
3:     Parameters params = context.getParameterlist("jndiParams").get(0);
4:  
5:     JndiAdapter jndiAdapter = (JndiAdapter)context.getAdapter("jndi",params, null, new JndiAdapter.Factory());
6: }
7: ...

On calling the method getAdapter() the framework first checks if there already is an instance of the adapter needed and, if not, creates a new one using the given factory.

For ease of use iValidator has the concept of adapter libraries.

Listing 15. The units can reference adapters

 1: <adapterlib>
 2:     <adapter name="jndiAdapter" class="JndiAdapter$Factory">
 3:         <parameters>
 4:             ...
 5:         </parameters>
 6:     </adapter>
 7:     ...
 8: </adapterlib>
 9: ...

As we see in Listing 15 the units can reference adapters from the adapterlib and they do not have to provide the parameters.

Reporting

Reports are written to XML files. The filename consists of a prefix and a timestamp of the beginning of the test execution. For every test the result, start time, end time, a comment and possibly exceptions are reported. The result of a suite comes from the results of all the tests in its flows. Always the worst is the winner. We have the following ranking:

• severe error

• error

• severe failure

• failure

• success

Repositories and Configuration

The iValidator framework uses XML as the default for configuration and reporting. But the frameworks architecture is independent of this. The framework can get test descriptions from any source, for instance, a database or a GUI. The same is true for reporting. We call these repositories; iValidator repositories are pluggable.

In Listing 4 we showed how to configure iValidator. There you can define the repositories and their properties.

What more?

There is much more to say about iValidator, have a look at http://www.ivalidator.org/,

for instance, parallel test execution, import functions, JMX management, graphical test language, U2TP conformance.

Conclusion

iValidator has proven to help implementing complex automatic integration tests that are easily maintainable and extensible (even by non-developers, at least to a certain extent). Therefore it supplements JUnit for test scenarios derived from business use cases. The automatic integration tests show that the system under test (still) fulfills the business requirements and drives the business processes, even when changed in unexpected ways.