PLANitIO Unit Tests and Integration Tests

[TOC]

1. Unit Tests and Integration Tests

Unit tests confirm that individual methods generate the correct result from a known set of input values. They are used to confirm that methods or classes are working correctly, rather than the whole application. If code changes cause a unit test to fail, the developer can immediately review the changes to see what caused the failure.

Integration tests test the whole application. They run the whole application using standard input files and confirm that the contents of the output files generated by the program match the expected results.

Unit testing is particularly useful when a team of several developers is working on the same piece of software. In that case each developer must write tests for his own code before integrating it into the larger codebase of the whole project.

We have not written unit tests at the method or class level for the PlanIt application. In the case of PLANit, the development team was small (2 people) and it was sufficient to use integration tests to find issues. Test failures could still be traced back to bugs which caused them.

All tests described in the rest of this document are integration tests.

2 Integration Tests

2.1 Overview

This section begins with a general description of how the integration tests work, which is applicable to all test cases. It then goes into more detail about the contents of the input and output files which are used in the tests.

2.2 General Structure of Integration Tests

Integration tests must use standard input files which define the network, demands and zoning for the test case. In the case of PLANitIO all of these are defined in one XML input file (usually named macroscopicinput.xml).

The expected results should have been generated by another method. These can include:-

• hand calculation (if the test case is simple);
• running the same test case in another application. All the tests in the “Route Choice” section were run on OmniTRANS, whose outputs were recorded for comparison;
• expected error messages, where a test has deliberately invalid inputs in order to test PLANit’s input validation methods.

The expected results generated from these alternative calculation methods are stored in two ways:-

• CSV and XML output files;
• Java data storage objects, usually Java Maps.

PLANitIO stores results from its runs using objects called output formatters. It uses two output formatters:

• PLANitOutputFormatter, which writes results to CSV and XML files;
• MemoryOutputFormatter, which stores results in Java memory.

The contents of the CSV results files generated by PLANitOutputFormatter can be compared to the standard results CSV files, and the values stored in Java memory objects are compared to those stored in MemoryOutputFormatter.

The XML results files generated by PLANitIO contain details of the output configuration used in the test, the output CSV files storing the results and the columns included in the output files. But they also contain a timestamp which gives the date and time of the PLANitIO run. This timestamp should never be the same for any two different runs. The tests must check that all the contents of the created XML file except the timestamp are the same as those in the standard results file, but the value of the timestamp in the generated file must always differ from that in the standard results file.

Each integration test must include:-

• an XML input file to define the inputs for the test case;
• Java code to run PLANit to read the input file, run the traffic assignment and save the results;
• CSV files to store the expected results (except for tests with invalid inputs to test PLANit’s input validation);
• XML output files;
• Java code to store the expected results in Java;
• Java code to compare the results from the test run with the expected results.

Java contains its own libraries which can read CSV files (Apache Commons CSV) and check that results from a test run match the expected values (JUnit).

The class PlanItIOTestHelper.java contains common code which is used to set up and run PLANit for these tests. This class defines:-

• default configuration methods which are used in many of the tests but not all of them;
• utility methods which are used for common file actions (e.g. comparing the contents of one file with another, deleting CSV results files after the test has finished with them).

The class PlanItIOIntegrationTest contains the individual test cases described in Section 3.

2.3 Input File Formats

Each test case has an XML input file called macroscopicinput.xml which contains its input data in the standard XML format (using the , and elements.

2.4 Standard Results Files – Naming Convention

CSV standard results files use the naming convention:

_Time Period _.csv

Where:

is one of “Link”, “Path” or “Origin-Destination”

is the appropriate time period number for the assignment execution

is whatever was defined as the maximum number of iterations in the test (for most tests this is either 1 or 500).

Note that Origin-Destination results files use an iteration number one below those for links and paths. This is because link costs are recalculated at the start of each iteration, using flows from the previous iteration.

So typical CSV output file names include Link_Time Period 1_500.csv,Origin-Destination_Time Period 2_1.csv etc.

XML output files follow a similar naming convention but do not include the number of iterations in their title (number of iterations is included in their content anyway). So the equivalent names to the above examples would be Link_Time Period 1.xml and Origin-Destination_Time Period 2.xml.

If a test case has more than one time period it will produce more than one set of output files. Several tests have three time periods; for these tests the generated results for all time periods are checked.

2.5 Storing Expected Results in Memory – Link Output

Tests of the contents of the MemoryOutputFormatter use data transfer objects (DTOs). These objects are populated with expected result values in the Java code, and then stored in Java Maps. After the traffic assignment run has finished, the values stored in the MemoryOutputFormatter can be compared to these standard results in the code.

Expected results for link output are stored in ResultDto objects. A ResultDto object is populated by its constructor call, which has the following arguments:

• startNodeId XML id of start node (used to define the link segment);
• endNodeId XML id of end node (used to define the link segment);
• linkFlow flow through link (output);
• linkCost cost (travel time) of link (output);
• capacity capacity of the link (input);
• length length of the link (input);
• speed travel speed of the link (input).

ResultDto objects are stored in a Java Map whose keys are run id, time period and mode. Test cases may have more than one run, time period or mode so this allows all the results to be stored for each.

The fifth argument in the ResultDto constructor, total cost to end node, is not currently used. This was originally included to provide a sort order for the ResultDto objects in the Map which stores them. This is quite helpful for human inspection of CSV output files but is not required for computerized testing.

2.6 Storing Expected Results in Memory – Path Output

2.7 Storing Expected Results in Memory – Origin-Destination Output

3 Test Cases

3.1 Explanatory Test

3.1.1 test_explanatory

Purpose:

This test corresponds to the basic example included in the ReadMe.Md file for the PLANitIO project. This illustrates the input and output files for the simplest possible network, one link with a demand of one unit across it. This allows modellers to relate the introductory example in the documentation to a real test case.

Description:

The link has a demand of 10 units from Node 1 to Node 2. The road is single-lane, one-way from Node 1 to Node 2. The capacity of the link is 2000.

Location:

src\test\resources\testcases\explanatory\xml

All input and output files are in this directory.

Notes:

This test has turned out to be very useful because it is so trivial. If you make a code change which causes this test to fail, you know you have made a mistake or mistype in your code changes to cause the failure. And it is usually easy to trace back to the coding error which causes a wrong result in this case.

3.1.2 test_explanatory_attempt_to_change_locked_formatter

Purpose:

This test validates that PLANit’s formatter locking validation is working.

When PLANit runs a traffic assignment execution, it locks the setup of all its output type configurations. Any attempt to add properties to or remove properties from an output type configuration after this should throw an exception. This test includes code to add properties to the link output type configuration after the traffic assignment has been run, which should generate an exception.

Description:

As “test_explanatory” above

Location:

src\test\resources\testcases\explanatory\xml

All input and output files are in this directory.

Notes:

If this test works correctly, it should run the traffic assignment once and then throw an exception. This first run will generate output files which should match “test_explanatory” above, but it should throw the exception without writing any results to the MemoryOutputFormatter.