Code coverage consists of a set of software metrics that can tell you how much of the production code is covered by a given test suite. It's purely quantitative, and does not say anything about the quality of either the production code or the test code. That said, the examination of code coverage reports will sometimes lead to the discovery of unreachable code which can be eliminated. But more importantly, such reports can be used as a guide for the discovery of missing tests. This is not only useful when creating tests for existing production code, but also when writing tests first, such as in the practice of TDD (Test Driven Development).
JMockit Coverage provides two different and complementary code coverage metrics: line coverage and path coverage. An example coverage report showing both metrics can be found here.
The line coverage metric tells us how much of the executable code in a source file has been exercised by tests. Each executable line of code can be uncovered, covered, or partially covered. In the first case, none of the executable code in it was executed at all. In the second, all of the code was fully executed at least once. In the third case, only part of the executable code in the line was executed. This can happen, for example, with lines of code containing multiple logical conditions in a complex boolean expression. JMockit Coverage identifies all three cases, computing the coverage percentage for each executable line of code accordingly: 0% for an uncovered line, 100% for a covered line, or some value in between for a partially covered line.
A branching point exists wherever the program makes a decision between two possible execution paths to follow. Any line of code containing a logical condition will be divided in at least two executable segments, each belonging to a separate branch. An executable line of source code with no branching points contains a single segment. Lines with one or more branching points contain two or more executable segments, separated by consecutive branching points in the line.
Lets say that NS >= 1 is the number of executable segments on a
given line.
If NE is the number of segments in that line which were executed at
least once during a test run (ie, they are covered segments), then we can calculate the
coverage percentage for the line as 100 * NE / NS.
Similarly, the line coverage percentage for a whole source file is calculated from the total number of executable segments and the total number of covered segments, considering all executable lines of code in the file. The percentage for a package, in turn, is calculated from the total and covered numbers of segments in the whole set of source files belonging to the package. Finally, the total code coverage percentage is computed by the same formula on the totals for all packages.
A completely different metric is path coverage, which is computed for method and constructor bodies, not for lines or segments of code. It tells us how many of the possible execution paths through a method or constructor, from entry to exit, have been executed at least once during the test run.
Note that each method or constructor has a single point of entry, but can have multiple exits.
An exit occurs when a return or throw statement is executed.
These are normal exits, of course. A method/constructor execution can also terminate
abruptly, by propagating an exception (or error) thrown as a result of a method call,
an attempt to access a null reference, or some other action which caused an
unintended program failure.
Each possible path can be either fully executed (covered) or not (uncovered). Paths that execute only partially (ie, they were terminated abruptly) are simply considered as uncovered.
The path coverage percentage for a method or constructor body is computed in a way
similar to the line coverage computation.
If NP is the number of possible paths through the implementation
body and NPE is the number of paths executed from entry to exit,
then the metric is computed as 100 * NPE / NP.
Also in the same way as the line coverage metric, we extend this formula to the whole source
file, the whole package, and the whole set of packages touched by the test run.
The JMockit Coverage tool can generate the following types of output:
For the first two types of output, there is optional "call point" information which may be included or not, as selected by the user. A call point is the point in the source test code from which an specific line of production code was exercised. Note that generating coverage with this extra information takes more time and produces significantly larger output. On the other hand, it can be useful to know which lines of test code caused a given line of production code to be executed during the test run. The XHTML report, in particular, makes this information easily viewable, though hiding it at first.
First of all, to enable the JMockit Coverage tool at least one of several coverage jars
needs to be added to the classpath.
The main jar file, which contains all of the tool implementation code, is
jmockit-coverage.jar.
Adding any of the other jmockit-coverage-xyz.jar files to the classpath will result
in having this one added implicitly (provided all jars are available in the same installation
directory).
Whatever the coverage jar(s) used, jmockit.jar (or a versioned jar obtained from the
Maven repository) does
not need to be added to the classpath, unless of course the suite contains tests which
use the JMockit API (it will also be implicitly added to the classpath).
Apart from enabling JMockit Coverage for a given test run, there are four aspects of the tool's behavior which can optionally be configured for the test run:
This mechanism applies when the -javaagent:jmockit.jar JVM parameter is not
used for running tests with JMockit. As such, it's only valid when running on JDK 1.6. (See the
relevant chapter for details.)
To select the desired output format(s) to be generated for a test run, add one or more of the following jar files to the classpath:
jmockit-coverage-htmlbasic.jar:
generates an XHTML report without call points.
jmockit-coverage-htmlfull.jar:
generates an XHTML report with call points.
jmockit-coverage-merge.jar:
generates a merged data file of name "coverage.ser".
The "htmlbasic" and "htmlfull" jars are mutually exclusive.
However, it is valid to have both jmockit-coverage-merge.jar and one of the
two "html" jars in the classpath at the same time.
In such a case, at the end of the test run both a "coverage.ser" file and a "coverage-report"
directory (unless another is specified) will be written.
Each one of these output specification jars is empty except for the standard
META-INF/MANIFEST.MF file.
Still, they depend on jmockit-coverage.jar and this is specified
through the Class-Path property inside MANIFEST.MF.
Consequently, if this last jar file is available in the same directory then it does not need to
be explicitly added to the classpath; otherwise, it (or an equivalent jar from the Maven
repository) will need to be added.
If none of the output specification jars is added to the classpath, but
jmockit-coverage.jar is, then the "htmlbasic" report will be generated.
To have control over all aspects of the code coverage tool, set one or more
system properties for the JVM instance that will run the test suite.
Note that you should be able to do this inside an Ant target, a Maven surefire
plugin configuration, or a test run configuration for your Java IDE of choice, using either
JUnit or TestNG.
The available system properties are:
jmockit-coverage-output: one or more comma-separated values
between html, html-nocp,
and merge.
The "nocp" suffix stands for "no call points".
The use of this system property takes precedence over the presence of coverage jars in the
classpath.
jmockit-coverage-outputDir: absolute or relative path to the
output directory, to be used for writing any "coverage.ser" or "index.html" files (plus the
remaining ".html" files of the XHTML report, in automatically created sub-directories).
jmockit-coverage-srcDirs: comma-separated list of Java source
directories to be searched when generating an XHTML report.
Each directory is specified by an absolute or relative path.
jmockit-coverage-classes:
a java.util.regex-conformable regular expression which will be used to select
the classes from production code which are considered for coverage.
jmockit-coverage-excludes:
a java.util.regex-conformable regular expression for class names which should be
excluded from consideration when deciding which classes will be modified to gather coverage.
This property can be used together with jmockit-coverage-classes or on its own.
-javaagent
Another way to have full control over the code coverage tool is to use a JVM command line with
-javaagent:jmockit.jar=coverage=<Coverage arguments> when running the test suite.
Note that you should be able to do this inside an Ant target, a Maven surefire
plug-in configuration, or in a test run configuration for your Java IDE of choice.
The "Coverage arguments" part consist of up to four different arguments, in the
format classSelectionRegex:outputFormat:outputDir:srcDirs, as follows.
Each of these four arguments can be left unspecified by leaving it empty in the full string.
After the last non-empty argument, the ":" characters can be omitted.
classSelectionRegex specifies the classes to be selected for coverage;
if empty, coverage will be gathered for all production classes loaded during the test which
are not inside jar files.
outputFormat specifies the format for the coverage output, and should be
one or more of: html, html-nocp,
merge ("nocp" stands for "no call points"); more than one value
can be specified by separating them with a comma; if no value is specified,
html-nocp is assumed by default.
outputDir specifies the destination directory for coverage output;
if empty, the current working directory is assumed, with a "coverage-report"
subdirectory being created for the XHTML report (if not already existing).
srcDirs is only applicable to the XHTML report, and is a comma-separated
list of Java source directories;
if empty, all "src" directories under the current working dir are automatically found
and searched for ".java" sources.
The classSelectionRegex argument should be a
java.util.regex-conformable regular expression specifying the classes and/or
packages in production code for which coverage is desired.
For example, "orderMngr.domain.*" selects all classes in the orderMngr.domain
package as well as in any sub-packages.
Note that both types ("merge" and XHTML) of output can be specified; if both "html"
and "html-nocp" are specified, the first occurrence will take precedence.
When one of these four arguments is left unspecified, the equivalent
jmockit-coverage-xyz system property will be respected, if specified.
If the argument is specified, though, it will take precedence over the property.
Suppose you have multiple test run configurations (in separate Ant targets, for example), and you want to generate a single XHTML report for the production code covered by the full set of tests. Normally, when JMockit Coverage generates a report at the end of a test run, it overwrites any previous report that may have been generated in the same output directory. So, instead of getting an aggregate report as desired, you would end up with the report for the last test run only (assuming all test runs used the same working dir). Here is where the third type of output, the merged data file (coverage.ser), comes in.
As seen in other sections, we activate the generation of this file by either adding
jmockit-coverage-merge.jar to the classpath or by using a JVM command line such as
"-javaagent:jmockit.jar=coverage=:html,merge".
At the end of any test run, and before any XHTML report is generated, an existing "coverage.ser" file in the current working dir will be read, with the coverage data in it merged with the current in-memory coverage data. The modified in-memory data is then used in the generation of the XHTML report, if it is enabled for the test run.
Note that these separate test runs may be for the same test suite, or for different test suites. The only thing in common between all test runs, if a single XHTML report is desired from them, is that the working directory should be the same. Otherwise, the shared "coverage.ser" file won't be found and a new "coverage-report" directory will be created. If different test runs must use different working dirs (perhaps because the source directories are not all under the same top-level directory, or for some other reason), then the "coverage.ser" file will have to be provided in each working dir by whatever means are available (for example, by moving the file through an Ant task, or by creating a link in the file system).
If you run tests with the "test" Maven goal, then you will need the following dependency in
pom.xml:
<dependency>
<groupId>mockit</groupId>
<artifactId>jmockit-coverage</artifactId>
<version>${jmockit.version}</version>
<scope>runtime</scope>
</dependency>
Notice that it will probably be useful to define a property for the JMockit version in use:
<properties>
<jmockit.version>0.995</jmockit.version>
</properties>
In Maven 2, the surefire plugin is the one usually responsible for actually running
tests.
To configure JMockit Coverage through JVM parameters, add something like the following:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>
-javaagent:"${settings.localRepository}"/mockit/jmockit/${jmockit.version}/jmockit-${jmockit.version}.jar=
coverage=:html:./target/coverage-report
</argLine>
</configuration>
</plugin>
A more convenient way to specify the output directory for files generated by JMockit Coverage
("coverage.ser", "index.html" and other XHTML pages) is to configure surefire with
the jmockit-coverage-outputDir system property (which goes inside the
configuration element shown above):
<systemProperties>
<property>
<name>jmockit-coverage-outputDir</name>
<value>target/my-coverage-report</value>
</property>
</systemProperties>
To facilitate switching on/off the generation of code coverage output, the read-only status of the relevant file is checked by JMockit at startup. The relevant file, always in the working directory, is "coverage.ser" for serialized output and "coverage-report/index.html" for XHTML output. (Note that the working directory can usually be selected separately for each test run configuration in the Java IDE.) If said file is currently marked as read-only in the OS, then the corresponding output is not generated for the next test run.
Notice that a Java IDE usually provides an easy mechanism to toggle the read-only status of a project file. In IntelliJ IDEA it is done by double clicking the status bar, with the desired file opened in the editor. In Eclipse there is a "Read only" check box in the "Properties" screen for the text file selected in the editor; this screen can be opened by typing "Alt + Enter".
jmockit-tools system property
Another way to switch coverage on/off is to use the jmockit-tools
system property, which can specify the "startup tools", between those available in the classpath
(if any), to be loaded by JMockit at startup.
For the coverage tool, the word "coverage" must be present in the value
specified for the property (which can contain two or more tool names separated by commas).