The JMockit Testing Toolkit

JMockit is open-source software licensed under the MIT License. It is a collection of tools and APIs for use in developer testing, that is, tests written by developers using a testing framework such as JUnit or TestNG. The tools rely on the Java 5 SE instrumentation feature (the java.lang.instrument package), internally using the ASM library to modify bytecode at runtime. Therefore, tests using JMockit must be run under a Java 5+ SE JVM.

This toolkit was created mainly as an attempt to overcome certain limitations found in "conventional" mocking tools, which prevented the creation of unit tests for code designed according to well-established OO practices. Another goal was to provide simpler and more succinct APIs for writing developer tests. In addition, and differently from other testing tools which specifically target the use of mocks, JMockit also includes other tools designed to support the creation of large test suites.

Conventional tools for mock objects

The JMockit approach is an alternative to the conventional use of "mock objects" as provided by tools such as EasyMock and jMock.

Both of those tools are based on java.lang.reflect.Proxy, which requires an interface to be implemented. Additionally, they support the creation of mock objects for classes through CGLIB subclass generation. Because of that, said classes cannot be final and only overridable instance methods can be mocked.
Most importantly, however, when using these tools the dependencies of code under test (that is, the objects of other classes on which a given class under test depends) must be controlled by the tests, so that mock instances can be passed to the clients of those dependencies. Therefore, dependencies can't simply be instantiated with the new operator in a client class for which we want to write unit tests.

Ultimately, the technical limitations of conventional mocking tools impose the following design restrictions on production code:

  1. Each class which may need to be mocked in a test must either implement a separate interface or not be final.
  2. The dependencies of each class to be tested must either be obtained through configurable instance creation methods (factories or a Service Locator), or be exposed for dependency injection. Otherwise, unit tests won't be able to pass mock implementations of dependencies to the unit under test.
  3. Since only instance methods can be mocked, classes to be unit tested cannot call any static methods on their dependencies, nor instantiate them using any of the constructors.

Design considerations for production code

The problem with imposing restrictions on the design of production code is that there are good reasons for making classes and methods final, for directly obtaining instances of collaborators with the new operator, and for using APIs based on static methods. There is nothing inherently wrong with using these three Java language keywords, after all.

Use of final

In the first case, declaring classes or methods final makes it clear that they are not intended for extension by subclassing. For example, in a correct application of the Template Method design pattern, the "template" method itself (which implements an algorithm calling one or more overridable methods) must not be overridable. Also, from the point of view of API design, final classes are the only ones which can be safely evolved. Otherwise, you risk running into the fragile base-class problem. For more on this, see this article or this one.

In practice, most classes and methods in an application or even in a reusable class library are not designed with extension by subclassing in mind, so it makes sense to declare them as final; probably that's why most other OO languages (C++, C#, etc.) make all methods non-overridable by default.

Another benefit of making Java classes or methods final is that a good static analysis tool will be able to provide more useful feedback. The code inspections associated with the use of final in IntelliJ IDEA, for example, led me numerous times to simplify and even eliminate unused parts of the code base, when developing large business applications. (For the curious, some of the relevant inspections are: "Class structure: 'protected' member in 'final' class", "Declaration redundancy: redundant throws declaration", and "Initialization issues: overridable method call during object construction".)

For an authoritative discussion on design for extension (which defends the judicious use of final for classes and methods), see "Item 17: Design and document for inheritance or else prohibit it" in the Effective Java book. Another book which strongly recommends the use of final for classes and methods is Practical API Design (see "Make Everything Final", in chapter 5). Yet another relevant book is API Design for C++, which is also largely applicable to Java (see sections "2.3.2 Add Virtual Functions Judiciously" and "4.6.3 Using Inheritance"). The site for this last book points to several interesting articles, including Java API Design Guidelines.

Use of new

In the second case, directly instantiating dependencies with new facilitates the use of stateful objects. In OO design, objects normally are not stateless. However, when dependencies are obtained or injected through external means (such as a "Service Locator" or a DI framework) there is a tendency to make them stateless, and with a single global instance. Such a practice leads to code that is more procedural and less object oriented. Of course, this applies to dependencies whose interfaces have only one implementation in production code, or when the selection of an implementation between many would not benefit from external configuration (for example, people typically instantiate List implementations directly, even though they code to the abstract interface only); situations where one implementation must be selected through external configuration tend to be quite rare (at least in my experience).

Related to this issue, I think, there is a widely observed misunderstanding of just what the phrase Program to an interface, not an implementation (from the famous "Design Patterns" book) actually means. Many seem to think that one should create a new separate Java interface (or abstract class) for any concrete class which doesn't yet implement one. In reality, it was only meant as a recommendation to avoid declaring variables, fields, parameters or return types as the implementation type when an abstract type already exists (for example, don't declare variables of type ArrayList if all you need is a List). The "Effective Java" book mentioned above also discusses this topic, in "Item 52: Refer to objects by their interfaces".

Use of static

In the third case, the use of classes containing only static methods is the best choice when none of the methods operate on any state (for example, the Math class), or the actual state is stored in a separate context object, such as the HTTP request context or the persistence context. For this last situation, consider an application that has a persistence subsystem which provides access to a relational database. One possibility is to inject instances of work unit objects (an Hibernate Session, or a JPA EntityManager) wherever they are needed, and use the persistence API directly. A better approach, in my experience, is to use a static facade which encapsulates all access from the application to the persistence subsystem. The benefits of this are many, not the least of which is that it actually provides more flexibility than the more direct, instance-based, approach.

Another recommended usage of static methods is presented in "Item 1: Consider static factory methods instead of constructors", again in the "Effective Java" book.

Testability

The limitations found in conventional mocking tools has come to be associated with the idea of "untestable code". Often, we see the restrictions resulting from those limitations considered as inevitable, or even as something that could be beneficial. The JMockit toolkit, which breaks away from these limitations and restrictions, shows that in fact there is no such thing as truly untestable code. There is, of course, code that is harder to test because it is too complicated and convoluted, lacks cohesion, and so on and so forth.

Therefore, by eliminating the technical limitations traditionally involved in the isolation of a unit from its dependencies, we get the benefit that no artificial design restrictions must be imposed on production code for the sake of unit testing. Additionally, it becomes possible to write unit tests for legacy code, without the need for any prior adaptation or refactoring. The ability to create automated regression tests for existing code before refactoring it is extremely valuable, as implied by the very definition of the term. In short, with a less restrictive mock testing tool the testability of production code becomes less of an issue, giving developers more freedom to use the language and more design choices when creating new code, while facilitating the initial creation of tests for legacy code.

Another way of thinking about testability is to differentiate between intrinsic and extrinsic testability. In the first case, we can conclude that whatever makes the code more or less maintainable also makes it more or less easily testable, and vice-versa. For example, methods with higher cyclomatic complexity (basically, the number of different execution paths through the method) require more tests to be fully covered, while at the same time being more difficult to understand and change; in other words, intrinsic testability is not particularly useful as a separate measure for code quality, if it is nothing more than maintainability. In the second case, extrinsic testability, we are not really considering properties of the production code, but instead properties of the tools used to test that code. So, if a tool for testing code in isolation can provide an easy to use API that is able to deal with all possible isolation scenarios, then such extrinsic concerns for testability completely disappear.

Design considerations for mocking APIs

The APIs for testing with mocks that are available in existing toolkits, even the more recent ones, have certain undesirable characteristics. Of course, this is partly a matter of personal taste, but some objective observations can certainly be made.

In the following discussion, five different mocking toolkits (besides JMockit) are considered, all in their latest stable releases as of Aug 24, 2013: EasyMock 3.2, jMock 2.6.0, Mockito 1.9.5, PowerMock 1.5, and Unitils 3.3. The code snippets below come from the sample JMockit test suites that compare the JMockit approach with those other toolkits.

There are other less dramatic differences in style between JMockit and other mocking APIs, but the above items should be enough to show that many possibilities exist in the realm of mocking API design.

Alternative mocking tools

There are now other mocking tools for Java which also overcome the limitations of the conventional ones, between them PowerMock, jEasyTest, and MockInject. The one that comes closest to the feature set of JMockit is PowerMock, so I will briefly evaluate it here (besides, the other two are more limited and don't seem to be actively developed anymore).

JMockit vs PowerMock

Another recent mocking tool is Mockito. Although it does not attempt to overcome the limitations of older tools (jMock, EasyMock), it does introduce a new style of behavior testing with mocks. JMockit also supports this alternative style, through the Verifications API.

JMockit vs Mockito

Finally, the JMockit Testing Toolkit has a wider scope and more ambitious goals than other mocking toolkits, in order to provide a complete and sophisticated developer testing solution. A good API for mocking, even without artificial limitations, is not enough for productive creation of tests. An IDE-agnostic, easy to use, and well integrated Code Coverage tool is also essential, and that's what JMockit Coverage aims to provide.

JMockit Coverage

Existing OpenSource code coverage tools, such as EMMA, Cobertura and JaCoCo are also less than ideal. Specifically, JMockit Coverage provides the following benefits.

  1. Bytecode modification performed only at runtime, therefore avoiding the creation of undesirable files. No extra source or class files are created, and no coverage data file is generated unless explicitly requested. The only files created or modified are those that the user really wants as the desired output.
    Apart from JaCoCo, all other coverage tools require an extra build step for code instrumentation before a test suite can be executed.
  2. Running tests with JMockit Coverage does not require the use of any particular command line script, Ant task, Maven plug-in, or IDE-specific plug-in. The tool applies the idea of convention over configuration, trying to make it as easy as possible for the developer to run tests with code coverage: by simply adding one jar file to the classpath, and optionally specifying certain initialization parameters to the JVM (system properties settable with "-D" or its Ant/Maven equivalent).
  3. No need to specify which classes should be considered for coverage and which should not (by default, as they can be explicitly specified if needed). All code under test will automatically be analyzed for coverage. Specifically, the tool will gather coverage data for all production code executed by the test suite, excluding classes defined inside jar files (on the assumption that such code belongs to libraries used by the code under test).
  4. An HTML report and/or a serialized output file can be easily generated at the end of each test run. The first requires no extra configuration as it is the default form of coverage output; the second only requires the trivial effort of setting the "coverage-output" system property. For the generation of the HTML report, Java source files are automatically searched in all "src" directories under the working directory.
  5. In code coverage reports, each line of production code can be accurately linked to the lines in test code which caused their execution, for each individual execution of each line of code. (This feature is inactive by default, however, because of the higher runtime cost and the larger resulting HTML report.)
  6. Besides a line coverage metric, a path coverage metric is also made available. Both metrics are calculated and shown at the source file, package, and global levels. For path coverage, each possible path through a method or constructor can be interactively displayed in the HTML report.
  7. Intra-line coverage is provided, with the appropriate red/green coloring in the HTML report for each conditionally executed line segment.

A note: at this time, JMockit Coverage is still at a pre-1.0 version. The current version of the tool doesn't fully deliver on the last two points above.