The JMockit Testing Toolkit

JMockit is open-source software licensed under the MIT License. It is a collection of tools and APIs for use in developer testing, that is, tests written by developers using a testing framework such as JUnit or TestNG. The tools rely on the Java 5 SE instrumentation feature (the java.lang.instrument package), internally using the ASM library to modify bytecode at runtime. Therefore, tests using JMockit must be run under a Java 5+ SE JVM.

The following links provide a short overview of the different tools and APIs in the toolkit. The APIs (first three items) can be used for writing unit and integration tests. They enable the isolation of code under test from its dependencies, no matter what form they take or how they are obtained at runtime. The last item is a separate tool delivered in its own jar file, which can optionally be used while running test suites. The figure below is a clickable map with links to detailed documentation pages.

JMockit Expectations
JMockit Verifications
JMockit Mockups
JMockit Coverage

Motivation

This toolkit was created mainly as an attempt to overcome certain limitations found in "conventional" mocking tools. Another goal was to provide simpler and more succinct APIs for writing developer tests. In addition, and differently from other testing tools which specifically target the use of mocks, JMockit also includes other tools designed to support the creation of large test suites.

Conventional tools for mock objects

The JMockit approach is an alternative to the conventional use of "mock objects" as provided by tools such as EasyMock and jMock.

Both of those tools are based on java.lang.reflect.Proxy, which requires an interface to be implemented. Additionally, they support the creation of mock objects for classes through CGLIB subclass generation. Because of that, said classes cannot be final and only overridable instance methods can be mocked.
Most importantly, however, when using these tools the dependencies of code under test (that is, the objects of other classes on which a given class under test depends) must be controlled by the tests, so that mock instances can be passed to the clients of those dependencies. Therefore, dependencies can't simply be instantiated with the new operator in a client class for which we want to write unit tests.

Ultimately, the technical limitations of conventional mocking tools impose the following design restrictions on production code:

  1. Each class which may need to be mocked in a test must either implement a separate interface or not be final.
  2. The dependencies of each class to be tested must either be obtained through configurable instance creation methods (factories or a Service Locator), or be exposed for dependency injection. Otherwise, unit tests won't be able to pass mock implementations of dependencies to the unit under test.
  3. Since only instance methods can be mocked, classes to be unit tested cannot call any static methods on their dependencies, nor instantiate them using any of the constructors.

Design considerations for production code

The problem with imposing restrictions on the design of production code is that there are good reasons for making classes and methods final, for directly obtaining instances of collaborators with the new operator, and for using APIs based on static methods. There is nothing inherently wrong with using these three Java language keywords, after all.

In the first case, declaring classes or methods final makes it clear that they are not intended for extension by subclassing. For example, in a correct application of the Template Method design pattern, the "template" method itself (which implements an algorithm calling one or more overridable methods) must not be overridable. Also, from the point of view of API design, final classes are the only ones which can be safely evolved. For more on this, see this article.
In practice, most classes and methods in an application or even in a reusable class library are not designed with extension by subclassing in mind, so it makes sense to declare them as final; probably that's why most other OO languages (C++, C#, etc.) make all methods non-overridable by default.
Another benefit of making Java classes or methods final is that a good static analysis tool will be able to provide more useful feedback. The code inspections associated with the use of final in IntelliJ IDEA, for example, led me numerous times to simplify and even eliminate unused parts of the code base, when developing large business applications. (For the curious, some of the relevant inspections are: "Class structure: 'protected' member in 'final' class", "Declaration redundancy: redundant throws declaration", and "Initialization issues: overridable method call during object construction".)
For an authoritative discussion on design for extension (which defends the judicious use of final for classes and methods), see "Item 17: Design and document for inheritance or else prohibit it" in the Effective Java book. Another book which strongly recommends the use of final for classes and methods is Practical API Design (see "Make Everything Final", in chapter 5). Yet another relevant book is API Design for C++, which is also largely applicable to Java (see sections "2.3.2 Add Virtual Functions Judiciously" and "4.6.3 Using Inheritance"). The site for this last book points to several interesting articles, including Java API Design Guidelines.

In the second case, directly instantiating dependencies with new facilitates the use of stateful objects. In OO design, objects normally are not stateless. However, when dependencies are obtained or injected through external means (such as a "Service Locator" or a DI framework) there is a tendency to make them stateless, and with a single global instance. Such a practice leads to code that is more procedural and less object oriented. Of course, this applies to dependencies whose interfaces have only one implementation in production code, or when the selection of an implementation between many would not benefit from external configuration (for example, people typically instantiate List implementations directly, even though they code to the abstract interface only); situations where one implementation must be selected through external configuration tend to be quite rare (at least in my experience).
Related to this issue, I think, there is an widely observed misunderstanding of just what the phrase Program to an interface, not an implementation (from the famous "Design Patterns" book) actually means. Many seem to think that one should create a new separate Java interface (or abstract class) for any concrete class which don't yet implement one. In reality, it was only meant as a recommendation to avoid declaring variables, fields, parameters or return types as the implementation type when an abstract type already exists (for example, don't declare variables of type ArrayList if all you need is a List). The "Effective Java" book mentioned above also discusses this topic, in "Item 52: Refer to objects by their interfaces".

In the third case, the use of classes containing only static methods is the best choice when none of the methods operate on any state (for example, the Math class), or the actual state is stored in a separate context object, such as the HTTP request context or the persistence context. For this last situation, consider an application that has a persistence subsystem which provides access to a relational database. One possibility is to inject instances of work unit objects (an Hibernate Session, or a JPA EntityManager) wherever they are needed, and use the persistence API directly. A better approach, in my experience, is to use a static facade which encapsulates all access from the application to the persistence subsystem. The benefits of this are many, not the least of which is that it actually provides more flexibility than the more direct, instance-based, approach.
Another recommended usage of static methods is presented in "Item 1: Consider static factory methods instead of constructors", again in the "Effective Java" book.

Testability

The limitations found in conventional mocking tools has come to be associated with the idea of "untestable code". Often, we see the restrictions resulting from those limitations considered as inevitable, or even as something that could be beneficial. The JMockit toolkit, which breaks away from these limitations and restrictions, shows that in fact there is no such thing as truly untestable code. There is, of course, code that is harder to test because it is too complicated and convoluted, lacks cohesion, and so on and so forth.

Therefore, by eliminating the technical limitations traditionally involved in the isolation of an unit from its dependencies, we get the benefit that no artificial design restrictions must be imposed on production code for the sake of unit testing. Additionally, it becomes possible to write unit tests for legacy code, without the need for any prior adaptation or refactoring. In short, with a less restrictive mock testing tool the testability of production code becomes much less of an issue, and developers get more freedom in using Java language features, as well as more design choices.

Another way of thinking about testability is to differentiate between intrinsic and extrinsic testability. In the first case, we can conclude that whatever makes the code more or less maintainable also makes it more or less easily testable, and vice-versa. For example, methods with higher cyclomatic complexity (basically, the number of different execution paths through the method) require more tests to be fully covered, while at the same time being more difficult to understand and change; in other words, intrinsic testability is not particularly useful as a separate measure for code quality, if it is nothing more than maintainability. In the second case, extrinsic testability, we are not really considering properties of the production code, but instead properties of the tools used to test that code. So, if a tool for testing code in isolation can provide an easy to use API that is able to deal with all possible isolation scenarios, then such extrinsic concerns for testability completely disappear.

Design considerations for mocking APIs

The APIs for testing with mocks that are available in existing toolkits, even the more recent ones, have certain undesirable characteristics. Of course, this is partly a matter of personal taste, but some objective observations can certainly be made.

In the following discussion, five different mocking toolkits (besides JMockit) are considered, all in their latest stable releases as of May 9, 2011: EasyMock 3.0, jMock 2.5.1, Mockito 1.8.5, PowerMock 1.4.9, and Unitils 3.1. The code snippets below come from the sample JMockit test suites that compare the JMockit approach with those other toolkits.

There are other less dramatic differences in style between JMockit and other mocking APIs, but the above items should be enough to show that many possibilities exist in the realm of mocking API design.

Alternative mocking tools

There are now other mocking tools for Java which also overcome the limitations of the conventional ones, between them PowerMock, jEasyTest, and MockInject. The one that comes closest to the feature set of JMockit is PowerMock, so I will briefly evaluate it here (besides, the other two are more limited and don't seem to be actively developed anymore).

JMockit vs PowerMock

Another recent mocking tool is Mockito. Although it does not attempt to overcome the limitations of older tools (jMock, EasyMock), it does introduce a new style of behavior testing with mocks. JMockit also supports this alternative style, through the Verifications API.

JMockit vs Mockito

Finally, the JMockit Testing Toolkit has a wider scope and more ambitious goals than other mocking toolkits, in order to provide a complete and sophisticated developer testing solution. A good API for mocking, even without artificial limitations, is not enough for productive creation of tests. An IDE-agnostic, easy to use, and well integrated Code Coverage tool is also essential, and that's what JMockit Coverage aims to provide.

JMockit Expectations

The JMockit Expectations mocking API provides a record-replay model for writing behavior-based tests. In this model, a test begins by setting one or more expectations on the invocations made from code under test to its collaborators (dependencies). The classes and instances for such dependencies are established through the declaration of one or more mocked types inside the test class/method. Such mocked types can be declared through instance fields of the test class or of an Expectations anonymous subclass inside a test method, and also through parameters of test methods (even though JUnit and TestNG don't allow regular test methods to have parameters). After expectations are defined in this recording phase, the test transitions to the replay phase, when the code under test is exercised. The invocations that actually occur on the mocked collaborators are handled according to the mocked type declarations and the corresponding expectations recorded on them (if any). At the end of the replay phase, those expectations for which one or more invocations were expected are automatically verified so that missing invocations can be detected.

Expectations can be specified on any kind of method invocation (on interfaces, abstract classes, concrete final or non-final classes, and on static methods), as well as on class instantiation through any constructors. Private methods/constructors can also have expectations defined.

This API provides several ways to specify argument matching constraints on invocations, including the use of custom Hamcrest matchers. There are special methods and fields through which values to return or exceptions to throw can be specified for all kinds of mock invocations. Other API fields can be used to specify lower and/or upper limits on the number of expected and/or allowed invocations, respectively, for a given expectation. 

public class JMockitExpectationsExampleTest
{
   // Common mock fields can be declared here, and must be annotated with @Mocked.
   
   @Test
   public void testDoOperationAbc()
   {
      new Expectations() {
         // This is a local mock field; it can optionally be annotated with @Mocked.
         DependencyXyz mock; // a mocked instance is automatically created and assigned

         {
            new DependencyXyz(); // records an expectation on a constructor invocation
            mock.doSomething("test"); result = 123;
   
            // The expectations above are strict, causing the whole dependency to be strictly verified.
            // Therefore, invocations not recorded here will be considered unexpected, causing the test
            // to fail if they occur while exercising the code under test.
            // Non-strict expectations are also supported.
         }
      };

      // In ServiceAbc#doOperationAbc(String s): "new DependencyXyz().doSomething(s);"
      Object result = new ServiceAbc().doOperationAbc("test");

      assertNotNull(result);

      // That all expected invocations were actually executed in the replay phase is automatically
      // verified at this point, through transparent integration with the JUnit/TestNG test runner.
   }
}
Tutorial sample Tutorial chapter API documentation
Sample for comparison with EasyMock Sample for comparison with jMock

JMockit Verifications

This API is a natural extension of the Expectations API, where the record-replay model gets an extra phase and becomes the record-replay-verify model for behavior-based testing.

By default, all expectations recorded inside an Expectations block are strict, which means that for each expectation recorded a matching invocation must be executed during the replay phase of the test, and in the same order as recorded. Additionally, if during the replay phase an unexpected invocation (ie, one not matching a recorded strict expectation) to one of the mocked types/instances is detected, then an assertion error will be thrown. (The number of invocations for each strict expectation is flexible, though, since lower/upper limits can be specified individually.)

Inside an Expectations block, however, individual expectations can be marked as being non-strict. When all invocations to a given mocked type should be non-strict, the @NonStrict annotation can be used. Finally, if all expectations to be recorded (on one or more mocked types/instances in scope for the same test) should be non-strict, the NonStrictExpectations class can be used instead of Expectations.

A non-strict expectation, then, is one that by default can be invoked any number of times (including zero) during the replay phase, and in arbitrary order. Similarly, any unexpected invocation to a non-strict mocked type/instance does not cause an error to be thrown.

In this model, invocations to non-void methods that occur in the replay phase and that need a return value different from the default one must still be recorded, as they normally would need if the expectations were strict. All other invocations, on the other hand, can be verified to have occurred (or not) after the replay phase, in the same place regular JUnit assertions would be. This is done in the verification phase of the test, which is demarcated by a Verifications block (or a variant, such as VerificationsInOrder). It's also possible to have no expectations recorded for a given test, in which case there will be no expectation block in the test method, but only the verification block. 

public class JMockitVerificationsExampleTest
{
   @Test // notice the "mock parameter", whose argument value will be created automatically
   public void testDoAnotherOperation(final AnotherDependency anotherMock)
   {
      new NonStrictExpectations() {
         DependencyXyz mock; // mock instance created and assigned automatically

         {
            mock.doSomething("test"); result = 123; times = 1;
         }
      };

      // In ServiceAbc#doAnotherOperationAbc(String s): "new DependencyXyz().doSomething(s);"
      // and "new AnotherDependency().complexOperation(1, obj);".
      new ServiceAbc().doAnotherOperation("test");

      new Verifications() {{
         anotherMock.complexOperation(anyInt, null);
      }};
   }
}
Tutorial sample Tutorial chapter API documentation
Sample for comparison with Mockito Sample for comparison with Unitils Mock

JMockit Mockups

This is a different kind of mocking API, which can be seen as complementary to the Expectations & Verifications API. Instead of specifying expectations on the invocations made from code under test to its collaborators, mock classes are defined and applied for the scope of a single test method or a whole test class. Inside these mock classes, mock methods (indicated as such with the @Mock annotation) are directly implemented with code that will be executed instead of the original code for the corresponding mocked method/constructor. In addition, constraints on the number of expected invocations for each mock method can be specified, and mock methods can be made to be re-entrant. 

public class JMockitMockupsExampleTest
{
   @Test
   public void testDoOperationAbc()
   {
      // A "mock-up" class, defined and applied at the same time:
      new MockUp<DependencyXyz>()
      {
         @Mock(invocations = 1)
         int doSomething(String value)
         {
            assertEquals("test", value);
            return 123;
         }
      };

      // In ServiceAbc#doOperationAbc(String s): "new DependencyXyz().doSomething(s);"
      Object result = new ServiceAbc().doOperationAbc("test");

      assertNotNull(result);
   }
}
Tutorial sample Tutorial chapter API documentation

JMockit Coverage

Existing OpenSource code coverage tools, such as Cobertura and EMMA are also less than ideal. Specifically, JMockit Coverage provides the following benefits.

  1. Bytecode modification performed only at runtime, therefore avoiding the creation of undesirable files. No extra source or class files are created, and no coverage data file is generated unless explicitly requested. The only files created or modified are those that the user really wants as the desired output.
  2. Running tests with JMockit Coverage does not require the use of any particular command line script, Ant task, or IDE plug-in. The tool applies the idea of convention over configuration, trying to make it as easy as possible for the developer to run tests with code coverage: by simply adding one jar file to the classpath, or by specifying certain initialization parameters to the JVM (plus having the main coverage jar file in the classpath).
  3. No need to specify which classes should be considered for coverage (it still can be specified if needed, though). All code under test will automatically be analyzed for coverage. Specifically, the tool will gather coverage data for all production code executed by the test suite, excluding classes defined inside jar files (on the assumption that such code belongs to libraries used by code under test).
  4. A serialized output file and/or an HTML report can be generated at the end of each test run with no extra configuration beyond adding the relevant jar files to the classpath (at least one jmockit-coverage-xyz.jar file). For the generation of the HTML report, Java source files are automatically searched in all "src" directories under the working directory.
  5. In code coverage reports, each line of production code can be accurately linked to the lines in test code which caused their execution, for each individual execution of each line of code. (This feature is optional, however, because of the higher runtime cost and the larger resulting HTML report.)
  6. Besides a line coverage metric, a path coverage metric is also made available. Both metrics are calculated and shown at the source file, package, and global levels. For path coverage, each possible path through a method or constructor can be interactively displayed in the HTML report.
  7. Intra-line coverage is provided, with the appropriate red/green coloring in the HTML report for each conditionally executed line segment.
Running tests with JMockit Coverage Sample coverage report