Plugin Test Projects

Overview

Eclipse plugins should be developed as seperate eclipse projects with the plugin id as project name. To ensure a good code quality it is important to provide unit tests for the plugins as for any other java project. However, tests are not part of the software itself and for this, they should not be shipped with the plugin itself. A good practice to handle this is to place the JUnit tests within a separate eclipse projects named as the plugin project with the suffix ".tests". For Palladio development, we also use plugin tests for automated execution during Palladio Nightly builds.

This article describes best practices for such test projects and how to integrate them into the Palladio Build Infrastructure.

Test Project Setup

Naming Convention

As mentioned before, test projects should be named like the plugin project they provide tests for, suffixed with ".tests" ("<<PluginID>>.tests").

Fragment Projects

New Fragment Project Wizard

Eclipse plugins are OSGi Bundles and provide a clear interface by explicitly defining the java packages accessible from other plugins / projects. To be able to provide unit tests also for packages and classes which are not published by the OSGi Bundle interface description, fragments have been introduced as a specific type of plugins. They are not standalone projects but associated to a specified host plugin. Such a host is specified in the manifest file with a property such as

Fragment-Host: de.uka.ipd.sdq.stoex.analyser

The above code is used in the de.uka.ipd.sdq.stoex.analyser.tests plugin indicating tests that are specified for the de.uka.ipd.sdq.stoex.analyser plugin.

JUnit Tests

JUnit has become a defacto standard for writing automatic tests for your java code (http://www.junit.org/).

A test case is specified for a single java class. They are java classes themselves. As described further down in the context of the Eclipse "New Junit Test Case" wizard, it is recommended to name the Test class like the class under test suffixed by the string "Test" and to place it inside the same package but in a separate test project. Within a test case, several tests can be implemented as test methods. Since JUnit 4, test methids are annotated with @Test. In theory, test methods can contain arbitrary code, but it should be related to the class under test such as the behaviour of an algorithm implemented by this class for specific parameters.

Creating an java.lang.AssertionError within a test method can be used to identify unexpected or unintended test results. JUnit contains the class org.unit.Assert to provide specific tests and queries which create an AssertionError if necessary:

assertEquals(...): Checks the equality respectively the similarity of fields and objects (probably by using a given tolerance range)
assertSame(...): Ensures that two object references point to the same object instance.
assertNotSame(...): Ensures that two object references do not point to the same object instance.
assertTrue(...): Checks that a given boolen expression results to true.
assertFalse(...): Checks that a given boolen expression results to false.
assertNotNull(...): Checks that an Object is not null.
assertNull(...): Checks that an Object is null.
fail(): Creates an AssertionError.

For example having a calculator class such as

package com.example.junit;

public class Calculator(){
  public int sum(int a, int b){
    return a+b;
  }
}

one can write a test such as

package com.example.junit;

import static org.junit.Assert.*;
import org.junit.Test;

public class CalculatorTest(){

  @Test
  public void testSum(){
    Calculator cal = new Calculator();
    int result = cal.sum(2,3);
    assertTrue("Addition failed",5, result);
  }
}

The test case can be automatically executed and it ensures that the calculator is able to sum up the integer values 2 and 3 and proves the result to be equal 5. One can imagine that such test cases can become arbitrary complex to validate a class.

JUnit Eclipse Integation

New Junit Test Step 1

New Junit Test Step 2

JUnit is integrated in the Eclipse Java Development Toolkit (Eclipse JDT). The integration provides a wizard to create new Test classes.

Right click on your test project and choose New -> JUnit Test Case
Activate the radio button "New JUnit 4 Test"
In the dialog, choose the "Class under test".
Provide a name for the test class which is the same as the class under test suffixed with the string "Test". (Not necessary for JUnit 4 but a naming convention for more intuitive code)
Set the package to the same package as the class under test. This is possible because we are working in the separate fragment test project.
Click the next button and select the methods of the class under test to generate test methods for.
Click finish and the skeleton for your new JUnit test is generated.

If you are interested in more details about test driven development and unit tests we recommend to take a look at the JUnit documentation and the wikipedia article about test-driven development as a starting point.

Standard JUnit Tests vs. Plugin JUnit Tests

It is always recommended to write your code as well as your Unit tests as independent from a running eclipse plugin environment as possible. The major advantage of this is the test performance. As long as you don't need to initialize an eclipse environment to run a plugin junit test, this will dramatically speed up your test cycle, and thus let your run your tests more frequently and also improve your overall velocity.

Nevertheless, try to write clean code and do not start developing workarounds to try to prevent any plugin junit tests. Sometimes you really need them.

Logging

Logging is an already established technique, available and configurable in productive environments and there is no need to use System.out.println() statements anymore. The most mature and recommendable logging infrastructure is the apache log4j framework. However, a big issue is that there might be no initialized logging environment when running your JUnit tests. Your code might run in production and logging is provided by the Eclipse environment, but in your JUnit test run, you might end up with an exception in your console output such as

log4j:WARN No appenders could be found for logger (org.splevo.diffing.MatchEngineDiffingService).
log4j:WARN Please initialize the log4j system properly.

To prevent this and get a valid logging output you could initialize a basic logger in your test setup method ("@Before" annotation in JUnit4). Log4j provides the BasicConfigurator class to do this within two simple lines of code:

/**
 * Prepare the test.
 * Initializes a log4j logging environment.
 */
@Before
public void setUp() {
  // set up a basic logging configuration for the test environment
  BasicConfigurator.resetConfiguration();
  BasicConfigurator.configure();
}

If you use java.util.logging and Log4J, the following code configures logging depending on a file logging.properties. A default value is used for Log4J. The file should be at the root of the project path.

	@BeforeClass
	public static void initializeLoggingInfrastructure() {
		// java.util.logging
		// see also http://openbook.galileocomputing.de/java7/1507_20_001.html
		try {
			System.setProperty( "java.util.logging.config.file", LOG_PROPERTIES_FILENAME );
			LogManager.getLogManager().readConfiguration();
		} catch (IOException e) {
			String msg = "Failed to initalizes logging infrastructure. File not found: " + LOG_PROPERTIES_FILENAME + ". Continuing with default setting.";
			logger.severe(msg);
		}
		// log4j
		BasicConfigurator.resetConfiguration();
		BasicConfigurator.configure();
		org.apache.log4j.Logger.getRootLogger().setLevel(Level.WARN);
	}

For further notes about logging in eclipse plugins such as more compact logging output, see: http://www.bar54.de/blog/2012/08/logging-in-eclipse-plugins-with-junit-support/

Project Publishing

SVN Commit

The test project should be placed in the same svn directory as the plugin project under test.

The svn directory must be configured in the according Jenkins build job to ensure Jenkins checks out the project during the build run. To do this, add the absolute svn path of the project to the checked out modules in the job configuration. Ensure that the repository URL contains the absolute path including https as protocol and verify that the input field for the local module directory is empty. (see screenshot)

Register in Buckminster Project

The Palladio Build Infrastructure makes use of the Buckminster tooling to realize the eclipse plugin and update site generation. To ensure the junit tests can be executed, the test projects should be registered separatly in the Buckminster project. This is required, while they do not contribute to any feature, but need to be referable in the execution context of the Buckminster run.

To do this

check out the Buckminster project from https://svnserver.informatik.kit.edu/i43/svn/code/Palladio/Core/trunk/Build/de.uka.ipd.sdq.palladio.buckminster
Add your test fragment project to the included plugins of the feature.xml (see screenshot on the right)
Save feature.xml and commit the changes.

Test Launching

Launching JUnit tests in eclipse can be done with the JUnit run configurations. To simplify the launch configuration, JUnit runs can be specified to execute all JUnit tests within a specified directory. To use this, select the radio button "Run all tests in the selected project, package or source folder:" and search the directory you want to execute all tests from. The source directory, e.g. "src", is recommended.

To enable the build server to automatically run those tests as part of the continous integration build, the launch configuration needs to be persisted as launch configuration file. This file needs to confirm to the following naming convention: The launch configuration file has to be stored within the base directory of the project and ends with the suffix "tests.launch". To activate the persistence of the launch configuration, activate the radio button "Shared file" on the "Common" tab of the launch configuration dialog. This will automatically persist the launch configuration in the file named like the launch configuration suffixed by ".launch". It is a good practice to name the launch configuration the same as the name of your plugin project which should already end with the string ".tests" (e.g. "de.uka.ipd.sdq.simucomframework.variables.tests.launch").

The build server automatically tracks how much and which parts of the code is covered by unit tests. The open source project emma is used for this. It is not necessary but a recommended practice to install the emma code coverage plugin available for the Eclipse IDE. With this in place, you can also run the JUnit tests inside your IDE as coverage runs to check how well your code is covered by your JUnit tests. Again, this is an optional best practice to improve your development and not necessary for the execution by the Jenkins build server.

ANTLR vs. Emma Code Coverage
Generated code sometimes lead to problems with emma code coverage when executed in the Buckminster context as done in the build environment. If you face such a problem, you need to specify the code which should be instrumented by emma and which should not. This can be done by manually specifying the byte code directories to instrument by emma. This can be done in the launch configuration file using the attribute com.mountainminds.eclemma.core.INSTRUMENTATION_PATHS. The example below will force emma on the build server to instrument only byte code located inside the stoex.analyzer, stochasticexpression and stochasticexpression.tests binary projects.

<listAttribute key="com.mountainminds.eclemma.core.INSTRUMENTATION_PATHS">
<listEntry value="/de.uka.ipd.sdq.stoex.analyzer/bin"/>
<listEntry value="/de.uka.ipd.sdq.pcm.stochasticexpressions/bin"/>
<listEntry value="/de.uka.ipd.sdq.pcm.stochasticexpressions.tests/bin"/>
</listAttribute>

Check also: http://www.bar54.de/blog/2012/10/jenkinsbuckminsteremma-nosuchmethoderror-specify-packages-to-instrument/