                             SpecJava V8 Release notes
                             -------------------------


This latest version of SpecJava contains many changes to the harness
and the way the benchmarks are run, but only minor no changes to the
benchmarks themselves.

The most important change is the way the benchmarks are timed. In
previous versions of SpecJava the timing was done within each benchmark
program. This was done in an attempt to eliminate the class loading and
initialization time. We have changed this for two reasons. Firstly it
did not work at all well, usually only a few classes of each benchmark
were loaded when the timer was started, and secondly it was generally
though more realistic to include these times in any case. These effects
will of course be totally absent on the second and subsequent
iterations of a benchmark, and this provides a far more accurate run
time for the raw benchmark code.

Another major change is how I/O is performed. Previous versions of
SpecJava have had I/O modifications to allow input data to be read from
an HTTP server. This version provides a far more complete
implementation of this so that programs such as javac can now be run as
applets. In all cases the output files of the benchmarks are discarded.
This can easily be reversed but this strategy makes it easy to run all
the benchmarks as applets. In addition to this we has also provided a
way of caching the input data, so that the second and subsequent
iterations of a benchmarks will be entirely CPU bound. This is a
selectable feature. The times for all the I/O and cache organisation is
measured and recorded.

There is a crude mechanism to allow for very short runs of all the
benchmarks. This is in the form of a way to select the percentage of
the run to be performed. The intention here is for the user to be able
to select a 1% run just to get confidence that the harness is going to
work, a 10% run to check this more convincingly and then a 100% run for
real measurement. This time for this 100% run should roughly be 300
seconds on a SparcStation 20. Most, but not all, of the benchmarks
support this features. _202_jess does not at all, and _204_netmst,
_211_anagram, and _213_javac only support it partly. In some cases the
benchmarks have been lengthened in a very crude way. Our preferred way
to do this is to make a benchmark perform a bigger problem. In several
cases however we have simply made the benchmark loop repeating the same
problem to achieve the time. The next version will address this issue
more completely.

An "Auto Run" feature is present that will repeatedly run each selected
benchmark until the time of a run it less than 5% faster than the
previous run. All these run results are recorded and reported.

The entire report generation system has been replaced. Firstly all the
parameters SpecJava requires are now kept in standard Java property
files. These are loaded when the program starts up, and the files used
can be selected by the user. The intention here is make it possible to
cut down on a lot of tedious form filling by allowing the user to
define a set of standard files that contain the information that does
not normally change.  These file are created and edited using any
convenient ASCII file editor.

The reports can now only be mailed. This works fine if SpecJava is run
as a Java application on a machine with an SMTP server, or as an applet
from a server which has an SMTP server. Unlike the previous version a
careful check is make of the mail server responses to ensure that the
data is transferred okay. All the report formatting has been removed
from SpecJava and this is now done in a program that processes the
output email files afterwards. The format of these email files is quite
standard with the benchmark information being coded in property file
format for easy subsequent loading. The report screen has a button that
will switch between a brief and full view of the benchmark results, but
the full version is always transmitted.


Known Problems
--------------

All of the benchmarks in the "All" group have successfully been run
with input from a local file and from an HTTP server on some system.
Two benchmarks are included in a "Broken" group which we intend to fix
as soon as possible. These are _209_db which cannot be made to run
successfully on some platforms, and there a new benchmark
_222_mpegaudio. This is a commercial application that decompressed
audio files that confirm to the ISO MPEG Layer-3 audio specification.
This benchmark currently has some class verification problems when load
from an HTTP server, which is being addressed by its authors. It does
however run from a local disk very well.

No system has successfully run all the benchmarks at 100% execution
time in "Auto Run" mode, so don't expect everything to work perfectly
because it won't. SpecJava appears to tax all systems to breaking point
so one of the main uses for this release will be for the developers to
look at this lack of robustness.

The other known problems are as follows. There will no doubt be many more.

     The harness components are not well formatted using MS SDK 2.0 

    _208_cst hangs MS SDK 2.0 appletviewer when run at 100%.

    _218_richards_dna crashes 1.1.3 JDK on Solaris in some circumstances

    _205_raytrace, _208_cst, and _210_si cause a security exception
    with Netscape 3.01 when running the applet from the local hard
    disk.

    _213_javac Does not work with Hotjava 1.0 when the harness runs as
    an applet.



Complete list of changes
------------------------


1,  Almost all benchmarks will run for ~300 seconds on a Sparc Server 20 (using 
    current VM/JIT technology). (Some of the benchmarks do this by crude 
    iterations, V9 should do this better.)

2,  The timing of the benchmarks has changed to include class loading and 
    initialization time.

3,  The harness and all benchmarks (including javac) can be run as an applet.

4,  The input data for all benchmarks can be read from an HTTP server (or
    local files).

5,  Data input time is measured (however class load time is not).

6,  Data output is not performed at all.

7,  Caching of input data can optionally be performed so all benchmark
    re-runs can be performed without any I/O. (We are not sure if we will
    use this.)

8,  Test configuration information is now stored in standard Java
    property files. These are loaded in to the harness at run time from
    local files or URLs.

9,  Results are output as email via an SMTP server.

10, Report generation has been completely removed from the harness and
    is done in a separate program that reads the emailed result data. This
    is largely automated and produces HTML output in the required format
    for SPEC.

11, The main outcome of the above is that the harness can run in any
    Java enabled browser simply by entering the correct URL.

12, A test modes are available that will only run only 1%, or 10% of
    each  benchmark. Additionally a 1000% mode is available in case we need
    to 'change gear' just before the final SpecJava release.

13, The benchmarks can also be run as a Java application. (This may
    require using    a simple dummy SMTP server to collect the output.)

14, The individual benchmarks can be run in a "command line" mode that
    does not involve any AWT support. (This option does not produce an
    email report.)

15, Report screen has a brief and full listing. The full version is
    always mailed.

