Java measurements

This table shows 4 measurements - CPU Time, Elapsed Time, Memory and Code.

Each row shows those measurements for a particular Java program with a particular command-line input value N.

This table shows the current Java programs.

Program Source Code  N  CPU secs Elapsed secs Memory KB Code B
binary-trees  #2  120.210.18?603
binary-trees  #2  160.960.7996,036603
binary-trees  #2  2018.8013.38475,860603
binary-trees  #3  120.210.18?584
binary-trees  #3  160.930.7594,308584
binary-trees  #3  2018.8013.20476,640584
binary-trees-redux  #2  120.220.18?607
binary-trees-redux  #2  161.010.7859,764607
binary-trees-redux  #2  20253.7174.14134,460607
chameneos-redux  #2  60,0000.540.282121429
chameneos-redux  #2  600,0004.341.5040,5961429
chameneos-redux  #2  6,000,00039.2213.0242,0041429
chameneos-redux  #3  60,0000.900.352401267
chameneos-redux  #3  600,0003.371.0839,2161267
chameneos-redux  #3  6,000,00026.347.8456,0081267
chameneos-redux   60,0000.380.232361770
chameneos-redux   600,0001.410.8426,8761770
chameneos-redux   6,000,0005.004.4231,4561770
chameneos-redux  #4  60,0000.700.312361607
chameneos-redux  #4  600,0002.400.7824,2641607
chameneos-redux  #4  6,000,00018.225.2726,3241607
fannkuch-redux   100.780.302401282
fannkuch-redux   114.201.1719,5401282
fannkuch-redux   1264.3816.3319,9081282
fannkuch-redux  #2  100.790.7419,760514
fannkuch-redux  #2  117.837.7719,928514
fannkuch-redux  #2  1275.3675.3220,052514
fasta  #5  250,0000.360.212402457
fasta  #5  2,500,0000.910.402122457
fasta  #5  25,000,0005.231.7329,1042457
fasta  #2  250,0000.310.252401240
fasta  #2  2,500,0001.121.0320,3041240
fasta  #2  25,000,0007.927.8122,6961240
fasta  #4  250,0000.250.222361507
fasta  #4  2,500,0000.780.7019,2601507
fasta  #4  25,000,0005.145.0622,0401507
fasta-redux  #3  250,0000.200.18?1443
fasta-redux  #3  2,500,0000.430.382321443
fasta-redux  #3  25,000,0001.921.8719,7921443
k-nucleotide  #7  250,0001.340.4930,0921844
k-nucleotide  #7  2,500,0004.701.49308,6241844
k-nucleotide  #7  25,000,00035.8910.661,200,3201844
k-nucleotide  #5  250,0002.671.14171,7642211
k-nucleotide  #5  2,500,00013.344.91156,8962211
k-nucleotide  #5  25,000,00077.9433.76197,5682211
k-nucleotide  #2  250,0002.080.7336,4561602
k-nucleotide  #2  2,500,0006.051.9984,0921602
k-nucleotide  #2  25,000,00045.0512.92516,2081602
k-nucleotide  #3  250,0001.570.5433,8801630
k-nucleotide  #3  2,500,0006.011.8086,1601630
k-nucleotide  #3  25,000,00045.5413.06516,7721630
k-nucleotide  #4  250,0001.861.05183,7321873
k-nucleotide  #4  2,500,0006.836.18183,2361873
k-nucleotide  #4  25,000,00052.5751.48195,5761873
mandelbrot   1,0000.350.33240665
mandelbrot   4,0002.902.8619,516665
mandelbrot   16,00043.4043.3619,500665
mandelbrot  #2  1,0000.340.19?796
mandelbrot  #2  4,0001.970.6021,076796
mandelbrot  #2  16,00027.036.9181,940796
mandelbrot  #6  1,0000.340.19?802
mandelbrot  #6  4,0002.250.6821,072802
mandelbrot  #6  16,00030.597.8081,652802
mandelbrot  #3  1,0000.470.23236903
mandelbrot  #3  4,0003.100.8921,540903
mandelbrot  #3  16,00042.7610.8583,828903
meteor-contest   2,0980.970.8422,8324368
meteor-contest  #2  2,0980.530.292405177
n-body  #2  500,0000.420.402361424
n-body  #2  5,000,0002.612.5819,4801424
n-body  #2  50,000,00024.4724.4519,5081424
pidigits   2,0002.091.13164,060800
pidigits   6,00010.269.21375,272800
pidigits   10,00026.4625.41376,628800
pidigits  #2  2,0000.340.34236938
pidigits  #2  6,0001.601.5120,016938
pidigits  #2  10,0004.154.0423,488938
pidigits  #3  2,0001.600.6233,7201826
pidigits  #3  6,0004.782.1038,0401826
pidigits  #3  10,0008.774.3647,0961826
pidigits  #4  2,0001.970.8431,6321808
pidigits  #4  6,0006.152.4641,0561808
pidigits  #4  10,00012.474.6447,0241808
regex-dna  #5  50,0000.830.36236929
regex-dna  #5  500,0003.571.22100,196929
regex-dna  #5  5,000,00026.349.02714,924929
regex-dna  #8  50,0000.680.4622,2561377
regex-dna  #8  500,0002.762.4030,5441377
regex-dna  #8  5,000,00021.4021.06563,0281377
regex-dna  #7  50,0000.660.4421,8321284
regex-dna  #7  500,0002.782.3930,8921284
regex-dna  #7  5,000,00021.2420.78562,9881284
regex-dna  #6  50,0000.780.342081410
regex-dna  #6  500,0003.200.9896,7201410
regex-dna  #6  5,000,000Failed  1410
regex-dna  #2  50,0001.250.4629,7561534
regex-dna  #2  500,0004.801.60115,1001534
regex-dna  #2  5,000,00039.8912.61731,6441534
regex-dna  #4  50,0000.840.5333,348921
regex-dna  #4  500,0003.582.9268,464921
regex-dna  #4  5,000,00029.4127.77739,244921
reverse-complement  #4  250,0000.270.20?592
reverse-complement  #4  2,500,0000.550.4830,800592
reverse-complement  #4  25,000,0002.872.78481,900592
reverse-complement  #3  250,0000.320.232081661
reverse-complement  #3  2,500,0000.570.352121661
reverse-complement  #3  25,000,0002.671.36306,1561661
reverse-complement  #7  250,0000.510.372081640
reverse-complement  #7  2,500,0000.800.9347,4881640
reverse-complement  #7  25,000,0002.908.78270,6881640
reverse-complement  #6  250,0000.260.20?745
reverse-complement  #6  2,500,0000.410.34240745
reverse-complement  #6  25,000,0001.681.40519,728745
spectral-norm  #2  5000.600.28208950
spectral-norm  #2  3,0005.371.4920,068950
spectral-norm  #2  5,50016.564.3020,064950
spectral-norm   5000.380.37240514
spectral-norm   3,0005.085.0219,748514
spectral-norm   5,50016.4816.4123,812514
thread-ring  #7  500,0005.434.7827,364473
thread-ring  #7  5,000,00049.1739.4627,460473
thread-ring  #7  50,000,000491.16403.4729,440473
thread-ring  #3  500,0005.684.46368,368530
thread-ring  #3  5,000,00050.8441.70368,144530
thread-ring  #3  50,000,000498.78412.00366,984530

 Java : ubiquitous jit server virtual machine 

java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) Server VM (build 25.25-b02, mixed mode)

Home Page: Java SE at a Glance

Download: Java SE Downloads

Let's see how much, or how little, the time taken to invoke the JVM might contribute to the usual Java program times shown in the benchmarks game. Here are some additional (Intel® Q6600® quad-core) elapsed time measurements, taken after the Java programs started and before they exited.

In the first case (Cold), we simply started and measured the program 66 times; and then discarded the first measurement leaving 65 data points.

   public static void main(String[] args){
      for (int i=0; i<1; ++i){ 
         System.gc(); 
         long t1 = System.nanoTime();
         nbody.program_main(args);
         long t2 = System.nanoTime();
         System.err.println( String.format( "%.6f", (t2 - t1) * 1e-9 ) );         
      }
   }

In the second case (Warmed), we started the program once and repeated measurements again and again and again 66 times without restarting the JVM; and then discarded the first measurement leaving 65 data points.

   public static void main(String[] args){
      for (int i=0; i<66; ++i){ 
         System.gc(); 
         long t1 = System.nanoTime();
         nbody.program_main(args);
         long t2 = System.nanoTime();
         System.err.println( String.format( "%.6f", (t2 - t1) * 1e-9 ) );         
      }
   }

Compare these additional measurements against the usual Java program measurements shown in the benchmarks game --

"1.7.0_06" Java HotSpot(TM) 64-Bit Server VM
System.nanoTime()  1) Cold   2) Warmed   
  mean σ mean σ   usual
meteor contest   0.0118s 0.0007 0.0016s 0.0002 0.22s
fasta-redux   2.45s 0.00 2.32s 0.00 2.51s
spectral-norm   4.44s 0.02 4.20s 0.16 4.51s
pidigits   4.69s 0.09 4.44s 0.05 4.61s
fasta   5.07s 0.46 4.84s 0.02 5.13s
chameneos-redux   5.84s 0.46 5.70s 0.48 5.65s
mandelbrot   7.93s 0.23 7.99s 0.01 7.02s
k-nucleotide   8.09s 0.28  --   --  8.05s
regex-dna   8.65s 0.27  --   --  8.61s
binary-trees   10.54s 0.28 7.66s 0.16 9.08s
fannkuch-redux   16.89s 1.32 17.26s 0.10 17.38s
nbody   22.43s 0.00 22.41s 0.00 22.50s
binary-trees-redux   34.15s 0.39 33.93s 0.31 33.38s

The largest and most obvious effects of bytecode loading and dynamic optimization can be seen with the meteor-contest program which only runs for a fraction of a second.

A more sophisticated way to make JVM timings following warmup iterations is to use JMH, see "Java Microbenchmark Harness (the lesser of two evils)" pdf.

"1.8.0" Java HotSpot(TM) 64-Bit Server VM
  Mode Samples Mean Mean error
spectralnorm.testMethod   ss 65   4.181 0.008
nbody.testMethod   ss 65   22.462 0.001

But please don't just assume those warmed-up timings will be a whole lot faster for the JVM programs shown here, at the workloads shown here.

Revised BSD license

  Home   Conclusions   License   Play