The Computer Language
Benchmarks Game

Why do people just make up stuff sometimes?

Trust, but verify

Sometimes people just make up stuff because the facts don't fit the story they want to tell you.

You will probably come across stuff that people say about the benchmarks game. Do they show that what they claim is true? If they provide a URL, does the content actually confirm what they say?

Perhaps they are genuinely mistaken. Perhaps the content changed.

Wtf kind of benchmark counts the jvm startup time?

Compare the times against [pdf] Java Microbenchmark Harness reports:

secs JMH Average
n-body 21.54 23.367 ± 0.062
spectral-norm 4.29 4.061 ± 0.054
meteor-contest 0.24 0.112 ± 0.001

JVM start-up, JIT, OSR… are quickly effective and typical cold / warmed-up comparison at most of these workloads will show miniscule difference. Note the exception.

(In stark contrast to the traditional expectation of warmup, some benchmarks exhibit slowdown, where the performance of in-process iterations drops over time.)

Прогрелось. Но работает минимум в 65x медленнее – Поэтому супер-оптимизация…

No. The repeated measurements of the Java pi-digits program without restarting the JVM, did not make the Java program seem 65x slower.

That really would have been ridiculous. The JavaOne expert's slides mysteriously fail to show that the sum was correctly divided by 65 to give an average.

…to dispute a decision you basically need to pray the maintainer reopens it for some reason.

No. Followup comments can always be made in the project ticket tracker. There's a public discussion forum, etc. etc.

Someone's brilliant hack was rejected. Someone took the opportunity to push traffic to their personal blog.

There's a reason they call it a game
It's a game

The name "benchmarks game" signifies nothing more than the fact that programmers contribute programs that compete (but try to remain comparable) for fun not money.

It's what you make of it.

Oct 2015 through Sep 2016, Google Analytics shows 354,872 users.

above average page views
(go.html python.html etc June 2017)
Go 8,428
C# 7,101
Python 5,904
Rust 5,050
JavaScript 4,573
C 4,497
Java 4,040
PHP 3,484
Ruby 2,470
C++ 2,455

Once upon a time…

Doug Bagley had a burst of crazy curiosity: When I started this project, my goal was to compare all the major scripting languages. Then I started adding in some compiled languages for comparison…

That project was abandoned in 2002, restarted in 2004 by Brent Fulgham and continued from 2008 by Isaac Gouy. Everything has changed; several times.