@stamourv if I wanted to write benchmarks for a package that are part of a test suite and don’t produce pretty plots, should I extend https://github.com/stamourv/racket-benchmark or would I be better off making my own package for that?

notjack: extending racket-benchmark would be cool

@ben do you use it often?

@notjack: Sure, I’m all for making racket-benchmark better!

We have a number of experiments that use it, so not breaking it is important, but improvements would be welcome!

@stamourv do you have links to some of the experiments?

Not directly, but I think some of our research artifacts may have some, lemme check.

never

but yeah I had put some benchmarks here: https://github.com/nuprl/gradual-typing-performance/tree/master/benchmarks

and then I put some others here: https://github.com/bennn/eidetic/tree/master/benchmarks/rkt

and this is getting silly. Usually I want some tooling around these benchmarks (to reproduce data in a paper), but the code should all be in 1 place

like the larceny benchmarks http://www.larcenists.org/benchmarksAboutR7.html

@notjack: Here’s one artifact: http://drops.dagstuhl.de/opus/frontdoor.php?source_opus=5514

It’s a large VM, that has a benchmark suite (and other stuff) inside.

thanks @ben and @stamourv for links :)

really the main thing I want is something like (check-faster (foo-benchmark foo-impl1) (foo-benchmark foo-impl2))

Oh, sure. That should be easy to do with the results data structure.

Beware of the case where the confidence intervals overlap, though. Can’t conclude that either is faster.

memory consumption benchmarks instead of timing benchmarks would also be useful

@notjack I have benchmark-ips… it might help normalize some of the error

and in minitest, I have minitest/benchmark which allows you to assert that the performance of a block of code fits well to a particular curve. Allows tests to pass regardless of the hardware you run on.

@zenspider I definitely want curve-fitting perf tests for racket. The lens library in particular could benefit from perf testing.

basically looks like:

    def bench_my_algorithm
      assert_performance_linear 0.9999 do \|n\| # n is a range value
        @obj.my_algorithm(n)
      end
    end

And asserts against the R^2 of the fit instead of actual measurements

https://github.com/seattlerb/minitest/blob/master/lib/minitest/benchmark.rb

benchmark-ips is my port of a ruby tool. Similar to racket’s benchmark, but it doesn’t graph and it is focused on iterations per second instead of some fixed number of iterations… I mostly use it when experimenting with N different implementations of something