A simple microbenchmarking function in Racket
In a previous post, I wrote a function to perform repeated timings of untyped and typed versions of the same Racket functions.
#lang racket (require math) (define (time-apply-cpu-old proc lst reps) (define out (for/list ([i (in-range reps)]) (define-values (results cpu-time real-time gc-time) (time-apply proc lst)) cpu-time)) (displayln (string-append "min: " (number->string (apply min out)) " mean: " (number->string (round (mean out))) " max: " (number->string (apply max out)) " function: " (symbol->string (object-name proc)))))
time-apply-cpu-old is a wrapper function for
racket/base that runs
time-apply repeatedly and prints the
max cpu time.
time-apply produces multiple output values that are not contained in a data structure.
define-values allows you to capture those outputs and bind them to names (in this case,
string-append is similar to
paste in R, but
string-append requires that all arguments are strings and, thus, requires some conversion (e.g.,
> (time-apply-cpu-old flnormal-sample (list 0.0 1.0 10000) 50) min: 1 mean: 2 max: 22 function: flnormal-sample > (time-apply-cpu-old flnormal-sample (list 0.0 1.0 100000) 50) min: 14 mean: 17 max: 45 function: flnormal-sample
time-apply-cpu-old only allows for evaluation of a single function at a time and the display of the output is ugly. I thought it would be a good exercise to try to address those two deficiencies. There was a recent discussion on the Racket mailing list that provided several options to allow for pretty printing of the timing output. I opted for a suggestion involving the
table function from the raart package.
> (require raart/draw) > (define example-list (list (list "col1" "col2" "col3") (list 1.001 2.002 3.003) (list 4.004 5.005 6.006))) > (draw-here (table (text-rows example-list) #:frames? #f #:inset-dw 1 #:halign 'right)) col1 col2 col3 1.001 2.002 3.003 4.004 5.005 6.006
In this example, I only changed a few of the default arguments to the
table function. I dropped the table borders, increased the horizontal spacing from 0 to 1, and changed the horizontal alignmental from
I’m modeling the format for my target output on the
microbenchmark function in the microbenchmark package for R.
> library(microbenchmark) > microbenchmark(rnorm(10000), rnorm(100000), unit = "ms") Unit: milliseconds expr min lq mean median uq max neval rnorm(10000) 0.575444 0.618125 0.7051323 0.6259115 0.6549435 4.466793 100 rnorm(1e+05) 5.741617 6.131754 6.3758384 6.1757695 6.4165000 11.183758 100
First, we are going to modify
time-apply-cpu-old to return a list of
cpu-time rather than displaying the
max cpu time.
(define (time-apply-cpu proc args reps) (for/list ([i (in-range reps)]) (define-values (results cpu-time real-time gc-time) (time-apply proc args)) cpu-time))
My new Racket function,
microbenchmark, requires similar arguments as
procs is a list,
args is a list of lists, and reps has a default value of 100.
microbenchmark starts with an expression to check that the lengths of
(unless (equal? (length procs) (length args)) is similar to
if (length(procs) != length(args)) in R. I don’t yet have a good sense for how much effort I should put into checking inputs, but my preliminary impression is that Racket is less likely than R to run successfully with unexpected inputs (and with potentially incorrect outputs).
(define (microbenchmark procs args [reps 100]) (unless (equal? (length procs) (length args)) (error "List of procedures is not same length as list of arguments.")) (define (create-timing-table procs args [result (list (list "expr" "args" "min" "lq" "mean" "median" "uq" "max" "neval"))]) (cond [(null? procs) (reverse result)] [else (define tmp (time-apply-cpu (first procs) (first args) reps)) (create-timing-table (rest procs) (rest args) (cons (list (symbol->string (object-name (first procs))) (first args) (apply min tmp) (quantile 0.25 < tmp) (round (mean tmp)) (quantile 0.5 < tmp) (quantile 0.75 < tmp) (apply max tmp) reps) result))])) (displayln "Units: milliseconds") (draw-here (table (text-rows (create-timing-table procs args)) #:frames? #f #:inset-dw 1 #:halign 'right)))
microbenchmark function, a recursive function,
create-timing-table, repeatedly calls
time-apply-cpu to build up a table of results. The results table is initialized as a list of lists where the first list contains the column headings.
create-timing-table is a list-eater function that passes the first item from
time-apply-cpu and then uses
cons to add the latest output to the front of the results list, which is why the results list needs to be reversed when the function exits.
To demonstrate the
microbenchmark output, I will compare two different functions for drawing random numbers from a normal distribution based on this post.
> (microbenchmark (list flnormal-sample rnorm flnormal-sample rnorm) (list (list 0.0 1.0 10000) (list 0.0 1.0 10000) (list 0.0 1.0 100000) (list 0.0 1.0 100000))) Units: milliseconds expr args min lq mean median uq max neval flnormal-sample (0.0 1.0 10000) 1 1 2 1 2 8 100 rnorm (0.0 1.0 10000) 4 5 7 5 9 34 100 flnormal-sample (0.0 1.0 100000) 14 15 15 15 16 21 100 rnorm (0.0 1.0 100000) 62 67 93 73 84 1772 100