Travis Hinkelman

Join dataframes in Scheme

Sun, 05 Apr 2026 00:00:00 GMT

This post is part of a series on the dataframe library for Scheme (R6RS). In this post, I will contrast functions for joining dataframes from the dataframe library with the dplyr package for R.

Set up

First, let’s create a couple of dataframes in both languages.

library(dplyr)

df1 <- data.frame(name = c("Alice", "Bob", "Carol"),
                  age = c(30, 25, 35))

df2 <- data.frame(name = c("Bob", "Carol", "Dave"),
                  dept = c("Sales", "Engineering", "Marketing"))

(import (dataframe))

(define df1
  (make-df* (name "Alice" "Bob" "Carol")
            (age 30 25 35)))

(define df2
  (make-df* (name "Bob" "Carol" "Dave")
            (dept "Sales" "Engineering" "Marketing")))

Left join

In R, left_join keeps all rows from the left dataframe and adds matching columns from the right dataframe, filling NA for non-matches.

> left_join(df1, df2)
   name age        dept
1 Alice  30        <NA>
2   Bob  25       Sales
3 Carol  35 Engineering

dataframe-left-join works similarly. When join-names is #f or omitted, the join columns are auto-detected from the common column names. Because Scheme doesn’t have explicit missing values, 'na is used by default, but a different fill value can be specified.

> (dataframe-display (dataframe-left-join df1 df2))

 dim: 3 rows x 3 cols
    name     age        dept 
              
   Alice      30.         na 
     Bob      25.      Sales 
   Carol      35. Engineering

Inner join

inner_join keeps only rows that have matches in both dataframes.

> inner_join(df1, df2)
   name age        dept
1   Bob  25       Sales
2 Carol  35 Engineering

dataframe-inner-join works the same way.

> (dataframe-display (dataframe-inner-join df1 df2))

 dim: 2 rows x 3 cols
    name     age        dept 
              
     Bob      25.      Sales 
   Carol      35. Engineering

Full join

full_join keeps all rows from both dataframes, filling missing values where there is no match.

> full_join(df1, df2)
   name age        dept
1 Alice  30        <NA>
2   Bob  25       Sales
3 Carol  35 Engineering
4  Dave  NA   Marketing

dataframe-full-join fills missing values with 'na by default.

> (dataframe-display (dataframe-full-join df1 df2))

 dim: 4 rows x 3 cols
    name     age        dept 
              
   Alice      30.         na 
     Bob      25.      Sales 
   Carol      35. Engineering 
    Dave       na   Marketing

Joining on different column names

In dplyr, when the join columns have different names in each dataframe, you specify the mapping with a named vector.

> df3 <- data.frame(person = c("Bob", "Carol", "Dave"),
                    dept = c("Sales", "Engineering", "Marketing"))

> left_join(df1, df3, by = c("name" = "person"))
   name age        dept
1 Alice  30        <NA>
2   Bob  25       Sales
3 Carol  35 Engineering

In the dataframe library, the join-names argument accepts a list of pairs mapping column names from df1 to df3.

> (define df3
    (make-df* (person "Bob" "Carol" "Dave")
              (dept "Sales" "Engineering" "Marketing")))

> (dataframe-display
   (dataframe-left-join df1 df3 '((name . person))))

 dim: 3 rows x 3 cols
    name     age        dept 
              
   Alice      30.         na 
     Bob      25.      Sales 
   Carol      35. Engineering

The join-names argument supports three forms: #f (or omitted) for auto-detecting common names, a list of symbols for same-named columns (e.g., '(name dept)), and a list of pairs for differently-named columns (e.g., '((name . person))).

Left join all

left_join only takes two dataframes at a time. To left join a list of dataframes, you could use purrr::reduce.

purrr::reduce(list(df1, df2), left_join)

dataframe-left-join-all takes a list of dataframes and sequentially left joins them.

(dataframe-display (dataframe-left-join-all (list df1 df2)))

Implementation

All four join procedures follow the same overall strategy: split both dataframes by the join columns, match groups between the two sets of splits, and then bind the results back together.

(define dataframe-left-join
  (case-lambda
    [(df1 df2)
     (dataframe-left-join df1 df2 #f 'na)]
    [(df1 df2 join-names)
     (dataframe-left-join df1 df2 join-names 'na)]
    [(df1 df2 join-names fill-value)
     (let* ([who "(dataframe-left-join df1 df2 join-names fill-value)"]
            [names-pair (parse-join-names df1 df2 join-names)]
            [names1 (car names-pair)]
            [names2 (cadr names-pair)]
            [df2-aligned (align-df2 df2 names2 names1 who)])
       (check-join df1 df2-aligned names1 who)
       (let ([slists1 (dataframe-split-helper df1 names1 who)]
             [slists2 (dataframe-split-helper df2-aligned names1 who)])
         (dataframe-bind-all
          (df-left-join-helper slists1 slists2 names1 fill-value))))]))

First, parse-join-names resolves the three supported forms of join-names into a pair of name lists, one for each dataframe. When join-names is #f, it finds the column names common to both dataframes. If the join columns have different names, align-df2 renames the columns in df2 to match df1 so the rest of the join machinery can work with a single set of names.

Both dataframes are then split into groups by the join columns using dataframe-split-helper. The split produces a list of slists (lists of series), one per unique combination of join-column values.

Matching groups

The core of each join variant is a helper that iterates over the groups from df1 and looks for matching groups in df2. A group’s identity is determined by get-join-group, which extracts the values from the first row of the join columns (all rows have the same values because the slist was split on those columns).

(define (get-join-group slist join-names)
  (apply append (map series-lst (slist-ref slist '(0) join-names))))

For the left join, each group from df1 is checked against an association list built from df2’s groups. If a match is found, the two groups are combined with join-match. If not, the non-join columns from df2 are filled with fill-value via join-no-match.

(define (df-left-join-helper slists1 slists2 join-names fill-value)
  (let* ([grps2 (map (lambda (x) (get-join-group x join-names)) slists2)]
         [grps2-slists2 (map cons grps2 slists2)])
    (map (lambda (slist)
           (let* ([grp (get-join-group slist join-names)]
                  [grp-match (assoc grp grps2-slists2)])
             (if grp-match
                 (join-match slist (cdr grp-match) join-names)
                 (join-no-match slist (car slists2) join-names fill-value))))
         slists1)))

The inner join helper is similar but simply drops groups from df1 that have no match, using a loop with an accumulator rather than map. The full join helper additionally tracks which df2 groups have been matched and appends the unmatched ones at the end, with the df1 non-join columns filled.

Combining matched groups

When two groups match, join-match drops the join columns from slist2 (to avoid duplication) and appends the remaining columns. If the two groups have different numbers of rows, the shorter one is repeated to match the longer one.

(define (join-match slist1 slist2 join-names)
  (let* ([n1 (series-length (car slist1))]
         [n2 (series-length (car slist2))]
         [slist2-drop (slist-drop slist2 join-names)]
         [slist1-new (if (>= n1 n2)
                         slist1
                         (slist-repeat-rows slist1 n2 'times))]
         [slist2-new (if (>= n2 n1)
                         slist2-drop
                         (slist-repeat-rows slist2-drop n1 'times))])
    (make-dataframe (append slist1-new slist2-new))))

This row-repeating behavior is how the library handles many-to-many relationships within a group. If df1 has 2 rows for a given group and df2 has 3, the df1 rows are repeated to length 3 (and vice versa). This is a simpler approach than producing a full cross product.

When there is no match, join-no-match retains all columns from the unmatched slist and fills the non-join columns from the other dataframe with fill-value.

(define (join-no-match slist1 slist2 join-names fill-value)
  (let* ([n (series-length (car slist1))]
         [slist2-names (map series-name slist2)]
         [slist2-names-sel (not-in slist2-names join-names)])
    (make-dataframe
     (append slist1 (slist-fill-missing slist2-names-sel n fill-value)))))

Conclusions

The join procedures follow the same split-apply-combine pattern used by other procedures in the dataframe library. Splitting both dataframes by join columns reduces the problem to matching and combining small groups, which makes the logic for each join variant straightforward. The parse-join-names and align-df2 steps handle the bookkeeping of resolving column name differences before the main join logic runs. The design is inspired by the dplyr family of join functions, though the handling of many-to-many matches uses row repetition rather than a full cross product.

Display and glimpse dataframes in Scheme

Wed, 01 Apr 2026 00:00:00 GMT

This post is part of a series on the dataframe library for Scheme (R6RS). In this post, I will describe dataframe-display and dataframe-glimpse, which are inspired by the default print format for tibbles in R and the dplyr::glimpse function. Both procedures rely heavily on Chez Scheme’s format procedure, so I will first give an overview of the format directives used in the dataframe library before showing how dataframe-display and dataframe-glimpse work.

Format directives

Chez Scheme’s format procedure is modeled on Common Lisp’s format and supports a rich set of directives for controlling output. The first argument to format is the destination: #t sends output to the current output port (i.e., prints to screen), #f returns the result as a string, and a port sends output to that port.

The general form is:

(format destination format-string arg ...)

Directives are embedded in the format string as ~ followed by a character, with optional numeric parameters before the character. Here are the directives used in dataframe-display and dataframe-glimpse.

~d prints an integer in decimal notation.

> (format #t "~d rows x ~d cols" 10 3)
10 rows x 3 cols

~a prints a value in human-readable form, e.g., strings without quotes, characters without the #\ prefix.

> (format #t "~a" "hello")
hello

> (format #t "~a" #\x)
x

~Na adds a minimum field width of N, padding on the right (left-aligning the value). ~N@a pads on the left instead, right-aligning the value in a field of width N. The @ modifier means “use left padding”, which produces right-alignment. This is used throughout dataframe-display to align columns.

> (format #t "~10a|" "hi")
hi        |

> (format #t "~10@a|" "hi")
        hi|

~N,Df prints a floating-point number in a field of width N with D digits after the decimal point.

> (format #t "~10,2f" 3.14159)
      3.14

~N,D,Ee prints a number in exponential notation in a field of width N, with D decimal digits and E digits in the exponent.

> (format #t "~12,3,1e" 0.000012345)
   1.234e-5

~& emits a newline only if the output is not already at the start of a line. ~% emits an unconditional newline. In practice, ~& is used at the start of format strings in this library to ensure each row of the table starts on its own line.

~{...~} is the iteration directive. It consumes one argument from the argument list, which must be a flat list, and consumes elements of the list as arguments to the directives in the body.

> (format #t "~{~a ~}" '(a b c))
a b c

~:{...~} is similar but iterates over a list of sublists. Each sublist provides the arguments for one iteration of the body.

> (format #t "~:{~& ~a ~a~}" '((a 1) (b 2) (c 3)))
 a 1
 b 2
 c 3

This distinction between ~{~} and ~:{~} is central to how dataframe-display works, as we’ll see below.

dataframe-display

dataframe-display prints a formatted table showing the dataframe dimensions, column names, column types, and up to n rows of values.

(define dataframe-display
  (case-lambda
    [(df) (df-display-helper df 10 76 7)]
    [(df n) (df-display-helper df n 76 7)]
    [(df n total-width) (df-display-helper df n total-width 7)]
    [(df n total-width min-width) (df-display-helper df n total-width min-width)]))

The defaults are 10 rows, a total width of 76 characters, and a minimum column width of 7. If a dataframe has more columns than fit in total-width, the extra columns are omitted and listed in a footer.

> (define df
    (make-df*
     (Boolean #t #f #t)
     (Char #\y #\e #\s)
     (String "these" "are" "strings")
     (Symbol 'these 'are 'symbols)
     (Exact 1/2 1/3 1/4)
     (Integer 1 -2 3)
     (Expt 1000000.0 -123456.0 1.2346e-6)
     (Dec4 132.1 -157.0 10.234)
     (Dec2 12.14 -9.54 100.01)
     (Other '(1 2 3) '#(a b) '#(1 2))))

> (dataframe-display df)

 dim: 3 rows x 10 cols
  Boolean    Char  String  Symbol   Exact Integer      Expt    Dec4 
                            
        #t       y   these   these     1/2      1.  1.000e6  132.1000 
        #f       e     are     are     1/3     -2. -1.235e5 -157.0000 
        #t       s strings symbols     1/4      3.  1.235e-6  10.2340 
 Columns not displayed: Dec2, Other

The output has four components. The first line shows the full dimensions. Then the column names and types, each right-aligned in a field sized to accommodate the widest value in that column. The table rows follow, with numeric columns formatted using ~f or ~e depending on the magnitude of the values. Any columns that don’t fit within total-width are listed in the footer.

Building the format strings

The most intricate part of dataframe-display is build-format-parts, which constructs three separate format strings (header row, the types row, and the table) by iterating over columns left to right and accumulating directives as long as the column fits within total-width.

The header and types rows use ~{...~} because they iterate over a flat list of names or type strings. The table uses ~:{...~} because it iterates over a list of rows (sublists).

The format strings are built up incrementally. For each column, a width-part is computed (e.g., "~10") and combined with "@a " for the header and types rows, or with a numeric directive suffix for the table. For example, a floating-point column of width 10 with 4 decimal places contributes "~10,4f " to the table format string and "~10@a " to the header and types format strings.

Here is a simplified illustration of what the final format strings look like for a dataframe with one string column and one numeric column:

;; header:  "~& ~{~8@a ~10@a ~}"  applied to  (Name Score)
;; types:   "~& ~{~8@a ~10@a ~}"  applied to  ("" "")
;; table:   "~:{~& ~8@a ~10,4f ~}" applied to  (("Alice" 98.5) ("Bob" 72.1))

The ~& at the start of each format string ensures each row begins on a new line. The ~} or ~} closes the iteration. The values are assembled in format-df:

(define (format-df df-names df-types ls-vals dim total-width min-width)
  (let* ([prep-vals (map prepare-non-numbers ls-vals)]
         [parts (build-format-parts df-names df-types prep-vals total-width min-width 2)])
    (format #t " dim: ~d rows x ~d cols" (car dim) (cdr dim))
    (format #t (cadr (assoc 'header parts)) (caddr (assoc 'header parts)))
    (format #t (cadr (assoc 'types parts))  (caddr (assoc 'types parts)))
    (format #t (cadr (assoc 'table parts))  (caddr (assoc 'table parts)))
    (newline)
    (display (cdr (assoc 'footer parts)))))

Each call to format here takes the format string (the cadr of the alist entry) and the corresponding list of values (the caddr).

Numeric formatting

For columns containing only numbers, dataframe-display chooses between floating-point (~f) and exponential (~e) notation based on the magnitude of the values. The logic is in compute-decimal, which returns the number of decimal places to use. If any value in the column is assigned the maximum decimal count (the threshold for “very large or very small”), the whole column is displayed in exponential notation; otherwise floating-point is used.

(define (compute-decimal x sigfig e e-dec)
  (let ([default 4]
        [x (abs x)])
    (cond [(or (< e -3) (> e 5)) e-dec]  ;; exponential notation
          [(integer? x) 0]
          [(> e 3) 2]                    ;; fewer decimals for large numbers
          [(and (< x 1) (> sigfig default)) sigfig]
          [else default])))

The e argument is the base-10 exponent (order of magnitude) of the value, computed via compute-expt. So a value like 1000000.0 has e = 6, which exceeds the threshold of 5 and triggers exponential notation for the entire column.

Column widths for numeric columns are computed separately from non-numeric ones. For a floating-point column, the width is neg + sig + dec + pad, where neg is 1 if any value is negative (to accommodate the minus sign), sig is the number of digits left of the decimal point, dec is the number of decimal places, and pad is spacing between columns.

Non-numeric values

Columns that are not purely numeric, or that contain exact fractions like 1/3, are handled by prepare-non-numbers. Compound objects like lists, vectors, and hashtables are replaced with placeholder strings such as or . Primitive non-numeric values (booleans, characters, strings, symbols) are displayed with ~a.

Exact fractions are routed through compute-object-width rather than compute-num-width because their string representation (e.g., "1/3") has a width that is easier to measure directly.

dataframe-glimpse

dataframe-glimpse provides a transposed summary of the dataframe: one row per column, showing the column name, type, and as many values as fit across the screen. It is inspired by dplyr::glimpse in R, which is useful for dataframes with many columns that don’t fit comfortably in a standard tabular display.

(define dataframe-glimpse
  (case-lambda
    [(df) (df-glimpse-helper df 76)]
    [(df total-width) (df-glimpse-helper df total-width)]))

> (dataframe-glimpse df)

 dim: 3 rows x 10 cols
 Boolean    #t #f #t
 Char        y e s
 String      these are strings
 Symbol      these are symbols
 Exact       1/2 1/3 1/4
 Integer     1 -2 3
 Expt        1000000.0 -123456.0 1.2346e-6
 Dec4        132.1 -157.0 10.234
 Dec2        12.14 -9.54 100.01
 Other

Unlike dataframe-display, glimpse shows all columns regardless of width, since each column occupies its own row. The values for each column are rendered as a truncated flat list (without parentheses) by prepare-lst, which appends values separated by spaces until the remaining width is exhausted, then appends ", ..." to signal truncation.

The glimpse format string

The glimpse format string is built by glimpse-format-string and uses ~:{...~} to iterate over a list of (name type values-string) triples. Each iteration prints one column’s row.

(define (glimpse-format-string name-width type-width list-width)
  (let* ([nw (number->string name-width)]
         [tw (number->string type-width)]
         [lw (number->string list-width)])
    (string-append "~:{~& ~" nw "a ~" tw "a ~" lw "a ~}")))

Notice that ~Na is used here (without @), producing left-aligned output. This gives each column’s name, type, and values left-aligned in their respective fields because it is a row-oriented display where values are a freeform list rather than values in a fixed-width column.

The three field widths are computed from the data: name-width is the width of the longest column name (minimum 7), type-width is the width of the longest type string (minimum 7), and list-width is whatever space remains after subtracting the other two from total-width.

The (name type values-string) triples are assembled by build-format-list and passed as a single argument to the ~:{...~} directive:

(define (build-format-list df-names df-types list-str)
  (map (lambda (n t ls) (list n t ls)) df-names df-types list-str))

For example, on a terminal 76 characters wide, glimpse-format-string might produce the string "~:{~& ~9a ~7a ~57a ~}", which is then applied to the list of triples in a single format call.

Conclusions

Implementing dataframe-display and dataframe-glimpse turned out to require the most complex code in the dataframe library. Getting the column widths right across different numeric types, handling exponential versus floating-point notation, and fitting columns within a total width all interact in ways that took considerable trial and error to get right. The design of dataframe-display was directly inspired by the way tibbles print in R. Tibbles show the dimensions, column types below the names, and truncate to a fixed number of rows by default. dataframe-glimpse follows dplyr::glimpse in transposing the view so that every column appears on its own line, making wide dataframes much easier to inspect at a glance.

Electric vehicles in California

Thu, 23 Oct 2025 00:00:00 GMT

I bought my first electric vehicle (EV) about a month ago and I thought it would be useful to capture my decision making around this purchase and initial impressions of driving an EV. I also thought it would be fun to use my Scheme dataframe library to explore data on EVs in California.

The decision to buy an EV was relatively abrupt. I wanted to replace the 2012 Honda Civic that my son was driving and considered buying an e-bike to fill our transportation needs by sharing my 2017 Toyota Prius, biking, and using public transportation. Our brief experiment with sharing one car left me feeling inconvenienced, though, and I wasn’t convinced that an e-bike would alleviate those feelings.

I had heard that used EVs were reasonably affordable right now and was persuaded by arguments about the practical advantages of used EVs (e.g., this video). My initial thinking was that I would take my chances on a relatively cheap used EV ($12-14k), but the caliber of the cars in that price range made me too nervous about battery health. In the $15-18k price range, some of the most commonly available EVs included the 2016-2018 BMW i3 and 2022-2024 Nissan LEAF. It’s easy to find highly favorable reviews of used BMW i3s and my affinity for small cars had me leaning in that direction. Ultimately, though, I couldn’t pass up on the peace of mind associated with buying a newer LEAF with lower specs but more remaining battery warranty. So I bought a 2024 Nissan LEAF S from Carvana for about $17k.

The 1st and 2nd generation LEAFs (which takes you through the 2025 model year) have two fatal flaws. One is that it uses a DC fast-charging standard (CHAdeMO) that is being phased out in North America. The other is that there is no active battery thermal management. The lack of battery conditioning means that when you operate the car outside of the optimal range for the battery you lose driving range and potentially increase the rate of battery degradation.

These flaws are not actually fatal if you can charge at home, don’t live in a hot climate, and are not planning to use the LEAF for road trips.

I have struggled a bit with the perceived limitations of driving a relatively short-range EV like the LEAF even though it arguably covers >90% of my usage needs. First of all, I was in the habit of refueling my gas tank when it approached 1/4 tank even though that likely equated to 100+ miles of range in the fuel efficient cars that I’ve been driving over the past 10+ years. With my LEAF, 100 miles of range is approaching the max that I would be willing to travel before charging. That requires a big shift in my thinking. Plus, the road trip limitations carry a general sense of a loss of freedom (which would not be a concern with a longer range EV).

The vast majority (93%) of U.S. vehicle trips are less than 30 miles.

I recently took my LEAF on a short road trip to gauge the experience. The trip was just under 200 miles each way. My car has a 40 kWh battery with 92% state of health. On the trip, I averaged 4 mi/kWh. Those numbers mean that with a fully charged battery I can get 147 miles of range [40 * 0.92 * 4]. But at 10% battery remaining the car starts warning you to charge, triggering the same irrational anxiety that had me filling my gas car at 1/4 tank, so we can knock another 14.7 miles off the practical range taking us down to 132 miles. Because fast charging rates decrease as the battery fills up, it is not recommended to top off the battery when DC fast charging. If I charge to 80%, I can expect to get about 103 miles of range [40 * 0.92 * 4 * (0.8 - 0.1)]. Put all of this together and I consider trips up to 235 miles to be a reasonable option in my LEAF. Of course, this assumes that there is a working DC fast charger at the right place along the route, that I will have access to destination charging at my lodging, and that temperatures are mild (even better if the terrain is flat and the winds are light).

[UPDATE 2025-11-30: I made another short road trip (~170 miles each way) in my LEAF. On this recent trip, I averaged 3.5 mi/kWh. At first, I thought that the lower efficiency was related to the colder temperatures, but my previous 4 mi/kWh average included all driving during that trip, not just highway driving. The EPA rates the 2024 Nissan LEAF S at 99 MPGe for highway driving, which is about 2.9 mi/kWh. If we use that value in my 80% calculation above, we get only 75 miles of highway range.

Along my route, the available fast charging stations were about 40-50 miles apart. That easily falls within my range, but, if the CHAdeMO charger at each stop (usually only one unit) was not working, then I would have been limping along to try to get to the next charger. On the way there, I took the conservative approach of stopping at all three chargers. Fortunately, I was able to charge successfully at each stop so I had no range anxiety, but all of that charging added roughly an hour compared to making the same trip in a gas car.

On my way there, I averaged only 3.3 mi/kWh while driving 70-75 mph uphill with the heat on. Because I made so many charging stops, though, I was always above 20% SOC when arriving at my next stop (even with the higher speeds and heat). On the way back, I was hoping to make the trip with only one charging stop, partly because I would be going mostly downhill and partly because I had more confidence that I would be able to charge successfully at my one planned stop. I drove the first 75 miles of the return trip at 70-75 mph with the heat and then charged to 80%. At that point, my efficiency was 3.5 mi/kWh. After leaving that charging stop, though, my range was dropping rapidly so I decided to reduce my speed down to 60-65 mph and turn off the heat. I also drove the last 5-10 miles on surface streets instead of the freeway. Those changes allowed me to make it home with 7% SOC. Based on the original ETA from the navigation, those changes also added 10-15 minutes to the drive. Arguably, I should have just made one more short charging stop (15-20 min) to make for a more comfortable drive. My overall return trip efficiency was 3.7 mi/kWh thanks to the more conservative driving on the second leg, though.]

Enough with the downsides. Let’s talk about why this car makes sense for me. The biggest factors are that I have a NEMA 14-50 outlet in my garage giving me access to level 2 charging at home and that I work from home with infrequent, short trips to the office. Moreover, I live in Sacramento with mild winters where I don’t expect cold weather to dramatically reduce my range. The extra planning required for EV ownership doesn’t feel particularly onerous to me and, in fact, I quite enjoy thinking about how to drive more efficiently and optimize charging. I love the experience of charging at home and the satisfying sound of plugging in my car. I thoroughly enjoy the EV driving experience, including one-pedal driving and the quiet but relatively powerful engine.

On the flip side, the hot summers might hasten my battery degradation.

This LEAF is the lowest mileage car that I have ever purchased (~6000 miles). The exterior and interior are in great shape and I’m quite enjoying having such a pristine car (while it lasts). The main feature upgrade in this car over my 2017 Toyota Prius is Apple CarPlay, but, because my LEAF is the S trim, it is also missing features that are present in my old Prius, e.g., adaptive cruise control, forward parking sensors, and more.

It is easy to imagine driving this LEAF for many years, but one unexpected side effect of becoming a first-time EV owner is how much more interested I’ve become in other EV models. I’ve never been a car guy so this newfound, and presumably short-lived, interest in cars is unusual. From where I sit right now, though, I think I’m most likely to replace my current LEAF with a newer LEAF because beginning with the 2026 model year they have fixed the fatal flaws of the previous generations of the LEAF. Of course, EV technology is evolving so rapdily that it is hard to predict which model will most appeal to me in 5+ years. I would like to be able to say that this experience has made me resolved to never buy a gas car again, but I loved my 2017 Toyota Prius so much that if my driving needs were to dramatically change I could see myself going back to a hybrid. However, I think I can say with some confidence that I will never buy a plug-in hybrid because it seems to me that they offer the worst of both worlds.

Now, let’s move on to the data. I had previously worked with data provided by the California Energy Commission on the light-duty vehicle population to make a Shiny app that overlaps heavily with the CEC’s own dashboard. In this post, I’m just presenting tabular summaries, but you can also plot the results with the gnuplot-pipe library (see this post).

I’ve copied the county-level data provided by the CEC to a CSV file and made it available for download here. First, let’s import the dataframe library, read the data, and do some preliminary processing.

(import (dataframe))

(define df
  (-> (csv->dataframe "light-duty.csv")
      (dataframe-drop* Fuel-Type)
      (dataframe-rename* (Data-Year Year)
                         (Dashboard-Fuel-Type-Group Fuel-Type)
                         (Number-of-Vehicles Count))))

(dataframe-display df)

 dim: 40877 rows x 6 cols
    Year   County               Fuel-Type    Make     Model    Count 
                                       
   2010.  Alameda  Battery Electric (BEV)    Ford    Ranger       3. 
   2010.  Alameda  Battery Electric (BEV)   Tesla  Roadster      17. 
   2010.  Alameda                  Diesel      na        na   10939. 
   2010.  Alameda                Gasoline      na        na   10974. 
   2010.  Alameda                Gasoline      na        na  840577. 
   2010.  Alameda         Gasoline Hybrid      na        na   22720. 
   2010.  Alameda                   Other      na        na     173. 
   2010.  Alameda                   Other      na        na      19. 
   2010.   Alpine                  Diesel      na        na      73. 
   2010.   Alpine                Gasoline      na        na      16.

The shiny app that I made doesn’t include any of the data on make and model so I will focus on make and model in this post. Make and model is provided for all zero-emission vehicles in the county-level dataset, but here I will work only with the BEVs.

(define df-bev
  (-> df
      (dataframe-filter*
       (Fuel-Type)
       (string=? Fuel-Type "Battery Electric (BEV)"))
      (dataframe-drop* Fuel-Type)))

As a starting point, let’s sum across counties and models to create year-make and then find the 10 makes with the highest average annual EV population. This average only includes years with non-zero counts for each make because the mean procedure provided by the dataframe library drops 'na by default.

(define year-make
  (dataframe-aggregate*
   df-bev
   (Year Make)
   (Count (Count) (sum Count))))

(define top-makes
  (-> year-make
      (dataframe-aggregate*
       (Make)
       (Avg (Count) (round (inexact (mean Count)))))
      (dataframe-sort* (> Avg))
      (dataframe-head 10)))
 
(dataframe-display top-makes)

 dim: 10 rows x 2 cols
        Make      Avg 
            
       Tesla  224091. 
   Chevrolet   28379. 
      Nissan   26785. 
     Hyundai   15330. 
  Volkswagen   13816. 
        Audi   11981. 
      Rivian   11654. 
        FIAT    9582. 
        Ford    9397. 
         Kia    8344.

Now, let’s see how the population of those makes has changed over time. Nissan led the pack from 2011-2015 before the Tesla takeover. As of 2024, Nissan had fallen into 5th place for the number of EVs registered in California.

(-> year-make
    (dataframe-filter* (Make) (member Make ($ top-makes 'Make)))
    (dataframe-spread 'Make 'Count)
    (dataframe-display 15 90))

 dim: 15 rows x 11 cols
    Year    Ford    Tesla  Nissan  Chevrolet    FIAT     Kia  Volkswagen  Hyundai    Audi  Rivian 
                                           
   2010.     51.     431.      na         na      na      na          na       na      na      na 
   2011.     45.     557.    3638         na      na      na          na       na      na      na 
   2012.    214.     673.    5899         na      na      na          na       na      na      na 
   2013.   1112.    7859.   12845        239     317      na          na       na      na      na 
   2014.   2095.   13673.   21623       1311    5868      30          na       na      na      na 
   2015.   3192.   23498.   28351       3087   12152     770        2158       na      na      na 
   2016.   3463.   39591.   31694       5425   16002    1617        5905       na      na      na 
   2017.   3726.   61747.   32853      17818   18748    2521        9745      342      na      na 
   2018.   3412.  129300.   32638      26570   15023    2653        8971      762      na      na 
   2019.   2817.  197453.   31600      33202   11322    3369       10081     3125    1449      na 
   2020.   1953.  261018.   30113      33065    8017    4169        8586     4540    4010      na 
   2021.   8297.  378014.   32841      39277    7430    6957       13461     8181    7152     265 
   2022.  20004.  550093.   34503      46680    7034   13791       17771    17693   11983    4817 
   2023.  36936.  761556.   36653      60958    6629   21534       29180    33672   19557   14940 
   2024.  53642.  935895.   39733      72921    6441   34371       32306    54326   27737   26592

Let’s look at the same data as proportions. Unlike R’s dplyr where you can pair group_by with mutate for grouped operations, in dataframe we need to explicitly split, apply, and combine. -> and ->> are the thread first and last operators, respectively. We use ->> to pass a list as the last argument to map. In this call to dataframe-modify*, we calculate a scalar value for each row of the dataframe and that scalar is automatically repeated for all rows in dfx.

(define (round-any x places)
  (let ([factor (expt 10 places)])
    (/ (round (* x factor)) factor)))

(-> year-make
    (dataframe-filter* (Make) (member Make ($ top-makes 'Make)))
    (dataframe-split 'Year)
    (->> (map
          (lambda (dfx)
            (dataframe-modify*
             dfx (Total () (sum ($ dfx 'Count)))))))
    (dataframe-bind-all)
    (dataframe-modify*
     (Prop (Count Total) (round-any (inexact (/ Count Total)) 3)))
    (dataframe-drop* Count Total)
    (dataframe-spread 'Make 'Prop)
    (dataframe-display 15 90))


 dim: 15 rows x 11 cols
    Year    Ford   Tesla  Nissan  Chevrolet    FIAT     Kia  Volkswagen  Hyundai    Audi  Rivian 
                                          
   2010.  0.1060  0.8940      na         na      na      na          na       na      na      na 
   2011.  0.0110  0.1310   0.858         na      na      na          na       na      na      na 
   2012.  0.0320  0.0990   0.869         na      na      na          na       na      na      na 
   2013.  0.0500  0.3510   0.574      0.011   0.014      na          na       na      na      na 
   2014.  0.0470  0.3070   0.485      0.029   0.132   0.001          na       na      na      na 
   2015.  0.0440  0.3210   0.387      0.042   0.166   0.011       0.029       na      na      na 
   2016.  0.0330  0.3820   0.306      0.052   0.154   0.016       0.057       na      na      na 
   2017.  0.0250  0.4190   0.223      0.121   0.127   0.017       0.066    0.002      na      na 
   2018.  0.0160  0.5900   0.149      0.121   0.068   0.012       0.041    0.003      na      na 
   2019.  0.0100  0.6710   0.107      0.113   0.038   0.011       0.034    0.011   0.005      na 
   2020.  0.0050  0.7340   0.085      0.093   0.023   0.012       0.024    0.013   0.011      na 
   2021.  0.0170  0.7530   0.065      0.078   0.015   0.014       0.027    0.016   0.014   0.001 
   2022.  0.0280  0.7590   0.048      0.064    0.01   0.019       0.025    0.024   0.017   0.007 
   2023.  0.0360  0.7450   0.036       0.06   0.006   0.021       0.029    0.033   0.019   0.015 
   2024.  0.0420  0.7290   0.031      0.057   0.005   0.027       0.025    0.042   0.022   0.021

I’m not generally one to care about the uniqueness of my car, but let’s see how common the few models that I considered were in my county in recent years. I was primarily choosing between the Nissan LEAF and the BMW i3, but I included a couple of others here. I would have considered the Hyundai IONIQ Electric more seriously, but they weren’t as easy to find. I never strongly considered the Chevy Bolt because of a battery recall related to battery fires, but the LEAF and the Bolt have dominated the affordable EV market.

(-> df-bev
    (dataframe-filter*
     (Year County Model)
     (and (member Year '(2022 2023 2024))
          (string=? County "Sacramento")
          (member Model '("LEAF" "i3" "IONIQ Electric" "Bolt EV"))))
    (dataframe-drop* County)
    (dataframe-sort* (string>? Make) (< Year))
    (dataframe-display 12))

 dim: 12 rows x 4 cols
    Year       Make           Model   Count 
                        
   2022.     Nissan            LEAF   1713. 
   2023.     Nissan            LEAF   1752. 
   2024.     Nissan            LEAF   1755. 
   2022.    Hyundai  IONIQ Electric     79. 
   2023.    Hyundai  IONIQ Electric     69. 
   2024.    Hyundai  IONIQ Electric     66. 
   2022.  Chevrolet         Bolt EV   1376. 
   2023.  Chevrolet         Bolt EV   1355. 
   2024.  Chevrolet         Bolt EV   1472. 
   2022.        BMW              i3     91. 
   2023.        BMW              i3     93. 
   2024.        BMW              i3     89.

Let’s look at the relative popularity over time of the different models offered by a few key manufacturers. Nissan is notable for not offering many options and my understanding is that Nissan will stop selling the ARIYA in North America after the introduction of the 3rd generation LEAF in 2026. I was disappointed to see that there are more Cybertrucks than IONIQ 6’s on California roads in 2024. The IONIQ 6 is a bigger car than I usually prefer, but I appreciate the emphasis on efficiency via the sleek shape.

(define (count-models df make)
  (-> df
      (dataframe-filter* (Make) (string=? Make make))
      (dataframe-aggregate*
       (Year Model)
       (Count (Count) (sum Count)))
      (dataframe-spread 'Model 'Count)))

(dataframe-display (count-models df-bev "Nissan") 15)

 dim: 14 rows x 3 cols
    Year    LEAF   ARIYA 
          
   2011.   3638.      na 
   2012.   5899.      na 
   2013.  12845.      na 
   2014.  21623.      na 
   2015.  28351.      na 
   2016.  31694.      na 
   2017.  32853.      na 
   2018.  32638.      na 
   2019.  31600.      na 
   2020.  30113.      na 
   2021.  32841.      na 
   2022.  34503.      na 
   2023.  33742.    2911 
   2024.  32501.    7232 

(dataframe-display (count-models df-bev "Tesla") 15)

 dim: 15 rows x 7 cols
    Year  Roadster  Model S  Model X  Model 3  Model Y  Cybertruck 
                                
   2010.      431.       na       na       na       na          na 
   2011.      557.       na       na       na       na          na 
   2012.      575.       98       na       na       na          na 
   2013.      568.     7291       na       na       na          na 
   2014.      559.    13114       na       na       na          na 
   2015.      562.    22936       na       na       na          na 
   2016.      568.    34343     4680       na       na          na 
   2017.      554.    47580    12737      876       na          na 
   2018.      541.    55816    20993    51950       na          na 
   2019.      517.    59798    27347   109791       na          na 
   2020.      460.    60869    33528   144311    21850          na 
   2021.      450.    65118    33483   197125    81838          na 
   2022.      417.    71617    43858   266246   167955          na 
   2023.      399.    74295    53424   338329   295097          12 
   2024.      379.    74678    60377   376489   414659        9313 

(dataframe-display (count-models df-bev "Hyundai") 15)

 dim: 8 rows x 5 cols
    Year  IONIQ Electric  KONA EV  IONIQ 5  IONIQ 6 
                           
   2017.            342.       na       na       na 
   2018.            762.       na       na       na 
   2019.           1428.     1697       na       na 
   2020.           1419.     3121       na       na 
   2021.           2031.     6150       na       na 
   2022.           1932.     7975     7786       na 
   2023.           1778.    10751    17069     4074 
   2024.           1611.    11209    33580     7926 

(dataframe-display (count-models df-bev "Chevrolet") 15)

 dim: 12 rows x 8 cols
    Year  Spark EV  Bolt EV  Bolt EUV    S-10  Blazer EV  Silverado EV  Equinox EV 
                                           
   2013.      239.       na        na      na         na            na          na 
   2014.     1311.       na        na      na         na            na          na 
   2015.     3087.       na        na      na         na            na          na 
   2016.     5425.       na        na      na         na            na          na 
   2017.     5273.    12545        na      na         na            na          na 
   2018.     4260.    22310        na      na         na            na          na 
   2019.     2762.    30440        na      na         na            na          na 
   2020.     2464.    30601        na      na         na            na          na 
   2021.     2356.    35870      1014      37         na            na          na 
   2022.     2207.    35094      9343      36         na            na          na 
   2023.     2051.    38280     20469      37         48            73          na 
   2024.     1864.    37837     21898      37       4217          1987        5081

The last tables made me curious about which models have had the biggest percent change from one year to the next. We use a split-apply-combine strategy again here to create a new column with the count for the next year. Lots of list reversing to create the Count-Next column. When displaying the results, I decided to constrain the output to only include models that had a count of at least 500 in the previous year. This approach eliminates examples like the Cybertruck which increased by 77508% from 2023 to 2024.

The Honda Clarity Electric is interesting for appearing on the list of biggest increases and then repeatedly appearing on the list of biggest decreases. The Tesla Model 3 made a huge splash when it went from less than 1000 cars in 2017 to over 50,000 in 2018.

(define df-bev-change
  (-> df-bev
      (dataframe-aggregate*
       (Year Make Model)
       (Count (Count) (sum Count)))
      (dataframe-split 'Make 'Model)
      (->> (map
            (lambda (dfx)
              (dataframe-modify*
               dfx (Count-Next () (reverse (cons 'na (reverse (cdr ($ dfx 'Count))))))))))
      (dataframe-bind-all)
      (dataframe-remove-na 'Count-Next)
      (dataframe-modify*
       (Change% (Count Count-Next) (inexact (* 100 (/ (- Count-Next Count) Count)))))))
               
(-> df-bev-change
    (dataframe-filter* (Count) (> Count 500))
    (dataframe-sort* (< Change%))
    (dataframe-display 20))

 dim: 253 rows x 6 cols
    Year           Make                  Model   Count  Count-Next   Change% 
                                               
   2013.            BMW                ActiveE    515.        251.  -51.2621 
   2022.          Honda       Clarity Electric    702.        425.  -39.4587 
   2017.  Mercedes-Benz                B-Class   2701.       1722.  -36.2458 
   2016.          Smart  Fortwo Electric Drive   3026.       1954.  -35.4263 
   2018.      Chevrolet               Spark EV   4260.       2762.  -35.1643 
   2020.          Honda       Clarity Electric   1613.       1050.  -34.9039 
   2021.          Honda       Clarity Electric   1050.        702.  -33.1429 
   2019.          Honda       Clarity Electric   2398.       1613.  -32.7356 
   2023.         Cruise              Cruise AV    930.        632.  -32.0430 
   2019.           Ford                  Focus   2745.       1913.  -30.3097 
   2019.           FIAT                   500e  11322.       8017.  -29.1910 
   2018.  Mercedes-Benz                B-Class   1722.       1230.  -28.5714 
   2017.          Honda                 Fit EV    667.        494.  -25.9370 
   2018.           FIAT                   500e  15023.      11322.  -24.6356 
   2019.  Mercedes-Benz                B-Class   1230.        939.  -23.6585 
   2019.            Kia                Soul EV   2221.       1712.  -22.9176 
   2016.         Toyota                RAV4 EV   2156.       1724.  -20.0371 
   2017.           FIAT                   500e  18748.      15023.  -19.8688 
   2017.      Chevrolet               Spark EV   5273.       4260.  -19.2111 
   2018.           Ford                  Focus   3328.       2745.  -17.5180 

(-> df-bev-change
    (dataframe-filter* (Count) (> Count 500))
    (dataframe-sort* (> Change%))
    (dataframe-display 20))

 dim: 253 rows x 6 cols
    Year           Make                  Model   Count  Count-Next    Change% 
                                                
   2017.          Tesla                Model 3    876.      51950.  5830.3653 
   2022.         Rivian                    R1S    579.       7659.  1222.7979 
   2021.      Chevrolet               Bolt EUV   1014.       9343.   821.4004 
   2022.           Audi              Q4 e-tron    645.       3593.   457.0543 
   2023.       Cadillac                  LYRIQ   1443.       6620.   358.7665 
   2022.            BMW                     iX   1510.       6281.   315.9603 
   2020.          Tesla                Model Y  21850.      81838.   274.5446 
   2022.            BMW                     i4   3207.      11647.   263.1743 
   2020.        Porsche                 Taycan   1047.       3710.   254.3457 
   2021.       Polestar                      2   1269.       4155.   227.4232 
   2022.          Lucid                    Air   1043.       3258.   212.3682 
   2022.  Mercedes-Benz                    EQS   3463.      10134.   192.6364 
   2013.          Smart  Fortwo Electric Drive    654.       1842.   181.6514 
   2019.           Audi                 e-tron   1449.       4010.   176.7426 
   2022.           Ford        F-150 Lightning   2331.       6383.   173.8310 
   2015.     Volkswagen                 e-Golf   2158.       5905.   173.6330 
   2016.          Tesla                Model X   4680.      12737.   172.1581 
   2017.          Honda       Clarity Electric    727.       1844.   153.6451 
   2023.         Nissan                  ARIYA   2911.       7232.   148.4370 
   2021.           Ford         Mustang Mach-E   6367.      15817.   148.4215

NetLogo: a retrospective

Sun, 18 May 2025 00:00:00 GMT

In this post, I briefly reflect on my experiences with NetLogo, which played a key role in my professional life.

My meandering graduate school research experiences were all broadly related to predator-prey interactions, foraging behavior, and animal movement. During my PhD (the 2nd try), I was studying the movement behavior of ladybug larvae after they consumed low-quality, potentially toxic, prey. I was interested in modeling their movement with the potential of eventually exploring how high-quality prey could benefit from the association with low-quality prey.

My PhD advisor, an R programmer, understandably encouraged me to pursue this modeling project with R, but I had stumbled across a paper with a simple animal movement or foraging model that included the NetLogo code in the appendix. I was primarily using R for statistics at the time and I had a clearer vision for how I would get started on this project in NetLogo than in R, which is not surprising because NetLogo is much better suited for 2D movement modeling than R.

I briefly scanned my library of old papers, but my memory is too hazy on the details to find the actual paper.

I was quickly hooked on NetLogo. The NetLogo environment is beginner friendly, the core ideas and approach are intuitive, and the built-in graphical approach provides quick visual feedback on whether your code is doing what you think it is doing. Detractors will say that NetLogo is only useful for toy models and prototyping, but it is a capable platform for agent-based models. I think it is fair to say that my current love of programming grew more out of my experiences with NetLogo than MATLAB or R.

My initial attempts to build a model of ladybird larvae movement centered on how to best implement the behavioral rules governing their movement. I was mostly flailing around while trying different ideas (which was fun!) and largely ignorant of the state of the art, particularly in the physics literature. After presenting on my initial modeling attempts to the grad students and faculty in my department, my dear, late friend, Ben Nolting, gently nudged me in the direction of more elegant approaches. Our work together on this project eventually led to this paper. We didn’t publish the NetLogo code with the paper, but I later put it on GitHub.

Ben also played a critical and supportive role in a short-lived consulting business.

Even after leaving academia for consulting, I hoped to work in my free time on extending our model to incorporate a second resource type to pursue my original interest in using the model to study shared predation. I made some progress [code], but my motivation was insufficient to stay on top of the literature and chase down any novel ideas that might arise from using our movement model as the mechanism of shared predation.

My last tango with NetLogo involved trying to replicate Hill et al. (2003). I had discovered ReScience C, which has the goal of encouraging replication of published computational research. Not only was I philosophically aligned with the ReScience mission, but I was excited at the prospect of doing research where the emphasis wasn’t on novelty and selling my ideas. I reached out to Ben about the possibility of collaborating on a side project for ReScience and proposed replicating Hill et al. (2003), which complemented our paper.

Our work, which represents very general search behavior, and that of Hill et al. (2003), which focused on a specific system, provide complementary evidence for the potential importance of composite foraging strategies that are not based on time.

Ben thought it sounded like a fun side project. We exchanged numerous emails about how to implement the model in NetLogo [code], but couldn’t reconcile some early discrepancies in the results and lost our momentum. Replication is difficult. Human language lacks the precision of programming languages and trying to figure out what someone else meant was not enough fun to keep pushing forward on this side project.

I spent many years (too many!) as a graduate student where I dreamed of a productive career as a research scientist. I have had a very hard time letting go of that dream. In my first job after my PhD, I worked for a company, Cramer Fish Sciences, that conducted original research, published papers, etc. I wasn’t doing much research myself, but the proximity to research slowed and masked the death of my research scientist dream. I’m finally comfortable acknowledging that original research is in my rearview mirror and fully embrace my future as a data scientist and programmer. With that, I say a fond farewell to NetLogo and the central role it played in my programming journey.

Updating Shiny Scorekeeper app with Shiny modules and bslib

Sun, 27 Apr 2025 00:00:00 GMT

Shiny Scorekeeper is a basketball scorekeeper app built with the Shiny web framework for R. The app was initially built in 2018, but I recently decided to update it to improve maintainability and provide a more modern look. It’s sort of a strange choice to invest time in this project in 2025. I built the app for scoring videos of my son’s youth basketball games, but my son is no longer a youth and I haven’t used the app in years. However, I have a fondness for the app because I had fun building it initially and it served me well for the years that I used it.

This new version is not about adding new features. The core functionality in the app is relatively unchanged, e.g., I’m still using editable DataTables from the DT package to create and manage rosters. One of the big changes was splitting the code among Shiny modules and an R package. In the previous version, the server.R file was over 800 lines of code that I didn’t find very easy to maintain or extend.

I moved some of the code into an R package partly because I’ve recently had fun playing with Tk in R and Scheme (see this post) and thought it might be fun to try to build a version of the app with a Tk interface (as an alternative to the Shiny interface). I had plenty of moments, though, where I regretted adding the extra overhead of having that code in a separate R package, especially considering the payoff is mostly hypothetical and delayed (e.g., future maintenance or different interface). I also didn’t make any effort to think about whether the functions in the package are written in a general way that will work equally well with a Tk interface.

Despite building many Shiny apps over the years, including some reasonably complex apps, I had never used Shiny modules. One one of the primary objectives in this Shiny Scorekeeper refresh project was to learn how to use Shiny modules. My use of modules here was limited to the simplest use case, though, i.e., splitting code among files. The app is comprised of three sections (Roster, Scorekeeper, Stats Viewer) that led to obvious breaks for splitting into modules.

The biggest friction that I encountered with using modules is that auto reload is not triggered by changes to module files. Fortunately, the auto reload behavior will be included in a future version of Shiny (see this pull request). The most common mistake that I made with modules was forgetting to namespace UI elements, which often produces unexpected behavior without any errors sent to the R console. The only other aspect that I found a little tricky was understanding how to pass values around among the modules.

The other big change was switching from shinydashboard to bslib. Design is not one of my strengths so I tend to stick with default themes and colors. By default, shinydashboard uses too much color for my liking.

Image of old version of Scorekeeper section

The bslib default uses very little color, which looks modern to me, but bslib also provides many themes.

Image of new version of Scorekeeper section

I’ve started using bslib in nearly all of my Shiny apps. The default layouts look better than stock Shiny (or shinydashboard) apps and the elements compose nicely to create more complex layouts. I think the new bslib version of Shiny Scorekeeper uses space more effectively, which followed naturally from the different default layout options (e.g., panel navigation in the top bar instead of the sidebar for page_navbar).

I decided to stick to more of the standard Shiny UI components in this version. That was partly because I perceived a more uniform look with those elements when using bslib. The one 3rd-party component that I couldn’t give up was shinyWidgets::pickerInput. When using multiple select, the Shiny alternative (selectInput) is uglier and provides far fewer options (e.g., select/deselect all, live search, etc.).

The Stats Viewer in Shiny Scorekeeper changed more than any other section. Previously, I used DT tables as the mechanism for sorting and filtering the team and game data (by selecting rows in the tables). This approach allowed for compact presentation of the details associated with each team and game, but provided an unconventional interface that I ended up not liking.

Image of old version of Stats Viewer section

For the new version, I have hierarchical inputs to filter down to the desired game stats. This approach leads to a relatively large hierarchy: leagues -> teams -> seasons -> scoring margin -> opponents -> dates. Changes at the top of the hierarchy are propagated dynamically to update all of the other inputs. This requires more code than my previous approach with tables and could lead to notable lag in updating of the game stats table as data size increases.

Image of new version of Stats Viewer section

I also replaced the DT game stats table with a reactable version. DT tables have been my go-to option for Shiny apps for many years, but I recently read that the DT author thinks reactable is generally better than DT. I would have used reactable for all tables in Shiny Scorekeeper, but reactable tables are not editable so I needed to stick with DT tables for the roster section. I also think it is nice that the game stats table has a different look to reinforce that it is not editable.

Exploratory data analysis with Scheme, Gnuplot, and Tk

Thu, 26 Dec 2024 00:00:00 GMT

In my second post on this blog, I expressed an interest in learning how to build desktop applications. I have yet to pursue that interest. Instead, I’ve primarily continued developing Shiny apps deployed on the web (but see Shiny Scorekeeper). Recently, though, I’ve spent some time learning about the Tk GUI toolkit for developing desktop applications. In this post, I revisit an old post using the dataframe and gnuplot-pipe libraries for Scheme to conduct simple exploratory data analysis (EDA) and add an interface with chez-tk.

Tcl/Tk

A few months ago I saw on Hacker News that Tcl/Tk accounced a major release. My interest was piqued by the generally favorable comments and comparisons to Lisp. I also like the generality that comes from the large number of languages that provide bindings to Tk. One of those languages is my primary programming language, R. It seems that the GUI options for R have largely collapsed to only Shiny and tcltk. To satisfy my curiousity, I tackled a little project to learn more about the tcltk package in R.

I was also interested in available bindings to Tk for my favorite programming language, Scheme, and found PS/Tk.

The PSTK library has had a long history in the Scheme community and, in one form or another, is available for many Scheme implementations. The current file includes its history starting from an implementation of Chicken/Tk by Wolf-Dieter Busch from 2004 based on earlier code by Sven Hartrumpf from 1997. Nils Holm made the library portable, and so created PSTK. Ken Dickey created an R6RS version.

PS/Tk communicates with Tcl/Tk through a process port. The versions of PS/Tk code that I found did not include R6RS compatibility so I added the code needed to open a process port in Chez Scheme, converted from R7RS to R6RS library, called it chez-tk, and submitted it to Akku. I’ve collected examples for using chez-tk based on existing PS/Tk examples or translating examples from TkDocs. In translating those examples, I was primarily using tkinter for Python to understand how they work. I was impressed by the autocompletion and documentation for tkinter available through VS Code. For tcltk and chez-tk, you have to learn the translation rules and use the Tcl/Tk documentation.

After working up a set of chez-tk examples, the only apparent bug that I found was related to the inclusion of parentheses in listbox choices. It was a small change in the code to fix that bug, and I didn’t notice any negative impacts when running through the examples after making the change, but I also don’t understand why that procedure was written that way in the first place. That’s an uneasy feeling.

App Overview

When I first started making Shiny apps 10+ years ago, I was drawn to the potential to make my work in R accessible to non-R users. I didn’t anticipate the extent to which I would find graphical user interfaces to multi-dimensional datasets useful for my own data exploration efforts. This chez-tk example is made in that spirit, i.e., the app isn’t packaged into a standalone desktop application for other users. The user needs to know how to use Chez Scheme and Akku and needs to install Tcl/Tk and Gnuplot.

The app allows for filtering on years, months, and cities and summarizes an annual or monthly time series grouped by city for the selected response variable. Each click of the button will generate a plot in a new window.

Data Preparation

Libraries

All of the libraries in the import statement below are available through Akku. dataframe is used for data manipulation. Only one procedure is imported from wax irregex to process a string that is returned from Tcl/Tk. gnuplot-pipe is used for plotting. chez-tk allows us to build a user interface. [All of the code is available in a single file in this gist.]

(import (dataframe)
        (only (wak irregex) irregex-split)
        (prefix (gnuplot-pipe) gp:)
        (chez-tk))

Data

We are using the Texas housing dataset included as part of the ggplot2 package for R. I’ve written that dataset to a CSV file for use in this post.

> (define df (csv->dataframe "txhousing.csv"))

> (dataframe-display df)

 dim: 8602 rows x 9 cols
     city    year   month   sales    volume  median  listings  inventory       date 
                                       
  Abilene   2000.      1.     72.  5.380E+6  71400.      701.     6.3000  2000.0000 
  Abilene   2000.      2.     98.  6.505E+6  58700.      746.     6.6000  2000.0833 
  Abilene   2000.      3.    130.  9.285E+6  58100.      784.     6.8000  2000.1667 
  Abilene   2000.      4.     98.  9.730E+6  68600.      785.     6.9000  2000.2500 
  Abilene   2000.      5.    141.  1.059E+7  67300.      794.     6.8000  2000.3333 
  Abilene   2000.      6.    156.  1.391E+7  66900.      780.     6.6000  2000.4167 
  Abilene   2000.      7.    152.  1.264E+7  73500.      742.     6.2000  2000.5000 
  Abilene   2000.      8.    131.  1.071E+7  75000.      765.     6.4000  2000.5833 
  Abilene   2000.      9.    104.  7.615E+6  64500.      771.     6.5000  2000.6667 
  Abilene   2000.     10.    101.  7.040E+6  59300.      764.     6.6000  2000.7500

Global Variables

We define global variables based on df for use in the app. First, though, we need to define a helper procedure to double quote any strings in a list that have spaces because chez-tk appends those strings into one big string for passing to Tcl/Tk. Ideally, chez-tk would handle this for us, but, for now, I’m reluctant to make too many changes to chez-tk (see above).

We get the list of cities from the dataframe and remove duplicates. For some reason, there is a problem with three of the cities causing the app to crash with a message about “invalid listvar values.” I have no idea why just those three cities cause a problem, but, for now, I’ve decided not to try to chase down that problem. The other thing to point out is that the options for the combobox are defined as a single string in vars-labs, which also shows the double quoting requirement. When a response variable is selected in the app, the column name is looked up using the associations in vars.

(define (double-quote lst)
  (map (lambda (x)
         (let ([x-list (string->list x)])
           (if (member #\space x-list)
               (string-append "\"" x "\"")
               x)))
       lst))

(define cities (remove-duplicates ($ df 'city)))
(define cities
  (filter
   (lambda (x) (not (member x '("Montgomery County"
                                "Port Arthur"
                                "Wichita Falls"))))
   cities))
(define cities-dq (double-quote cities))

(define min-yr (apply min ($ df 'year)))
(define max-yr (apply max ($ df 'year)))
(define months '(Jan Feb Mar Apr May Jun
                     Jul Aug Sep Oct Nov Dec))
                     
(define vars '(("Median Sale Price" median)
               ("Sales" sales)
               ("Volume" volume)
               ("Listings" listings)
               ("Inventory" inventory)))
               
(define vars-labs "\"Median Sale Price\" Sales Volume Listings Inventory")

Filter

We create a small wrapper procedure around a standard dataframe-filter*. The month column in df is represented by numeric months. If we add one to the indices of the selected months, then we get the numeric months for use in the filter. We use the cities indices to get the city names from cities (not cities-dq). We remove all rows with missing values in resp-var with dataframe-remove-na. If there were missing values in year, month, and city columns, then we would need to add them to the dataframe-remove-na.

(define (filter-data df min-yr max-yr months-idx cities-idx cities resp-var)
  (let ([months-sel (map add1 months-idx)]
        [cities-sel (map (lambda (x) (list-ref cities x)) cities-idx)])
    (-> df
        (dataframe-remove-na resp-var)
        (dataframe-filter*
         (city year month)
         (and (>= year min-yr)
              (<= year max-yr)
              (member month months-sel)
              (member city cities-sel))))))

Aggregate

As with filtering, we are wrapping dataframe-aggregate. In this case, though, we can’t use the macro version (indicated with a trailing *). The macro versions of the dataframe verbs are intended for interactive use and provide simpler syntax. The grouping variables are city and xvar. The new column is named mean-rv where ‘rv’ stands for response variable. (list (list resp-var)) provides the names of the columns used in the lambda expressions (one sub-list for each expression).

(define (agg-data df xvar resp-var)
  (dataframe-aggregate
   df
   (list 'city xvar)
   '(mean-rv)
   (list (list resp-var))
   (lambda (resp-var) (exact->inexact (mean resp-var)))))

Plot

gp:send sends commands to Gnuplot as strings. When setting the axis labels, we need to surround the label with single quotation marks to distinguish the label from the rest of the command string. To plot multiple sets of data, gp:plot accepts a list where the first item is a string with optional properties (e.g., title provides a label for the legend), the second is a list with x-coordinates, and the third is a list with y-coordinates.

(define (plot-data df x y xvar-str resp-var-str)
  (gp:call/gnuplot
   (gp:send "set key top left")
   (gp:send (string-append "set xlabel \'" xvar-str "\'"))
   (gp:send (string-append "set ylabel \'Avg. " resp-var-str "\'"))
   (gp:send "set style data linespoints")
   (gp:plot
    (map (lambda (c)
           (let ([df-sub (dataframe-filter*
                          df
                          (city)
                          (string=? c city))])
             (list
              (string-append "title '" c "'")
              ($ df-sub x)
              ($ df-sub y))))
         (remove-duplicates ($ df 'city))))))

App Details

Named Frames and Widgets

Tile provides reimplementations of many classic widgets in the ttk namespace. In the first line in the code block below, we opt to use the Tile versions over the classic versions for all widgets with ttk-map-widgets. tk-start initializes the Tk shell.

In chez-tk, widgets are represented as procedures that can be used to configure the widget. In this app, we use frames, labels, spinboxes, listboxes, radiobuttons, a combobox, and a button, but only a few of them require names for subsequent configuration. The named procedures also specify the relationship of the frames and widgets, e.g., tk creates frame and frame creates months-lb, cities-lb, vars-cb, and all other widgets.

Commands are represented as symbols (e.g., 'create-widget) whereas parameters are represented as symbols with trailing colons (e.g., 'height:). Scheme symbols can be used in place of strings and Scheme values such as #f are converted to the Tcl/Tk equivalent. tk-var associates a Tk variable name with a widget.

When 'exportselection: is set to #t, clicking outside of the listbox deselects any listbox selections. For multiple selections in listboxes, the 'selectmode: needs to be set to 'multiple or 'extended.

If the selection mode is multiple or extended, any number of elements may be selected at once, including discontiguous ranges. In multiple mode, clicking button 1 on an element toggles its selection state without affecting any other elements. In extended mode, pressing button 1 on an element selects it, deselects everything else, and sets the anchor to the element under the mouse; dragging the mouse with button 1 down extends the selection to include all the elements between the anchor and the element under the mouse, inclusive.

(ttk-map-widgets 'all)
(define tk (tk-start))
(define frame (tk 'create-widget 'frame 'padding: '(10 10 10 10)))
(define months-lb
  (frame 'create-widget 'listbox 'listvariable: (tk-var 'months-tk)
         'height: 5 'exportselection: #f 'selectmode: 'extended))
(define cities-lb
  (frame 'create-widget 'listbox 'listvariable: (tk-var 'cities-tk)
         'height: 10 'exportselection: #f 'selectmode: 'extended))
(define vars-cb
  (frame 'create-widget 'combobox 'values: vars-labs 'state: 'readonly))

App Layout

We are using the grid geometry manager with a simple layout of three columns and nine rows all contained in a single frame. Widgets are sized to the content so a long label like Response Variable should be set to span multiple columns to prevent undesirable extra space. The 'sticky: parameter uses cardinal directions ('nwes) for alignment of widgets. Most widgets can be created within a call to tk/grid because there is no subsequent configuration of those widgets, just setting and getting the Tk variable.

(tk/grid frame)
(tk/grid (frame 'create-widget 'label 'text: "Years")
         'column: 0 'row: 0 'sticky: 'w 'pady: 5)
(tk/grid (frame 'create-widget 'spinbox 'from: min-yr 'to: max-yr
                'textvariable: (tk-var 'min-yr-tk) 'width: 5)
         'column: 1 'row: 0 'sticky: 'w)
(tk/grid (frame 'create-widget 'spinbox 'from: min-yr 'to: max-yr
                'textvariable: (tk-var 'max-yr-tk) 'width: 5)
         'column: 2  'row: 0 'sticky: 'w)

(tk/grid (frame 'create-widget 'label 'text: "Months")
         'column: 0 'row: 1 'sticky: 'w)
(tk/grid months-lb 'column: 0 'row: 2 'columnspan: 3 'sticky: 'we 'pady: 5)

(tk/grid (frame 'create-widget 'label 'text: "Cities")
         'column: 0 'row: 3 'sticky: 'w)
(tk/grid cities-lb 'column: 0 'row: 4 'columnspan: 3 'sticky: 'we 'pady: 5)

(tk/grid (frame 'create-widget 'label 'text: "X Variable")
         'column: 0 'row: 5 'sticky: 'w 'pady: 5)
(tk/grid (frame 'create-widget 'radiobutton 'text: "Year" 'value: "Year"
                'variable: (tk-var 'xvar-tk))
         'column: 1 'row: 5 'sticky: 'e)
(tk/grid (frame 'create-widget 'radiobutton 'text: "Month" 'value: "Month"
                'variable: (tk-var 'xvar-tk))
         'column: 2 'row: 5 'sticky: 'e)

(tk/grid (frame 'create-widget 'label 'text: "Response Variable")
         'column: 0 'row: 6 'columnspan: 3 'sticky: 'w)
(tk/grid vars-cb 'column: 0 'row: 7 'columnspan: 3 'sticky: 'we)

Commands

In this app, we only have one command, plot-cmd, which filters, aggregates, and plots data. plot-cmd is associated with the plot button via the 'command: parameter. A command procedure takes no arguments. Within plot-cmd, we retrieve the state of all of the app widgets at the time that the button was clicked and then pass those values to the procedures that we described above, i.e., filter-data, agg-data, and plot-data.

For several widgets, we use tk-get-var with the Tk variable name to get the current widget value and convert it to the appropriate type. For the combobox, we use the Scheme name (vars-cb) with 'get. Similarly, for listboxes, we use the Scheme name with 'curselection, which returns a string of the selected indices, e.g., "0 3 4 8". prepare-curselection splits that string into a list of numeric indices for use in filter-data.

If the filtering and aggregating steps produce an empty dataframe, then clicking on the plot button has no effect (because otherwise the app would crash). Ideally, the user would receive feedback on why the plot isn’t displayed, but, unfortunately, that feature is not a simple addition to the app (based on my current understanding of Tcl/Tk).

(define (prepare-curselection x)
  (map string->number (irregex-split " " x)))

(define plot-cmd
  (lambda ()
    (let* ([xvar-str (tk-get-var 'xvar-tk)]
           [xvar (if (string=? xvar-str "Year") 'year 'month)]
           [rv-str (vars-cb 'get)]
           [rv (cadr (assoc rv-str vars))]
           [df-sub (filter-data
                    df
                    (string->number (tk-get-var 'min-yr-tk))
                    (string->number (tk-get-var 'max-yr-tk))
                    (prepare-curselection (months-lb 'curselection))
                    (prepare-curselection (cities-lb 'curselection))
                    cities
                    rv)])
      ;; can't aggregate empty dataframe
      (when (> (car (dataframe-dim df-sub)) 0)
        (plot-data (agg-data df-sub xvar rv) xvar 'mean-rv xvar-str rv-str)))))

(tk/grid (frame 'create-widget 'button 'text: "Plot" 'command: plot-cmd)
         'column: 0 'row: 8 'columnspan: 3 'sticky: 'we 'pady: 5)

Initial Values

For spinboxes and radiobuttons, tk-set-var! sets the initial value. For the combobox, the initial value is set with the Scheme name, vars-cb, and 'set. For listboxes, tk-set-var! sets the options, but the initial values are set with the Scheme name. Here is the Tcl/Tk documentation for listbox selection set:

pathName selection set first ?last?
Selects all of the elements in the range between first and last, inclusive, without affecting the selection state of elements outside that range.

This nicely illustrates the translation between Tk and chez-tk where the Scheme name is used in place of the pathName and the rest of the expression is almost identical. For months-lb, we initially select all months. For cities-lb, we are selecting multiple elements that are not part of an inclusive range so we set the selected values iteratively. We use a helper procedure, get-idx, to get indices for a subset of the cities.

(tk-set-var! 'min-yr-tk min-yr)
(tk-set-var! 'max-yr-tk max-yr)
(tk-set-var! 'months-tk months)
(tk-set-var! 'cities-tk cities-dq)
(tk-set-var! 'xvar-tk "Year")
(vars-cb 'set "Median Sale Price")
(months-lb 'selection 'set 0 11)

(define (get-idx lst lst-sub)
  ;; get indices of lst-sub from lst
  (let* ([idx (iota (length lst))]
         [lst-idx (map (lambda (x i) (cons x i)) lst idx)])
    (map (lambda (y) (cdr (assoc y lst-idx))) lst-sub)))

(for-each (lambda (x) (cities-lb 'selection 'set x))
          (get-idx cities '("Austin" "Dallas" "El Paso" "Houston"
                            "Lubbock" "San Antonio")))

Conclusions

I have enjoyed learning the basics of making GUIs with Tk in Scheme and R. I don’t mind the outdated look of the widgets and I like the compactness of the interface (compared to a Shiny app). I think it would be fun to make a chez-tk version of my Shiny-Scorekeeper app (or even a tcltk version in R). I’m also interested in the possibility of packaging chez-tk and tcltk apps into standalone executables.

Reading JSON files as Scheme dataframes

Wed, 06 Nov 2024 00:00:00 GMT

In a previous post, I wrote about reading and writing JSON files in R and Chez Scheme. After updating that post, I was curious about how much code it would take to convert a Scheme object read from a JSON file into a dataframe. It is arguably a rare circumstance that a dataframe is the best data representation for JSON. In this post, we will tackle the limited case of reading JSON files that were created by writing R dataframes to JSON.

First, let’s use the jsonlite package for R to write the palmerpenguins dataset to JSON. toJSON provides the option to convert to row- or column-based JSON formats. [The JSON files created with the R code below are available here and here.]

library(jsonlite)
library(palmerpenguins)

writeLines(toJSON(penguins, dataframe = "rows"), 
           "penguins-row.json")
writeLines(toJSON(penguins, dataframe = "columns"), 
           "penguins-col.json")

Next, let’s import our Scheme libraries and read the data. We are using the json-tools library for reading JSON.

(import (json)
        (dataframe))

(define json-row
  (call-with-input-file "penguins-row.json" json-read))

(define json-col
  (call-with-input-file "penguins-col.json" json-read))

Tabular data in JSON can be oriented by rows or columns.

However, unfortunately R is an exception in its preference for column-based storage: most languages, systems, databases, APIs, etc, are optimized for record based operations. For this reason, the conventional way to store and communicate tabular data in JSON seems to almost exclusively row based.

Given that, let’s start with json-row. json-read converts the row-based format into a list of vectors where each vector represents a row comprised of pairs with the column name and row value.

> (car json-row)

#(("species" . "Adelie") ("island" . "Torgersen")
  ("bill_length_mm" . 39.1) ("bill_depth_mm" . 18.7)
  ("flipper_length_mm" . 181) ("body_mass_g" . 3750)
  ("sex" . "male") ("year" . 2007))

toJSON handles missing values in the row-based format by not including the name-value pair.

> (list-ref json-row 3)

#(("species" . "Adelie") ("island" . "Torgersen")
  ("year" . 2007))

To convert to a dataframe, we need the column names, but we can’t just use (car json-row) because it might not include all of the column names. The following procedure gets the length of each row n and finds the number of columns n-max. We cons the n values onto an index ind because we use list-ref to get one of the rows from json-row that includes all columns. From there, we extract the column names from that row. Of course, if you know, for example, that the first row contains all columns, then you can just use (map car (vector->list (car json-row))).

(define (get-names-str json-row)
  (let* ([ind (iota (length json-row))]
         [n (map vector-length json-row)]
         [n-max (apply max n)]
         [n-sub (filter (lambda (pair)
                          (= (car pair) n-max))
                        (map cons n ind))])
    (map car (vector->list (list-ref json-row (cdr (car n-sub)))))))

> (get-names-str json-row)

("species" "island" "bill_length_mm" "bill_depth_mm"
  "flipper_length_mm" "body_mass_g" "sex" "year")

The row-based JSON values are extracted into a row-based list of lists with get-vals. We use nested maps to map over all rows and all column names. If a column name is not found (via assoc), then we return ’na (symbol used for missing values in the dataframe library) to create sub-lists that all have the same length.

(define (get-vals json-row names-str)
  (map (lambda (vec)
         (map (lambda (nm)
                (let ([pair (assoc nm (vector->list vec))])
                  (if pair (cdr pair) 'na)))
              names-str))
       json-row))

> (car (get-vals json-row (get-names-str json-row)))

("Adelie" "Torgersen" 39.1 18.7 181 3750 "male" 2007)

> (list-ref (get-vals json-row (get-names-str json-row)) 3)

("Adelie" "Torgersen" na na na na na 2007)

We transpose the values into a list of columns that is the same length as our column names. A dataframe in Scheme is comprised of a list of series that are all the same length. make-series takes a name as symbol and a list of values to make a new series. transpose is provided by the dataframe library.

(define row-df
  (let ([names-str (get-names-str json-row)])
    (make-dataframe
     (map (lambda (nm lst)
            (make-series nm lst))
          (map string->symbol names-str)
          (transpose (get-vals json-row names-str))))))

> (dataframe-glimpse row-df)

 dim: 344 rows x 8 cols
 species               Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, ... 
 island                Torgersen, Torgersen, Torgersen, Torgersen, ...     
 bill_length_mm        39.1, 39.5, 40.3, na, 36.7, 39.3, 38.9, 39.2, ...   
 bill_depth_mm         18.7, 17.4, 18, na, 19.3, 20.6, 17.8, 19.6, ...     
 flipper_length_mm     181, 186, 195, na, 193, 190, 181, 195, 193, ...     
 body_mass_g           3750, 3800, 3250, na, 3450, 3650, 3625, 4675, ...   
 sex                   male, female, female, na, female, male, female, ... 
 year                  2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, ...

Let’s switch gears to working with the column-based case and the json-col object that we created earlier. json-read converts the column-based format into a vector of lists where the first item of each list is the column name and the other values are the column values.

> (list-head (list-ref (vector->list json-col) 5) 10)

("body_mass_g" 3750 3800 3250 "NA" 3450 3650 3625 4675 3475)

> (list-head (list-ref (vector->list json-col) 6) 10)

("sex" "male" "female" "female" null "female" "male"
  "female" "male" null)

Because the column-based format is similar to the structure of a dataframe, the code to convert json-col to a dataframe is simpler.

(define col-df
  (make-dataframe
   (map (lambda (col)
          (make-series
           (string->symbol (car col))
           (cdr col)))
        (vector->list json-col))))

> (dataframe-glimpse col-df)

 dim: 344 rows x 8 cols
 species               Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, ... 
 island                Torgersen, Torgersen, Torgersen, Torgersen, ...     
 bill_length_mm        39.1, 39.5, 40.3, na, 36.7, 39.3, 38.9, 39.2, ...   
 bill_depth_mm         18.7, 17.4, 18, na, 19.3, 20.6, 17.8, 19.6, ...     
 flipper_length_mm     181, 186, 195, na, 193, 190, 181, 195, 193, ...     
 body_mass_g           3750, 3800, 3250, na, 3450, 3650, 3625, 4675, ...   
 sex                   male, female, female, null, female, male, ...       
 year                  2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, ...

For missing values in the column-based format, toJSON uses NA for numbers and null for strings. json-read leaves the NA as is and converts the null to 'null, but make-dataframe automatically converts NA to 'na (missing value representation) and 'null to "null" (because of implicit conversion to maintain same type throughout the column). It isn’t strictly necessary to convert "null" to 'na, but we will do that below. If you already know that only the sex column has null values, then you can modify just that column. Alternatively, we can identify the names of the string columns with some mapping and filtering and apply the same procedure to all string columns.

(define col-df2
  (dataframe-modify*
   col-df
   (sex (sex) (if (string=? sex "null") 'na sex))))

(define col-df3
  (let* ([name-type (map (lambda (series)
                           (cons (series-type series)
                                 (series-name series)))
                         (dataframe-slist col-df))]
         [str-names (map cdr (filter
                              (lambda (pair)
                                (symbol=? (car pair) 'str))
                              name-type))])
    (apply
     dataframe-modify-at
     col-df
     (lambda (x) (if (string=? x "null") 'na x))
     str-names)))

> (dataframe-equal? row-df col-df)
#f
> (dataframe-equal? row-df col-df2 col-df3)
#t

I thought about adding this functionality to the dataframe library, but I don’t think it is a common enough use case to warrant the additional code maintenance and dependency on json-tools.

Horse racing game win probability

Sat, 26 Oct 2024 00:00:00 GMT

When gathering with family, we like to play yard, card, and board games. On a recent visit, one of the favorite games was the Across the Board Kentucky Derby Horse Racing Game. The game produced a lot of cheers and jeers and provided a fun diversion with no skill and little concentration required. After losing a little money, though, I lost interest in playing the game and decided to write some R code to simulate the game and generate win probabilities.

The game involves dealing a deck of cards (with Kings and Aces removed) to any number of players (well, up to 44 players where each player would be dealt one card). The cards in a player’s hand represent wagers on the winning horse where horses are numbered 2-12 and the jack and queen cards represent 11 and 12, respectively. Two dice are rolled and the total of the dice determines which horse moves around the board. The number of steps for a horse to win is scaled roughly in proportion to the probability of that number being rolled. Below we enumerate all of the possible rolls with expand.grid, calculate the probability of each roll, and compare those probabilities to the probabilities based on the number of steps included on the game board.

rolls_df = expand.grid(dice1 = 1:6, dice2 = 1:6) |>
  dplyr::mutate(roll = dice1 + dice2) |>
  dplyr::count(roll) |>
  dplyr::mutate(
    steps = c(3, 6, 8, 11, 14, 16, 14, 11, 8, 6, 3),
    prob = n / sum(n),
    prob_steps = steps / sum(steps)
  )

> rolls_df

   roll n steps       prob prob_steps
1     2 1     3 0.02777778       0.03
2     3 2     6 0.05555556       0.06
3     4 3     8 0.08333333       0.08
4     5 4    11 0.11111111       0.11
5     6 5    14 0.13888889       0.14
6     7 6    16 0.16666667       0.16
7     8 5    14 0.13888889       0.14
8     9 4    11 0.11111111       0.11
9    10 3     8 0.08333333       0.08
10   11 2     6 0.05555556       0.06
11   12 1     3 0.02777778       0.03

We can use the sample function to simulate rolling the dice. sample.int provides better performance, but requires adding 1L to shift our samples from 1:11 to our horse numbers of 2:12.

roll <- function(n, replace = TRUE, prob = rolls_df$prob) {
  sample.int(11, size = n, replace = replace, prob = prob) + 1L
}

> round(table(roll(500000))/500000, 3)

    2     3     4     5     6     7     8     9    10    11    12 
0.028 0.055 0.084 0.111 0.139 0.167 0.139 0.111 0.083 0.056 0.028

With this board setup, all horses have a roughly equal probability of winning because the horses that are rolled more rarely have fewer steps to move around the board. The game dynamics are made more interesting by introducing scratches, i.e., horses that are not able to participate in a race. We can use our roll function to generate the scratches. The number of scratches is always four different horses.

> roll(4, replace = FALSE)
[1] 8 5 4 3

The scratches determine how much players must contribute to the kitty. The first scratch horse is 1x and the last is 4x. Players pay the scratch value for every card in their hand that matches a scratch horse. We played with quarters so the starting kitty was always $10. Every time a player rolls a value that matches the scratch horse, they have to contribute that scratch amount to the kitty. The following function calculates the value of the kitty given a vector of dice rolls. In the example below, each rolled 7 would require the player to put a quarter in the kitty and the 10 would require $0.75.

get_kitty <- function(base_value, scratches, rolls = NULL) {
  init = 4 * base_value * (4 + 3 + 2 + 1) # multiply by 4 for 4 suits in deck
  vals = NULL
  if (!is.null(rolls) && length(rolls) > 0) {
    vals = sapply(rolls, function(x) {
      mult = which(scratches == x) # index in scratches indicates multiplier
      if (length(mult) > 0) mult * base_value else 0
    })
  }
  sum(c(init, vals))
}

> base_value = 0.25
> scratches = roll(4, FALSE)
> rolls = roll(5)
> scratches
[1]  7  3 10  9
> rolls
[1]  7  4  6  7 10
> get_kitty(base_value, scratches)
[1] 10
> get_kitty(base_value, scratches, rolls)
[1] 11.25

Rather than trying to do the math to determine how likely each horse is to win. I used Monte Carlo simulations to find the win probabilities for any point in a game (described by vector of rolls). I’ve covered most of the basic logic of the calculations here, but all of the code is available through GitHub.

sim_one_game <- function(scratches, rolls, winner_only = TRUE) {
  steps_remain = get_steps_remain(scratches, rolls)
  active = !is.na(steps_remain)
  sr = steps_remain
  if (any(sr[active] < 0)) {
    stop("Winner was already determined")
  }
  # conservative choice; highly unlikely to need 200 rolls to finish a game
  sim_rolls_pool = roll(200)
  i = 1
  while (all(sr[active] > 0)) {
    r = sim_rolls_pool[i] # integer 2–12
    i = i + 1
    idx = r - 1L # integer 1–11
    if (active[idx]) {
      sr[idx] = sr[idx] - 1L
    }
  }
  winner = names(which(sr == 0))
  if (winner_only) {
    winner
  } else {
    sim_rolls = sim_rolls_pool[seq_len(i - 1)] # trim to actual length
    list("winner" = winner, "rolls" = c(rolls, sim_rolls))
  }
}

I ran 100,000 simulations of the game to determine which horses win most often. 2 or 12 each win about 20% of the time, 3/11 and 4/10 ~ 9.5%, 5/9 = 5.5%, 6/7/8 = 3.5%. The primary driver behind that outcome is that 2 and 12 are less likely to end up as scratches than 3 and 11 and so on. 4/10 have roughly the same win probability as 3/11, despite 4/10 requiring more steps to move around the board than 3/11, which is a byproduct of how the board is discretized and is evident in rolls_df where the prob is higher than the prob_steps for 4/10. Same idea applies to why 7 has a similar win probability to 6/8.

The figure below shows the kitty distribution paneled by the winning horse for those 100,000 simulated games with a base value of $0.25. The overall average kitty was $24.45 with the winner-specific averages ranging from $22.30 (12) to $26.80 (8). The kitty grows largest in games where 6/7/8 win because it takes the most steps for those horses to make it around the board.

Lastly, I thought it would be fun to record the game live and provide updates on which horse was most likely to win as the game progressed. I was initially doing that within my original R script, but I decided to make a Shiny app to facilitate that process. The app layout is not great and it is missing some useful functionality (e.g., roll history not displayed and not able to undo if wrong button was clicked).

UPDATE (2026-04-01): I have been resistant to adopting AI. I would like to say that my resistance was born of noble intentions based on environmental and ethical concerns, but it arguably arose more from my negative emotions related to AI potentially taking away my favorite tasks at work. Recently, though, I have started experimenting with Claude and found myself enjoying finishing long languishing side projects. One example was improving my little Shiny app for the horse game. It is now much more compact and useable on a mobile device. The new version includes the roll history, an undo button, and disabling of inputs for invalid states. I also dropped the plots in favor of a cleaner approach with value boxes.

My initial version performed a lot of the calculations within dataframes using dplyr functionality. I found that approach intuitive, but it was slow so I rewrote it with the vector approach (see repo).

RPI and SOS in Scheme, Python, and Elixir

Mon, 02 Sep 2024 00:00:00 GMT

Last spring, I played in a 3x3 basketball leage with 14 teams and only 6 regular-season games. The unbalanced schedule made me wonder if we would end up with wonky playoff seeding. I thought it would be fun to calculate the Rating Percentage Index (RPI) and Strength Of Schedule (SOS) for each team to assess discrepancies between W-L record and team rating. I was mostly following the R code for RPI in this post and the SOS calculations here. For comparison, I wrote code in Scheme (dataframe), Python (Polars), and Elixir (Explorer), which is also using Polars as the backend. All code and data are available here. Disclaimer: the Python and Elixir code works, but it may not be the most idiomatic or performant way to write that code.

First, let’s load libraries and read the data.

;; Scheme dataframe
(import (dataframe))

(define df (csv->dataframe "GameResults.csv"))

# Python Polars
import polars as pl
import statistics as stats

df = pl.read_csv("GameResults.csv")

# Elixir Explorer
require Explorer.DataFrame, as: DF
require Explorer.Series, as: Series

df = DF.from_csv!("GameResults.csv")

Every game was comprised of three sets to 21 points (win by two points). The league tracked W-L record by game, but I used sets to increase the amount of data. The actual RPI calculation includes a weighting factor for home and away games, but all of these games were played at a neutral site, which simplified the calculations. There were several forfeits, which the league tracked as losses, but I decided to exclude them from my dataset.

This is the format of the raw data…

> (dataframe-display df)
 dim: 132 rows x 6 cols
    week     set       winner         loser  winner_score  loser_score 
                                         
      1.      1.   Above ParR  CityOfThrees           22.          16. 
      1.      2.   Above ParR  CityOfThrees           23.          21. 
      1.      3.   Above ParR  CityOfThrees           21.          15. 
      1.      1.        Kangz          Tsaf           21.          17. 
      1.      2.         Tsaf         Kangz           21.          19. 
      1.      3.        Kangz          Tsaf           21.          11. 
      1.      1.  Team Avatar        Motley           22.          14. 
      1.      2.       Motley   Team Avatar           21.           0. 
      1.      3.       Motley   Team Avatar           23.          21. 
      1.      1.     Chow Men      Bob Ross           21.          10.

The approach to calculating W-L record involves two passes through the data, which could prove costly with a larger dataset. The type parameter refers to the winner or loser column in the dataframe and is passed as a symbol (Scheme), string (Python), or atom (Elixir). Polars and Explorer both use square brackets for extracting a dataframe column whereas $ is used in Scheme dataframe.

;; Scheme dataframe
(define (calc-wl game-data team type)
  (length (filter (lambda (x) (string=? team x)) ($ game-data type))))

# Python Polars
def calc_wl(game_data, team, type):
  return pl.Series.sum(game_data[type] == team)

# Elixir Explorer
def calc_wl(game_data, team, type) do
  Series.sum(Series.equal(game_data[type], team))
end

All of the calculations involve filtering the dataset to the focal team and potentially excluding games played against another team (opp). Perhaps the most notable difference here (other than the verbosity of the Scheme code) is the different ways that dataframe columns are referenced. Scheme dataframe and Explorer use unquoted column names. Polars uses the col function and quoted names. In Explorer, a query expression expects unquoted names to refer to dataframe columns and the ^ is needed to indicate that a variable is defined outside of the query (e.g., ^team and ^opp). In Scheme dataframe, column names are specified separately (i.e., (winner loser)) to create the distinction between column names and other variables. This approach was taken to mirror lambda syntax, but maybe I should explore whether I could use the ^ inside the filter expression to avoid the redundant listing of column names.

;; Scheme dataframe
(define (filter-team game-data team)
  (dataframe-filter*
   game-data
   (winner loser)
   (or (string=? winner team)
       (string=? loser team))))

(define (filter-team-opp game-data team opp)
  (dataframe-filter*
   (filter-team game-data team)
   (winner loser)
   (and (not (string=? winner opp))
        (not (string=? loser opp)))))

# Python Polars
def filter_team(game_data, team):
    return pl.DataFrame.filter(
        game_data, 
        (pl.col("winner") == team) | (pl.col("loser") == team))


def filter_team_opp(game_data, team, opp):
    return pl.DataFrame.filter(
        filter_team(game_data, team),
        (pl.col("winner") != opp) & (pl.col("loser") != opp))

# Elixir Explorer
def filter_team(game_data, team) do
  DF.filter(game_data, winner == ^team or loser == ^team)
end

def filter_team_opp(game_data, team, opp) do
  DF.filter(filter_team(game_data, team), winner != ^opp and loser != ^opp)
end

I’ve arguably opted to split the code into too many function as indicated by the difficulty I had in naming this next function. This code takes the dataframe column of winners and finds the proportion that match the specified team. This function is only applied to a filtered dataset for the specified team.

;; Scheme
(define (wp winners team)
  (let ([team-winners (filter (lambda (x) (string=? team x)) winners)])
    (inexact (/ (length team-winners) (length winners)))))

# Python Polars
def wp(winners, team):
  return pl.Series.sum(winners == team) / pl.Series.count(winners)

# Elixir Explorer
def wp(winners, team) do
  Series.sum(Series.equal(winners, team)) / Series.count(winners)
end

In calc-wp, we are combining our previous functions so the code is similar in all languages.

;; Scheme dataframe
(define (calc-wp game-data team)
  (let ([games-played (filter-team game-data team)])
    (wp ($ games-played 'winner) team)))

# Python Polars
def calc_wp(game_data, team):
  games_played = filter_team(game_data, team)
  return wp(games_played["winner"], team)

# Elixir Explorer
def calc_wp(game_data, team) do
  games_played = filter_team(game_data, team)
  wp(games_played[:winner], team)
end

I was also interested in point differential (PD) as a metric, but it became clear as the season wore on that the scores were not very accurate. The league used set differential as a tiebreaker so the scores were largely irrelevant as long as they recorded the correct set winners and losers. I’ve included two versions of the Scheme code for comparison. While legible, I find the Polars code a bit ugly here.

;; Scheme
(define (calc-pd game-data team)
  (let* ([games-played (filter-team game-data team)])
    (sum (map (lambda (w ws ls) (if (string=? team w) (- ws ls) (- ls ws)))
              ($ games-played 'winner)
              ($ games-played 'winner_score)
              ($ games-played 'loser_score)))))

;; Scheme dataframe
(define (calc-pd game-data team)
  (-> game-data
      (filter-team team)
      (dataframe-modify*
       (pd (winner winner_score loser_score)
           (if (string=? team winner)
               (- winner_score loser_score)
               (- loser_score winner_score))))
      ($ 'pd)
      (sum)))

# Python Polars
def calc_pd(game_data, team):
    pd = pl.DataFrame.with_columns(
        filter_team(game_data, team),
        pl.when(pl.col("winner") == team)
        .then(pl.col("winner_score") - pl.col("loser_score"))
        .otherwise(pl.col("loser_score") - pl.col("winner_score"))
        .alias("pd"))
    return pl.Series.sum(pd["pd"])

# Elixir Explorer
  def calc_pd(game_data, team) do
    game_data
    |> filter_team(team)
    |> DF.mutate(
      pd: if(winner == ^team, 
        do: winner_score - loser_score, 
        else: loser_score - winner_score)
    )
    |> DF.pull("pd")
    |> Series.sum()
  end

In calculating the opponents’ winning percentage, we need to exclude games played against the focal team with filter-team-opp. The way the parameters are named is potentially confusing. In (map (lambda (x) (calc-wp-owp game-data x team)) opps), we are iterating through all opponents of team and calculating their WP against everybody except team and then calculating the mean of all opponents’ WP. The function for calc-oowp is is nearly the same as calc-owp so is not shown here.

;; Scheme
(define (calc-owp game-data team)
  (let* ([opp-games (filter-team game-data team)]
         [opps (map (lambda (w l) (if (string=? team w) l w))
                    ($ opp-games 'winner)
                    ($ opp-games 'loser))]
         [owp (map (lambda (x) (calc-wp-owp game-data x team)) opps)])
    (mean owp)))

(define (calc-wp-owp game-data team opp)
  (let ([games-played (filter-team-opp game-data team opp)])
    (wp ($ games-played 'winner) team)))

# Python Polars
def calc_owp(game_data, team):
    opp_games = pl.DataFrame.filter(
        game_data, (pl.col("winner") == team) | (pl.col("loser") == team))
    opp = pl.DataFrame.with_columns(
        opp_games,
        pl.when(pl.col("winner") == team)
        .then(pl.col("loser"))
        .otherwise(pl.col("winner"))
        .alias("opp"))
    return stats.mean([calc_wp_owp(game_data, x, team) for x in opp["opp"]])

def calc_wp_owp(game_data, team, opp):
  games_played = filter_team_opp(game_data, team, opp)
  return wp(games_played["winner"], team)

# Elixir Explorer
def calc_owp(game_data, team) do
  game_data
  |> DF.filter(winner == ^team or loser == ^team)
  |> DF.mutate(opp: if(winner == ^team, do: loser, else: winner))
  |> DF.pull("opp")
  # transform is computationally expensive b/c of type conversion
  |> Series.transform(fn x -> calc_wp_owp(game_data, x, team) end)
  |> Series.mean()
end

def calc_wp_owp(game_data, team, opp) do
  games_played = filter_team_opp(game_data, team, opp)
  wp(games_played[:winner], team)
end

The next functions are simple arithmetic, but are included to show Scheme’s prefix mathematical operations and how the \ is used in Python to break an operation across lines.

;; Scheme
(define (calc-sos game-data team)
  (/ (+ (* 2 (calc-owp game-data team)) (calc-oowp game-data team)) 3))

(define (calc-rpi game-data team)
  (+ (* 0.25 (calc-wp game-data team))
     (* 0.5 (calc-owp game-data team))
     (* 0.25 (calc-oowp game-data team))))

# Python
def calc_sos(game_data, team):
  return (2 * calc_owp(game_data, team) + calc_oowp(game_data, team)) / 3

def calc_rpi(game_data, team):
  rpi = 0.25 * calc_wp(game_data, team) + \
    0.5 * calc_owp(game_data, team) + \
    0.25 * calc_oowp(game_data, team)
  return rpi

# Elixir
def calc_sos(game_data, team) do
  (2 * calc_owp(game_data, team) + calc_oowp(game_data, team)) / 3
end

def calc_rpi(game_data, team) do
  0.25 * calc_wp(game_data, team) +
    0.5 * calc_owp(game_data, team) +
    0.25 * calc_oowp(game_data, team)
end

We need a list of all teams for iteration. Even though both Polars and Explorer are using Rust polars as a backend, my understanding is that Python Polars is using similar naming to Rust polars, but Explorer is creating an API that is more similar to R dyplr (e.g., unique vs distinct). Scheme dataframe returns a list whereas Polars and Explorer both return Series. The Scheme list is used in a mapping operation and the Polars Series can be used in a list comprehension, but the Explorer Series is converted to an enumerable for mapping.

;; Scheme
(define teams (remove-duplicates (append ($ df 'winner) ($ df 'loser))))

# Python Polars
teams = df["winner"].append(df["loser"]).unique()

# Elixir Explorer
teams =
  Series.concat(df[:winner], df[:loser])
  |> Series.distinct()
  |> Series.to_enum()

Finally, we put it all together and create a dataframe with all of our calculated columns sorted by descending RPI. The process is similar in all languages (but, again, most verbose in Scheme) and involves mapping (Scheme, Elixir) or list comprehensions (Python) to iterate over all the teams. In Scheme and Python, we set the number of rows displayed equal to the number of teams. I ran the Elixir code in Livebook and interactively changed the number of rows to display.

I made no effort to write performant code because the dataset is so small, but the Python and Elixir code take 2-3 seconds on my machine while the Scheme code is effectively instanteous. I’m assuming that is mostly overhead of dealing with the Rust polars backend and would be a neglible element of the overall compute time with a larger dataset.

;; Scheme dataframe
(-> (make-dataframe
     (list (make-series 'Team teams)
           (make-series 'Win (map (lambda (x) (calc-wl df x 'winner)) teams))
           (make-series 'Loss (map (lambda (x) (calc-wl df x 'loser)) teams))
           (make-series 'WP (map (lambda (x) (calc-wp df x)) teams))
           (make-series 'PD (map (lambda (x) (calc-pd df x)) teams))
           (make-series 'SOS (map (lambda (x) (calc-sos df x)) teams))
           (make-series 'RPI (map (lambda (x) (calc-rpi df x)) teams))))
    (dataframe-sort* (> RPI))
    (dataframe-display (length teams)))

# Python Polars
pl.Config.set_tbl_rows(len(teams))
pl.DataFrame(
  {
    "team": teams,
    "win": [calc_wl(df, x, "winner") for x in teams],
    "loss": [calc_wl(df, x, "loser") for x in teams],
    "wp": [calc_wp(df, x) for x in teams],
    "pd": [calc_pd(df, x) for x in teams],
    "sos": [calc_sos(df, x) for x in teams],
    "rpi": [calc_rpi(df, x) for x in teams]
  }
).sort("rpi", descending = True)

# Elixir Explorer
DF.new(
  team: teams,
  win: Enum.map(teams, fn x -> RPI.calc_wl(df, x, :winner) end),
  loss: Enum.map(teams, fn x -> RPI.calc_wl(df, x, :loser) end),
  wp: Enum.map(teams, fn x -> RPI.calc_wp(df, x) end),
  pd: Enum.map(teams, fn x -> RPI.calc_pd(df, x) end),
  sos: Enum.map(teams, fn x -> RPI.calc_sos(df, x) end),
  rpi: Enum.map(teams, fn x -> RPI.calc_rpi(df, x) end)
)
|> DF.sort_by(desc: rpi)

Here were the RPI rankings at the end of the regular season:

 dim: 14 rows x 7 cols
             Team     Win    Loss      WP      PD     SOS     RPI 
                               
         Spartans     16.      2.  0.8889    141.  0.5971  0.6701 
         Chow Men     14.      4.  0.7778    103.  0.5274  0.5900 
  The Free Agents     11.      7.  0.6111     59.  0.5385  0.5567 
             >30%      8.      7.  0.5333     -7.  0.5464  0.5432 
       Above ParR      8.      7.  0.5333    -15.  0.5364  0.5357 
        The Big 3     11.      7.  0.6111     20.  0.4905  0.5206 
     Net Positive      6.      6.  0.5000    -10.  0.4956  0.4967 
         Bob Ross      8.     10.  0.4444    -23.  0.4968  0.4837 
           Motley      8.      7.  0.5333     17.  0.4603  0.4786 
       The A-Team      1.      8.  0.1111    -83.  0.5855  0.4669 
      Team Avatar      9.      9.  0.5000    -24.  0.4242  0.4431 
            Kangz      7.     11.  0.3889    -12.  0.4501  0.4348 
             Tsaf      4.     14.  0.2222    -92.  0.4563  0.3978 
     CityOfThrees      3.     15.  0.1667    -74.  0.4432  0.3741

The top 3 ranked teams were also seeded as teams 1-3 in the playoffs based only on game record and set differential (not set record, RPI, etc.). >30% was seeded at #11 because they forfeited twice, which I didn’t attempt to account for. Team Avatar was the #6 seed because they benefited from forfeits that aren’t reflected in the rankings. Apart from the effects of forfeits, the RPI rankings for each team were generally within 1 or 2 places from the playoff seeding. I was surprised that only six regular season games produced playoff seedings that so reasonably reflected the teams’ rankings.

Proof of Kaprekar’s constant in Scheme, Python, and Elixir

Thu, 30 May 2024 00:00:00 GMT

I thoroughly enjoyed this delightful post on Kaprekar’s constant, which was new to me. Similar to the post author, I also like to use programming to understand mathematical concepts. I thought it would be a fun exercise to translate the example Ruby code to my favorite language, Scheme, and two other languages that I’m learning, Python and Elixir.

Kaprekar’s routine is an iterative process that starts with a 4-digit number and (usually) ends at 6174. Here is an example starting with 55:

5500 - 0055 = 5445
5544 - 4455 = 1089
9810 - 0189 = 9621
9621 - 1269 = 8352
8532 - 2358 = 6174
7641 - 1467 = 6174

In Scheme and Python, the approach is to convert the number to a string, left pad with zeros (if necessary), split into a list of characters (Scheme) or strings (Python), sort the list, convert back to string, and then convert to number.

Python provides a function for left padding a string.

>>> str.rjust("55", 4, "0")
'0055'

In Scheme, we use format to write a left pad procedure. Format directives are written as strings that start with ~. The ~a directive prints the object as with display. Commas separate four optional arguments: mincol, colinc, minpad, padchars. The width of the output is specified with mincol. We use the default values for colinc and minpad, which is why we need ,,,. The padchars are preceded by '. The @ indicates that the padding is placed on the left.

(define (string-pad-left x mincol padchars)
  (format
   (string-append "~" (number->string mincol) ",,,'" padchars "@a")
   x))

> (string-pad-left "55" 4 "0")
"0055"

In Elixir, we are only working with integers; no conversion to characters or strings. We “pad” the list instead of the string. ++ concatenates two lists.

def pad_left(lst, width, value) do
  if length(lst) < width do
    List.duplicate(value, width - length(lst)) ++ lst
  else
    lst
  end
end

> pad_left([5, 5], 4, 0)
[0, 0, 5, 5]

In Scheme and Python, we create functions that convert a number to a list of characters (Scheme) or strings (Python) and back again. In both cases, we build the 4-digit left padding into the conversion to lists and, thus, number->list and num2list are not as general as their names imply. num2list takes advantage of strings being iterable and uses a list comprehension to pull apart the newly created string (from n) into a list. The join syntax is a bit weird to me as a Python novice. Elixir provides digits and undigits that provide the functionality we need without working with strings.

;; Scheme
(define (number->list n)
  (string->list (string-pad-left (number->string n) 4 "0")))

(define (list->number lst)
  (string->number (list->string lst)))

# Python
def num2list(n):
    return [x for x in str.rjust(str(n), 4, "0")]

def list2num(lst):
    return int("".join(str(x) for x in lst))

# Elixir
> Integer.digits(1234)
[1, 2, 3, 4]
> Integer.undigits([1, 2, 3, 4])
1234

For repdigits (e.g., 1111, 2222), the result of the Kaprekar routine is 0, not 6174. Although not strictly necessary, I will follow the original post and create functions for detecting repdigits. The Python and Elixir functions use the same logic but different approachs, i.e., strings and sets in Python and unique list values in Elixir. The Scheme version iterates over all characters (after converting the number to list of characters) to check if each character is the same as the first. Scheme and Elixir both have the convention of using ? in boolean functions, which I think pops out nicely when reading code (arguably better than the is_ prefix).

;; Scheme
(define (repdigit? n)
  (let ([lst (number->list n)])
    (for-all (lambda (x) (char=? x (car lst))) lst)))

# Python
def is_repdigits(n):
    return len(set(str(n))) == 1

# Elixir
def repdigits?(n) do
  length(Enum.uniq(Integer.digits(n))) == 1
end

As pointed out in the original post, for the Kaprekar routine, we are only working with 4-digit numbers, which allows for a simpler approach using modulo.

;; Scheme
(define (repdigit? n)
  (= (modulo n 1111) 0))

# Python
def is_repdigits(n):
    return n % 1111 == 0

# Elixir
def repdigits?(n) do
  Integer.mod(n, 1111) == 0
end

Our functions for sorting digits allow for specifying sort direction (via pred, rev, and sorter). As a reminder, padding is included in number->list and num2list, but first appears for Elixir in sort_digits.

;; Scheme
(define (sort-digits n pred)
  (list->number (sort pred (number->list n))))

# Python
def sort_digits(n, rev):
    return list2num(sorted(num2list(n), reverse = rev))

# Elixir
def sort_digits(n, sorter) do
  Integer.undigits(Enum.sort(pad_left(Integer.digits(n), 4, 0), sorter))
end

We use recursion to implement the Kaprekar routine, which is similar in all three languages. None of these functions have guards against numbers bigger than four digits so you can end up in an infinte loop.

;; Scheme
(define (kap n)
  (let ([d (- (sort-digits n char>?) (sort-digits n char))])
    (if (= n d) n (kap d))))

# Python
def kap(n):
    d = sort_digits(n, True) - sort_digits(n, False)
    if (n == d):
        result = n
    else:
        result = kap(d)
    return result

# Elixir
def kap(n) do
  d = sort_digits(n, :desc) - sort_digits(n, :asc)
  if n == d, do: n, else: kap(d)
end

The last step is to iterate through all integers from 0 to 9999 and raise an exception if the routine doesn’t result in 6174 (or is a repdigit). These blocks of code all execute silently and provide proof of Kaprekar’s constant. Note that repdigits?(n) can be replaced by kap(n) == 0 (or similar for the other languages).

;; Scheme
(for-each
 (lambda (n)
   (unless (or (repdigit? n) (= (kap n) 6174))
     (assertion-violation
      "kap loop"
      (string-append
       (number->string n)
       " is not a repdigit nor does it converge to 674"))))
  (iota 10000))

# Python
for n in range(0, 10000):
    if not (is_repdigits(n) or kap(n) == 6174):
        raise Exception(str(n) + " is not a repdigit nor does it converge to 674")

# Elixir
Enum.each(
  0..9999,
  fn n ->
    unless repdigits?(n) or kap(n) == 6174 do
      raise("#{n} is not a repdigit nor does it converge to 674")
    end
  end
)

Dataframe manipulation in Scheme

Mon, 22 Apr 2024 00:00:00 GMT

This is yet another post where I try out my dataframe library for Scheme (R6RS) on examples found in blog posts. This blog post demonstrates data manipulation with dplyr (R). I provide only the Scheme code and some commentary. You will have to click through to the original post to see how it compares to R.

Library and Data

First, we import the dataframe library and the shuffle procedure from chez-stats. The original post uses the Palmer Penguins dataset, which I wrote to a csv file from R. We read the data from file and select only the columns used in this post. dataframe-display is similar to head in R and dataframe-glimpse is similar to str, but dataframe currently provides no functionality comparable to summary in R.

(import (dataframe)
        (only (chez-stats)
              shuffle))

(define dat
  (-> (csv->dataframe "penguins.csv")
      (dataframe-select* species body_mass_g sex year)))

(dataframe-display dat)

 dim: 344 rows x 4 cols
  species  body_mass_g     sex    year 
                   
   Adelie         3750    male   2007. 
   Adelie         3800  female   2007. 
   Adelie         3250  female   2007. 
   Adelie           na      na   2007. 
   Adelie         3450  female   2007. 
   Adelie         3650    male   2007. 
   Adelie         3625  female   2007. 
   Adelie         4675    male   2007. 
   Adelie         3475      na   2007. 
   Adelie         4250      na   2007. 

(dataframe-glimpse dat)

 dim: 344 rows x 4 cols
 species         Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, ...       
 body_mass_g     3750, 3800, 3250, na, 3450, 3650, 3625, 4675, 3475, ...   
 sex             male, female, female, na, female, male, female, male, ... 
 year            2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, ...

Filter Observations

The dataframe library includes some handling of missing values, represented as 'na, but nothing currently that will automatically drop 'na values while filtering as in dplyr::filter. A couple of options for handling 'na are presented in the code below. In the original blog post, they built up to a compound filter with pipe operators, but here I will just cut to the chase and show the final version.

;; this version is more readable, but requires two passes through the dataframe
;;  i.e., first with dataframe-remove-na and then with dataframe-filter
(-> dat
    (dataframe-remove-na 'body_mass_g 'sex)
    (dataframe-filter*
     (body_mass_g sex)
     (and (> body_mass_g 4000)
          (string=? sex "female")))
    (dataframe-display))

;; this version is more awkward to read (and write)
;;  but does all the filtering in one pass
(-> dat
    (dataframe-filter*
     (body_mass_g sex)
     (and (and (not (na? body_mass_g))
               (> body_mass_g 4000))
          (and (not (na? sex))
               (string=? sex "female"))))
    (dataframe-display))

 dim: 58 rows x 4 cols
  species  body_mass_g     sex    year 
                   
   Gentoo        4500.  female   2007. 
   Gentoo        4450.  female   2007. 
   Gentoo        4550.  female   2007. 
   Gentoo        4800.  female   2007. 
   Gentoo        4400.  female   2007. 
   Gentoo        4650.  female   2007. 
   Gentoo        4650.  female   2007. 
   Gentoo        4200.  female   2007. 
   Gentoo        4150.  female   2007. 
   Gentoo        4800.  female   2007.

Extract Observations

The dataframe library doesn’t include a slice function, but we can accomplish similar functionality with other dataframe procedures. You can slice by row indices using dataframe-ref. dataframe-head is similar to slice_head. dataframe-tail works similarly to list-tail in Scheme but differently than slice_tail in dplyr and requires jumping through some extra hoops to replicate slice_tail.

(dataframe-display (dataframe-ref dat '(1 4 36)))

 dim: 3 rows x 4 cols
  species  body_mass_g     sex    year 
                   
   Adelie        3800.  female   2007. 
   Adelie        3450.  female   2007. 
   Adelie        3950.    male   2007. 

(dataframe-display (dataframe-head dat 3))

 dim: 3 rows x 4 cols
  species  body_mass_g     sex    year 
                   
   Adelie        3750.    male   2007. 
   Adelie        3800.  female   2007. 
   Adelie        3250.  female   2007. 

(dataframe-display (dataframe-tail dat (- (car (dataframe-dim dat)) 3)))

 dim: 3 rows x 4 cols
    species  body_mass_g     sex    year 
                     
  Chinstrap        3775.    male   2009. 
  Chinstrap        4100.    male   2009. 
  Chinstrap        3775.  female   2009.

It takes considerably more code to replicate slice_min and slice_max from dplyr.

;; equivalent of `slice_min` example in original post
(-> (dataframe-filter*
     dat
     (body_mass_g)
     (and (not (na? body_mass_g))
          (<= body_mass_g (quantile ($ dat 'body_mass_g) 0.25 7))))
    (dataframe-sort* (< body_mass_g))
    (dataframe-display))

;; previous version only works with filter as the first step in the pipe
;;  this version works with filter step anywhere in the pipe
(-> dat
    (dataframe-remove-na 'body_mass_g)
    (->>
     ((lambda (dfx)
        (dataframe-filter*
         dfx
         (body_mass_g)
         (<= body_mass_g (quantile ($ dfx 'body_mass_g) 0.25 7))))))
    (dataframe-sort* (< body_mass_g))
    (dataframe-display))

 dim: 89 rows x 4 cols
    species  body_mass_g     sex    year 
                     
  Chinstrap        2700.  female   2008. 
     Adelie        2850.  female   2008. 
     Adelie        2850.  female   2008. 
     Adelie        2900.  female   2008. 
     Adelie        2900.  female   2008. 
     Adelie        2900.  female   2009. 
  Chinstrap        2900.  female   2007. 
     Adelie        2925.  female   2009. 
     Adelie        2975.      na   2007. 
     Adelie        3000.  female   2007.

;; equivalent of `slice_max` example in original post
(-> (dataframe-filter*
     dat
     (body_mass_g)
     (and (not (na? body_mass_g))
          (>= body_mass_g (quantile ($ dat 'body_mass_g) 0.75 7))))
    (dataframe-sort* (< body_mass_g))
    (dataframe-display))

 dim: 90 rows x 4 cols
    species  body_mass_g     sex    year 
                     
     Gentoo        4750.    male   2008. 
     Gentoo        4750.  female   2008. 
     Gentoo        4750.  female   2008. 
     Gentoo        4750.  female   2009. 
     Gentoo        4750.  female   2009. 
     Adelie        4775.    male   2009. 
     Gentoo        4800.  female   2007. 
     Gentoo        4800.  female   2007. 
  Chinstrap        4800.    male   2008. 
     Gentoo        4850.  female   2008.

Sample Observations

The equivalent of sample_n and sample_frac don’t currently exist in dataframe, but it doesn’t take much code to reproduce that functionality. Here, we lean on shuffle from chez-stats to randomly arrange the indices and then call dataframe-ref.

(define (get-first-n lst n)
  (let loop ([lst lst]
             [out '()]
             [count 0])
    (if (or (null? lst) (= count n))
        (reverse out)
        (loop (cdr lst) (cons (car lst) out) (add1 count)))))

(define (sample-n df n)
  (let* ([ind1 (iota (car (dataframe-dim df)))]
         [ind2 (get-first-n (shuffle ind1) n)])
    (dataframe-ref df ind2)))

(define (sample-frac df frac)
  (let* ([rows (car (dataframe-dim df))]
         [n (floor (inexact (* rows frac)))])
    (sample-n df n)))

(dataframe-display (sample-n dat 3))

 dim: 3 rows x 4 cols
  species  body_mass_g     sex    year 
                   
   Adelie        4250.    male   2009. 
   Gentoo        5250.    male   2009. 
   Gentoo        4600.  female   2008. 

 (dataframe-display (sample-frac dat 1/2))

 dim: 172 rows x 4 cols
    species  body_mass_g     sex    year 
                     
     Adelie        3325.  female   2007. 
     Adelie        3275.  female   2009. 
     Adelie        3475.  female   2009. 
  Chinstrap        3650.    male   2007. 
     Gentoo        4550.  female   2007. 
  Chinstrap        3675.  female   2009. 
     Adelie        4600.    male   2007. 
     Gentoo        4650.  female   2007. 
  Chinstrap        3325.  female   2009. 
     Adelie        3075.  female   2009.

Sort Observations

Sorting also requires explicitly handling missing values. dataframe-sort first sorts based on the expression on the left and then moves right through all other expressions (in this case, it means first sorting by body mass and then sex).

(-> dat
    (dataframe-remove-na 'body_mass_g 'sex)
    (dataframe-sort* (< body_mass_g) (string sex))
    (dataframe-display))

 dim: 333 rows x 4 cols
    species  body_mass_g     sex    year 
                     
  Chinstrap        2700.  female   2008. 
     Adelie        2850.  female   2008. 
     Adelie        2850.  female   2008. 
     Adelie        2900.  female   2008. 
     Adelie        2900.  female   2008. 
     Adelie        2900.  female   2009. 
  Chinstrap        2900.  female   2007. 
     Adelie        2925.  female   2009. 
     Adelie        3000.  female   2007. 
     Adelie        3000.  female   2009.

Select Variables

dplyr’s select provides particularly rich functionality for selecting columns from a dataframe. In the dataframe library, you can only select or drop columns by exact column names.

(dataframe-display (dataframe-select* dat body_mass_g year))

 dim: 344 rows x 2 cols
  body_mass_g    year 
            
         3750   2007. 
         3800   2007. 
         3250   2007. 
           na   2007. 
         3450   2007. 
         3650   2007. 
         3625   2007. 
         4675   2007. 
         3475   2007. 
         4250   2007. 

(dataframe-display (dataframe-drop* dat body_mass_g year))

 dim: 344 rows x 2 cols
  species     sex 
        
   Adelie    male 
   Adelie  female 
   Adelie  female 
   Adelie      na 
   Adelie  female 
   Adelie    male 
   Adelie  female 
   Adelie    male 
   Adelie      na 
   Adelie      na 

;; re-arrange column order
(dataframe-display (dataframe-select* dat sex species body_mass_g year))

 dim: 344 rows x 4 cols
     sex  species  body_mass_g    year 
                   
    male   Adelie         3750   2007. 
  female   Adelie         3800   2007. 
  female   Adelie         3250   2007. 
      na   Adelie           na   2007. 
  female   Adelie         3450   2007. 
    male   Adelie         3650   2007. 
  female   Adelie         3625   2007. 
    male   Adelie         4675   2007. 
      na   Adelie         3475   2007. 
      na   Adelie         4250   2007.

Rename Variables

Renaming columns works similarly to dplyr, but the order is (old-name new-name) not new_name = old_name.

(dataframe-display
 (dataframe-rename* dat (body_mass_g body_mass) (year study_year)))

 dim: 344 rows x 4 cols
  species  body_mass     sex  study_year 
                     
   Adelie       3750    male       2007. 
   Adelie       3800  female       2007. 
   Adelie       3250  female       2007. 
   Adelie         na      na       2007. 
   Adelie       3450  female       2007. 
   Adelie       3650    male       2007. 
   Adelie       3625  female       2007. 
   Adelie       4675    male       2007. 
   Adelie       3475      na       2007. 
   Adelie       4250      na       2007.

Create or Modify Variables

Again, we have to explicitly handle 'na values because / and < don’t handle symbols and dataframe doesn’t provide alternatives. In the second example, we can create a new column that is not based on any columns in the dataframe by not specifying any columns () to use in the expression.

(dataframe-display
 (dataframe-modify*
  dat
  (body_mass_kg
   (body_mass_g)
   (if (na? body_mass_g) 'na (inexact (/ body_mass_g 1000))))))

 dim: 344 rows x 5 cols
  species  body_mass_g     sex    year  body_mass_kg 
                            
   Adelie         3750    male   2007.          3.75 
   Adelie         3800  female   2007.           3.8 
   Adelie         3250  female   2007.          3.25 
   Adelie           na      na   2007.            na 
   Adelie         3450  female   2007.          3.45 
   Adelie         3650    male   2007.          3.65 
   Adelie         3625  female   2007.         3.625 
   Adelie         4675    male   2007.         4.675 
   Adelie         3475      na   2007.         3.475 
   Adelie         4250      na   2007.          4.25 

(dataframe-display
 (dataframe-modify*
  dat
  (id () (iota (car (dataframe-dim dat))))))

 dim: 344 rows x 5 cols
  species  body_mass_g     sex    year      id 
                      
   Adelie         3750    male   2007.      0. 
   Adelie         3800  female   2007.      1. 
   Adelie         3250  female   2007.      2. 
   Adelie           na      na   2007.      3. 
   Adelie         3450  female   2007.      4. 
   Adelie         3650    male   2007.      5. 
   Adelie         3625  female   2007.      6. 
   Adelie         4675    male   2007.      7. 
   Adelie         3475      na   2007.      8. 
   Adelie         4250      na   2007.      9. 

(dataframe-display
 (dataframe-modify*
  dat
  (body_mass_cat
   (body_mass_g)
   (if (na? body_mass_g)
       'na
       (if (> body_mass_g 4000) "High" "Low")))))

 dim: 344 rows x 5 cols
  species  body_mass_g     sex    year  body_mass_cat 
                             
   Adelie         3750    male   2007.            Low 
   Adelie         3800  female   2007.            Low 
   Adelie         3250  female   2007.            Low 
   Adelie           na      na   2007.             na 
   Adelie         3450  female   2007.            Low 
   Adelie         3650    male   2007.            Low 
   Adelie         3625  female   2007.            Low 
   Adelie         4675    male   2007.           High 
   Adelie         3475      na   2007.            Low 
   Adelie         4250      na   2007.           High 

(dataframe-display
 (dataframe-modify*
  dat
  (body_mass_cat
   (body_mass_g)
   (cond [(na? body_mass_g) 'na]
         [(< body_mass_g 3500) "Low"]
         [(> body_mass_g 4750) "High"]
         [else "Medium"]))))

 dim: 344 rows x 5 cols
  species  body_mass_g     sex    year  body_mass_cat 
                             
   Adelie         3750    male   2007.         Medium 
   Adelie         3800  female   2007.         Medium 
   Adelie         3250  female   2007.            Low 
   Adelie           na      na   2007.             na 
   Adelie         3450  female   2007.            Low 
   Adelie         3650    male   2007.         Medium 
   Adelie         3625  female   2007.         Medium 
   Adelie         4675    male   2007.         Medium 
   Adelie         3475      na   2007.            Low 
   Adelie         4250      na   2007.         Medium

Identify Distinct Values

(-> dat
    (dataframe-select* species)
    (dataframe-unique)
    (dataframe-display))

 dim: 3 rows x 1 cols
    species 
       
     Adelie 
     Gentoo 
  Chinstrap 

(-> dat
    (dataframe-select* species year)
    (dataframe-unique)
    (dataframe-display))

 dim: 9 rows x 2 cols
    species    year 
          
     Adelie   2007. 
     Adelie   2008. 
     Adelie   2009. 
     Gentoo   2007. 
     Gentoo   2008. 
     Gentoo   2009. 
  Chinstrap   2007. 
  Chinstrap   2008. 
  Chinstrap   2009.

Aggregate Observations

(dataframe-display
 (dataframe-aggregate*
  dat
  (species sex)
  (mean (body_mass_g) (inexact (mean body_mass_g)))
  (sd (body_mass_g) (standard-deviation body_mass_g))))

 dim: 8 rows x 4 cols
    species     sex       mean        sd 
                     
     Adelie    male  4043.4932  346.8116 
     Adelie  female  3368.8356  269.3801 
     Adelie      na  3540.0000  477.1661 
     Gentoo  female  4679.7414  281.5783 
     Gentoo    male  5484.8361  313.1586 
     Gentoo      na  4587.5000  338.1937 
  Chinstrap  female  3527.2059  285.3339 
  Chinstrap    male  3938.9706  362.1376 

(dataframe-display
 (dataframe-aggregate*
  dat
  (species)
  (n_obs (species) (length species))))

 dim: 3 rows x 2 cols
    species   n_obs 
          
     Adelie    152. 
     Gentoo    124. 
  Chinstrap     68.

Comparing dataframe operations in Scheme, Python, and R

Wed, 27 Mar 2024 00:00:00 GMT

I recently came across this blog post that calls pandas (Python) “clunky” compared to the “silky smooth” dplyr (R). No objections from me. dplyr is my favorite R package. I thought it would fun to compare the relative clunkiness of my dataframe library for Scheme (R6RS) to pandas and dplyr. I will include the R and Python code here to save some clicking back and forth, but will mostly be commenting on the Scheme code.

Reading in the data

# R 
library(dplyr)
purchases <- read.csv("purchases.csv")

# Python
import pandas as pd
purchases = pd.read_csv("purchases.csv")

;; Scheme
(import (dataframe))
(define purchases (csv->dataframe "purchases.csv"))

“How much do we sell..? Let’s take the total sum!”

# R 
sum(purchases$amount)

# Python
sum(purchases["amount"])

;; Scheme
(sum ($ purchases 'amount))

“Ah, they wanted it by country…”

dataframe-aggregate* is a macro to slightly reduce the verbosity of the Scheme code, but it still falls short of the brevity and readability of the dplyr code.

# R 
purchases |>
  group_by(country) |>
  summarize(total = sum(amount))

# Python
(purchases
  .groupby("country")
  .agg(total=("amount", "sum")) 
  .reset_index()                
)

;; Scheme
(dataframe-aggregate* purchases (country) (total (amount) (sum amount)))

“And I guess I should deduct the discount.”

In Version 1, subtracting the discount is included by mapping over the columns within dataframe-aggregate*. In Version 2, dataframe-modify* handles mapping down the columns and produces more compact code. Version 2 illustrates the similarities in dataframe-modify* and dataframe-aggregate* with the primary difference being that you need to specify the grouping columns, e.g., (country) in dataframe-aggregate*, but the general form is (new-name (names) (expression)).

# R 
purchases |> 
  group_by(country) |> 
  summarize(total = sum(amount - discount))

# Python
(purchases
  .groupby("country")
  .apply(lambda df: (df["amount"] - df["discount"]).sum()) 
  .reset_index()
  .rename(columns={0: "total"})                            
)

;; Scheme
;; Version 1
(-> purchases
    (dataframe-aggregate*
     (country)
     (total
      (amount discount)
      (sum (map (lambda (amt dis) (- amt dis)) amount discount)))))

;; Version 2
(-> purchases
    (dataframe-modify* (diff (amount discount) (- amount discount)))
    (dataframe-aggregate* (country) (total (diff) (sum diff))))

“Oh, and Maria asked me to remove any outliers.”

The Scheme version only works because the filter step is the first in the pipeline, i.e., the pipe is passing the same purchases that is referred to in ($ purchases 'amount).

# R 
purchases |>
  filter(amount <= median(amount) * 10) |> 
  group_by(country) |> 
  summarize(total = sum(amount - discount))

# Python
(purchases
  .query("amount <= amount.median() * 10")
  .groupby("country")
  .apply(lambda df: (df["amount"] - df["discount"]).sum())
  .reset_index()
  .rename(columns={0: "total"})
)

;; Scheme
(-> purchases
    (dataframe-filter* (amount) (<= amount (* 10 (median ($ purchases 'amount)))))
    (dataframe-modify* (diff (amount discount) (- amount discount)))
    (dataframe-aggregate* (country) (total (diff) (sum diff))))

“I probably should use the median within each country.”

In Version 1, the outlier filtering and discount subtraction happens within the hidden split-apply-combine approach in dataframe-aggregate*. In Version 2, the split-apply-combine is made explicit for outlier filtering. Personally, I find Version 2 more readable, but it should be avoided for large dataframes because it involves splitting twice (dataframe-split and dataframe-aggregate*). Of course, neither are as elegant as the dplyr version.

# R 
purchases |>
  group_by(country) |>                    
  filter(amount <= median(amount) * 10) |> 
  summarize(total = sum(amount - discount))

# Python
(purchases
  .groupby("country")                                               
  .apply(lambda df: df[df["amount"] <= df["amount"].median() * 10]) 
  .reset_index(drop=True)                                           
  .groupby("country")
  .apply(lambda df: (df["amount"] - df["discount"]).sum())
  .reset_index()
  .rename(columns={0: "total"})
)

;; Scheme
;; Version 1
(-> purchases
    (dataframe-aggregate*
     (country)
     (total
      (amount discount)
      (sum (map (lambda (amt dis)
                  (let ([multi (if (<= amt (* 10 (median amount))) 1 0)])
                    (* multi (- amt dis))))
                amount
                discount)))))

;; Version 2
(-> purchases
    (dataframe-split 'country)
    (->> (map
          (lambda (dfx)
            (dataframe-filter*
             dfx (amount) (<= amount (* 10 (median ($ dfx 'amount))))))))
    (dataframe-bind-all)
    (dataframe-modify* (diff (amount discount) (- amount discount)))
    (dataframe-aggregate* (country) (total (diff) (sum diff)))
    (dataframe-display))

Conclusion

I’m clearly biased, but I think a case could be made that my dataframe library for Scheme is at least not more clunky than pandas.

Exploratory data analysis in Elixir

Sat, 08 Jul 2023 00:00:00 GMT

I have been tinkering with lots of different programming languages (see here and here) over the last few years. Scheme is the only language so far that I have enjoyed enough to write a decent amount of code. Elixir first caught my eye back in April 2020, but I’ve only recently tried to write more than ‘hello world’ with it. So far, I think it is great and I’m excited to learn more. I haven’t previously been a fan of code notebooks, but I think Livebook is amazing.

I’m using Elixir on Ubuntu. There are lots of different ways to install Elixir on Linux. I opted to first install Erlang with sudo apt install erlang. Then I installed the Elixir precompiled package for the version of Erlang that I just installed (get Erlang/OTP version by running erl -s erlang halt). I moved the unzipped folder to my home directory and added the following line to .zshrc: export PATH="$PATH:/home/username/elixir-otp-25/bin".

I installed Livebook by running mix escript.install hex livebook in the terminal and added the following line to .zshrc: export PATH="$PATH:/home/username/.mix/escripts". After running source .zshrc, I was able to launch Livebook with livebook server.

As with learning Scheme, I like to first try to recreate examples that I’ve written in other programming languages. Below, I’ve written Elixir code that corresponds to the Scheme examples in this blog post based on the Texas housing dataset that is included as part of the ggplot2 package for R. I wrote that post to try out my dataframe library for Scheme, but below I will focus on the comparison with R. I will highlight snippets of code in this post, but the full notebook is here.

One of the great features of Livebook is that the user-friendly GUI elements create code that is then easy to edit. It dramatically lowers the learning curve for beginners. At the top of every notebook is a setup block that includes an Add package button that allows for searching of available packages and generates the following code.

Mix.install([
  {:kino, "~> 0.16.1"},
  {:kino_explorer, "~> 0.1.24"},
  {:kino_vega_lite, "~> 0.1.13"},
  {:req, "~> 0.5.15"}
])

Kino is behind the magic of Livebook. KinoExplorer and KinoVegaLite provide nice tables for viewing dataframes and data visualizations, respectively. Those packages include Explorer and VegaLite as dependencies so they do not need to be installed separately. I’m using req to get data directly from a URL.

Explorer is a dataframe library for Elixir. I love the choice to mostly follow dplyr, which is my favorite R package. We require these packages because they require compilation and include aliases with as:.

require Explorer.DataFrame, as: DF
require Explorer.Series, as: Series

In R, read.csv allows for reading directly from a URL. This Elixir code composes well, though. It didn’t occur to me at first (but should have) that I should be searching for how to do a GET request. load_csv is used here because there is already a representation of the CSV in memory. To read a file from disk, use from_csv. The ! in load_csv! indicates that a problem with the file will raise an exception, which is arguably the preferred behavior for an interactive use case like this notebook.

txhousing =
  Req.get!("https://www.travishinkelman.com/data/txhousing.csv").body
  |> DF.load_csv!(nil_values: ["", "NA"])

This block of code is almost exactly the same as what you would write in dplyr. I love that Elixir has a pipe operator (and it even uses the same characters as the pipe operator in base R!). It looks a little weird that pipe operators are placed at the beginning of lines after many years of using pipes in R.

df_agg_year =
  txhousing
  |> DF.group_by("year")
  |> DF.summarise(
    avg_sales: mean(sales),
    avg_volume: mean(volume),
    avg_median: mean(median)
  )

In group_by, the column month can be written as a string ("month") or an atom (:month) but it can’t be bare (month). In summarise and arrange, the column names must be bare, not a string or atom. Without reading the documentation or source code, I would guess that summarise and arrange are macros and group_by is not.

df_agg_month =
  txhousing
  |> DF.group_by(:month)
  |> DF.summarise(
    avg_sales: mean(sales),
    avg_volume: mean(volume),
    avg_median: mean(median)
  )
  |> DF.sort_by(month)

One of the convenient features of dplyr::mutate is that expressions can perform calculations on columns that were created within that mutate. Here we need to move that last expression to its own DF.mutate.

txhousing
|> DF.group_by(["city", "year"])
|> DF.mutate(
  total_sales: sum(sales),
  total_volume: sum(volume)
)
# need a new mutate when working with newly calculated column
|> DF.mutate(prop_sales: sales / total_sales)

Data transformation in Scheme

Fri, 07 Jul 2023 00:00:00 GMT

I have done some recent work on my dataframe library for Scheme (R6RS) and thought I would run through the examples in the Data Transformation chapter of R for Data Science (R4DS). In this post, I won’t reproduce any of the R code and will provide limited commentary on the Scheme code (which is also available via this gist).

Setup

The nycflights13::flights dataset is used for all of the examples shown below. I’ve written it to a file and posted it here.

(import (dataframe))

(define flights (tsv->dataframe "nycflights.tsv"))

The flights dataset has 336,776 rows and 19 columns. Datasets of this size strain the dataframe library and provide a suboptimal experience, especially compared to R. Skipped sections indicate that the dataframe library has no equivalent capabilities.

3.1 Introduction

3.1.3

(-> flights
    (dataframe-filter* (dest) (string=? dest "IAH")) 
    (dataframe-aggregate*
     (year month day)
     (arr_delay (arr_delay) (inexact (mean arr_delay))))
    (dataframe-display))

3.2 Rows

3.2.1

Piping all of the output to a dataframe-display will often not show clearly the operation was successful. The one liners below show that the operation yielded the expected result. Need to remove 'na from the dep_delay column or the > operation will fail. The version with dataframe-remove-na is more readable, but requires two passes through the data.

;; filter only version
(define delayed-flights
  (-> flights
      (dataframe-filter*
       (dep_delay)
       (and (not (na? dep_delay))
            (> dep_delay 120)))))

;; remove na first
(define delayed-flights
  (-> flights
      (dataframe-remove-na 'dep_delay)
      (dataframe-filter* (dep_delay) (> dep_delay 120))))

(apply min ($ delayed-flights 'dep_delay))

(-> flights
    (dataframe-remove-na 'month 'day)
    (dataframe-filter* (month day) (and (= month 1) (= day 1)))
    (dataframe-glimpse))

(define jan-feb-flights
  (-> flights
      (dataframe-remove-na 'month)
      (dataframe-filter* (month) (or (= month 1) (= month 2)))))

(remove-duplicates ($ jan-feb-flights 'month))

(-> flights
    (dataframe-remove-na 'month)
    (dataframe-filter* (month) (member month '(1 2)))
    (dataframe-glimpse))

3.2.3

(-> flights
    (dataframe-remove-na 'year 'month 'day 'dep_time)
    (dataframe-sort* (< year) (< month) (< day) (< dep_time))
    (dataframe-glimpse))

(-> flights
    (dataframe-remove-na 'dep_delay)
    (dataframe-sort* (> dep_delay))
    (dataframe-glimpse))

3.2.4

dplyr::distinct has a parameter, .keep_all, that keeps all columns for the first occurrence of each unique combo. dataframe-unique does not have that functionality.

(-> flights
    (dataframe-unique)
    (dataframe-glimpse))

(-> flights
    (dataframe-select* origin dest)
    (dataframe-unique)
    (dataframe-glimpse))

The dataframe library does not include a procedure comparable to dplyr::count, but the same result can be obtained with dataframe-aggregate*.

(-> flights
    (dataframe-aggregate*
     (origin dest)
     (n (origin) (length origin)))
    (dataframe-sort*
     (> n))
    (dataframe-display))

3.3 Columns

3.3.1

(-> flights
    (dataframe-remove-na 'dep_delay 'arr_delay 'distance 'air_time)
    (dataframe-modify*
     (gain (dep_delay arr_delay) (- dep_delay arr_delay))
     (speed (distance air_time) (* (/ distance air_time) 60)))
    (dataframe-glimpse))

dataframe-modify* has no equivalent to the .before and .after parameters of dplyr::mutate. Using a dataframe-select to get focal columns but this drops all other columns.

(-> flights
    (dataframe-remove-na 'dep_delay 'arr_delay 'distance 'air_time)
    (dataframe-modify*
     (gain (dep_delay arr_delay) (- dep_delay arr_delay))
     (speed (distance air_time) (inexact (* (/ distance air_time) 60))))
    (dataframe-select* year month day gain speed)
    (dataframe-display))

3.3.2

dplyr::select has many wonderful helper functions to do complicated select operations easily. dataframe-select* is very simple.

(-> flights
    (dataframe-select* year month day)
    (dataframe-display))

3.3.3

(-> flights
    (dataframe-rename* (tailnum tail_num))
    (dataframe-glimpse))

3.4 The Pipe

(-> flights
    (dataframe-remove-na 'dest 'distance 'air_time)
    (dataframe-filter* (dest) (string=? dest "IAH"))
    (dataframe-modify*
     (speed
      (distance air_time)
      (inexact (* (/ distance air_time) 60))))
    (dataframe-select*
     year month day dep_time carrier flight speed)
    (dataframe-sort*
     (> speed))
    (dataframe-display))

(dataframe-display
 (dataframe-sort*
  (dataframe-select*
   (dataframe-modify*
    (dataframe-filter*
     (dataframe-remove-na flights 'dest 'distance 'air_time)
     (dest) (string=? dest "IAH"))
    (speed
     (distance air_time)
     (inexact (* (/ distance air_time) 60))))
   year month day dep_time carrier flight speed)
  (> speed)))

(define flights1
  (dataframe-filter*
   (dataframe-remove-na flights 'dest 'distance 'air_time)
   (dest distance air_time)
   (string=? dest "IAH")))

(define flights2
  (dataframe-modify*
   flights1
   (speed
    (distance air_time)
    (inexact (* (/ distance air_time) 60)))))

(define flights3
  (dataframe-select*
   flights2
   year month day dep_time carrier flight speed))

(dataframe-display
 (dataframe-sort* flights3 (> speed)))

3.5 Groups

3.5.2

dplyr::group_by automatically sorts by grouping variables, but that step needs to be done explicitly in dataframe-aggregate*.

(-> flights
    (dataframe-aggregate*
     (month)
     (avg_delay (dep_delay) (inexact (mean dep_delay))))
    (dataframe-sort*
     (< month))
    (dataframe-display))

(-> flights
    (dataframe-remove-na 'dep_delay)
    (dataframe-aggregate*
     (month)
     (avg_delay (dep_delay) (inexact (mean dep_delay)))
     (n (dep_delay) (length dep_delay)))
    (dataframe-sort*
     (< month))
    (dataframe-display))

4.5.5

dataframe-aggregate* struggles when splitting a large dataframe into many groups (in this case, 365). The operation to summarize monthly results (not shown) is reasonably quick. In this case, I didn’t bother to filter because I knew there were no missing values for year, month, and day.

(define daily_flights
  (dataframe-aggregate*
   flights
   (year month day)
   (n (dep_time) (length dep_time))))

From Python to NumPy: random walk example in R and Scheme

Sun, 21 Aug 2022 00:00:00 GMT

As a learning exercise, I decided to translate examples from the book, From Python to NumPy, into R and Chez Scheme. This post describes the random walk example from Chapter 2. All of the code is in this repository so I will only highlight a few pieces of code below. For context, I am a long-time R programmer who only periodically pokes at Python and dabbles in Scheme for fun. Because performance is the primary motivation of vectorizing code with NumPy in Python, I will be loosely comparing timings between Python, R, and Chez Scheme. Take these timings with a large grain of salt. I don’t know how comparable the different timings are.

The simple example to motivate the book involves a 1D random walk. The book starts with a for loop example, reports a 7x speed up over the for loop by vectorizing the code with itertools, and reports a 500x improvement from for loop to NumPy (but, by my math, the reported timings show 1000x). On my machine, I also observe a 7x speed up from for loop to itertools but only a 80x jump from for loop to NumPy. The for loop in R has comparable peformance to the for loop in Python, but vectorized R was about 2x as fast as NumPy. Here are vectorized versions of the functions in Python (NumPy) and R:

# Python
def random_walk_fastest(n=1000):
    steps = np.random.choice([-1,+1], n)
    return np.cumsum(steps)

# R
random_walk_v <- function(n = 1000) {
  steps <- sample(c(-1, 1), size = n, replace = TRUE)
  cumsum(steps)
}

Chez Scheme doesn’t have the option to speed up code by vectorizing it, but Chez is known as one of the most performant Scheme implementations. I tried a couple of versions of the procedure in Chez. In one, I iterated over a vector with a do loop. In the other, I used recursion on a list. Both performed similarly at about 1.5x as fast as the vectorized R version. Here is the recursive version:

(define (random-help)
  ;; (random 2) returns 0 or 1
  (- (* 2 (random 2)) 1))

(define (random-walk-lst n)
  (let loop ([step 0]
             [position (random-help)]
             [walk '()])
    (if (= step n)
        (reverse walk)
        (loop (add1 step)
              (+ position (random-help))
              (cons position walk)))))

In the next example, the author makes the point that the best NumPy performance often comes at the expense of readability. The example involves returning the starting index for all occurrences of a sub-sequence that are found in the random walk list. The book indicates that the NumPy version was 10x faster than pure Python (7x on my machine). In trying to identify the best way to implement this in R, I found this mailing list thread with numerous solutions that vary widely in speed. The fastest R version (of the ones that I tried) was nearly 8x faster than the next best solution. I implemented that same algorithm in Python (+ a little NumPy). It was nearly 4x as fast as the NumPy example from the book and comparably fast to the R version. Here is that algorithm in Python and R (apologies for the inconsistent naming between my R and Python files):

# Python
def find_crossing_3(seq, sub):
    n = len(seq)
    m = len(sub)
    candidate = np.arange(n-m)
    for i in range(m):
        candidate = candidate[sub[i] == seq[candidate + i]]
    return candidate

# R
find_crossing_2 <- function(seq, sub) {
  n <- length(seq)
  m <- length(sub)
  candidate <- seq_len(n - m + 1)
  for (i in seq_len(m)) {
    candidate <- candidate[sub[i] == seq[candidate + i - 1]]
  }
  candidate
}

Unfortunately, it is not possible (as far as I understand) to reproduce this algorithm in Chez Scheme and my alternative was about 50x slower than the Python and R versions.

;; https://stackoverflow.com/a/28034455
(define (first-n lst n)
  (if (zero? n)            
      '()                
      (cons (car lst)         
            (first-n (cdr lst)    
                     (- n 1)))))

(define (find-crossing-lst seq sub)
  (let loop ([index 0]
             [lst seq]
             [results '()])
    (if (< (length lst) (length sub))
        (reverse results)
        (if (equal? (first-n lst (length sub)) sub)
            (loop (add1 index) (cdr lst) (cons index results))
            (loop (add1 index) (cdr lst) results)))))

After asking on the Scheme Discord server, oaktownsam provided the code below, which is 35x faster than my original version.

(define (same-head? seq sub)
  (cond
    [(and (null? seq) (null? sub)) #t]
    [(null? sub)                   #t]
    [(null? seq)                   #f]
    [(equal? (car seq) (car sub))  (same-head? (cdr seq) (cdr sub))]
    [else #f]))

(define (find-crossing-lst-2 seq sub)
  (define (step index seq results)
    (cond
      [(null? seq) (reverse results)]
      [(same-head? seq sub) (step (add1 index) (cdr seq) (cons index results))]
      [else (step (add1 index) (cdr seq) results)]))
  (step 0 seq '()))

Visualizing Scheme library procedures with an interactive network graph in R

Sat, 05 Jun 2021 00:00:00 GMT

As a learning exercise, I wrote a dataframe library for Scheme (R6RS). Because I was learning Scheme while I wrote dataframe, I did not prioritize performance. However, as I’ve tried to use the dataframe library (exploratory data analysis, spam simulation, gapminder), I’ve encountered performance pitfalls that make dataframe largely unusable for datasets with more than a few thousand rows. I have a rough idea of where the bottlenecks are, but I thought it would be a useful to take a step back and visualize the dataframe procedures as a network graph.

UPDATE: I improved dataframe such that performance is reasonable on datasets up to roughly one hundred thousand rows.

I had some experience with the R package, pkgnet, which allows for exploring the structure of a package by building a graph representation. I had also spent a little time using the package, visNetwork, that pkgnet uses to build the function graph. Moreover, because code is data in Scheme, it is relatively straightforward to analyze Scheme code and I had briefly experimented with that in a previous blog post.

Prepare Data

All of the Scheme code to analyze the dataframe procedures is found here. Below I will walk through the main ideas.

First, let’s create a silly example of Scheme library code. example is the list that would be created as the result of reading a file called example-library.sls.

(define example
  '(library (example-library)
     (export exported-proc)
     (import (rnrs))
     (define exported-proc
       (case-lambda
         [(x1) (exported-proc-helper x1 10)]
         [(x1 x2) (exported-proc-helper x1 x2)]))
     (define (exported-proc-helper x1 x2)
       (let ([x-sum (sum2 x1 x2)])
         (map add1 (iota x-sum))))
     (define (sum2 x1 x2)
       (+ x1 x2))
     (define (add1 x)
       (+ x 1))
     (define (iota count)
       (define start 0)
       (define step 1)
       (let loop ((n 0) (r '()))
         (if (= n count)
         (reverse r)
         (loop (+ 1 n)
               (cons (+ start (* n step)) r)))))))

We can work with example in the same way as any other Scheme list.

> (car example)
library
> (length example)
9

The following two procedures are used to extract the procedure names from example.

;; get all procedure definitions
(define (get-defs lst)
  (filter (lambda (x)
            (and (pair? x)
                 (or (symbol=? (car x) 'define)
                     (symbol=? (car x) 'define-syntax))))
          lst))

;; name is the procedure name
;; def is one element of the list from get-defs
(define (get-name def)
  (if (and (pair? (cadr def)) (not (symbol=? (car def) 'define-syntax)))
      (caadr def)
      ;; cadr version for definitions using lambda, case-lambda, and define-syntax
      (cadr def)))

We map get-name across the list from get-defs to get our list of procedure names.

> (define defs (get-defs example))
> (define names (map get-name defs))
> names
(exported-proc exported-proc-helper sum2 add1 iota)

The procedure names are the nodes in our network graph. The connections between the procedures are the edges. visNetwork requires that the edge list is defined by ID numbers, not procedure names. Next we create a list of pairs with each procedure assigned an ID number.

> (define names-nums 
      (map (lambda (name num) (cons name num)) names (enumerate names)))
                                                          
> names-nums
((exported-proc . 0)
  (exported-proc-helper . 1)
  (sum2 . 2)
  (add1 . 3)
  (iota . 4))

Now that we have enumerated our procedures, we need to iterate through all of the procedure definitions to identify which other procedures are called from within each procedure. Even though recursion is great for working with nested data structures, the get-edges procedure is the first time that I had ever used deep recursion (i.e., recursing on both the car and cdr of a list).

(define (get-edges def names-nums)
  (let* ([name (get-name def)]
         [num (cdr (assoc name names-nums))]
         [out (let loop ([body (cddr def)]
                         [results '()])
                (cond [(null? body)
                       results]
                      [(not (pair? body))
                       (let ([name-num (assoc body names-nums)])
                         (if name-num (cons (cdr name-num) results) results))]
                      [else
                       (loop (car body) (loop (cdr body) results))]))])
    (map (lambda (x) (cons num x)) (remove-duplicates out))))

get-edges returns a list of pairs where the car is the ID of def and the cdr is the ID of the procedures called by def. Here is the output of get-edges when applied to exported-proc-helper:

> (cadr defs)
(define (exported-proc-helper x1 x2)
  (let ([x-sum (sum2 x1 x2)]) (map add1 (iota x-sum))))
> (get-edges (cadr defs) names-nums)
((1 . 2) (1 . 3) (1 . 4))

That covers the main ideas in preparing the data. The rest of the code just applies those ideas across multiple files and writes the data for use by R.

Visualize Data

The network graph is visualized in a Shiny app with visNetwork. The code for the app can be found in this gist. The live app can be viewed here. [Note, it takes several seconds to build and display the graph in the Shiny app.] It requires remarkably little code to visualize the network graph. For example, this is all that is required for the server code in the Shiny app.

output$networkGraph <- renderVisNetwork({
    nodes %>% 
        visNetwork(edges) %>%
        visEdges(arrows = "to") %>% 
        visOptions(highlightNearest = TRUE, 
                    nodesIdSelection = TRUE)
})

Conclusions

Here are a couple of observations from the network graph:

Unsurprisingly, a few procedures are called by a lot of other procedures, e.g., make-dataframe, dataframe-slist, check-dataframe.
A few procedures (make-series, make-dataframe, etc.) are created automatically as part of defining the record types. I added them to the list of procedure names by appending the list of exported procedure names to the list of extracted procedure names (and removing duplicates from the appended list).

Stochastic population model in R, Rcpp, and Fortran

Sun, 09 May 2021 00:00:00 GMT

The stochastic logistic population model described in this blog post has become my default exercise when I’m exploring a new programming language (Racket, F#, Rust). I ended that Rust post by noting that I was interested in Rust as a modern alternative for scientific computing and thought it would be a good learning exercise to re-write small legacy Fortran programs in Rust. In the process of looking for Fortran programs to translate to Rust, though, I found myself becoming more interested in the idea of learning Fortran than Rust, particularly after learning about the efforts to improve the tooling around Fortran (e.g., here and here). So, here we are…exploring Fortran via the stochastic population model exercise.

Standalone Fortran Program

First, I wrote a standalone Fortran program of the stochastic population model. The main thing that tripped me up was reading arguments from the command line. I initialized a 1D array for holding the values of the arguments.

character(len=20), dimension(5) :: args

I initially missed the len argument for character and was confused by the results I was getting until I realized that the program was only reading the first character provided for each argument.

We can straightforwardly fill the args array with a loop.

do i=1,command_argument_count()
  call get_command_argument(i, args(i))
end do

But the syntax for converting the arguments from strings to the correct types is unusual.

read(args(1), *) t

t is the first value in args (indexed with args(1)). The * indicates the default format of the value, which the compiler infers is an integer because that is the type of t.

Fortran Subroutines

I also created a file that only contains subroutines for calling from R. Two of the subroutines (rstduniform and rnormal) are mostly the same as from the main program, but the names are changed because the R help for the .Fortran function recommended not using underscores in the subroutine names and the type of several parameters is double precision, not real, for compatibility with R. One of the subroutines, rnorm, is only included as a test of rnormal.

subroutine rnorm(reps, mu, sigma, arr)
  implicit none
  integer :: i
  integer, intent(in) :: reps
  double precision, intent(in) :: mu, sigma
  double precision, intent(inout) :: arr(reps)
  do i=1,reps
    call rnormal(mu, sigma, arr(i))
  end do
end subroutine rnorm

The arr parameter is declared as intent(inout) because on the R side we will pass in a vector that is filled on the Fortran side and then returned again to R. There are performance pitfalls with this approach if you are passing large vectors back-and-forth between Fortran and R, but those can be partly mitigated with the use of the dotCall64 package.

We compile this file into a shared object for use in R with

R CMD SHLIB stochastic-logistic-subroutines.f90

Calling Fortran from R

In the R code, we load the shared object with

dyn.load("stochastic-logistic-subroutines.so")

To test that the rnormal subroutine is giving reasonable results, we call the rnorm subroutine as the first argument to .Fortran. In this example, we draw 1 million random variates from a normal distribution with a mean of 7.8 and standard deviation of 8.3.

> n = 1000000L
> rnorm_result <- .Fortran("rnorm", n, 7.8, 8.3, vector("numeric", n))
# .Fortran returns a list; 4th element of list in this case contains the vector
> mean(rnorm_result[[4]])
[1] 7.798748
> sd(rnorm_result[[4]])
[1] 8.302705

Following the original blog post, we benchmark the R, Rcpp, and Fortran versions.

> library(microbenchmark)
> microbenchmark(
    Rcpp = logmodc(t, yinit, r, k, thetasd),
    Fortran = .Fortran("logmodf", t, yinit, r, k, thetasd, vector("numeric", t)),
    R = logmodr(t, yinit, r, k, thetasd),
    times = 500L
  )
Unit: microseconds
    expr    min      lq     mean  median      uq     max neval
    Rcpp 13.235 14.6260 17.11903 15.5280 16.8100  86.612   500
 Fortran 21.950 22.8965 26.76053 23.8760 25.3160  72.528   500
       R 56.994 58.8775 67.25860 59.9085 62.3655 243.965   500

The original blog post (from 2017) observed a 19x speed up of the Rcpp version over the R version. Here we only get a 4x improvement presumably related to performance improvements in R. Our Fortran version is 2.5x faster than the R version.

Again, following the original blog post (with a slight modification), we benchmark running multiple simulations.

> rseq <- seq(1.1, 2.2, length.out = 10)
> microbenchmark(
    Rcpp = lapply(rseq, function(x) logmodc(t, yinit, x, k, thetasd)),
    Fortran = lapply(rseq, function(x){
      .Fortran("logmodf", t, yinit, x, k, thetasd, vector("numeric", t))}),
    R = lapply(rseq, function(x) logmodr(t, yinit, x, k, thetasd)),
    times = 500L)

Unit: microseconds
    expr     min       lq     mean   median       uq      max neval
    Rcpp 159.578 166.4295 214.4782 198.2240 202.9475 8265.555   500
 Fortran 233.397 239.9620 264.1335 256.8515 262.1665 2994.857   500
       R 621.029 635.1115 676.0114 665.7525 674.2735 2952.635   500

The relative performance gains for multiple simulations are similar to the single simulation.

Conclusion

This post doesn’t make a resounding case for using Fortran over C++ for speeding up simulations in R. The C++ code is shorter and runs faster. My C++ avoidance has centered on the complexity of the language. Perhaps there is a simpler subset of C++ that would go a long way towards speeding up my R code. However, this post makes the case that Fortran is still easier to learn than C++ for physics simulations.

Analyzing gapminder dataset with base R and Scheme

Fri, 30 Apr 2021 00:00:00 GMT

I keep my eye out for blog posts illustrating data analysis tasks in R that I can use to test the functionality of my chez-stats and dataframe libraries for Scheme (R6RS). A post comparing pandas (Python) and dplyr (R) in a basic analysis of the gapminder dataset provides a nice little test case. In this post, I will also include base R code used to accomplish the same tasks as a contrast to both the Scheme code and the dplyr code from the other post.

Data Loading

The R package, gapminder, provides an excerpt of the data available at Gapminder.org. I’ve written the data from that package to a CSV file (available here).

Base R

I’ve added a little function, head10, to simplify subsequent code. It is not necessary. By default, head provides six rows. I’ve changed it to 10 to match the default for dataframe-display. The main thing to note here is that because we are using base R, we don’t need to load any packages.

> gapminder <- read.csv("gapminder.csv")
> head10 <- function(data) head(data, n = 10)
> head10(gapminder)

   continent     country year lifeExp      pop gdpPercap
1       Asia Afghanistan 1952  28.801  8425333  779.4453
2       Asia Afghanistan 1957  30.332  9240934  820.8530
3       Asia Afghanistan 1962  31.997 10267083  853.1007
4       Asia Afghanistan 1967  34.020 11537966  836.1971
5       Asia Afghanistan 1972  36.088 13079460  739.9811
6       Asia Afghanistan 1977  38.438 14880372  786.1134
7       Asia Afghanistan 1982  39.854 12881816  978.0114
8       Asia Afghanistan 1987  40.822 13867957  852.3959
9       Asia Afghanistan 1992  41.674 16317921  649.3414
10      Asia Afghanistan 1997  41.763 22227415  635.3414

Scheme

We need to first import the dataframe library. Otherwise, the code is similar.

> (import (dataframe))

> (define gapminder (csv->dataframe "gapminder.csv"))

> (dataframe-display gapminder)

 dim: 1704 rows x 6 cols
  continent      country    year  lifeExp       pop  gdpPercap 
                                 
       Asia  Afghanistan   1952.  28.8010  8.425E+6   779.4453 
       Asia  Afghanistan   1957.  30.3320  9.241E+6   820.8530 
       Asia  Afghanistan   1962.  31.9970  1.027E+7   853.1007 
       Asia  Afghanistan   1967.  34.0200  1.154E+7   836.1971 
       Asia  Afghanistan   1972.  36.0880  1.308E+7   739.9811 
       Asia  Afghanistan   1977.  38.4380  1.488E+7   786.1134 
       Asia  Afghanistan   1982.  39.8540  1.288E+7   978.0114 
       Asia  Afghanistan   1987.  40.8220  1.387E+7   852.3959 
       Asia  Afghanistan   1992.  41.6740  1.632E+7   649.3414 
       Asia  Afghanistan   1997.  41.7630  2.223E+7   635.3414

Filtering

Problem 1

Filter the dataset to retain only rows where year is 2007.

Base R

> head10(gapminder[gapminder$year == 2007, ])

    continent     country year lifeExp       pop  gdpPercap
12       Asia Afghanistan 2007  43.828  31889923   974.5803
24     Europe     Albania 2007  76.423   3600523  5937.0295
36     Africa     Algeria 2007  72.301  33333216  6223.3675
48     Africa      Angola 2007  42.731  12420476  4797.2313
60   Americas   Argentina 2007  75.320  40301927 12779.3796
72    Oceania   Australia 2007  81.235  20434176 34435.3674
84     Europe     Austria 2007  79.829   8199783 36126.4927
96       Asia     Bahrain 2007  75.635    708573 29796.0483
108      Asia  Bangladesh 2007  64.062 150448339  1391.2538
120    Europe     Belgium 2007  79.441  10392226 33692.6051

Scheme

This example introduces the thread-first operator (->), which takes the result of the previous procedure and passes it to the first argument of the next procedure. For data analysis, I strongly prefer the threading (or piping) approach over writing the code inside-out or creating lots of intermediate datasets.

> (-> gapminder
      (dataframe-filter* (year) (= year 2007))
      dataframe-display)

 dim: 142 rows x 6 cols
  continent      country    year  lifeExp       pop  gdpPercap 
                                 
       Asia  Afghanistan   2007.  43.8280  3.189E+7   974.5803 
     Europe      Albania   2007.  76.4230  3.601E+6  5937.0295 
     Africa      Algeria   2007.  72.3010  3.333E+7  6223.3675 
     Africa       Angola   2007.  42.7310  1.242E+7  4797.2313 
   Americas    Argentina   2007.  75.3200  4.030E+7 12779.3796 
    Oceania    Australia   2007.  81.2350  2.043E+7 34435.3674 
     Europe      Austria   2007.  79.8290  8.200E+6 36126.4927 
       Asia      Bahrain   2007.  75.6350  7.086E+5 29796.0483 
       Asia   Bangladesh   2007.  64.0620  1.504E+8  1391.2538 
     Europe      Belgium   2007.  79.4410  1.039E+7 33692.6051

Problem 2

Filter the dataset to retain only rows where year is 2007 and continent is Americas.

Base R

In this case, I’ve used subset rather than the more conventional [ subsetting used in Problem 1. The advantage of subset is that I only need to type gapminder once instead of three times.

> head10(subset(gapminder, year == 2007 & continent == "Americas"))

    continent            country year lifeExp       pop gdpPercap
60   Americas          Argentina 2007  75.320  40301927 12779.380
144  Americas            Bolivia 2007  65.554   9119152  3822.137
180  Americas             Brazil 2007  72.390 190010647  9065.801
252  Americas             Canada 2007  80.653  33390141 36319.235
288  Americas              Chile 2007  78.553  16284741 13171.639
312  Americas           Colombia 2007  72.889  44227550  7006.580
360  Americas         Costa Rica 2007  78.782   4133884  9645.061
396  Americas               Cuba 2007  78.273  11416987  8948.103
444  Americas Dominican Republic 2007  72.235   9319622  6025.375
456  Americas            Ecuador 2007  74.994  13755680  6873.262

Scheme

dataframe-filter* was designed to avoid having to repeat the dataframe name in each sub-expression, i.e., (= year 2007) and (string=? continent "Americas"), but requires typing year and continent two times each in this example.

> (-> gapminder
      (dataframe-filter*
       (year continent)
       (and (= year 2007)
            (string=? continent "Americas")))
      dataframe-display)

 dim: 25 rows x 6 cols
  continent             country    year  lifeExp       pop  gdpPercap 
                                        
   Americas           Argentina   2007.  75.3200  4.030E+7 12779.3796 
   Americas             Bolivia   2007.  65.5540  9.119E+6  3822.1371 
   Americas              Brazil   2007.  72.3900  1.900E+8  9065.8008 
   Americas              Canada   2007.  80.6530  3.339E+7 36319.2350 
   Americas               Chile   2007.  78.5530  1.628E+7 13171.6389 
   Americas            Colombia   2007.  72.8890  4.423E+7  7006.5804 
   Americas          Costa Rica   2007.  78.7820  4.134E+6  9645.0614 
   Americas                Cuba   2007.  78.2730  1.142E+7  8948.1029 
   Americas  Dominican Republic   2007.  72.2350  9.320E+6  6025.3748 
   Americas             Ecuador   2007.  74.9940  1.376E+7  6873.2623

Problem 3

Filter the dataset to retain only rows where year is 2007 and continent is Americas and country is United States. This last filter is a little silly because the country condition makes the continent condition unnecessary, but the point is to show how code complexity increases with additional conditions.

Base R

Thanks to subset, adding additional conditions is straightforward with zero redundancy.

> subset(gapminder, 
         year == 2007 & 
           continent == "Americas" &
           country == "United States")

     continent       country year lifeExp       pop gdpPercap
1620  Americas United States 2007  78.242 301139947  42951.65

Scheme

> (-> gapminder
      (dataframe-filter*
       (year continent country)
       (and (= year 2007)
            (string=? continent "Americas")
            (string=? country "United States")))
      dataframe-display)

 dim: 1 rows x 6 cols
  continent        country    year  lifeExp       pop  gdpPercap 
                                   
   Americas  United States   2007.  78.2420  3.011E+8   42951.65

Summary Statistics

Problem 1

Calculate the average life expectancy worldwide in 2007.

Base R

> mean(subset(gapminder, year == 2007)$lifeExp)
[1] 67.00742

Scheme

This example introduces the $ operator, which was inspired by R, to extract the values of a column (e.g., lifeExp). The difference in verbosity between base R and Scheme in this example is related to the filtering step. Extracting a column and calculating the mean involve nearly the same number of characters.

> (-> gapminder
      (dataframe-filter* (year) (= year 2007))
      ($ 'lifeExp)
      (mean))

67.00742253521126

Problem 2

Calculate the average life expectancy for every continent in 2007.

Base R

aggregate allows for use of the formula syntax (e.g., lifeExp ~ continent) to concisely describe the summarized value (lifeExp) and grouping variable(s).

> aggregate(lifeExp ~ continent, 
           data = subset(gapminder, year == 2007),
           FUN = mean)

  continent  lifeExp
1    Africa 54.80604
2  Americas 73.60812
3      Asia 70.72848
4    Europe 77.64860
5   Oceania 80.71950

Scheme

One difference to note here is that dataframe-aggregate doesn’t automatically sort by the grouping variable. We would have to explicitly add that sorting step.

> (-> gapminder
      (dataframe-filter* (year) (= year 2007))
      (dataframe-aggregate*
       (continent)
       (mean-lifeExp (lifeExp) (mean lifeExp)))
      (dataframe-display))

 dim: 5 rows x 2 cols
  continent  mean-lifeExp 
                
       Asia       70.7285 
     Europe       77.6486 
     Africa       54.8060 
   Americas       73.6081 
    Oceania       80.7195

Problem 3

Calculate the total population per continent in 2007 and sort the results in descending order of total population.

Base R

> total_pop = aggregate(pop ~ continent, 
                        data = subset(gapminder, year == 2007),
                        FUN = sum)
> total_pop[order(-total_pop$pop),]

  continent        pop
3      Asia 3811953827
1    Africa  929539692
2  Americas  898871184
4    Europe  586098529
5   Oceania   24549947

Scheme

> (-> gapminder
      (dataframe-filter* (year) (= year 2007))
      (dataframe-aggregate*
       (continent)
       (total-pop (pop) (sum pop)))
      (dataframe-sort* (> total-pop))
      (dataframe-display))

 dim: 5 rows x 2 cols
  continent  total-pop 
             
       Asia   3.812E+9 
     Africa   9.295E+8 
   Americas   8.989E+8 
     Europe   5.861E+8 
    Oceania   2.455E+7

Creating Derived Columns

Problem 1

Calculate the total GDP by multiplying pop and gdpPercap.

Base R

The help page for transform advises that it is only intended for interactive use. Alternatively, you could use: gapminder$GPD = gapminder$pop * gapminder$gdpPercap.

> head10(transform(gapminder, GDP = pop * gdpPercap))

   continent     country year lifeExp      pop gdpPercap         GDP
1       Asia Afghanistan 1952  28.801  8425333  779.4453  6567086330
2       Asia Afghanistan 1957  30.332  9240934  820.8530  7585448670
3       Asia Afghanistan 1962  31.997 10267083  853.1007  8758855797
4       Asia Afghanistan 1967  34.020 11537966  836.1971  9648014150
5       Asia Afghanistan 1972  36.088 13079460  739.9811  9678553274
6       Asia Afghanistan 1977  38.438 14880372  786.1134 11697659231
7       Asia Afghanistan 1982  39.854 12881816  978.0114 12598563401
8       Asia Afghanistan 1987  40.822 13867957  852.3959 11820990309
9       Asia Afghanistan 1992  41.674 16317921  649.3414 10595901589
10      Asia Afghanistan 1997  41.763 22227415  635.3414 14121995875

Scheme

> (-> gapminder
      (dataframe-modify*
       (GDP (pop gdpPercap) (* pop gdpPercap)))
      (dataframe-display))

 dim: 1704 rows x 7 cols
  continent      country    year  lifeExp       pop  gdpPercap        GDP 
                                       
       Asia  Afghanistan   1952.  28.8010  8.425E+6   779.4453  6.567E+09 
       Asia  Afghanistan   1957.  30.3320  9.241E+6   820.8530  7.585E+09 
       Asia  Afghanistan   1962.  31.9970  1.027E+7   853.1007  8.759E+09 
       Asia  Afghanistan   1967.  34.0200  1.154E+7   836.1971  9.648E+09 
       Asia  Afghanistan   1972.  36.0880  1.308E+7   739.9811  9.679E+09 
       Asia  Afghanistan   1977.  38.4380  1.488E+7   786.1134  1.170E+10 
       Asia  Afghanistan   1982.  39.8540  1.288E+7   978.0114  1.260E+10 
       Asia  Afghanistan   1987.  40.8220  1.387E+7   852.3959  1.182E+10 
       Asia  Afghanistan   1992.  41.6740  1.632E+7   649.3414  1.060E+10 
       Asia  Afghanistan   1997.  41.7630  2.223E+7   635.3414  1.412E+10

Problem 2

Find the top 10 countries in percentile of gdpPercap.

Base R

> percentile <- function(x){
    rank_x = rank(x)
    rank_x/max(rank_x)
  }

> gapminder2007 = transform(subset(gapminder, year == 2007),
                            percentile = percentile(gdpPercap))
> head10(gapminder2007[order(-gapminder2007$percentile),])

     continent          country year lifeExp       pop gdpPercap percentile
1152    Europe           Norway 2007  80.196   4627926  49357.19  1.0000000
864       Asia           Kuwait 2007  77.588   2505559  47306.99  0.9929577
1368      Asia        Singapore 2007  79.972   4553009  47143.18  0.9859155
1620  Americas    United States 2007  78.242 301139947  42951.65  0.9788732
756     Europe          Ireland 2007  78.885   4109086  40676.00  0.9718310
672       Asia Hong Kong, China 2007  82.208   6980412  39724.98  0.9647887
1488    Europe      Switzerland 2007  81.701   7554661  37506.42  0.9577465
1092    Europe      Netherlands 2007  79.762  16570613  36797.93  0.9507042
252   Americas           Canada 2007  80.653  33390141  36319.24  0.9436620
696     Europe          Iceland 2007  81.757    301931  36180.79  0.9366197

Scheme

This example reveals a weak spot in the dataframe API. The issue is that rank% operates on a list whereas dataframe-modify* takes only scalars as inputs because the expr is mapped over all rows in a dataframe. The workaround to avoid the mapping is to specify that no columns (i.e., ()) from the dataframe are used, then the list created in the subsequent expression (e.g., (rank% ($ df 'gdpPercap))) will be used as the column values.

Because the dataframe library only has thread-first (->) and thread-last (->>) operators, we have to create the awkward lambda procedure to get the output of dataframe-filter* to the right place in dataframe-modify*. SRFI 197 provides a pipeline operator that requires that the location of the output of the previous procedure is explicitly specified with _, which would likely simplify this code. An earlier, simpler version of SRFI 197 was the source for -> and ->> in the dataframe library.

We import only the rank procedure from chez-stats because there are name conflicts between chez-stats and dataframe. The conflicting procedures (e.g., mean, median, sum, etc.) in dataframe provide handling of missing values.

(import (only (chez-stats) rank))

(define (rank% lst)
  (let* ([rank-lst (rank lst 'mean)]
         [max-rank (apply max rank-lst)])
    (map (lambda (x) (inexact (/ x max-rank))) rank-lst)))

> (-> gapminder
      (dataframe-filter* (year) (= year 2007))
      (->> ((lambda (df)
              (dataframe-modify*
               df
               (percentile () (rank% ($ df 'gdpPercap)))))))
      (dataframe-sort* (> gdpPercap))
      (dataframe-display))

 dim: 142 rows x 7 cols
  continent           country   year  lifeExp       pop  gdpPercap  percentile 
     Europe            Norway  2007.  80.1960  4.628E+6   49357.19      1.0000 
       Asia            Kuwait  2007.  77.5880  2.506E+6   47306.99      0.9930 
       Asia         Singapore  2007.  79.9720  4.553E+6   47143.18      0.9859 
   Americas     United States  2007.  78.2420  3.011E+8   42951.65      0.9789 
     Europe           Ireland  2007.  78.8850  4.109E+6   40676.00      0.9718 
       Asia  Hong Kong, China  2007.  82.2080  6.980E+6   39724.98      0.9648 
     Europe       Switzerland  2007.  81.7010  7.555E+6   37506.42      0.9577 
     Europe       Netherlands  2007.  79.7620  1.657E+7   36797.93      0.9507 
   Americas            Canada  2007.  80.6530  3.339E+7   36319.24      0.9437 
     Europe           Iceland  2007.  81.7570  3.019E+5   36180.79      0.9366

If we create an intermediate dataframe, we get simpler code.

(define gapminder2007
  (dataframe-filter* gapminder (year) (= year 2007)))

(-> (dataframe-modify*
     gapminder2007
     (percentile () (rank% ($ gapminder2007 'gdpPercap))))
    (dataframe-sort* (> gdpPercap))
    (dataframe-display))

Conclusion

The main outcome of writing this post was that it led me to completely rewrite dataframe-display because it previously didn’t handle large numbers well, which was made abundantly clear in working with the gapminder data. The new version of dataframe-display is much improved and I gained a greater appreciation for all of the decisions that go into how to print a representation of a data structure to the screen.

If you are not familiar with Scheme code, you might find it to be unacceptably verbose in the examples above and, especially, when compared to dplyr code. Because of Scheme’s macro system, dataframe could be written in a more terse style. But I made the decision early on to stick to relatively simple macro usage and write dataframe in a way that I thought would be familiar to Scheme programmers. And that often involves more verbose code. For example, nearly all of the dataframe procedures have dataframe in the name, e.g., dataframe-filter**, dataframe-modify*, etc. This is following the example for hashtables, e.g., hashtable-ref, hashtable-values, etc. I also hope that experienced Scheme programmers can see that the dataframe macros mostly exist to reduce the number of times that lambda is written but the shape of the code should still feel familiar. While Scheme code might be more verbose in the small, I find it extremely expressive in the large because the core ideas compose so well.

Guess the number game in Fortran

Wed, 07 Apr 2021 00:00:00 GMT

I recently came across this blog post on writing simple test programs in different programming languages as a way to get a feel for a language. At the bottom of the post, there were links to articles on implementations of a number guessing game in 13 different languages, including Fortran. I was curious about the Fortran example because I’ve been interested in learning a little Fortran after hearing about efforts to improve the tooling around Fortran (e.g., here and here). Well, the Fortran example was written in Fortran 77 so I decided that re-writing it in modern Fortran would be a nice little exercise.

I’m not going to write about the Fortran 77 version. It is well described in the other article. Before I show the code for the “modern” version, I will introduce a couple of subroutines that I plucked from this random number tutorial.

subroutine random_stduniform(u)
   implicit none
   real,intent(out) :: u
   real :: r
   call random_number(r)
   u = 1 - r
end subroutine random_stduniform

! assuming a
subroutine random_uniform(a,b,x)
   implicit none
   real,intent(in) :: a,b
   real,intent(out) :: x
   real :: u
   call random_stduniform(u)
   x = (b - a) * u + a
end subroutine random_uniform

The first thing to note is that Fortran has both subroutines and functions. I have only a fuzzy understanding of when you would choose one over the other. In this case, it seems that random_stduniform and random_uniform could have also been written as functions (but I didn’t try). Perhaps the fact that random_number is a subroutine makes it more intuitive to build subroutines on top of it. When calling a subroutine, you need to preface the statement with call, which is not required for functions. Also, in a subroutine, assignment is handled by passing the variable name as an argument. For example, in random_stduniform, the variable r is declared with type real and then assigned a value when passed to the random_normal subroutine.

One of the other unusual (to me) components of these subroutines is implicit none, which means that the types of variables need to be explicitly declared and not implicitly inferred from the variable names. Here is an argument for embracing implicit none rather than seeking to change it in a future version of the Fortran Standard.

Armed with those two sub-routines, here is the full program for the number guessing game:

program guessnum
  implicit none

  real :: number
  integer :: number_int, guess

  call random_seed()
  call random_uniform(0.999, 100.0, number)
  number_int = int(number)
  guess = 0

  print *, "Guess a number between 1 and 100"

  do while (guess /= number_int) 
      read *, guess
      if (guess < number_int) then
        print *, "too low"
      else if (guess > number_int) then
        print *, "too high"
      end if
  end do 

  print *, "correct!"
 
end program guessnum

After declaring the types of our variables, we set a random seed and generate a number from a random uniform distribution that is > 0.999 and <= 100. random_uniform assigns a real number to the variable number so we coerce it to an integer with int, which has the same effect as floor or truncate operators in other languages. guess is initialized at zero to get us into our while loop because number_int should never be equal to zero in this program.

We print the guessing instructions, enter the while loop, and read the input from the user. Then, we compare the user’s guess to number_int, give feedback, and read their next guess. When guess matches number_int, we break out of the while loop and print correct! before exiting the program.

If you have the GNU Fortran compiler installed, and the code above in a file called guess.f90, you can compile the program with gfortran guess.f90 -o guess and run the program with ./guess. Here is example output from running the program:

 Guess a number between 1 and 100
50
 too high
25
 too low
37
 too low
43
 too high
40
 too high
39
 too high
38
 correct!

That was a fun first experience with Fortran. I’m happy to trade some verbosity for simplicity. I think the Fortran example is a lot easier to understand than the Rust example. Despite working for many years as a scientific programmer (but never officially my job title), I had never considered learning Fortran. I assumed it was an ugly relict hanging on in legacy codebases, but I no longer think it is ugly and it even moved back into the top 20 of the Tiobe index.

Spam simulation in Scheme

Sat, 12 Dec 2020 00:00:00 GMT

I learned a lot about Scheme (R6RS) by writing a few libraries and I expect that there is more to learn by trying to use those libraries (e.g., EDA in Scheme). A blog post about a stochastic simulation of spam comments in R caught my eye as an interesting example to test my dataframe library.

I was drawn to the spam simulation post because it describes using the crossing function from the tidyr package as a convenient way to examine the parameter space of a simulation under the familar framework of a dataframe, which is an approach that I’ve used in my own work. One of the challenges for me in learning new programming languages is that my R experience has led me to primarily think in terms of dataframes.

The takeaway from implementing the spam simulation in Scheme is that my dataframe library is not well suited to this task. That is not entirely surprising. dataframe was written for ease of implementation, not performance. More importantly, for a simple simulation like this, plain Scheme code provides a more straightforward solution than the dataframe approach.

In this post, I will focus on explaining the Scheme code rather than describing the simulation. In short, the idea is that you can build up your intuition about a problem with simulations, which might eventually lead you to a more concise mathematical solution to the problem. The objective is to determine the number of spam comments after three days given that spammers follow a Poisson process.

Dataframe approach

First, let’s import the dataframe library and a couple of procedures from the chez-stats library (after following installation instructions at the library repos). We import only the specific procedures from chez-stats because there are name conflicts between chez-stats and dataframe. The conflicting procedures (e.g., mean, median, sum, etc.) in dataframe provide handling of missing values.

(import (dataframe)
        (only (chez-stats)
              random-exponential
              repeat))

dataframe-crossing takes either dataframes or series and returns the cartesian product of the objects. The original post used 25,000 trials. We are only using 100 trials because the performance of dataframe-split (see below) is poor with large numbers of groups. We are using the -> operator to build up a chain of operations in a more readable way.

(define sim-waiting
  (-> (dataframe-crossing (make-series 'trial (map add1 (iota 100)))
                          (make-series 'observation (map add1 (iota 300))))))

> (dataframe-display sim-waiting)

 dim: 30000 rows x 2 cols
   trial  observation 
            
      1.           1. 
      1.           2. 
      1.           3. 
      1.           4. 
      1.           5. 
      1.           6. 
      1.           7. 
      1.           8. 
      1.           9. 
      1.          10.

We continue to build up the sim-waiting dataframe by adding a column with waiting times by drawing from an exponential distribution with the observation as the rate parameter (used by rexp in R). However, random-exponential from chez-stats takes the mean of the distribution as the parameter, which is equal to 1/rate.

(define sim-waiting
  (-> (dataframe-crossing (make-series 'trial (map add1 (iota 100)))
                          (make-series 'observation (map add1 (iota 300))))
      (dataframe-modify*
       (waiting (observation) (random-exponential (/ 1 observation))))))

> (dataframe-display sim-waiting)

 dim: 30000 rows x 3 cols
   trial  observation  waiting 
                
      1.           1.   1.4662 
      1.           2.   0.0535 
      1.           3.   0.3442 
      1.           4.   0.3174 
      1.           5.   0.8945 
      1.           6.   0.1303 
      1.           7.   0.0622 
      1.           8.   0.1867 
      1.           9.   0.1254 
      1.          10.   0.0213

The next step uses the split-apply-combine strategy to return the cumulative sum of the waiting column for each trial. -> pipes into the first argument of the next procedure whereas ->> pipes into the last. In the apply step, we add a new column for each dataframe that came out of dataframe-split with (cumulative () (cumulative-sum ($ df 'waiting))). If a dataframe-modify* contains a list of the same length as the number of rows in the dataframe, then it is added to the dataframe as a column with the specified name, which is cumulative in this example.

(define sim-waiting
  (-> (dataframe-crossing (make-series 'trial (map add1 (iota 100)))
                          (make-series 'observation (map add1 (iota 300))))
      (dataframe-modify*
       (waiting (observation) (random-exponential (/ 1 observation))))
      (dataframe-split 'trial)
      (->> (map (lambda (df)
              (dataframe-modify*
               df
           (cumulative () (cumulative-sum ($ df 'waiting)))))))
      (dataframe-bind-all)))

> (dataframe-display sim-waiting)

 dim: 30000 rows x 4 cols
   trial  observation  waiting  cumulative 
                       
      1.           1.   0.8186      0.8186 
      1.           2.   0.5609      1.3795 
      1.           3.   0.2155      1.5950 
      1.           4.   0.1990      1.7940 
      1.           5.   0.2132      2.0072 
      1.           6.   0.0494      2.0566 
      1.           7.   0.1209      2.1775 
      1.           8.   0.0454      2.2230 
      1.           9.   0.0384      2.2614 
      1.          10.   0.0183      2.2797

We cross sim-waiting with a new time column to find the number of spam comments within each trial and time combination. The cumulative column gives the total time that has elapsed. We are counting the number of rows where cumulative is less than time to determine the number of comments received in a specified time. The last step is to calculate the average number of spam comments for each time across all trials.

(define average-over-time
  (-> sim-waiting
      (dataframe-crossing (cons 'time (map (lambda (x) (* x 0.25)) (iota 13))))
      (dataframe-modify*
       (comment (cumulative time) (< cumulative time)))
      (dataframe-aggregate*
       (trial time)
       (num-comments (comment) (sum comment)))
      (dataframe-aggregate*
       (time)
       (average (num-comments) (exact->inexact (mean num-comments))))))

> (dataframe-display average-over-time 13)

 dim: 13 rows x 2 cols
    time  average 
        
  0.0000   0.0000 
  0.2500   0.1400 
  0.5000   0.3900 
  0.7500   0.8100 
  1.0000   1.3000 
  1.2500   2.1100 
  1.5000   2.9000 
  1.7500   3.8700 
  2.0000   5.4000 
  2.2500   7.3600 
  2.5000   9.5500 
  2.7500  12.1200 
  3.0000  16.0000

Idiomatic Scheme approach

As I mentioned above, the dataframe approach is inefficient and more verbose than an idiomatic Scheme approach. One source of inefficiency is generating 300 observations per trial when the average for three days is only 19. This inefficiency is not expensive in R because it is well optimized for these types of vectorized operations. However, in Scheme, we can write a simple recursive function that doesn’t bother to build up a list of all of the waiting times and stops as soon as the number of comments exceeding the waiting time is known.

The bigger inefficiency is the dataframe library, particularly dataframe-split.

> (define (get-num-events max-obs max-time)
    (let loop ([obs 1]
               [time 0]
               [events 0])
      (if (or (> obs max-obs) (> time max-time))
          (sub1 events) ;; sub1 to find num events less than max-time threshold
          (let ([exp-draw (random-exponential (/ 1 obs))])
            (loop (add1 obs) (+ time exp-draw) (add1 events))))))

> (get-num-events 300 3)
16
> (get-num-events 300 3)
1
> (get-num-events 300 3)
24
> (get-num-events 300 3)
28

The repeat procedure from chez-stats uses recursion to repeat a thunk n times, which allows us to build up a set of replications.

> (define sim (repeat 1e5 (lambda () (get-num-events 300 3))))
> (inexact (mean sim))
19.03991

The last step is to vary the times used in the simulation.

> (define max-times (map (lambda (x) (* x 0.25)) (iota 13)))

> (define sim-times
   (map (lambda (mt)
          (repeat 1e5 (lambda () (get-num-events 300 mt))))
        max-times))

> (define sim-times-mean (map (lambda (x) (inexact (mean x))) sim-times))

> max-times
(0 0.25 0.5 0.75 1.0 1.25 1.5 1.75 2.0 2.25 2.5 2.75 3.0)
> sim-times-mean
(0.0 0.28459 0.64931 1.11663 1.71885 2.48855 3.47136 4.74836
     6.38984 8.48071 11.13153 14.64021 19.07148)

This approach uses 100,000 trials and is effectively instantaneous whereas only 50 trials in the dataframe approach took several seconds and hundreds of trials was likely to freeze Emacs. The dataframe library is not capable of working with hundreds of thousands of row. Fortunately, many data analysis projects involve datasets much smaller than that. More importantly, idiomatic Scheme code is well suited for simulations like this spam comments problem.