Access Chez Scheme documentation from the REPL

In the process of learning Chez Scheme, I missed R’s ability to quickly pull up documentation from the console via help or ?. I was interested in the idea of trying to format the contents of the Chez Scheme User’s Guide (CSUG) and The Scheme Programming Language (TSPL) for display in the REPL (similar to Clojure Docs). That was a bigger task than I wanted to tackle and the first version of chez-docs simply opened links to the relevant sections of CSUG and TSPL from the REPL. Eventually, I more fully scraped the HTML documentation pages and displayed more information in the REPL while still providing the option to open a link to the online documentation.

The initial web scraping was all done in R, but I eventually moved all of that code to Chez Scheme. The scraping code lives in a separate library, chez-docs-scrape, and a short script, scrape-script.ss, runs the whole process. Neither CSUG nor TSPL provides a sitemap.xml, so crawl-index downloads each site’s index.html, parses it with html->sxml from the wak library, and uses sxpath to extract every href. The fragments (#...) and leading ./ are stripped from the hrefs, which are then filtered to local .html pages and deduplicated. download-pages shells out to curl for each page and pauses for one second between requests.

The CSUG Summary of Forms page links to both CSUG and TSPL and is used to build a lookup table mapping each procedure name (the key) to its anchor and URL. The HTML is parsed with html->sxml and sxpath is used to extract the table rows (after dropping the column headers and horizontal rule in the first two rows). The key is the first token of the form with parentheses removed, and relative CSUG URLs are expanded to full URLs. Rows are split by source, and two edge cases are handled manually: alias appears twice in CSUG as both a procedure and a keyword for import. The keyword version is dropped. let appears twice in TSPL pointing to two different anchors. One is renamed "named let".

Before parsing each downloaded HTML page, read-and-clean uses irregex to fix a couple of problems in the source HTML that html->sxml doesn’t handle well. Stray <p> tags that immediately precede a <p>, <li>, or <ul> tag are removed. Some <dl> blocks omit the closing </dt> before <dd>, which causes html->sxml to nest <dd> inside <dt> rather than treating them as siblings, so the missing </dt> tags are inserted (and any resulting </dt></dt> is collapsed back to a single </dt>).

(define (read-and-clean path)
  (let* ([raw     (call-with-input-file path get-string-all)]
         [step1   (irregex-replace/all "<p>[ \t\r\n]*(?=<(p|li|ul)>)" raw "")]
         [step2   (irregex-replace/all "<dd>" step1 "</dt><dd>")]
         [step3   (irregex-replace/all "</dt></dt>" step2 "</dt>")])
    (open-string-input-port step3)))

For each cleaned page, sxpath is used to extract <p> elements and walk through them looking for formdef blocks, the marked-up sections in the documentation that describe each procedure. Everything between a formdef and the next formdef, header, or footer is collected as the documentation for that anchor. Each <p> element is converted to plain text by render-sxml, which recursively walks the SXML tree. HTML entities (e.g.,  , <, é) are translated with a lookup table, inline images are replaced with their text equivalents (e.g., math GIFs become =>, ->, λ, ∞), attribute subtrees are skipped, <br> becomes a newline, list items are rendered as * bullets, definition details are indented, superscripts are prefixed with ^, and tables are replaced with [table not shown].

The trickiest part was whitespace. The line breaks and indentation in the HTML source are invisible in a browser, but they show up as stray newlines and spaces in plain text. Before rendering the children of a node, normalize-children cleans them up in five steps.

(define (normalize-children children)
  (let* ([step1 (strip-embedded-newlines children)]
         [step2 (collapse-br-clusters step1)]
         [step3 (drop-leading-noise step2)]
         [step4 (drop-trailing-noise step3)])
    (normalize-inter-sibling-whitespace step4)))

strip-embedded-newlines removes the trailing newline from each string and, when the string isn’t the last child, inserts an empty-string sentinel in its place. collapse-br-clusters drops the sentinels adjacent to <br> nodes. drop-leading-noise and drop-trailing-noise trim empty strings, empty anchors, and dangling <br> nodes from the ends of the list. Finally, normalize-inter-sibling-whitespace replaces any sentinel sitting between two pieces of content with a single space and trims the leading indentation from the string that follows it. The result is a nested list keyed by source and anchor.

The script writes the summary table and the documentation to summary-data.scm and chez-docs-data.scm, respectively. The chez-docs library loads these data files directly using include, which requires summary-data.scm and chez-docs-data.scm to be in the same directory as the library file.

The main procedure in chez-docs is doc, which uses case-lambda to handle optional arguments with default values.

(define doc
  (case-lambda
    [(proc) (doc-helper proc 'display 'both)]
    [(proc action) (doc-helper proc action 'both)]
    [(proc action source) (doc-helper proc action source)]))

doc-helper validates arguments, then calls get-doc to look up the documentation text for proc in the given source. get-doc uses get-anchor to map the procedure name to its anchor via summary-data, then looks up the full documentation text in chez-docs-data. If proc isn’t found, doc-helper raises an error that suggests the closest match based on a fuzzy search (Levenshtein distance) of all procedure names.

(define (get-doc proc source)
  (let* ([anchor (get-anchor proc source)]
         [src-data (cdr (assoc source chez-docs-data))]
         [doc (assoc anchor src-data)])
    (if doc (cdr doc) doc)))

(define (get-anchor proc source)
  (let* ([src-data (cdr (assoc source summary-data))]
         [row (assoc proc src-data)])
    (if row (cadr row) row)))

When doc is called with 'both, documentation from both sources is displayed when available.

> (doc "<" 'display)

CHEZ SCHEME USER'S GUIDE

(< real1 real2 real3 ...)
...

THE SCHEME PROGRAMMING LANGUAGE

(< real1 real2 real3 ...)
...

action-helper displays the documentation and optionally opens a link to the relevant section in your default browser. When action is 'display, only the text is shown; 'open launches the browser link without displaying text; 'both does both. Opening a link makes a system call to open (macOS), xdg-open (Linux), or start (Windows) and requires an internet connection.

(define (action-helper proc doc action source)
  (let ([srclu '((csug "\nCHEZ SCHEME USER'S GUIDE\n\n")
                 (tspl "\nTHE SCHEME PROGRAMMING LANGUAGE\n\n"))])
    (when (not (symbol=? action 'open))
      (display (cadr (assoc source srclu)))
      (for-each display doc))
    (when (not (symbol=? action 'display))
      (launch-doc-link proc source))))

(machine-type) is used to determine the system-specific string, open-string, for use in the system call. If the machine type isn’t recognized, an error is raised that includes the URL so that you can open it yourself.

(define open-string
  (case (machine-type)
    ;; windows
    [(i3nt ti3nt a6nt ta6nt arm64nt tarm64nt)
     "start "]   
    ;; mac
    [(i3osx ti3osx a6osx ta6osx arm64osx tarm64osx ppc32osx tppc32osx)
     "open "]  
    ;; linux
    [(i3le ti3le a6le ta6le arm64le tarm64le arm32le tarm32le
       rv64le trv64le la64le tla64le ppc32le tppc32le)
     "xdg-open "]                      
    [else "undefined"]))

This was a fun little project. When I first had the idea, I was really excited because I worked out all of the initial code in less than 2 hours. But, when I started to write the first version of this blog post, I started to discover all of the little problems that didn’t occur to me initially. Nonetheless, I produced something incredibly useful to myself as an aid for learning and programming in Chez Scheme.