Access Chez Scheme documentation from the REPL
In the process of learning Chez Scheme, I missed R’s ability to quickly pull up documentation from the console via help or ?. I was interested in the idea of trying to format the contents of the Chez Scheme User’s Guide (CSUG) and The Scheme Programming Language (TSPL) for display in the REPL (similar to Clojure Docs). That was a bigger task than I wanted to tackle and the first version of chez-docs simply opened links to the relevant sections of CSUG and TSPL from the REPL. Eventually, I more fully scraped the HTML documentation pages and displayed more information in the REPL while still providing the option to open a link to the online documentation.
The initial web scraping was all done in R, but I eventually moved all of that code to Chez Scheme. The scraping code lives in a separate library, chez-docs-scrape, and a short script, scrape-script.ss, runs the whole process. Neither CSUG nor TSPL provides a sitemap.xml, so crawl-index downloads each site’s index.html, parses it with html->sxml from the wak library, and uses sxpath to extract every href. The fragments (#...) and leading ./ are stripped from the hrefs, which are then filtered to local .html pages and deduplicated. download-pages shells out to curl for each page and pauses for one second between requests.
The CSUG Summary of Forms page links to both CSUG and TSPL and is used to build a lookup table mapping each procedure name (the key) to its anchor and URL. The HTML is parsed with html->sxml and sxpath is used to extract the table rows (after dropping the column headers and horizontal rule in the first two rows). The key is the first token of the form with parentheses removed, and relative CSUG URLs are expanded to full URLs. Rows are split by source, and two edge cases are handled manually: alias appears twice in CSUG as both a procedure and a keyword for import. The keyword version is dropped. let appears twice in TSPL pointing to two different anchors. One is renamed "named let".
Before parsing each downloaded HTML page, read-and-clean uses irregex to fix a couple of problems in the source HTML that html->sxml doesn’t handle well. Stray <p> tags that immediately precede a <p>, <li>, or <ul> tag are removed, as are stray <p> tags before an anchored list item (e.g., <p><a name="g32"></a><li>), which otherwise leave the <li> unclosed and nest the rest of the page inside it (as in foreign.html in CSUG 10.3). Some <dl> blocks omit the closing </dt> before <dd>, which causes html->sxml to nest <dd> inside <dt> rather than treating them as siblings, so the missing </dt> tags are inserted (and any resulting </dt></dt> is collapsed back to a single </dt>).
(define (read-and-clean path)
(let* ([raw (call-with-input-file path get-string-all)]
[step0 (irregex-replace/all "<p>[ \t\r\n]*(?=<a name=\"[^\"]*\"></a><li>)" raw "")]
[step1 (irregex-replace/all "<p>[ \t\r\n]*(?=<(p|li|ul)>)" step0 "")]
[step2 (irregex-replace/all "<dd>" step1 "</dt><dd>")]
[step3 (irregex-replace/all "</dt></dt>" step2 "</dt>")])
(open-string-input-port step3)))For each cleaned page, sxpath is used to extract <p> elements and walk through them looking for formdef blocks, the marked-up sections in the documentation that describe each procedure. Everything between a formdef and the next formdef, header, or footer is collected as the documentation for that anchor.
Originally, each <p> element was converted directly to a plain-text string. That threw away the markup (e.g., code, metavariables, and links) that would be needed to display the documentation with any formatting. Now, sxml->doc recursively walks the SXML tree and converts each <p> element into a small semantic tree that keeps only the markup worth rendering. A doc node is either a string or a tagged list.
;; (code node ...) <tt>
;; (var node ...) <i>, <it> (metavariables)
;; (bold node ...) <b>
;; (sub node ...) <sub>
;; (sup node ...) <sup>
;; (link href node ...) <a href=...>
;; (math string) gif images and <=, >= entities
;; (br) <br>
;; (ul node ...) <ul>
;; (li node ...) <li>
;; (dt node ...) <dt>
;; (dd node ...) <dd>
;; (table row ...) <table>, where row is (row cell ...)
;; and cell is (cell node ...)Tags not in that list (e.g., p, span, div) are spliced into their parent, attribute subtrees are skipped, and adjacent strings are merged to keep the tree compact. HTML entities (e.g., , <, é) are translated with a lookup table, and inline images are replaced with their text equivalents (e.g., math GIFs become =>, ->, λ, ∞).
The trickiest part was whitespace. The line breaks and indentation in the HTML source are invisible in a browser, but they show up as stray newlines and spaces in plain text. Before rendering the children of a node, normalize-children cleans them up in five steps.
(define (normalize-children children)
(let* ([step1 (strip-embedded-newlines children)]
[step2 (collapse-br-clusters step1)]
[step3 (drop-leading-noise step2)]
[step4 (drop-trailing-noise step3)])
(normalize-inter-sibling-whitespace step4)))strip-embedded-newlines removes the trailing newline from each string and, when the string isn’t the last child, inserts an empty-string sentinel in its place. collapse-br-clusters drops the sentinels adjacent to <br> nodes. drop-leading-noise and drop-trailing-noise trim empty strings, empty anchors, and dangling <br> nodes from the ends of the list. Finally, normalize-inter-sibling-whitespace replaces any sentinel sitting between two pieces of content with a single space and trims the leading indentation from the string that follows it. The result is a nested list keyed by source and anchor, where each anchor is followed by a list of paragraphs and each paragraph is a list of doc nodes.
The script writes the summary table and the documentation to summary-data.scm and chez-docs-data.scm, respectively. The chez-docs library loads these data files directly using include, which requires summary-data.scm and chez-docs-data.scm to be in the same directory as the library file.
The main procedure in chez-docs is doc, which uses case-lambda to handle optional arguments with default values.
(define doc
(case-lambda
[(proc) (doc-helper proc 'display 'both)]
[(proc action) (doc-helper proc action 'both)]
[(proc action source) (doc-helper proc action source)]))doc-helper validates arguments, then calls get-doc to look up the documentation text for proc in the given source. get-doc uses get-anchor to map the procedure name to its anchor via summary-data, then looks up the full documentation text in chez-docs-data. If proc isn’t found, doc-helper raises an error that suggests the closest match based on a fuzzy search (Levenshtein distance) of all procedure names. If proc is documented only in the other book, the error says so instead, e.g., (doc "fxsll" 'display 'tspl) reports fxsll not found in tspl but is found in csug.
(define (get-doc proc source)
(let* ([anchor (get-anchor proc source)]
[src-data (cdr (assoc source chez-docs-data))]
[doc (assoc anchor src-data)])
(if doc (cdr doc) doc)))
(define (get-anchor proc source)
(let* ([src-data (cdr (assoc source summary-data))]
[row (assoc proc src-data)])
(if row (cadr row) row)))When doc is called with 'both, documentation from both sources is displayed when available.
> (doc "<" 'display)
CHEZ SCHEME USER'S GUIDE
(< real1 real2 real3 ...)
...
THE SCHEME PROGRAMMING LANGUAGE
(< real1 real2 real3 ...)
...action-helper displays the documentation and optionally opens a link to the relevant section in your default browser. When action is 'display, only the text is shown; 'open launches the browser link without displaying text; 'both does both. Opening a link makes a system call to open (macOS), xdg-open (Linux), or start (Windows) and requires an internet connection.
(define (action-helper proc doc action source)
(let ([srclu '((csug "CHEZ SCHEME USER'S GUIDE")
(tspl "THE SCHEME PROGRAMMING LANGUAGE"))])
(when (not (symbol=? action 'open))
(let ([render (current-renderer source)])
(display (string-append "\n" (render (list (list 'bold (cadr (assoc source srclu)))))
"\n\n"))
(for-each (lambda (para) (display (string-append (render para) "\n\n"))) doc)))
(when (not (symbol=? action 'display))
(launch-doc-link proc source))))Because the data is now a doc tree rather than plain strings, rendering happens in chez-docs. current-renderer returns a procedure, built by make-renderer, that walks a list of doc nodes and returns a string. The style is controlled by the doc-style parameter, which defaults to 'plain. Plain rendering produces the same ASCII text as the original string-based scrape: <br> becomes a newline, list items are rendered as * bullets, definition details are indented, and superscripts are prefixed with ^. Tables, which were previously replaced with [table not shown], are now laid out in aligned columns.
(define doc-style
(make-parameter
'plain
(lambda (x)
(unless (memq x '(plain rich))
(assertion-violation "(doc-style style)" "style not one of 'plain, 'rich" x))
x)))Setting (doc-style 'rich) uses ANSI escape codes to render labels in bold, metavariables in italics, and code in cyan. It also substitutes Unicode symbols, e.g., ⇒ for =>, ≤ for <=, obj₁ for obj1, and • for list bullets. Cross-references become OSC 8 hyperlinks to the online documentation, which are clickable in terminals that support them. The links in the HTML are relative to the page, so they are resolved against each book’s base URL, which is taken from the summary data so that it follows the scraped version of the docs. If the NO_COLOR environment variable is set, the escape codes (including hyperlinks) are omitted but the Unicode symbols are kept. To use rich output in every session, add (doc-style 'rich) to your .chezschemerc file after (import (chez-docs)), or use parameterize to change the style for a single call.
> (parameterize ([doc-style 'rich]) (doc "eq?"))(machine-type) is used to determine the system-specific string, open-string, for use in the system call. If the machine type isn’t recognized, an error is raised that includes the URL so that you can open it yourself.
(define open-string
(case (machine-type)
;; windows
[(i3nt ti3nt a6nt ta6nt arm64nt tarm64nt)
"start "]
;; mac
[(i3osx ti3osx a6osx ta6osx arm64osx tarm64osx ppc32osx tppc32osx)
"open "]
;; linux
[(i3le ti3le a6le ta6le arm64le tarm64le arm32le tarm32le
rv64le trv64le la64le tla64le ppc32le tppc32le)
"xdg-open "]
[else "undefined"]))This was a fun little project. When I first had the idea, I was really excited because I worked out all of the initial code in less than 2 hours. But, when I started to write the first version of this blog post, I started to discover all of the little problems that didn’t occur to me initially. Nonetheless, I produced something incredibly useful to myself as an aid for learning and programming in Chez Scheme.