#+AUTHOR: Mark Wooding
#+LaTeX_CLASS: strayman
#+LaTeX_HEADER: \usepackage{tikz, gnuplot-lua-tikz}
+#+EXPORT_FILE_NAME: doc/README.pdf
~runlisp~ is a small C program intended to be run from a script ~#!~
line. It selects and invokes a Common Lisp implementation, so as to run
+ Embeddable Common Lisp (~ecl~), and
+ Steel Bank Common Lisp (~sbcl~).
-I'm happy to take patches to support additional free Lisp
-implementations. I'm not interested in supporting non-free Lisp
-systems.
+Adding more Lisps is simply a matter of writing the necessary runes in a
+configuration file. Of course, there's a benefit to having a collection
+of high-quality configuration runes curated centrally, so I'm happy to
+accept submissions in support of any free[fn:free] Lisp implementations.
+[fn:free] Here I mean free as in freedom.
-* Writing scripts in Lisp
+
+* Writing scripts in Common Lisp
** Basic use
** Where =runlisp= looks for configuration
You can influence which Lisp implementations are chosen by ~runlisp~ by
-writing a configuration file, and/or setting an environment variable.
+writing configuration files, and/or setting environment variables.
-~runlisp~ looks for configuration in ~~/.runlisprc~, and in
-~~/.config/runlisprc~. You could put configuration in both, but that
-doesn't seem like a great idea. A configuration file just contains
-blank lines, comments, and command-line options, just as you'd write
-them to the shell. Simple quoting and escaping is provided: see the
-manual page for the full details. Each line is processed independently,
-so it doesn't work to write an option on one line and then its argument
-on the next.
+The ~runlisp~ program looks for configuration in a number of places.
-The environment variable ~RUNLISP_OPTIONS~ is processed /after/ reading
-the configuration file(s), if any. Again, it should contain
-command-line options, as you'd write them to the shell.
+ + There's a system-global directory ~SYSCONFDIR/runlisp/runlisp.d/~.
+ All of the files in this directory named ~SOMETHING.conf~ are read,
+ in increasing lexicographical order by name. The package comes with
+ a file ~0base.conf~ intended to be read first, so that it can be
+ overridden if necessar. This sets up basic definitions, and defines
+ the necessary runes for those Lisp implementations which are
+ supported `out of the box'. New Lisp packages might come with
+ additional files to drop into this directory.
-** Deciding which Lisp implementation to use
+ + There's a system-global file ~SYSCONFDIR/runlisp/runlisp.conf~ which
+ is intended to be edited by the system administrator to account for
+ any local quirks. This is read /after/ the directory, which is
+ intended to be used by distribution packages, so that the system
+ administrator can override them.
+
+ + Users can create files ~$HOME/.runlisp.conf~ and/or
+ ~$HOME/.config/runlisp.conf~[fn:xdg-config] in their home
+ directories to add support for privately installed Lisp systems, or
+ to override settings made by earlier configuration files.
+
+The configuration syntax is complicated, and explained in detail in the
+*runlisp.conf* manpage.
-The most useful option to use here is ~-P~, which builds up a
-/preference list/, in order. The argument to ~-P~ is a comma-separated
-list of Lisp implementation names, just like you'd give to ~-L~.
+Configuration options can also be set on the command line, though the
+effects are subtly different. Again, see the manual pages for details.
+
+[fn:xdg-config] More properly, in ~$XDG_CONFIG_HOME/runlisp.conf~, if
+you set that.
+
+
+** Deciding which Lisp implementation to use
-If you provide multiple ~-P~ options (e.g., on different lines of your
-configuration file, or separately in the configuration file and
-environment variable, then the lists are concatenated. Since the
-environment variable is processed after the configuration file, this
-means that
+The ~prefer~ option specifies a /preference list/ of Lisp
+implementations. The value is a list of Lisp implementation names, as
+you'd give to ~-L~, separated by commas and/or spaces. If the
+environment variable ~RUNLISP_PREFER~ is set, then this overrides any
+value found in the configuration files.
When deciding which Lisp implementation to use, ~runlisp~ works as
follows. It builds a list of /acceptable/ Lisp implementations from the
-~-L~ options, and a list of /preferred/ Lisp implementations from the
-~-P~ options. If there aren't any ~-L~ options, then it assumes that
-/all/ Lisp implementations are acceptable; but if there are no ~-P~
-options then it assumes that /no/ Lisp implementations are preferred.
-It then works through the preferred list in order: if it finds an
-implementation which is installed and acceptable, then it uses that one.
-If that doesn't work, then it works through the acceptable
-implementations that it hasn't tried yet, in order, and if it finds one
-of those that's installed, then it runs that one. Otherwise it reports
-an error and gives up.
-
-** Clearing the preferred list
-
-Since the environment variable is processed after the configuration
-files, it can only append more Lisp implementations to the end of the
-preferred list, which may well not be so helpful. There's an additional
-option ~-C~, which completely clears the preferred list. The idea is
-that you can write ~-C~ at the start of your ~RUNLISP_OPTIONS~
-environment variable to temporarily override your usual configuration
-for some special effect.
+~-L~ command-line option, and a list of /preferred/ Lisp implementations
+from the ~prefer~ configuration option (or environment variable). If
+there aren't any ~-L~ options, then it assumes that /all/ Lisp
+implementations are acceptable; if no ~prefer~ option is set then it
+assumes that /no/ Lisp implementations are preferred. It then works
+through the preferred list in order: if it finds an implementation which
+is installed and acceptable, then it uses that one. If that doesn't
+work, then it works through the acceptable implementations that it
+hasn't tried yet, in order, and if it finds one of those that's
+installed, then it runs that one. Otherwise it reports an error and
+gives up.
+
+
+** Supporting new Lisp implementations
+
+~runlisp~ tries hard to make adding support for a new Lisp as painless
+as possible. An awkward Lisp will of course cause trouble, but
+~runlisp~ itself is easy.
+
+As a simple example, let's add support for the 32-bit version of
+Clozure\nbsp{}CL. The source code for Clozure\nbsp{}CL easily builds
+both 32- and 64-bit binaries in either 32- or 64-bit userlands, and one
+might reasonably want to use the 32-bit CCL for some reason. The
+following configuration stanza is sufficient
+
+: [ccl32]
+: @PARENTS = ccl
+: command = ${@ENV:CCL32?ccl32}
+
+ + The first line heads a configuration section, providing the name
+ which will be used for this Lisp implementation, e.g., in ~-L~
+ options or ~prefer~ lists.
+
+ + The second line tells ~runlisp~ that configuration settings not
+ found in this section should be looked up in the ~ccl~ section
+ instead.
+
+ + The third line defines the command to be used to invoke the Lisp
+ system. It tries to find an environment variable named ~CCL32~,
+ falling back to looking up ~ccl32~ in the path otherwise.
+
+And, err..., that's it. The ~@PARENTS~ setting uses the detailed
+command-line runes for ~ccl~, so they don't need to be written out
+again.
+
+That was rather anticlimactic, because all of the work got done
+somewhere else. So let's look at a complete example: Steel Bank Common
+Lisp. (SBCL's command-line interface is well thought-out, so this is an
+ideal opportunity to explain how ~runlisp~ configuration works, without
+getting bogged down in the details of fighting less amenable Lisps.)
+
+The provided ~0base.conf~ file defines SBCL as follows.
+
+: [sbcl]
+:
+: command = ${@ENV:SBCL?sbcl}
+: image-file = ${@NAME}+asdf.core
+:
+: run-script =
+: ${command} --noinform
+: $?@IMAGE{--core "${image-path}" --eval "${image-restore}" |
+: --eval "${run-script-prelude}"}
+: --script "${@SCRIPT}"
+:
+: dump-image =
+: ${command} --noinform --no-userinit --no-sysinit --disable-debugger
+: --eval "${dump-image-prelude}"
+: --eval "(sb-ext:save-lisp-and-die \"${@IMAGENEW|q}\")"
+
+Let's take this in slightly larger pieces.
+
+ + We see the ~[sbcl]~ section heading, and the ~command~ setting
+ again. These should now be unsurprising.
+
+ + There's no ~@PARENTS~ setting, so by default the ~sbcl~ section
+ inherits settings from the ~@COMMON~ section, defined in
+ ~0base.conf~. We shall use a number of definitions from this
+ section.
+
+ + The ~image-file~ gives the name of the custom image file to look for
+ when trying to start SBCL, but not the directory. (The directory is
+ named by the ~image-dir~ configuration setting.) The image file
+ will be named ~sbcl+asdf.core~, but this isn't what's written.
+ Instead, it uses ~${@NAME}~, which is replaced by the name of the
+ section being processed. When we're running SBCL, this does the
+ same thing; but if someone wants to configure a new ~foo~ Lisp and
+ set ~@PARENTS~ to ~sbcl~, then the image file for ~foo~ will be
+ named ~foo+asdf.core~ by default. You needn't take such care when
+ configuring Lisp implementations for your own purposes, but it's
+ important for configurations which will be widely used.
+
+ + The ~run-script~ setting explains how to get SBCL to run a script.
+ This string is broken into words at (unquoted) spaces.
+
+ The syntax ~$?VAR{CONSEQ|ALT}~ means: if a configuration setting
+ ~VAR~ is defined, then expand to ~CONSEQ~; otherwise, expand to
+ ~ALT~. In this case, if the magic setting ~@IMAGE~ is defined, then
+ we add the tokens ~--core "${image-path}" --eval "${image-restore}"~
+ to the SBCL command line; otherwise, we add ~--eval
+ "${run-script-prelude}"~. The ~@IMAGE~ setting is defined by
+ ~runlisp~ only if (a)\nbsp{}a custom image was found in the correct
+ place, and (b)\nbsp{}use of custom images isn't disabled on its
+ command line.
+
+ The ~${image-path}~ token expands to the full pathname to the custom
+ image file; ~image-restore~ is a predefined Lisp expression to be
+ run when starting from a dumped image (e.g., to get ASDF to refresh
+ its idea of which systems are available).
+
+ The ~run-script-prelude~ is another (somewhat involved) Lisp
+ expression which sets up a Lisp environment suitable for running
+ scripts -- e.g., by arranging to ignore ~#!~ lines, and pushing
+ ~:runlisp-script~ onto ~*features*~.
+
+ Finally, regardless of whether we're using a custom or vanilla
+ image, we add the tokens ~--script "${@SCRIPT}"~ to the command
+ line. The ~${@SCRIPT}~ token is replaced by the actual script
+ pathname. ~runlisp~ then appends further arguments from its own
+ command line and runs the command. (For most Lisps, ~uiop~ needs a
+ ~--~ marker before the user arguments, but not for SBCL.)
+
+ + Finally, ~dump-image~ defines a command line for dumping a custom
+ images. The ~dump-image-prelude~ setting is a Lisp expression for
+ setting up a Lisp so that it will be in a useful state when dumped:
+ it's very similar to ~run-script-prelude~, and is built out of many
+ of the same pieces.
+
+ The thing we haven't seen before is ~${@IMAGENEW|q}~. The
+ ~@IMAGENEW~ setting is defined by the ~dump-runlisp-image~ program
+ the name the file in which the new image should be
+ saved.[fn:image-rename] The ~|q~ `filter' is new: it means that the
+ filename should be escaped suitable for inclusion in a Lisp quoted
+ string, by prefixing each ~\~ or ~"~ with a ~\~.
+
+That's more or less all there is. SBCL is a particularly simple
+example, but mostly because other Lisp implementations require fancier
+stunts /at the Lisp level/. The ~runlisp~-level configuration isn't any
+more complicated than SBCL.
+
+[fn:image-rename] ~dump-runlisp-image~ wants to avoid clobbering an
+existing image with a half-finished one, so it tries to arrange for the
+new image to be written to a different file, and then renames it once
+it's been created successfully.)
* What's wrong with =cl-launch=?
implementations, and compared them to how long ~cl-launch~ took: the
results are shown in table [[tab:runlisp-vanilla]]. ~runlisp~ is /at least/
two and half times faster at running this script than ~cl-launch~ on all
-implementations except Clozure CL[fn:slow-ccl], and approaching four and
-a half times faster on SBCL.
+implementations except Clozure\nbsp{}CL[fn:slow-ccl], and approaching
+four and a half times faster on SBCL.
#+CAPTION: ~cl-launch~ vs ~runlisp~ (with vanilla images)
#+NAME: tab:runlisp-vanilla
|------------------+-------------------+-----------------+----------------------|
| *Implementation* | *~cl-launch~ (s)* | *~runlisp~ (s)* | *~runlisp~ (factor)* |
|------------------+-------------------+-----------------+----------------------|
-| ABCL | 7.3036 | 2.6027 | 2.806 |
-| Clozure CL | 1.2769 | 0.9678 | 1.319 |
-| GNU CLisp | 1.2498 | 0.2659 | 4.700 |
-| CMU CL | 0.9665 | 0.3065 | 3.153 |
-| ECL | 0.8025 | 0.3173 | 2.529 |
-| SBCL | 0.3266 | 0.0739 | 4.419 |
+| ABCL | 7.3378 | 2.6474 | 2.772 |
+| Clozure CL | 1.2888 | 0.9742 | 1.323 |
+| GNU CLisp | 1.2405 | 0.2703 | 4.589 |
+| CMU CL | 0.9521 | 0.3097 | 3.074 |
+| ECL | 0.8020 | 0.3236 | 2.478 |
+| SBCL | 0.3205 | 0.0874 | 3.667 |
|------------------+-------------------+-----------------+----------------------|
#+TBLFM: $4=$2/$3;%.3f
|------------------+-------------------+-----------------+----------------------|
| *Implementation* | *~cl-launch~ (s)* | *~runlisp~ (s)* | *~runlisp~ (factor)* |
|------------------+-------------------+-----------------+----------------------|
-| ABCL | 7.3036 | 2.5873 | 2.823 |
-| Clozure CL | 1.2769 | 0.0088 | 145.102 |
-| GNU CLisp | 1.2498 | 0.0146 | 85.603 |
-| CMU CL | 0.9665 | 0.0063 | 153.413 |
-| ECL | 0.8025 | 0.3185 | 2.520 |
-| SBCL | 0.3266 | 0.0077 | 42.416 |
+| ABCL | 7.3378 | 2.7023 | 2.715 |
+| Clozure CL | 1.2888 | 0.0371 | 34.739 |
+| GNU CLisp | 1.2405 | 0.0191 | 64.948 |
+| CMU CL | 0.9521 | 0.0060 | 158.683 |
+| ECL | 0.8020 | 0.3275 | 2.449 |
+| SBCL | 0.3205 | 0.0064 | 50.078 |
|------------------+-------------------+-----------------+----------------------|
#+TBLFM: $4=$2/$3;%.3f
|------------------------------+-------------|
| *Implementation* | *Time (ms)* |
|------------------------------+-------------|
-| Clozure CL | 8.8 |
-| GNU CLisp | 14.6 |
-| CMU CL | 6.3 |
-| SBCL | 7.7 |
+| Clozure CL | 37.1 |
+| GNU CLisp | 19.1 |
+| CMU CL | 6.0 |
+| SBCL | 6.4 |
|------------------------------+-------------|
-| Perl | 1.2 |
-| Python | 10.3 |
+| Perl | 1.1 |
+| Python | 6.8 |
|------------------------------+-------------|
-| Debian Almquist shell (dash) | 1.4 |
-| GNU Bash | 2.0 |
-| Z Shell | 4.1 |
+| Debian Almquist shell (dash) | 1.2 |
+| GNU Bash | 1.5 |
+| Z Shell | 3.1 |
|------------------------------+-------------|
-| Tiny C (compile & run) | 1.2 |
-| GCC (precompiled) | 0.5 |
+| Tiny C (compile & run) | 1.6 |
+| GCC (precompiled) | 0.6 |
|------------------------------+-------------|
#+CAPTION: Comparison of ~runlisp~ and other script interpreters
measurements are included in the source distribution, in the ~bench/~
subdirectory.)
-[fn:slow-ccl] I don't know why Clozure CL shows such a small difference
-here.
+[fn:slow-ccl] I don't know why Clozure\nbsp{}CL shows such a small
+difference here.
** It's inconvenient
`software system', though this notion doesn't appear to be well-defined,
but this all works by editing a single ~$LISPS~ shell variable. By
contrast, ~runlisp~ has a ~-L~ option with which scripts can specify the
-Lisp systems they support (in a preference order), and a ~-P~ option
-with which users can express their own preferences (e.g., in the
-environment or a configuration file): ~runlisp~ will never choose a Lisp
-system which the script can't deal with, but it will respect the user's
-relative preferences.
+Lisp systems they support (in a preference order), and a ~prefer~
+configuration setting with which users can express their own
+preferences: ~runlisp~ will never choose a Lisp system which the script
+can't deal with, but it will respect the user's relative preferences.
+
+Also, ~cl-launch~ is a monolith. Adding a new Lisp implementation to
+it, or changing how a particular implementation is invoked, is rather
+involved. By contrast, ~runlisp~ makes this remarkably easy, as
+described in [[Supporting new Lisp implementations]].
** It doesn't establish a (useful) common environment
~mdw / runlisp / blobdiff