Inside the machine
One of the most interesting aspects of programming is the ability to inspect and modify programs while they are running. We all know debuggers, but there are lots of programs which let you interact with them directly while they are running. Some programs let you run scripts via embedded interpreters, which let you extend the program, but in some instances the programs themselves are the interpreters.
The famous story of the Deep Space 1 probe is an interesting one:
The Remote Agent software, running on a custom port of Harlequin Common Lisp, flew aboard Deep Space 1 (DS1), the first mission of NASA's New Millennium program. Remote Agent controlled DS1 for two days in May of 1999. During that time we were able to debug and fix a race condition that had not shown up during ground testing. (Debugging a program running on a $100M piece of hardware that is 100 million miles away is an interesting experience. Having a read-eval-print loop running on the spacecraft proved invaluable in finding and fixing the problem. The story of the Remote Agent bug is an interesting one in and of itself.) — Ron Garret, Lisping at JPL, (2002)
Examples: Emacs, StumpWM, Nyxt
Famous examples of programs being nothing but interpreters with a function are the Emacs family of text editors. Essentially, Emacs itself is a Lisp interpreter acting as a text editor. You may at any time open a read-eval-print loop inside the editor and actually interact with the program directly.
For instance, if I open an Emacs Lisp REPL inside GNU Emacs using M-x ielm,
opening and type (next-buffer), I actually end up selecting the next buffer!
Something similar to this is StumpWM, a tiling X11 window manager that is built entirely in Common Lisp. Somewhat similar to Emacs, you can install Swank, a REPL over a network connection, as an extension inside StumpWM, and then you can connect to the REPL using your client of choice. From there, any function inside the StumpWM Lisp image is available to you!
Another program that you can REPL into is Nyxt, an extensible web browser built
on WebKit. By running C-h s you can start up a Swank session and then you can
connect to it from, e.g., SLIME. Nyxt itself is written in Common Lisp, so
offering REPLability via Swank is pretty much a no-brainer. I’ve only tried Nyxt
very briefly, but from what I’ve seen it’s pretty interesting!
I actually tried something similar years ago with Common Lisp, it turned out to be super simple. I was building a Lisp-customizable i3 status bar program (I ended up writing it writing it in CHICKEN Scheme, though). Turns it didn’t require much: Swank is available via Quicklisp, so you can just add it to your Common Lisp program and voilà, you have a REPL connection available. I recorded the thing in action via video to demonstrate how simple and powerful this concept is.
These days I work with a multitude of application platforms and the Java Virtual Machine is one of them. Java has powerful introspectability, from its native debugger support to protocols like JMX and even the more modern (and seriously amazing!) Java Flight Recorder. I’ve actually been able to find bugs using JMX, a web server was routinely freezing during high traffic hours – but only for 0.5% of the requests or so, still, this (barely) under the SLA we had for the service. Turns out there was a thread starvation issue: when traffic was sufficiently large, other background processes were exhausting the thread pool reserved for incoming connections, causing the web server to not be able to receive incoming connections for a short period of time. Long story short, I found these incidents via JMX, and was able to pinpoint the issue. The fix was to use separate thread pools for the web server HTTP traffic and then background I/O.
Of course the Java Virtual Machine is not a Lisp interpreter, far from it, but it offers things that are similar. Hot code swapping, dynamic class reloading, etc. make it quite a powerful platform in terms of introspectability. But it’s a far cry from being a Lisp interpreter and a text editor, or a window manager, or a web browser.
Blazing fast development cycles
I think there is a crucial difference between allowing extensibility via scripts (or plugins) and the ability to actually interact completely via a live process. The former usually does not offer total control over the program, the latter offers total control over the process, because the interpreter is the program.
There is something admittedly fascinating about programs being, in a certain manner of speaking, living things. Not only does it give you a feeling of control, but it’s even more personal: you’re not just using the program, you’re inside it.
Having tried Clojure and Figwheel where you can directly REPL into your browser, the speed of development if you’re building some application while it’s running is quite something. I’ve also tried Fennel, a Lisp that compiles to Lua, using the LÖVE game engine using a Fennel REPL connected directly into the LÖVE game engine. The result is quite unfinished but the speed of development was astounding – I could change the game physics on the fly just by evaluating Lisp expressions in Emacs and the game would change instantly!
If I’m ever after building some sort of hyper-customizable program I will definitely try building it around a Lisp interpreter, possibly using Common Lisp.