Add lisp: icing or cake? post.

1 files changed, 284 insertions, 0 deletions

diff --git a/posts/2024-06-01-lisp-icing-or-cake.md b/posts/2024-06-01-lisp-icing-or-cake.md
new file mode 100644
index 0000000..34aaa91
--- /dev/null
+++ b/posts/2024-06-01-lisp-icing-or-cake.md
@@ -0,0 +1,284 @@
+title: Lisp: Icing or Cake?
+date: 2024-06-01 16:15:00
+tags: lisp, scheme, guile, gamedev, gamejam, chickadee
+summary: Is Lisp a thin layer on top of the stack or can it be (more of) the stack?
+---
+The [Spring Lisp Game Jam
+2024](https://itch.io/jam/spring-lisp-game-jam-2024) ended one week
+ago.  48 games were submitted, a new record for the jam!  This past
+week has been a time for participants to play and rate each other’s
+games.  As I explored the entries, I noticed two distinct
+meta-patterns in how people approached building games with Lisp.  I
+think this patterns apply more broadly to all applications of Lisp.
+This blog post will talk about these patterns in some detail, with
+examples.
+
+## But first!
+
+Here’s the breakdown of the [jam
+submissions](https://itch.io/jam/spring-lisp-game-jam-2024/entries) by
+language:
+
+```
+lang       entries    % (rounded)
+----       -------    -----------
+guile      15         31
+fennel     10         21
+clojure    5          10
+cl         5          10
+racket     4          8
+elisp      4          8
+s7         3          6
+kawa       1          2
+owl        1          2
+```
+
+I haven’t rolled up the various Schemes (Guile, Racket, S7, Kawa) into
+a general `scheme` category because Scheme is so minimally specified
+and they are all very distinct implementations for different purposes,
+not to mention that Racket has a lot more going on than just Scheme.
+
+For the first time ever, [Guile](https://gnu.org/software/guile) came
+out on top with the most submissions!  There’s a very specific reason
+for this outcome.  11 out of the 15 Guile games were built for the web
+with [Hoot](https://spritely.institute/hoot), a Scheme to WebAssembly
+compiler that I work on at the [Spritely
+Institute](https://spritely.institute).  2 of those 11 were official
+Spritely projects.  We [put out a
+call](https://spritely.institute/news/make-a-game-with-hoot-for-the-lisp-game-jam.html)
+for people to try making games with Hoot before the jam started, and a
+lot of people took us up on it!  Very cool!
+
+The next most popular language, which is typically *the* most popular
+language in these jams, is [Fennel](https://fennel-lang.org/).  Fennel
+is a Lisp that compiles to Lua.  It’s very cool!
+
+Also of note, at least to me as a Schemer, is that 3 games used
+[S7](https://ccrma.stanford.edu/software/snd/snd/s7.html).  Hmm, there
+might be something relevant to this post going on there.
+
+The patterns I’m about to talk about could sort of be framed as “The
+Guile Way vs. The Fennel Way”, but I don’t want to do that.  It's not
+an “us vs. them” thing.  It’s wonderful that there are so many flavors
+of Lisp these days that anyone can find a great implementation that
+suits their preferences.  Not only that, but many of these
+implementations can be used to make games that anyone can easily play
+in their web browser!  That was *not* the case several years ago.
+Incredible!
+
+I want to preface the rest of this post by saying that **both patterns
+are valid**, and while I prefer one over the other, that is not to say
+that the other is inferior.  I'll also show how these patterns can be
+thought of two ends of a spectrum and how, in the end, compromises
+must be made.  Okay, let’s get into it!
+
+## Lisp as icing
+
+The icing pattern is using Lisp as a “scripting” language on top of a
+cake that is made from C, Rust, and other static languages.  The
+typical way to do this is by embedding a Lisp interpreter into the
+larger program.  If you’re most interested in writing the high-level
+parts of an application in Lisp then this pattern is the fastest way
+to get there.  All you need is a suitable interpreter/compiler and a
+way to add the necessary hooks into your application.  Since the
+program is mainly C/Rust/whatever, you can then use emscripten to
+compile it to WebAssembly and deploy to the web.  Instant
+gratification, but strongly tied to static languages and their
+toolchains.
+
+S7 is an example of an embeddable Scheme.  Guile is also used for
+extending C programs, though typically that involves dynamically
+linking to libguile rather than embedding the interpreter into the
+program’s executable.  Fennel takes a different approach, recognizing
+that there are many existing applications that are already extensible
+through Lua, and provides a lispy language that *compiles to* Lua.
+
+## Lisp as cake
+
+The cake pattern is using Lisp to implement as much of the software
+stack as possible.  It’s Lisp all the way down... sorta.  Rather than
+embedding Lisp into a non-Lisp program, the cake pattern does the
+inverse: the majority of the program is written in Lisp.  When
+necessary, shared libraries can be called via a foreign function
+interface, but this should be kept to a minimum.  This approach takes
+longer to yield results.  Time is spent implementing missing libraries
+for your Lisp of choice and writing wrappers around the C shared
+libraries you can’t avoid using.  Web deployment gets trickier, too,
+since the project is not so easily emscriptenable.
+
+(You may recognize this as the classic embed vs. extend debate.
+You’re correct!  I'm just adding my own thoughts and applying it
+specifically to some real-world Lisp projects.)
+
+I mentioned Guile as an option for icing, but Guile really shines best
+as cake.  The initial vision for Guile was to Emacsify other programs
+by adding a Scheme interpreter to them.  These days, the best practice
+is to write your program in Scheme to begin with.  Common Lisp is
+probably the best example, though.  Implementations like SBCL have
+good C FFIs and can compile efficient native executables, minimizing
+the desire to use some C for performance reasons.
+
+## Case studies
+
+Let’s take a look at some of the languages and libraries used for the
+Lisp Game Jam and evaluate their icing/cake-ness.
+
+### Fennel + love2d
+
+[love2d](https://love2d.org/) has been a popular choice for solo or
+small team game development for many years.  It is a C++ program that
+embeds a Lua interpreter, which means its a perfect target for Fennel.
+Most Linux distributions package love2d, so it’s easy to run `.love`
+files natively.  Additionally, thanks to emscripten, love2d games can
+be deployed to the web.  Thus, most Fennel games use love2d.
+[./soko.bin](https://jleightcap.itch.io/sokobin) and [Gnomic
+Vengeance](https://alexjgriffith.itch.io/gnomic-vengeance) are two
+games that use this stack.
+
+Fennel + love2d is a perfect example of Lisp as icing.  Fennel sits at
+the very top of the stack, but there’s not really path to get Lisp
+into the layers below.  It is also the most successful Lisp games
+software stack to date.
+
+### S7 + raylib
+
+This stack is new to me, but two games used it this time around:
+[GhostHop](https://gcmas.itch.io/ghosthop) and [Life
+Predictor](https://illusion-fisherman.itch.io/life-predictor).  (You
+really gotta play, GhostHop, btw.  It’s a great little puzzle game and
+it is playable on mobile devices.)  [Raylib](https://www.raylib.com/)
+is a C library with bindings for many higher-level languages that has
+become quite popular in recent years.  S7 is also implemented in C and
+is easily embeddable.  This makes the combination easy to deploy on
+the web with emscripten.
+
+S7 + raylib is another example of Lisp as icing.  I’m curious to see
+if this stack becomes more popular in future jams.
+
+### Guile + Chickadee
+
+This is the stack that I helped build.
+[Chickadee](/projects/chickadee.html) is a game library for Guile that
+implements almost all of the interesting parts in Scheme, including
+rendering.  Two games were built with Chickadee in the most recent
+jam: [Turbo Racer 3000](https://etenil.itch.io/turbo-racer-3000) and
+[Bloatrunner](https://snamellit.itch.io/bloatrunner).
+
+Guile + Chickadee is an example of Lisp as cake.  Chickadee wraps some
+C libraries for low-level tasks such as loading images, audio, and
+fonts, but it is written in pure Scheme.  All the matrix and vector
+math is in Scheme.  Chickadee comes with a set of rendering primitives
+comparable to love2d and raylib but they’re all implemented in Scheme.
+I’ve even made progress on rendering vector graphics with Scheme,
+something that most other Lisp game libraries call out to C to do.
+Chickadee has pushed the limits of Guile’s compiler and virtual
+machine, and Guile has been improved as a result.  But it’s the long
+road.  Chickadee is mostly developed by just me in my very limited
+spare time, so it is taking a long time to reach feature parity with
+more popular game development libraries.
+
+### Hoot + HTML5 canvas
+
+I also helped build this one.  Hoot is a Scheme-to-WebAssembly
+compiler.  Rather than compile the Guile VM (written in C) to Wasm
+using emscripten, Hoot implements a complete Wasm toolchain and a new
+backend for Guile’s compiler that emits Wasm directly.  Hoot is
+written entirely in Scheme.  Unlike C programs compiled with
+emscripten that target Wasm 1.0 with linear memory, Hoot targets Wasm
+2.0 with GC managed heap types.  This gives Hoot a significant
+advantage: Hoot binaries **do not ship a garbage collector** and thus
+are much smaller than Lisp runtimes compiled via emscripten.
+Cirkoban’s Wasm binary weighs in at < 2MiB whereas `love.wasm` is
+almost 6MiB when I checked.  Hoot programs can also easily
+interoperate with JavaScript.  Scheme objects can easily be passed to
+JavaScript, and vice versa, as they are managed in the same heap.
+With all of the browser APIs just a Wasm import away, an obvious
+choice for games was the built-in HTML5 canvas API for easy 2D
+rendering.
+
+11 games used Hoot in the jam, including (shameless plug)
+[Cirkoban](https://davexunit.itch.io/cirkoban) and [Lambda
+Dungeon](https://fluxharmonic.itch.io/lambda-dungeon).
+
+Hoot + HTML5 canvas is mostly dense cake with a bit of icing.  On one
+hand, it took a year and significant funding to boot Hoot.  We said
+“no” to emscripten, built a completely separate toolchain, and
+extended Guile’s compiler.  It's Lisp all the way until you hit the
+browser runtime!  We even have a Wasm interpreter that runs on the
+Guile VM!  Hoot rules!  It was a risk but it paid off.  On the other
+hand, the canvas API is very high level.  The more cake thing to do
+would be to use Hoot’s JS FFI to call WebGL and/or WebGPU.  Indeed,
+this is the plan for the future!  Wasm GC needs some improvements to
+make this feasible, but my personal goal is to get Chickadee ported to
+Hoot.  I want Chickadee games to be easy to play natively and in
+browsers, just like love2d games.
+
+## The cake/icing spectrum
+
+I must acknowledge the limitations of the cake approach.  We’re not
+living in a world of Lisp machines, but a world of glorified PDP-11s.
+Even the tallest of Lisp cakes sits atop an even larger cake made
+mostly of C.  All modern Lisp systems bottom out at some point.  Emacs
+rests on a C core.  Guile’s VM is written in C.  Hoot runs on mammoth
+JavaScript engines written in C++ like V8.  Games on Hoot currently
+render with HTML5 canvas rather than the lower level WebGL/WebGPU.
+Good luck using OpenGL without libGL; Chickadee uses guile-opengl
+which uses the C FFI to call into libGL.  Then there’s libpng,
+FreeType, and more.  Who the heck wants to rewrite all this in Lisp?
+Who even has the resources?  Does spending all this time taking the
+scenic route matter at all, or are we just deluding ourselves because
+we have fun writing Lisp code?
+
+I think it *does* matter.  Every piece of the stack that can be
+reclaimed from the likes of C is a small victory.  The parts written
+in Lisp are much easier to hack on, and some of those things become
+**live hackable** while our programs are running.  They are also
+memory safe, typically, thanks to GC managed runtimes.  Less FFI calls
+means less overhead from traversing the Lisp/C boundary and more
+safety.  As more of the stack becomes Lisp, it starts looking less
+like icing and more like cake.
+
+Moving beyond games, we can look to the [Guix](https://guix.gnu.org)
+project as a great example of just how tasty the cake can get.  Guix
+took the functional packaging model from the Nix project and made a
+fresh implementation, replacing the Nix language with Guile.  Why?
+For code staging, code sharing, and improved hackability.  Guix also
+uses an init system written in Guile rather than systemd.  Why?  For
+code staging, code sharing, and improved hackability.  These are real
+advantages that make the trade-off of not using the industry-standard
+thing worth it.
+
+I’ve been using Guix since the early days, and back then it was easy
+to make the argument that Guix was just reinventing wheels for no
+reason.  But now, over 10 years later, the insistence on maximizing
+the usage of Lisp has been key to the success of the project.  As a
+user, once you learn the Guix idioms and a bit of Guile, you unlock
+extraordinary power to craft your OS to your liking.  It’s the closest
+thing you can get to a Lisp machine on modern hardware.  The cake
+approach paid off for Guix, and it could pay off for other projects,
+too.
+
+If Common Lisp is more your thing, and even if it isn’t, you’ll be
+amazed by the [Trial](https://shirakumo.github.io/trial/) game engine
+and how much of it is implemented in Common Lisp rather than wrapping
+C libraries.
+
+There’s also projects like [Pre-Scheme](https://prescheme.org/) that
+give me hope that one day the layers below the managed GC runtime can
+be implemented in Lisp.  Pre-Scheme was developed and successfully
+used for [Scheme 48](https://s48.org/) and I am looking forward to a
+modern revival of it thanks to an [NLnet
+grant](https://nlnet.nl/project/Pre-Scheme/).
+
+## I'm a cake boy
+
+That’s right, I said it: I’m a cake boy.  I want to see projects
+continue to push the boundaries of what Lisp can do.  When it comes to
+the Lisp Game Jam, what excites me most are not the games themselves,
+but the small advances made to reclaim another little slice of the
+cake from stale, dry C.  I intend to keep pushing the limits for Guile
+game development with my Chickadee project.
+
+It’s *not* a piece of cake to bake a lispy cake, and the way is often
+hazy, but I know we can’t be lazy and just do the cooking by the book.
+Rewrite it in Rust?  No way!  Rewrite it in Lisp!