diff options
| -rw-r--r-- | posts/2024-06-01-lisp-icing-or-cake.md | 284 |
1 files changed, 284 insertions, 0 deletions
diff --git a/posts/2024-06-01-lisp-icing-or-cake.md b/posts/2024-06-01-lisp-icing-or-cake.md new file mode 100644 index 0000000..34aaa91 --- /dev/null +++ b/posts/2024-06-01-lisp-icing-or-cake.md @@ -0,0 +1,284 @@ +title: Lisp: Icing or Cake? +date: 2024-06-01 16:15:00 +tags: lisp, scheme, guile, gamedev, gamejam, chickadee +summary: Is Lisp a thin layer on top of the stack or can it be (more of) the stack? +--- +The [Spring Lisp Game Jam +2024](https://itch.io/jam/spring-lisp-game-jam-2024) ended one week +ago. 48 games were submitted, a new record for the jam! This past +week has been a time for participants to play and rate each other’s +games. As I explored the entries, I noticed two distinct +meta-patterns in how people approached building games with Lisp. I +think this patterns apply more broadly to all applications of Lisp. +This blog post will talk about these patterns in some detail, with +examples. + +## But first! + +Here’s the breakdown of the [jam +submissions](https://itch.io/jam/spring-lisp-game-jam-2024/entries) by +language: + +``` +lang entries % (rounded) +---- ------- ----------- +guile 15 31 +fennel 10 21 +clojure 5 10 +cl 5 10 +racket 4 8 +elisp 4 8 +s7 3 6 +kawa 1 2 +owl 1 2 +``` + +I haven’t rolled up the various Schemes (Guile, Racket, S7, Kawa) into +a general `scheme` category because Scheme is so minimally specified +and they are all very distinct implementations for different purposes, +not to mention that Racket has a lot more going on than just Scheme. + +For the first time ever, [Guile](https://gnu.org/software/guile) came +out on top with the most submissions! There’s a very specific reason +for this outcome. 11 out of the 15 Guile games were built for the web +with [Hoot](https://spritely.institute/hoot), a Scheme to WebAssembly +compiler that I work on at the [Spritely +Institute](https://spritely.institute). 2 of those 11 were official +Spritely projects. We [put out a +call](https://spritely.institute/news/make-a-game-with-hoot-for-the-lisp-game-jam.html) +for people to try making games with Hoot before the jam started, and a +lot of people took us up on it! Very cool! + +The next most popular language, which is typically *the* most popular +language in these jams, is [Fennel](https://fennel-lang.org/). Fennel +is a Lisp that compiles to Lua. It’s very cool! + +Also of note, at least to me as a Schemer, is that 3 games used +[S7](https://ccrma.stanford.edu/software/snd/snd/s7.html). Hmm, there +might be something relevant to this post going on there. + +The patterns I’m about to talk about could sort of be framed as “The +Guile Way vs. The Fennel Way”, but I don’t want to do that. It's not +an “us vs. them” thing. It’s wonderful that there are so many flavors +of Lisp these days that anyone can find a great implementation that +suits their preferences. Not only that, but many of these +implementations can be used to make games that anyone can easily play +in their web browser! That was *not* the case several years ago. +Incredible! + +I want to preface the rest of this post by saying that **both patterns +are valid**, and while I prefer one over the other, that is not to say +that the other is inferior. I'll also show how these patterns can be +thought of two ends of a spectrum and how, in the end, compromises +must be made. Okay, let’s get into it! + +## Lisp as icing + +The icing pattern is using Lisp as a “scripting” language on top of a +cake that is made from C, Rust, and other static languages. The +typical way to do this is by embedding a Lisp interpreter into the +larger program. If you’re most interested in writing the high-level +parts of an application in Lisp then this pattern is the fastest way +to get there. All you need is a suitable interpreter/compiler and a +way to add the necessary hooks into your application. Since the +program is mainly C/Rust/whatever, you can then use emscripten to +compile it to WebAssembly and deploy to the web. Instant +gratification, but strongly tied to static languages and their +toolchains. + +S7 is an example of an embeddable Scheme. Guile is also used for +extending C programs, though typically that involves dynamically +linking to libguile rather than embedding the interpreter into the +program’s executable. Fennel takes a different approach, recognizing +that there are many existing applications that are already extensible +through Lua, and provides a lispy language that *compiles to* Lua. + +## Lisp as cake + +The cake pattern is using Lisp to implement as much of the software +stack as possible. It’s Lisp all the way down... sorta. Rather than +embedding Lisp into a non-Lisp program, the cake pattern does the +inverse: the majority of the program is written in Lisp. When +necessary, shared libraries can be called via a foreign function +interface, but this should be kept to a minimum. This approach takes +longer to yield results. Time is spent implementing missing libraries +for your Lisp of choice and writing wrappers around the C shared +libraries you can’t avoid using. Web deployment gets trickier, too, +since the project is not so easily emscriptenable. + +(You may recognize this as the classic embed vs. extend debate. +You’re correct! I'm just adding my own thoughts and applying it +specifically to some real-world Lisp projects.) + +I mentioned Guile as an option for icing, but Guile really shines best +as cake. The initial vision for Guile was to Emacsify other programs +by adding a Scheme interpreter to them. These days, the best practice +is to write your program in Scheme to begin with. Common Lisp is +probably the best example, though. Implementations like SBCL have +good C FFIs and can compile efficient native executables, minimizing +the desire to use some C for performance reasons. + +## Case studies + +Let’s take a look at some of the languages and libraries used for the +Lisp Game Jam and evaluate their icing/cake-ness. + +### Fennel + love2d + +[love2d](https://love2d.org/) has been a popular choice for solo or +small team game development for many years. It is a C++ program that +embeds a Lua interpreter, which means its a perfect target for Fennel. +Most Linux distributions package love2d, so it’s easy to run `.love` +files natively. Additionally, thanks to emscripten, love2d games can +be deployed to the web. Thus, most Fennel games use love2d. +[./soko.bin](https://jleightcap.itch.io/sokobin) and [Gnomic +Vengeance](https://alexjgriffith.itch.io/gnomic-vengeance) are two +games that use this stack. + +Fennel + love2d is a perfect example of Lisp as icing. Fennel sits at +the very top of the stack, but there’s not really path to get Lisp +into the layers below. It is also the most successful Lisp games +software stack to date. + +### S7 + raylib + +This stack is new to me, but two games used it this time around: +[GhostHop](https://gcmas.itch.io/ghosthop) and [Life +Predictor](https://illusion-fisherman.itch.io/life-predictor). (You +really gotta play, GhostHop, btw. It’s a great little puzzle game and +it is playable on mobile devices.) [Raylib](https://www.raylib.com/) +is a C library with bindings for many higher-level languages that has +become quite popular in recent years. S7 is also implemented in C and +is easily embeddable. This makes the combination easy to deploy on +the web with emscripten. + +S7 + raylib is another example of Lisp as icing. I’m curious to see +if this stack becomes more popular in future jams. + +### Guile + Chickadee + +This is the stack that I helped build. +[Chickadee](/projects/chickadee.html) is a game library for Guile that +implements almost all of the interesting parts in Scheme, including +rendering. Two games were built with Chickadee in the most recent +jam: [Turbo Racer 3000](https://etenil.itch.io/turbo-racer-3000) and +[Bloatrunner](https://snamellit.itch.io/bloatrunner). + +Guile + Chickadee is an example of Lisp as cake. Chickadee wraps some +C libraries for low-level tasks such as loading images, audio, and +fonts, but it is written in pure Scheme. All the matrix and vector +math is in Scheme. Chickadee comes with a set of rendering primitives +comparable to love2d and raylib but they’re all implemented in Scheme. +I’ve even made progress on rendering vector graphics with Scheme, +something that most other Lisp game libraries call out to C to do. +Chickadee has pushed the limits of Guile’s compiler and virtual +machine, and Guile has been improved as a result. But it’s the long +road. Chickadee is mostly developed by just me in my very limited +spare time, so it is taking a long time to reach feature parity with +more popular game development libraries. + +### Hoot + HTML5 canvas + +I also helped build this one. Hoot is a Scheme-to-WebAssembly +compiler. Rather than compile the Guile VM (written in C) to Wasm +using emscripten, Hoot implements a complete Wasm toolchain and a new +backend for Guile’s compiler that emits Wasm directly. Hoot is +written entirely in Scheme. Unlike C programs compiled with +emscripten that target Wasm 1.0 with linear memory, Hoot targets Wasm +2.0 with GC managed heap types. This gives Hoot a significant +advantage: Hoot binaries **do not ship a garbage collector** and thus +are much smaller than Lisp runtimes compiled via emscripten. +Cirkoban’s Wasm binary weighs in at < 2MiB whereas `love.wasm` is +almost 6MiB when I checked. Hoot programs can also easily +interoperate with JavaScript. Scheme objects can easily be passed to +JavaScript, and vice versa, as they are managed in the same heap. +With all of the browser APIs just a Wasm import away, an obvious +choice for games was the built-in HTML5 canvas API for easy 2D +rendering. + +11 games used Hoot in the jam, including (shameless plug) +[Cirkoban](https://davexunit.itch.io/cirkoban) and [Lambda +Dungeon](https://fluxharmonic.itch.io/lambda-dungeon). + +Hoot + HTML5 canvas is mostly dense cake with a bit of icing. On one +hand, it took a year and significant funding to boot Hoot. We said +“no” to emscripten, built a completely separate toolchain, and +extended Guile’s compiler. It's Lisp all the way until you hit the +browser runtime! We even have a Wasm interpreter that runs on the +Guile VM! Hoot rules! It was a risk but it paid off. On the other +hand, the canvas API is very high level. The more cake thing to do +would be to use Hoot’s JS FFI to call WebGL and/or WebGPU. Indeed, +this is the plan for the future! Wasm GC needs some improvements to +make this feasible, but my personal goal is to get Chickadee ported to +Hoot. I want Chickadee games to be easy to play natively and in +browsers, just like love2d games. + +## The cake/icing spectrum + +I must acknowledge the limitations of the cake approach. We’re not +living in a world of Lisp machines, but a world of glorified PDP-11s. +Even the tallest of Lisp cakes sits atop an even larger cake made +mostly of C. All modern Lisp systems bottom out at some point. Emacs +rests on a C core. Guile’s VM is written in C. Hoot runs on mammoth +JavaScript engines written in C++ like V8. Games on Hoot currently +render with HTML5 canvas rather than the lower level WebGL/WebGPU. +Good luck using OpenGL without libGL; Chickadee uses guile-opengl +which uses the C FFI to call into libGL. Then there’s libpng, +FreeType, and more. Who the heck wants to rewrite all this in Lisp? +Who even has the resources? Does spending all this time taking the +scenic route matter at all, or are we just deluding ourselves because +we have fun writing Lisp code? + +I think it *does* matter. Every piece of the stack that can be +reclaimed from the likes of C is a small victory. The parts written +in Lisp are much easier to hack on, and some of those things become +**live hackable** while our programs are running. They are also +memory safe, typically, thanks to GC managed runtimes. Less FFI calls +means less overhead from traversing the Lisp/C boundary and more +safety. As more of the stack becomes Lisp, it starts looking less +like icing and more like cake. + +Moving beyond games, we can look to the [Guix](https://guix.gnu.org) +project as a great example of just how tasty the cake can get. Guix +took the functional packaging model from the Nix project and made a +fresh implementation, replacing the Nix language with Guile. Why? +For code staging, code sharing, and improved hackability. Guix also +uses an init system written in Guile rather than systemd. Why? For +code staging, code sharing, and improved hackability. These are real +advantages that make the trade-off of not using the industry-standard +thing worth it. + +I’ve been using Guix since the early days, and back then it was easy +to make the argument that Guix was just reinventing wheels for no +reason. But now, over 10 years later, the insistence on maximizing +the usage of Lisp has been key to the success of the project. As a +user, once you learn the Guix idioms and a bit of Guile, you unlock +extraordinary power to craft your OS to your liking. It’s the closest +thing you can get to a Lisp machine on modern hardware. The cake +approach paid off for Guix, and it could pay off for other projects, +too. + +If Common Lisp is more your thing, and even if it isn’t, you’ll be +amazed by the [Trial](https://shirakumo.github.io/trial/) game engine +and how much of it is implemented in Common Lisp rather than wrapping +C libraries. + +There’s also projects like [Pre-Scheme](https://prescheme.org/) that +give me hope that one day the layers below the managed GC runtime can +be implemented in Lisp. Pre-Scheme was developed and successfully +used for [Scheme 48](https://s48.org/) and I am looking forward to a +modern revival of it thanks to an [NLnet +grant](https://nlnet.nl/project/Pre-Scheme/). + +## I'm a cake boy + +That’s right, I said it: I’m a cake boy. I want to see projects +continue to push the boundaries of what Lisp can do. When it comes to +the Lisp Game Jam, what excites me most are not the games themselves, +but the small advances made to reclaim another little slice of the +cake from stale, dry C. I intend to keep pushing the limits for Guile +game development with my Chickadee project. + +It’s *not* a piece of cake to bake a lispy cake, and the way is often +hazy, but I know we can’t be lazy and just do the cooking by the book. +Rewrite it in Rust? No way! Rewrite it in Lisp! |