#+TITLE: Functional Package and Configuration Management with GNU Guix #+AUTHOR: David Thompson #+EMAIL: davet@gnu.org #+DATE: Wednesday, January 20th, 2016 #+DESCRIPTION: #+KEYWORDS: #+LANGUAGE: en #+OPTIONS: H:1 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t #+OPTIONS: TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc #+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js #+EXPORT_SELECT_TAGS: export #+EXPORT_EXCLUDE_TAGS: noexport #+LINK_UP: #+LINK_HOME: #+startup: beamer #+LaTeX_CLASS: beamer #+LaTeX_CLASS_OPTIONS: [bigger] #+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %4BEAMER_col(Col) %10BEAMER_extra(Extra) #+LATEX_HEADER: \beamertemplatenavigationsymbolsempty #+BEAMER_THEME: metropolis * About me GNU project volunteer GNU Guile user and contributor since 2012 GNU Guix contributor since 2013 Day job: Ruby + JavaScript web development / “DevOps” * Overview - Problems with application packaging and deployment - Intro to functional package and configuration management - Towards the future - How you can help * User autonomy and control It is becoming increasingly difficult to have control over your own computing: - GNU/Linux package managers not meeting user needs - Self-hosting web applications requires too much time and effort - Growing number of projects recommend installation via =curl | sudo bash= [fn:2] or otherwise avoid using system package managers - Users unable to verify that a given binary corresponds to the source code * User autonomy and control #+latex: \huge{ “Debian and other distributions are going to be that thing you run Docker on, little more.” [fn:7] #+latex: } * User autonomy and control This is very bad for desktop users and system administrators alike. We must regain control! * What’s wrong with Apt/Yum/Pacman/etc.? Global state (=/usr=) that prevents *multiple versions* of a package from coexisting. *Non-atomic* installation, removal, upgrade of software. No way to *roll back*. *Nondeterminstic* package builds and maintainer-uploaded binaries. (though this is changing!) Reliance on pre-built binaries provided by a *single point of trust*. Requires *superuser* privileges. * The problem is bigger Proliferation of *language-specific package managers* and *binary bundles* that complicate secure system maintenance. * Web applications Web applications are particularly painful. * Web applications It’s common for today’s web applications to require *two or more package managers* to get all dependencies. * Web applications Importing a web application available only for a language-specific manager into a distribution proves difficult. NodeJS is particularly frightening. [fn:6] * Web applications There’s a growing number of popular web applications (Hadoop, Chef Server, Cloudera, etc.) that *no one knows how to build from source*! [fn:3] * Deployment How do we automate application deployment without going crazy? * Chef/Puppet/Ansible/etc. are pretty good, right? Building on top of mainstream package managers and distros yields an unstable foundation. * Problems with configuration management software - Imperative config management is overly-complex and brittle (idempotence is hard) - More reliable builds require spawning new machines and building from scratch each time. (sledgehammer) - Made primarily for developers for server maintenance, but all types of users could benefit. * Docker? Surely Docker addresses these issues? * Docker? \center{I’m afraid not.} \begin{center} \includegraphics[height=7cm]{containers.jpg} \end{center} * Problems with Docker - Still imperative (though resulting images are immutable) - Dockerfile DSL is not expressive - Promotes one disk image per application to cover up underlying package management mess [fn:4] - No provenance - Image layering is an ineffective caching strategy - Does not compose (what about the host?) * Problems with Docker Reliance on DockerHub binaries proves to be insecure [fn:5] \begin{center} \includegraphics[width=\textwidth]{docker-insecure.png} \end{center} * Well that was pessimistic Computers are hard. Let’s just look at cat pictures, instead. \begin{center} \includegraphics[width=8cm]{cons-cats.jpg} \end{center} * Meet GNU Guix \begin{center} \includegraphics[width=5cm]{guix-logo.png} \end{center} Guix is the functional package management tool for the GNU system. It is based on the pioneering work of the Nix project. [fn:8] * Meet GuixSD \begin{center} \includegraphics[width=4cm]{guixsd-logo.png} \end{center} GuixSD is the GNU/Linux distribution that uses Guix as its package manager. * What does “functional” mean? “Functional” in this context means treating package builds as functions, in the mathematical sense. =emacs = f(gcc,make,coreutils,…)= * Functional package management Benefits: - Build reproducibility - No single point of trust - Unprivileged package management - Atomic upgrades and roll backs - Multiple variants of the same software may coexist * Functional package management The *complete dependency graph* is captured, precisely, down to the *bootstrap binaries*. No SAT solver or other complex algorithm for dependency resolution. * Functional package management To view package builds this way, Guix performs builds in an *isolated container* in which *only the specified dependencies* are accessible. Build results are *immutable*. This maximizes *build reproducibility*. * Reproducible builds Reproducible builds produce *bit-identical binaries* when performed multiple times under the same conditions. Requires fixing issues in upstream build systems that are nondeterministic. * Why? “With reproducible builds, multiple parties can *redo this process independently* and ensure they *all get /exactly/ the same result*. We can thus *gain confidence* that a distributed binary code is indeed coming from a given source code.” [fn:9] * Use cases \begin{center} \includegraphics[width=\textwidth]{nsa-vw.png} \end{center} * Transparent Guix is a *source-based* package manager, but will *transparently* download pre-built binaries from a trusted party, if available. Otherwise, it will simply build from source. * Decentralized In Guix, there is *no central point of trust* for receiving pre-built binaries (substitutes). * Decentralized Guix provides http://hydra.gnu.org, but it is optional. Users may authorize zero or more substitute servers, or even publish their own substitutes for others to use via =guix publish=. * Challenge authority When builds are reproducible, users may *challenge* their substitute providers by building locally and comparing the results. * Unprivileged Users can build and install software *without root privileges*. * Unprivileged Each user may have one or more “profiles”, a union of many packages. Use cases: - Alyssa and Ben use different versions of Emacs - Alyssa hacks on 2 Ruby projects that require different versions * Atomic Package installation/removal and full-system updates are *atomic* operations, meaning that either the operation succeeds, or nothing happens. * Roll back /Any/ package transaction may be *rolled back*, likewise for full-system upgrades. If a full-system update goes wrong, just boot into the previous working generation! * Coexistence Each package has its own *unique* directory in the store that contains its build artifacts. You can have every version of Ruby, Python, and Perl under the sun and that’s OK! * Demo! =guix package= =guix challenge= * Hacking Guix is made to be maximally hackable, taking inspiration from Emacs. We seek to intentionally blur the line between user and developer. * Choice of language Guix is rather special in its choice of implementation language. * Philosophy It’s better to *extend an existing programming language* for package recipes and configuration files rather than making a new, domain-specific one. * Embedded vs. External DSLs Using an extensible programming language as a host has several advantages compared to external DSLs: - No new parser, interpreter/compiler, editor tools, etc. to maintain - Access to all available libraries of the host language - Extensions to the host language can be used as a library by others Not all general-purpose programming languages are suitable for embedding new languages, [fn:1] so which did we choose? * Guile Scheme \begin{center} \includegraphics[width=4cm]{guile-logo.png} \end{center} GNU Guile is a Scheme implementation and the official extension language of the GNU project. It’s a great choice for EDSLs because of Scheme’s *hygienic macro system*. It’s a great choice for Guix because *purely functional* programming is well-supported in Scheme. * Guile goes with everything Guix uses Guile for nearly everything: - Initial RAM disk - Init system (GNU Shepherd, formerly GNU dmd) - Package recipes (including build scripts!) - Command line tools - Low-level POSIX/Linux utilities (such as =call-with-container=) * Guix as a library Guix is a big collection of Guile modules. Packages are first-class Scheme objects. Anyone can use Guix as a library to write new Guile programs that manipulate package recipes, create new user interfaces (like a web UI), etc. * Example package recipe #+latex: \tiny{ #+BEGIN_SRC scheme (define-public livestreamer (package (name "livestreamer") (version "1.12.2") (source (origin (method url-fetch) (uri (string-append "https://github.com/chrippa/livestreamer/archive/v" version ".tar.gz")) (file-name (string-append "livestreamer-" version ".tar.gz")) (sha256 (base32 "1fp3d3z2grb1ls97smjkraazpxnvajda2d1g1378s6gzmda2jvjd")))) (build-system python-build-system) (arguments '(#:tests? #f)) ; tests rely on external web servers (native-inputs `(("python-setuptools" ,python-setuptools))) (propagated-inputs `(("python-requests" ,python-requests) ("python-singledispatch" ,python-singledispatch))) (synopsis "Internet video stream viewer") (description "Livestreamer is a command-line utility that extracts streams from various services and pipes them into a video playing application.") (home-page "http://livestreamer.io/") (license license:bsd-2))) #+END_SRC #+latex: } * Dependency graph \includegraphics[width=\textwidth]{livestreamer-graph.png} * Demo! Emacs + Geiser * Other user interfaces Besides the CLI, there’s also an Emacs interface, naturally. Proof of concept web interface. (not in Guix core) * Demo! Emacs * Importing packages The =guix import= tool that can *automatically generate code snippets* for packages found in foreign systems. Supported systems include: PyPI, RubyGems, CPAN, Hackage, ELPA, and CRAN. * Auto-updating The =guix refresh= tool can automatically find the latest release of certain software. For example, Python packages can be updated by querying PyPI for information on the latest release. * Demo! =guix import= * Reproducible development environments Getting the dependencies needed to create development environments can be tough. Many languages invent their own solution, but this is a general problem. * Reproducible development environments Guix has a tool for this: =guix environment= Think of it like a language-agnostic version of Python’s =virtualenv=. * Reproducible development environments Environments can be *purified* via standard environment variables or, for better isolation, Linux containers. This allows developers to have confidence that potential contributors will be able to build their software. * Demo! =guix environment= * Full-system configuration The Guix System Distribution supports a *consistent whole-system configuration mechanism*. All aspects of a system configuration are *declared* in a single place. * Advantages Easy to replicate configuration on different machines *without resorting to additional tools* layered on top. System upgrades are atomic and can be rolled back. * Example system configuration #+latex: \tiny{ #+BEGIN_SRC scheme (operating-system (host-name "izanagi") (timezone "America/New_York") (locale "en_US.UTF-8") (bootloader (grub-configuration (device "/dev/sda"))) (file-systems (cons (file-system (device "root") (title 'label) (mount-point "/") (type "ext4")) %base-file-systems)) (users (list (user-account (name "dave") (comment "David Thompson") (group "users") (supplementary-groups '("wheel" "netdev" "audio" "video" "cdrom")) (home-directory "/home/dave")))) (packages (cons* adwaita-icon-theme avahi dbus gnome-terminal htop less man-db nss-certs openssh pulseaudio wicd unzip rsync xfce %base-packages)) (services %desktop-services) (name-service-switch %mdns-host-lookup-nss)) #+END_SRC #+latex: } * Service graph \includegraphics[width=\textwidth]{service-graph.png} * Demo! =guix system vm= * Project status - Full-featured package manager - 3,000 packages, 4 platforms - Guix System Distribution in beta - Binaries at http://hydra.gnu.org - Variety of useful tools * Project status \includegraphics[width=\textwidth]{stats.png} \center\url{https://www.openhub.net/p/gnuguix} * Project status \begin{center} \includegraphics[width=\textwidth]{contributor-graph.png} \end{center} * The people have spoken \begin{center} \includegraphics[width=4cm]{awesome-rating.png} \end{center} * Project status \approx200–500 new packages per release. *More needed!* * Future I intend to focus on: - A cluster deployment tool: =guix deploy= - Improved support for GuixSD containers * Future More generally: - Stronger build farm - More packages that are reproducible - GNOME - LVM - Encrypted root for everyone * Join us! - Use Guix on top of your existing distro - Use the distribution - Add new packages or upgrade existing ones - Write system services - Add new translations - Tell us your ideas! * Join us! We are currently collecting donations via the FSF to purchase new servers for our build farm! Since mid-Decemeber, $8,200 USD has been raised. https://gnu.org/software/guix/donate/ * Join us! Chat with us in the =#guix= channel on Freenode or on the =guix-devel@gnu.org= and =help-guix@gnu.org= mailing lists. * LibrePlanet 2016 Christopher Webber of the GNU MediaGoblin project and myself will be co-presenting “Solving the Deployment Crisis with GNU Guix” at LibrePlanet 2016 on March 19th or 20th. Visit https://libreplanet.org/2016 for full details. * Thank you! Visit https://gnu.org/software/guix for source code, documentation, past talks, etc. \begin{center} \huge{Questions?} \end{center} * Legal © 2016 David Thompson == This presentation is licensed under the Creative Common Attribute Share-Alike 4.0 International license. GNU Guix and GuixSD logo, GFDL, http://gnu.org/s/guix/graphics Copyright of other images included in this document is held by their respective owners. * Footnotes [fn:1] “How to be a good host: miniKanren as a case study” https://www.youtube.com/watch?v=b9C3r3dQnNY [fn:2] http://curlpipesh.tumblr.com/ [fn:3] “Your big data toolchain is a big security risk!” http://www.vitavonni.de/blog/201504/2015042601-big-data-toolchains-are-a-security-risk.html [fn:4] “The sad state of sysadmin in the age of containers” http://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html [fn:5] http://www.banyanops.com/blog/analyzing-docker-hub/ [fn:6] “Let’s Package jQuery: A Javascript Packaging Dystopian Novella” http://dustycloud.org/blog/javascript-packaging-dystopia/ [fn:7] “ownCloud and distribution packaging” http://lwn.net/Articles/670566/ [fn:8] http://nixos.org/nix/ [fn:9] https://reproducible-builds.org/