#+TITLE: Functional Package and Configuration Management with GNU Guix #+AUTHOR: David Thompson #+EMAIL: davet@gnu.org #+DATE: Wednesday, January 20th, 2016 #+DESCRIPTION: #+KEYWORDS: #+LANGUAGE: en #+OPTIONS: H:1 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t #+OPTIONS: TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc #+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js #+EXPORT_SELECT_TAGS: export #+EXPORT_EXCLUDE_TAGS: noexport #+LINK_UP: #+LINK_HOME: #+startup: beamer #+LaTeX_CLASS: beamer #+LaTeX_CLASS_OPTIONS: [bigger] #+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %4BEAMER_col(Col) %10BEAMER_extra(Extra) #+LATEX_HEADER: \beamertemplatenavigationsymbolsempty #+BEAMER_THEME: metropolis * About me GNU project volunteer GNU Guile user and contributor since 2012 GNU Guix contributor since 2013 Day job: Ruby + JavaScript web development / “DevOps” * Overview - Problems with application packaging and deployment - Intro to functional package and configuration management - Towards the future - How you can help * User autonomy and control It is becoming increasingly difficult to have control over your own computing: - GNU/Linux package managers not meeting user needs - Self-hosting web applications requires too much time and effort - Growing number of projects recommend installation via =curl | sudo bash= [fn:2] or otherwise avoid using system package managers - Users unable to verify that a given binary corresponds to the source code * User autonomy and control #+latex: \huge{ “Debian and other distributions are going to be that thing you run Docker on, little more.” [fn:7] #+latex: } * User autonomy and control This is very bad for desktop users and system administrators alike. We must regain control! * What’s wrong with Apt/Yum/Pacman/etc.? Global state (=/usr=) that prevents *multiple versions* of a package from coexisting *Non-atomic* installation, removal, upgrade of software No way to *roll back* *Nondeterminstic* package builds and maintainer-uploaded binaries (though this is changing!) Reliance on pre-built binaries provided by a *single point of trust* Requires *superuser* privileges * The problem is bigger Proliferation of *language-specific package managers* and *binary bundles* that complicate secure system maintenance. * Web applications Web applications are particularly painful. * Web applications It’s common for today’s web applications to require *two or more package managers* to get all dependencies. * Web applications Integrating a web application packaged only for a language-specific manager into a system package manager proves difficult. NodeJS is particularly frightening. [fn:6] * Web applications There’s a growing number of popular web applications (Hadoop, Chef Server, Cloudera, etc.) that *no one knows how to build from source*! [fn:3] * Deployment How do we automate application deployment without going crazy? * Chef/Puppet/Ansible/etc. are pretty good, right? Building on top of mainstream package managers and distros yields an unstable foundation. * Problems with configuration management software Imperative config management is overly-complex and brittle (idempotence is hard) More reliable builds require spawning new machines and building from scratch each time (sledgehammer) Made primarily for developers for server maintenance, but all users could benefit * Docker? Surely Docker addresses these issues? * Docker? \center{I’m afraid not.} \begin{center} \includegraphics[height=7cm]{containers.jpg} \end{center} * Problems with Docker Still imperative (though resulting images are immutable) Dockerfile DSL is not expressive Promotes one disk image per application to cover up underlying package management mess [fn:4] No provenance Image layering is an ineffective caching strategy Does not compose (what about the host?) * Problems with Docker Reliance on DockerHub binaries proves to be insecure [fn:5] \begin{center} \includegraphics[width=\textwidth]{docker-insecure.png} \end{center} * Well that was pessimistic Computers are hard. Let’s just look at cat pictures, instead. \begin{center} \includegraphics[width=8cm]{cons-cats.jpg} \end{center} * Meet GNU Guix \begin{center} \includegraphics[width=5cm]{guix-logo.png} \end{center} Guix is the functional package management tool for the GNU system. It is based on the pioneering work of the Nix project. [fn:8] * Meet GuixSD \begin{center} \includegraphics[width=4cm]{guixsd-logo.png} \end{center} GuixSD is the GNU/Linux distribution that uses Guix as its package manager. * What does “functional” mean? “Functional” in this context means treating package builds as functions, in the mathematical sense. =emacs = f(gcc,make,coreutils,…)= * Functional package management Benefits: - Build reproducibility - No single point of trust - Unprivileged package management - Atomic upgrades and roll backs - Multiple variants of the same software may coexist * Functional package management The *complete dependency graph* is captured, precisely, down to the *bootstrap binaries*. No SAT solver or other complex algorithm for dependency resolution. * Functional package management To view package builds this way, Guix performs builds in an *isolated container* in which *only the specified dependencies* are accessible. Build results are *immutable*. This maximizes *build reproducibility*. * Reproducible builds Reproducible builds produce *bit-identical binaries* when performed multiple times under the same conditions. Requires fixing issues in upstream build systems that are nondeterministic. * Why? “With reproducible builds, multiple parties can *redo this process independently* and ensure they *all get /exactly/ the same result*. We can thus *gain confidence* that a distributed binary code is indeed coming from a given source code.” [fn:9] * Use cases \begin{center} \includegraphics[width=\textwidth]{nsa-vw.png} \end{center} * Transparent Guix is a *source-based* package manager, but will *transparently* download pre-built binaries from a trusted party, if available. Otherwise, it will simply build from source. * Decentralized In Guix, there is *no central point of trust* for receiving pre-built binaries (substitutes). * Decentralized Guix provides http://hydra.gnu.org, but it is optional. Users may authorize zero or more substitute servers, or even publish their own substitutes for others to use via =guix publish=. * Challenge authority When builds are reproducible, users may *challenge* their substitute providers by building locally and comparing the results. * Unprivileged Users can build and install software *without root privileges*. * Unprivileged Each user may have one or more “profiles”, a union of many packages. Use cases: - Eva and Ben use different versions of Emacs - Eva hacks on 2 Ruby projects that require different versions * Atomic Package installation/removal and full-system updates are *atomic* operations, meaning that either the operation succeeds, or nothing happens. * Roll back /Any/ package transaction may be *rolled back*, likewise for full-system upgrades. If a full-system update goes wrong, just boot into the previous working generation! * Coexistence Each package has its own *unique* directory in the store that contains its build artifacts. You can have every version of Ruby, Python, and Perl under the sun and that’s OK! * Demo! =guix package= =guix challenge= * Hacking Guix is made to be maximally hackable, taking inspiration from Emacs. We seek to intentionally blur the line between user and developer. * Choice of language Guix is rather special in its choice of implementation language. * Philosophy It’s better to *extend an existing programming language* for package recipes and configuration files rather than making a new, domain-specific one. * Embedded vs. External DSLs Using an extensible programming language as a host has several advantages compared to external DSLs: - No new parser, interpreter/compiler, editor tools, etc. to maintain - Access to all available libraries of the host language - Extensions to the host language can be used as a library by others Not all general-purpose programming languages are suitable for embedding new languages, [fn:1] so which did we choose? * Guile Scheme \begin{center} \includegraphics[width=4cm]{guile-logo.png} \end{center} GNU Guile is a Scheme implementation and the official extension language of the GNU project. It’s a great choice for EDSLs because of Scheme’s *hygienic macro system*. It’s a great choice for Guix because *purely functional* programming is well-supported in Scheme. * Guile goes with everything Guix uses Guile for nearly everything: - Initial RAM disk - Init system (GNU Shepherd, formerly GNU dmd) - Package recipes (including build scripts!) - Command line tools - Low-level POSIX/Linux utilities (such as =call-with-container=) * Guix as a library Guix is a big collection of Guile modules. Packages are first-class Scheme objects. Anyone can use Guix as a library to write new Guile programs that manipulate package recipes, create new user interfaces (like a web UI), etc. * Example package recipe #+latex: \tiny{ #+BEGIN_SRC scheme (define-public livestreamer (package (name "livestreamer") (version "1.12.2") (source (origin (method url-fetch) (uri (string-append "https://github.com/chrippa/livestreamer/archive/v" version ".tar.gz")) (file-name (string-append "livestreamer-" version ".tar.gz")) (sha256 (base32 "1fp3d3z2grb1ls97smjkraazpxnvajda2d1g1378s6gzmda2jvjd")))) (build-system python-build-system) (arguments '(#:tests? #f)) ; tests rely on external web servers (native-inputs `(("python-setuptools" ,python-setuptools))) (propagated-inputs `(("python-requests" ,python-requests) ("python-singledispatch" ,python-singledispatch))) (synopsis "Internet video stream viewer") (description "Livestreamer is a command-line utility that extracts streams from various services and pipes them into a video playing application.") (home-page "http://livestreamer.io/") (license license:bsd-2))) #+END_SRC #+latex: } * Dependency graph \includegraphics[width=\textwidth]{livestreamer-graph.png} * Demo! Emacs + Geiser * Other user interfaces Demo Emacs UI, web prototype * Importing packages The =guix import= tool that can *automatically generate code snippets* for packages found in foreign systems. Supported systems include: PyPI, RubyGems, CPAN, Hackage, ELPA, and CRAN. * Auto-updating The =guix refresh= tool can automatically find the latest release of certain software. For example, Python packages can be updated by querying PyPI for information on the latest release. * Demo! =guix import= * Reproducible development environments Getting the dependencies needed to create development environments can be tough. Many languages invent their own solution, but this is a general problem. * Reproducible development environments Guix has a tool for this: =guix environment= Think of it like a language-agnostic version of Python’s =virtualenv=. * Reproducible development environments Environments can be *purified* via standard environment variables or, for better isolation, Linux containers. This allows developers to have confidence that potential contributors will be able to build their software. * Demo! =guix environment= * Full-system configuration The Guix System Distribution supports a *consistent whole-system configuration mechanism*. All aspects of a system configuration are *declared* in a single place. * Advantages Easy to replicate configuration on different machines *without resorting to additional tools* layered on top. System upgrades are atomic and can be rolled back. * Example system configuration #+latex: \tiny{ #+BEGIN_SRC scheme (operating-system (host-name "izanagi") (timezone "America/New_York") (locale "en_US.UTF-8") (bootloader (grub-configuration (device "/dev/sda"))) (file-systems (cons (file-system (device "root") (title 'label) (mount-point "/") (type "ext4")) %base-file-systems)) (users (list (user-account (name "dave") (comment "David Thompson") (group "users") (supplementary-groups '("wheel" "netdev" "audio" "video" "cdrom")) (home-directory "/home/dave")))) (packages (cons* adwaita-icon-theme avahi dbus gnome-terminal htop less man-db nss-certs openssh pulseaudio wicd unzip rsync xfce %base-packages)) (services %desktop-services) (name-service-switch %mdns-host-lookup-nss)) #+END_SRC #+latex: } * Service graph \includegraphics[width=\textwidth]{service-graph.png} * Demo! =guix system vm= * Project status - Full-featured package manager - 3,000 packages, 4 platforms - Guix System Distribution in beta - Binaries at http://hydra.gnu.org - tooling: auto-update, “linting”, etc. * Project status \includegraphics[width=\textwidth]{stats.png} \center\url{https://www.openhub.net/p/gnuguix} * Project status \begin{center} \includegraphics[width=\textwidth]{contributor-graph.png} \end{center} * The people have spoken \begin{center} \includegraphics[width=4cm]{awesome-rating.png} \end{center} * Project status \approx200–500 new packages per release. *More needed!* * Future I intend to focus on: - A cluster deployment tool: =guix deploy= - Improved support for GuixSD containers * Future More generally: - Stronger build farm - More packages that are reproducible - GNOME - LVM - Encrypted root * Join us! - Use Guix on top of your existing distro - Use the distribution - Add new packages or upgrade existing ones - Write system services - Add new translations - Tell us your ideas! * Join us! We are currently collecting donations via the FSF to purchase new servers for our build farm! https://gnu.org/software/guix/donate/ * Join us! Chat with us in the =#guix= channel on Freenode or on the =guix-devel@gnu.org= and =help-guix@gnu.org= mailing lists. * Thank you! Visit https://gnu.org/software/guix for source code, documentation, past talks, etc. \begin{center} \huge{Questions?} \end{center} * Legal © 2016 David Thompson == This presentation is licensed under the Creative Common Attribute Share-Alike 4.0 International license. GNU Guix and GuixSD logo, GFDL, http://gnu.org/s/guix/graphics Copyright of other images included in this document is held by their respective owners. * Footnotes [fn:1] “How to be a good host: miniKanren as a case study” https://www.youtube.com/watch?v=b9C3r3dQnNY [fn:2] http://curlpipesh.tumblr.com/ [fn:3] “Your big data toolchain is a big security risk!” http://www.vitavonni.de/blog/201504/2015042601-big-data-toolchains-are-a-security-risk.html [fn:4] “The sad state of sysadmin in the age of containers” http://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html [fn:5] http://www.banyanops.com/blog/analyzing-docker-hub/ [fn:6] “Let’s Package jQuery: A Javascript Packaging Dystopian Novella” http://dustycloud.org/blog/javascript-packaging-dystopia/ [fn:7] “ownCloud and distribution packaging” http://lwn.net/Articles/670566/ [fn:8] http://nixos.org/nix/ [fn:9] https://reproducible-builds.org/