path: root/guix-blu-2016-01-20.org

#+TITLE: Functional Package and Configuration Management with GNU Guix
#+AUTHOR: David Thompson
#+EMAIL: davet@gnu.org
#+DATE: Wednesday, January 20th, 2016
#+DESCRIPTION:
#+KEYWORDS:
#+LANGUAGE:  en
#+OPTIONS:   H:1 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
#+OPTIONS:   TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
#+EXPORT_SELECT_TAGS: export
#+EXPORT_EXCLUDE_TAGS: noexport
#+LINK_UP:
#+LINK_HOME:
#+startup: beamer
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [bigger]
#+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %4BEAMER_col(Col) %10BEAMER_extra(Extra)
#+LATEX_HEADER: \beamertemplatenavigationsymbolsempty
#+BEAMER_THEME: metropolis

* About me

  GNU project volunteer

  GNU Guile user and contributor since 2012

  GNU Guix contributor since 2013

  Day job: Ruby + JavaScript web development / “DevOps”

* Overview

  - Problems with application packaging and deployment
  - Intro to functional package and configuration management
  - Towards the future
  - How you can help

* User autonomy and control

  It is becoming increasingly difficult to have control over your own
  computing:

  - GNU/Linux package managers not meeting user needs
  - Self-hosting web applications requires too much time and effort
  - Growing number of projects recommend installation via =curl | sudo
    bash= [fn:2] or otherwise avoid using system package managers
  - Users unable to verify that a given binary corresponds to the
    source code

* User autonomy and control

  #+latex: \huge{
  “Debian and other distributions are going to be that thing you run
  Docker on, little more.” [fn:7]
  #+latex: }

* User autonomy and control

  This is very bad for desktop users and system administrators alike.
  We must regain control!

* What’s wrong with Apt/Yum/Pacman/etc.?

  Global state (=/usr=) that prevents *multiple versions* of a package
  from coexisting.

  *Non-atomic* installation, removal, upgrade of software.

  No way to *roll back*.

  *Nondeterminstic* package builds and maintainer-uploaded binaries.
  (though this is changing!)

  Reliance on pre-built binaries provided by a *single point of trust*.

  Requires *superuser* privileges.

* The problem is bigger

  Proliferation of *language-specific package managers* and *binary
  bundles* that complicate secure system maintenance.

* Web applications

  Web applications are particularly painful.

* Web applications

  It’s common for today’s web applications to require *two or more
  package managers* to get all dependencies.

* Web applications

  Importing a web application available only for a language-specific
  manager into a distribution proves difficult.  NodeJS is
  particularly frightening. [fn:6]

* Web applications

  There’s a growing number of popular web applications (Hadoop, Chef
  Server, Cloudera, etc.) that *no one knows how to build from
  source*! [fn:3]

* Deployment

  How do we automate application deployment without going crazy?

* Chef/Puppet/Ansible/etc. are pretty good, right?

  Building on top of mainstream package managers and distros yields an
  unstable foundation.

* Problems with configuration management software

  - Imperative config management is overly-complex and brittle
    (idempotence is hard)

  - More reliable builds require spawning new machines and building
    from scratch each time.  (sledgehammer)

  - Made primarily for developers for server maintenance, but all
    types of users could benefit.

* Docker?

  Surely Docker addresses these issues?

* Docker?

  \center{I’m afraid not.}

  \begin{center}
  \includegraphics[height=7cm]{containers.jpg}
  \end{center}

* Problems with Docker

  - Still imperative (though resulting images are immutable)

  - Dockerfile DSL is not expressive

  - Promotes one disk image per application to cover up underlying
    package management mess [fn:4]

  - No provenance

  - Image layering is an ineffective caching strategy

  - Does not compose (what about the host?)

* Problems with Docker

  Reliance on DockerHub binaries proves to be insecure [fn:5]

  \begin{center}
  \includegraphics[width=\textwidth]{docker-insecure.png}
  \end{center}

* Well that was pessimistic

  Computers are hard.  Let’s just look at cat pictures, instead.

  \begin{center}
  \includegraphics[width=8cm]{cons-cats.jpg}
  \end{center}

* Meet GNU Guix

  \begin{center}
  \includegraphics[width=5cm]{guix-logo.png}
  \end{center}

  Guix is the functional package management tool for the GNU system.

  It is based on the pioneering work of the Nix project. [fn:8]

* Meet GuixSD

  \begin{center}
  \includegraphics[width=4cm]{guixsd-logo.png}
  \end{center}

  GuixSD is the GNU/Linux distribution that uses Guix as its package
  manager.

* What does “functional” mean?

   “Functional” in this context means treating package builds as
   functions, in the mathematical sense.

   =emacs = f(gcc,make,coreutils,…)=

* Functional package management

  Benefits:

  - Build reproducibility

  - No single point of trust

  - Unprivileged package management

  - Atomic upgrades and roll backs

  - Multiple variants of the same software may coexist

* Functional package management

  The *complete dependency graph* is captured, precisely, down to the
  *bootstrap binaries*.

  No SAT solver or other complex algorithm for dependency resolution.

* Functional package management

  To view package builds this way, Guix performs builds in an
  *isolated container* in which *only the specified dependencies* are
  accessible.

  Build results are *immutable*.

  This maximizes *build reproducibility*.

* Reproducible builds

  Reproducible builds produce *bit-identical binaries* when performed
  multiple times under the same conditions.

  Requires fixing issues in upstream build systems that are
  nondeterministic.

* Why?

  “With reproducible builds, multiple parties can *redo this process
  independently* and ensure they *all get /exactly/ the same result*.
  We can thus *gain confidence* that a distributed binary code is
  indeed coming from a given source code.” [fn:9]

* Use cases

  \begin{center}
  \includegraphics[width=\textwidth]{nsa-vw.png}
  \end{center}

* Transparent

  Guix is a *source-based* package manager, but will *transparently*
  download pre-built binaries from a trusted party, if available.

  Otherwise, it will simply build from source.

* Decentralized

  In Guix, there is *no central point of trust* for receiving
  pre-built binaries (substitutes).

* Decentralized

  Guix provides http://hydra.gnu.org, but it is optional.

  Users may authorize zero or more substitute servers, or even publish
  their own substitutes for others to use via =guix publish=.

* Challenge authority

  When builds are reproducible, users may *challenge* their substitute
  providers by building locally and comparing the results.

* Unprivileged

  Users can build and install software *without root privileges*.

* Unprivileged

  Each user may have one or more “profiles”, a union of many packages.

  Use cases:

  - Alyssa and Ben use different versions of Emacs
  - Alyssa hacks on 2 Ruby projects that require different versions

* Atomic

  Package installation/removal and full-system updates are *atomic*
  operations, meaning that either the operation succeeds, or nothing
  happens.

* Roll back

  /Any/ package transaction may be *rolled back*, likewise for
  full-system upgrades.

  If a full-system update goes wrong, just boot into the previous
  working generation!

* Coexistence

  Each package has its own *unique* directory in the store that
  contains its build artifacts.

  You can have every version of Ruby, Python, and Perl under the sun
  and that’s OK!

* Demo!

  =guix package=

  =guix challenge=

* Hacking

  Guix is made to be maximally hackable, taking inspiration from
  Emacs.

  We seek to intentionally blur the line between user and developer.

* Choice of language

  Guix is rather special in its choice of implementation language.

* Philosophy

  It’s better to *extend an existing programming language* for package
  recipes and configuration files rather than making a new,
  domain-specific one.

* Embedded vs. External DSLs

   Using an extensible programming language as a host has several
   advantages compared to external DSLs:

   - No new parser, interpreter/compiler, editor tools, etc. to
     maintain

   - Access to all available libraries of the host language

   - Extensions to the host language can be used as a library by
     others

   Not all general-purpose programming languages are suitable for
   embedding new languages, [fn:1] so which did we choose?

* Guile Scheme

  \begin{center}
  \includegraphics[width=4cm]{guile-logo.png}
  \end{center}

   GNU Guile is a Scheme implementation and the official extension
   language of the GNU project.

   It’s a great choice for EDSLs because of Scheme’s *hygienic macro
   system*.

   It’s a great choice for Guix because *purely functional*
   programming is well-supported in Scheme.

* Guile goes with everything

   Guix uses Guile for nearly everything:

   - Initial RAM disk

   - Init system (GNU Shepherd, formerly GNU dmd)

   - Package recipes (including build scripts!)

   - Command line tools

   - Low-level POSIX/Linux utilities (such as =call-with-container=)

* Guix as a library

  Guix is a big collection of Guile modules.

  Packages are first-class Scheme objects.

  Anyone can use Guix as a library to write new Guile programs that
  manipulate package recipes, create new user interfaces (like a web
  UI), etc.

* Example package recipe

  #+latex: \tiny{
  #+BEGIN_SRC scheme
    (define-public livestreamer
      (package
        (name "livestreamer")
        (version "1.12.2")
        (source (origin
                  (method url-fetch)
                  (uri (string-append
                        "https://github.com/chrippa/livestreamer/archive/v"
                        version ".tar.gz"))
                  (file-name (string-append "livestreamer-" version ".tar.gz"))
                  (sha256
                   (base32
                    "1fp3d3z2grb1ls97smjkraazpxnvajda2d1g1378s6gzmda2jvjd"))))
        (build-system python-build-system)
        (arguments
         '(#:tests? #f)) ; tests rely on external web servers
        (native-inputs
         `(("python-setuptools" ,python-setuptools)))
        (propagated-inputs
         `(("python-requests" ,python-requests)
           ("python-singledispatch" ,python-singledispatch)))
        (synopsis "Internet video stream viewer")
        (description "Livestreamer is a command-line utility that extracts streams
    from various services and pipes them into a video playing application.")
        (home-page "http://livestreamer.io/")
        (license license:bsd-2)))
  #+END_SRC
  #+latex: }

* Dependency graph

  \includegraphics[width=\textwidth]{livestreamer-graph.png}

* Demo!

  Emacs + Geiser

* Other user interfaces

  Besides the CLI, there’s also an Emacs interface, naturally.

  Proof of concept web interface. (not in Guix core)

* Demo!

  Emacs

* Importing packages

  The =guix import= tool that can *automatically generate code
  snippets* for packages found in foreign systems.

  Supported systems include: PyPI, RubyGems, CPAN, Hackage, ELPA, and
  CRAN.

* Auto-updating

  The =guix refresh= tool can automatically find the latest release of
  certain software.

  For example, Python packages can be updated by querying PyPI for
  information on the latest release.

* Demo!

  =guix import=

* Reproducible development environments

  Getting the dependencies needed to create development environments
  can be tough.

  Many languages invent their own solution, but this is a general
  problem.

* Reproducible development environments

  Guix has a tool for this: =guix environment=

  Think of it like a language-agnostic version of Python’s
  =virtualenv=.

* Reproducible development environments

  Environments can be *purified* via standard environment variables
  or, for better isolation, Linux containers.

  This allows developers to have confidence that potential
  contributors will be able to build their software.

* Demo!

  =guix environment=

* Full-system configuration

  The Guix System Distribution supports a *consistent whole-system
  configuration mechanism*.

  All aspects of a system configuration are *declared* in a single
  place.

* Advantages

  Easy to replicate configuration on different machines *without
  resorting to additional tools* layered on top.

  System upgrades are atomic and can be rolled back.

* Example system configuration

  #+latex: \tiny{
  #+BEGIN_SRC scheme
    (operating-system
      (host-name "izanagi")
      (timezone "America/New_York")
      (locale "en_US.UTF-8")
      (bootloader (grub-configuration (device "/dev/sda")))
      (file-systems (cons (file-system
                            (device "root")
                            (title 'label)
                            (mount-point "/")
                            (type "ext4"))
                          %base-file-systems))
      (users (list (user-account
                    (name "dave")
                    (comment "David Thompson")
                    (group "users")
                    (supplementary-groups '("wheel" "netdev" "audio"
                                            "video" "cdrom"))
                    (home-directory "/home/dave"))))
      (packages (cons* adwaita-icon-theme avahi dbus gnome-terminal
                       htop less man-db nss-certs openssh pulseaudio
                       wicd unzip rsync xfce
                       %base-packages))
      (services %desktop-services)
      (name-service-switch %mdns-host-lookup-nss))
  #+END_SRC
  #+latex: }

* Service graph

  \includegraphics[width=\textwidth]{service-graph.png}

* Demo!

  =guix system vm=

* Project status

  - Full-featured package manager
  - 3,000 packages, 4 platforms
  - Guix System Distribution in beta
  - Binaries at http://hydra.gnu.org
  - Variety of useful tools

* Project status

  \includegraphics[width=\textwidth]{stats.png}

  \center\url{https://www.openhub.net/p/gnuguix}

* Project status

  \begin{center}
  \includegraphics[width=\textwidth]{contributor-graph.png}
  \end{center}

* The people have spoken

  \begin{center}
  \includegraphics[width=4cm]{awesome-rating.png}
  \end{center}

* Project status

  \approx200–500 new packages per release.  *More needed!*

* Future

  I intend to focus on:

  - A cluster deployment tool: =guix deploy=
  - Improved support for GuixSD containers

* Future

  More generally:

  - Stronger build farm
  - More packages that are reproducible
  - GNOME
  - LVM
  - Encrypted root for everyone

* Join us!

  - Use Guix on top of your existing distro
  - Use the distribution
  - Add new packages or upgrade existing ones
  - Write system services
  - Add new translations
  - Tell us your ideas!

* Join us!

  We are currently collecting donations via the FSF to purchase new
  servers for our build farm!

  Since mid-Decemeber, $8,200 USD has been raised.

  https://gnu.org/software/guix/donate/

* Join us!

  Chat with us in the =#guix= channel on Freenode or on the
  =guix-devel@gnu.org= and =help-guix@gnu.org= mailing lists.

* LibrePlanet 2016

  Christopher Webber of the GNU MediaGoblin project and myself will be
  co-presenting “Solving the Deployment Crisis with GNU Guix” at
  LibrePlanet 2016 on March 19th or 20th.

  Visit https://libreplanet.org/2016 for full details.

* Thank you!

  Visit https://gnu.org/software/guix for source code, documentation,
  past talks, etc.

  \begin{center}
    \huge{Questions?}
  \end{center}

* Legal

  © 2016 David Thompson =<davet@gnu.org>=

  This presentation is licensed under the Creative Common Attribute
  Share-Alike 4.0 International license.

  GNU Guix and GuixSD logo, GFDL, http://gnu.org/s/guix/graphics

  Copyright of other images included in this document is held by their
  respective owners.

* Footnotes

[fn:1] “How to be a good host: miniKanren as a case study”
https://www.youtube.com/watch?v=b9C3r3dQnNY

[fn:2] http://curlpipesh.tumblr.com/

[fn:3] “Your big data toolchain is a big security risk!”
http://www.vitavonni.de/blog/201504/2015042601-big-data-toolchains-are-a-security-risk.html

[fn:4] “The sad state of sysadmin in the age of containers”
http://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html

[fn:5] http://www.banyanops.com/blog/analyzing-docker-hub/

[fn:6] “Let’s Package jQuery: A Javascript Packaging Dystopian Novella” http://dustycloud.org/blog/javascript-packaging-dystopia/

[fn:7] “ownCloud and distribution packaging” http://lwn.net/Articles/670566/

[fn:8] http://nixos.org/nix/

[fn:9] https://reproducible-builds.org/