summaryrefslogtreecommitdiff
path: root/2016-01-20-guix-blu/guix-blu-2016-01-20.org
blob: 8f4547eb01ef757d71d7bf5c2e02ff3900a756be (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
#+TITLE: Functional Package and Configuration Management with GNU Guix
#+AUTHOR: David Thompson
#+EMAIL: davet@gnu.org
#+DATE: Wednesday, January 20th, 2016
#+DESCRIPTION:
#+KEYWORDS:
#+LANGUAGE:  en
#+OPTIONS:   H:1 num:t toc:nil \n:nil @:t ::t |:t ^:t -:t f:t *:t <:t
#+OPTIONS:   TeX:t LaTeX:t skip:nil d:nil todo:t pri:nil tags:not-in-toc
#+INFOJS_OPT: view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
#+EXPORT_SELECT_TAGS: export
#+EXPORT_EXCLUDE_TAGS: noexport
#+LINK_UP:
#+LINK_HOME:
#+startup: beamer
#+LaTeX_CLASS: beamer
#+LaTeX_CLASS_OPTIONS: [bigger]
#+COLUMNS: %40ITEM %10BEAMER_env(Env) %9BEAMER_envargs(Env Args) %4BEAMER_col(Col) %10BEAMER_extra(Extra)
#+LATEX_HEADER: \beamertemplatenavigationsymbolsempty
#+BEAMER_THEME: metropolis

* About me

  GNU project volunteer

  GNU Guile user and contributor since 2012

  GNU Guix contributor since 2013

  Day job: Ruby + JavaScript web development / “DevOps”

* Overview

  - Problems with application packaging and deployment
  - Intro to functional package and configuration management
  - Towards the future
  - How you can help

* User autonomy and control

  It is becoming increasingly difficult to have control over your own
  computing:

  - GNU/Linux package managers not meeting user needs
  - Self-hosting web applications requires too much time and effort
  - Growing number of projects recommend installation via =curl | sudo
    bash= [fn:2] or otherwise avoid using system package managers
  - Users unable to verify that a given binary corresponds to the
    source code

* User autonomy and control

  #+latex: \huge{
  “Debian and other distributions are going to be that thing you run
  Docker on, little more.” [fn:7]
  #+latex: }

* User autonomy and control

  This is very bad for desktop users and system administrators alike.
  We must regain control!

* What’s wrong with Apt/Yum/Pacman/etc.?

  Global state (=/usr=) that prevents *multiple versions* of a package
  from coexisting.

  *Non-atomic* installation, removal, upgrade of software.

  No way to *roll back*.

  *Nondeterminstic* package builds and maintainer-uploaded binaries.
  (though this is changing!)

  Reliance on pre-built binaries provided by a *single point of trust*.

  Requires *superuser* privileges.

* The problem is bigger

  Proliferation of *language-specific package managers* and *binary
  bundles* that complicate secure system maintenance.

* Web applications

  Web applications are particularly painful.

* Web applications

  It’s common for today’s web applications to require *two or more
  package managers* to get all dependencies.

* Web applications

  Importing a web application available only for a language-specific
  manager into a distribution proves difficult.  NodeJS is
  particularly frightening. [fn:6]

* Web applications

  There’s a growing number of popular web applications (Hadoop, Chef
  Server, Cloudera, etc.) that *no one knows how to build from
  source*! [fn:3]

* Deployment

  How do we automate application deployment without going crazy?

* Chef/Puppet/Ansible/etc. are pretty good, right?

  Building on top of mainstream package managers and distros yields an
  unstable foundation.

* Problems with configuration management software

  - Imperative config management is overly-complex and brittle
    (idempotence is hard)

  - More reliable builds require spawning new machines and building
    from scratch each time.  (sledgehammer)

  - Made primarily for developers for server maintenance, but all
    types of users could benefit.

* Docker?

  Surely Docker addresses these issues?

* Docker?

  \center{I’m afraid not.}

  \begin{center}
  \includegraphics[height=7cm]{containers.jpg}
  \end{center}

* Problems with Docker

  - Still imperative (though resulting images are immutable)

  - Dockerfile DSL is not expressive

  - Promotes one disk image per application to cover up underlying
    package management mess [fn:4]

  - No provenance

  - Image layering is an ineffective caching strategy

  - Does not compose (what about the host?)

* Problems with Docker

  Reliance on DockerHub binaries proves to be insecure [fn:5]

  \begin{center}
  \includegraphics[width=\textwidth]{docker-insecure.png}
  \end{center}

* Well that was pessimistic

  Computers are hard.  Let’s just look at cat pictures, instead.

  \begin{center}
  \includegraphics[width=8cm]{cons-cats.jpg}
  \end{center}

* Meet GNU Guix

  \begin{center}
  \includegraphics[width=5cm]{guix-logo.png}
  \end{center}

  Guix is the functional package management tool for the GNU system.

  It is based on the pioneering work of the Nix project. [fn:8]

* Meet GuixSD

  \begin{center}
  \includegraphics[width=4cm]{guixsd-logo.png}
  \end{center}

  GuixSD is the GNU/Linux distribution that uses Guix as its package
  manager.

* What does “functional” mean?

   “Functional” in this context means treating package builds as
   functions, in the mathematical sense.

   =emacs = f(gcc,make,coreutils,…)=

* Functional package management

  Benefits:

  - Build reproducibility

  - No single point of trust

  - Unprivileged package management

  - Atomic upgrades and roll backs

  - Multiple variants of the same software may coexist

* Functional package management

  The *complete dependency graph* is captured, precisely, down to the
  *bootstrap binaries*.

  No SAT solver or other complex algorithm for dependency resolution.

* Functional package management

  To view package builds this way, Guix performs builds in an
  *isolated container* in which *only the specified dependencies* are
  accessible.

  Build results are *immutable*.

  This maximizes *build reproducibility*.

* Reproducible builds

  Reproducible builds produce *bit-identical binaries* when performed
  multiple times under the same conditions.

  Requires fixing issues in upstream build systems that are
  nondeterministic.

* Why?

  “With reproducible builds, multiple parties can *redo this process
  independently* and ensure they *all get /exactly/ the same result*.
  We can thus *gain confidence* that a distributed binary code is
  indeed coming from a given source code.” [fn:9]

* Use cases

  \begin{center}
  \includegraphics[width=\textwidth]{nsa-vw.png}
  \end{center}

* Transparent

  Guix is a *source-based* package manager, but will *transparently*
  download pre-built binaries from a trusted party, if available.

  Otherwise, it will simply build from source.

* Decentralized

  In Guix, there is *no central point of trust* for receiving
  pre-built binaries (substitutes).

* Decentralized

  Guix provides http://hydra.gnu.org, but it is optional.

  Users may authorize zero or more substitute servers, or even publish
  their own substitutes for others to use via =guix publish=.

* Challenge authority

  When builds are reproducible, users may *challenge* their substitute
  providers by building locally and comparing the results.

* Unprivileged

  Users can build and install software *without root privileges*.

* Unprivileged

  Each user may have one or more “profiles”, a union of many packages.

  Use cases:

  - Alyssa and Ben use different versions of Emacs
  - Alyssa hacks on 2 Ruby projects that require different versions

* Atomic

  Package installation/removal and full-system updates are *atomic*
  operations, meaning that either the operation succeeds, or nothing
  happens.

* Roll back

  /Any/ package transaction may be *rolled back*, likewise for
  full-system upgrades.

  If a full-system update goes wrong, just boot into the previous
  working generation!

* Coexistence

  Each package has its own *unique* directory in the store that
  contains its build artifacts.

  You can have every version of Ruby, Python, and Perl under the sun
  and that’s OK!

* Demo!

  =guix package=

  =guix challenge=

* Hacking

  Guix is made to be maximally hackable, taking inspiration from
  Emacs.

  We seek to intentionally blur the line between user and developer.

* Choice of language

  Guix is rather special in its choice of implementation language.

* Philosophy

  It’s better to *extend an existing programming language* for package
  recipes and configuration files rather than making a new,
  domain-specific one.

* Embedded vs. External DSLs

   Using an extensible programming language as a host has several
   advantages compared to external DSLs:

   - No new parser, interpreter/compiler, editor tools, etc. to
     maintain

   - Access to all available libraries of the host language

   - Extensions to the host language can be used as a library by
     others

   Not all general-purpose programming languages are suitable for
   embedding new languages, [fn:1] so which did we choose?

* Guile Scheme

  \begin{center}
  \includegraphics[width=4cm]{guile-logo.png}
  \end{center}

   GNU Guile is a Scheme implementation and the official extension
   language of the GNU project.

   It’s a great choice for EDSLs because of Scheme’s *hygienic macro
   system*.

   It’s a great choice for Guix because *purely functional*
   programming is well-supported in Scheme.

* Guile goes with everything

   Guix uses Guile for nearly everything:

   - Initial RAM disk

   - Init system (GNU Shepherd, formerly GNU dmd)

   - Package recipes (including build scripts!)

   - Command line tools

   - Low-level POSIX/Linux utilities (such as =call-with-container=)

* Guix as a library

  Guix is a big collection of Guile modules.

  Packages are first-class Scheme objects.

  Anyone can use Guix as a library to write new Guile programs that
  manipulate package recipes, create new user interfaces (like a web
  UI), etc.

* Example package recipe

  #+latex: \tiny{
  #+BEGIN_SRC scheme
    (define-public livestreamer
      (package
        (name "livestreamer")
        (version "1.12.2")
        (source (origin
                  (method url-fetch)
                  (uri (string-append
                        "https://github.com/chrippa/livestreamer/archive/v"
                        version ".tar.gz"))
                  (file-name (string-append "livestreamer-" version ".tar.gz"))
                  (sha256
                   (base32
                    "1fp3d3z2grb1ls97smjkraazpxnvajda2d1g1378s6gzmda2jvjd"))))
        (build-system python-build-system)
        (arguments
         '(#:tests? #f)) ; tests rely on external web servers
        (native-inputs
         `(("python-setuptools" ,python-setuptools)))
        (propagated-inputs
         `(("python-requests" ,python-requests)
           ("python-singledispatch" ,python-singledispatch)))
        (synopsis "Internet video stream viewer")
        (description "Livestreamer is a command-line utility that extracts streams
    from various services and pipes them into a video playing application.")
        (home-page "http://livestreamer.io/")
        (license license:bsd-2)))
  #+END_SRC
  #+latex: }

* Dependency graph

  \includegraphics[width=\textwidth]{livestreamer-graph.png}

* Demo!

  Emacs + Geiser

* Other user interfaces

  Besides the CLI, there’s also an Emacs interface, naturally.

  Proof of concept web interface. (not in Guix core)

* Demo!

  Emacs

* Importing packages

  The =guix import= tool that can *automatically generate code
  snippets* for packages found in foreign systems.

  Supported systems include: PyPI, RubyGems, CPAN, Hackage, ELPA, and
  CRAN.

* Auto-updating

  The =guix refresh= tool can automatically find the latest release of
  certain software.

  For example, Python packages can be updated by querying PyPI for
  information on the latest release.

* Demo!

  =guix import=

* Reproducible development environments

  Getting the dependencies needed to create development environments
  can be tough.

  Many languages invent their own solution, but this is a general
  problem.

* Reproducible development environments

  Guix has a tool for this: =guix environment=

  Think of it like a language-agnostic version of Python’s
  =virtualenv=.

* Reproducible development environments

  Environments can be *purified* via standard environment variables
  or, for better isolation, Linux containers.

  This allows developers to have confidence that potential
  contributors will be able to build their software.

* Demo!

  =guix environment=

* Full-system configuration

  The Guix System Distribution supports a *consistent whole-system
  configuration mechanism*.

  All aspects of a system configuration are *declared* in a single
  place.

* Advantages

  Easy to replicate configuration on different machines *without
  resorting to additional tools* layered on top.

  System upgrades are atomic and can be rolled back.

* Example system configuration

  #+latex: \tiny{
  #+BEGIN_SRC scheme
    (operating-system
      (host-name "izanagi")
      (timezone "America/New_York")
      (locale "en_US.UTF-8")
      (bootloader (grub-configuration (device "/dev/sda")))
      (file-systems (cons (file-system
                            (device "root")
                            (title 'label)
                            (mount-point "/")
                            (type "ext4"))
                          %base-file-systems))
      (users (list (user-account
                    (name "dave")
                    (comment "David Thompson")
                    (group "users")
                    (supplementary-groups '("wheel" "netdev" "audio"
                                            "video" "cdrom"))
                    (home-directory "/home/dave"))))
      (packages (cons* adwaita-icon-theme avahi dbus gnome-terminal
                       htop less man-db nss-certs openssh pulseaudio
                       wicd unzip rsync xfce
                       %base-packages))
      (services %desktop-services)
      (name-service-switch %mdns-host-lookup-nss))
  #+END_SRC
  #+latex: }

* Service graph

  \includegraphics[width=\textwidth]{service-graph.png}

* Demo!

  =guix system vm=

* Project status

  - Full-featured package manager
  - 3,000 packages, 4 platforms
  - Guix System Distribution in beta
  - Binaries at http://hydra.gnu.org
  - Variety of useful tools

* Project status

  \includegraphics[width=\textwidth]{stats.png}

  \center\url{https://www.openhub.net/p/gnuguix}

* Project status

  \begin{center}
  \includegraphics[width=\textwidth]{contributor-graph.png}
  \end{center}

* The people have spoken

  \begin{center}
  \includegraphics[width=4cm]{awesome-rating.png}
  \end{center}

* Project status

  \approx200–500 new packages per release.  *More needed!*

* Future

  I intend to focus on:

  - A cluster deployment tool: =guix deploy=
  - Improved support for GuixSD containers

* Future

  More generally:

  - Stronger build farm
  - More packages that are reproducible
  - GNOME
  - LVM
  - Encrypted root for everyone

* Join us!

  - Use Guix on top of your existing distro
  - Use the distribution
  - Add new packages or upgrade existing ones
  - Write system services
  - Add new translations
  - Tell us your ideas!

* Join us!

  We are currently collecting donations via the FSF to purchase new
  servers for our build farm!

  Since mid-Decemeber, $8,200 USD has been raised.

  https://gnu.org/software/guix/donate/

* Join us!

  Chat with us in the =#guix= channel on Freenode or on the
  =guix-devel@gnu.org= and =help-guix@gnu.org= mailing lists.

* LibrePlanet 2016

  Christopher Webber of the GNU MediaGoblin project and myself will be
  co-presenting “Solving the Deployment Crisis with GNU Guix” at
  LibrePlanet 2016 on March 19th or 20th.

  Visit https://libreplanet.org/2016 for full details.

* Thank you!

  Visit https://gnu.org/software/guix for source code, documentation,
  past talks, etc.

  \begin{center}
    \huge{Questions?}
  \end{center}

* Legal

  © 2016 David Thompson =<davet@gnu.org>=

  This presentation is licensed under the Creative Common Attribute
  Share-Alike 4.0 International license.

  GNU Guix and GuixSD logo, GFDL, http://gnu.org/s/guix/graphics

  Copyright of other images included in this document is held by their
  respective owners.

* Footnotes

[fn:1] “How to be a good host: miniKanren as a case study”
https://www.youtube.com/watch?v=b9C3r3dQnNY

[fn:2] http://curlpipesh.tumblr.com/

[fn:3] “Your big data toolchain is a big security risk!”
http://www.vitavonni.de/blog/201504/2015042601-big-data-toolchains-are-a-security-risk.html

[fn:4] “The sad state of sysadmin in the age of containers”
http://www.vitavonni.de/blog/201503/2015031201-the-sad-state-of-sysadmin-in-the-age-of-containers.html

[fn:5] http://www.banyanops.com/blog/analyzing-docker-hub/

[fn:6] “Let’s Package jQuery: A Javascript Packaging Dystopian Novella” http://dustycloud.org/blog/javascript-packaging-dystopia/

[fn:7] “ownCloud and distribution packaging” http://lwn.net/Articles/670566/

[fn:8] http://nixos.org/nix/

[fn:9] https://reproducible-builds.org/