Group: Guix/GSoC-2019

This page is for tracking ideas and process for participating in Google Summer of Code 2019.

Project ideas for Guix and GuixSD

Guile based build-tool

Guile-based build tool for advanced software developers who do not want to deal with the complexity of multi-platform deployment and dependency hell. By default runs inside the GNU Guix build system and effectively replaces the likes of autotools, cmake, etc.

Build tools such as automake and cmake are complex beasts. What they try to do is target multiple build environments by generating make files using a complex list of string expansions. Complexity, in general, means hard to understand, change and maintain. The bad thing is that almost every software project has to deal with this complexity to make the software deployable.

What is wrong with these build tools? In short:

1. Complex - and therefore hard to understand and maintain

2. Simplistic language - simple string expansion systems are not nice and not DRY

3. Race conditions - make's time stamps are not suitable for building across servers

4. Different combinations of options are badly handled.

5. No intermediate representation

Alternatives, such as Java ant, Scala sbt, and Ruby rake, try to resolve similar issues but are specialised for their environments. Here we have a *fresh* start of a build tool which should ultimately be able to run complex workflows on multiple servers and make use of opportunistic execution, i.e. jobs may be distributed multiple times and the first one that comes back wins.

We primarily targets GNU Guix because Guix already solves the complexity of handling dependencies and software deployment. This simplifies things dramatically because the build system does not have to deal with platform or architecture targets. Even so, in time, we can target out of GNU Guix builds.

Mentors: Pjotr Prins and other Guix members

Continue rewrite build daemon in Guile Scheme

(This is the continuation of a GSOC-2017 effort; see the 'guile-daemon' branch.) Currently the build daemon of Guix is written in C++, inherited from Nix. It works fine but is not as hackable as we'd like, and has poor integration with the rest of Guix. For instance, the daemon calls out to the 'guix substitute', 'guix authenticate', and 'guix offload' commands, but its interface to these commands is very limited. Furthermore, a large part of the daemon's code is already implemented in Scheme: container functionality is available with 'call-with-container', archive creation is implemented in (guix nar), writing derivation files (.drv) is done in (guix derivations), and so on.

The goal of this project would be to rewrite the daemon in Guile Scheme using the building blocks already available. Important missing bits include the garbage collector and its scanner (which scans files for references to /gnu/store items), the scheduler, which schedules derivation builds in topological order, using the specified number of cores, etc.

Mentors: Ludovic (+ David, +...?)

Content-addressed protocol for substitutes

The goal of this project would be to reduce the amount of data that needs to be transferred when retrieving substitutes. The rationale and design idea is described in this message.

Remote machine deployment

Server administrators want to be able to automate system updates across the entire fleet of servers under their control. Running 'guix system reconfigure' manually on each server is tedious. In the world of traditional distros with imperative package managers, Chef and Ansible (and others) fill this role. The goal of this project is provide a high-level management tool for any number of GuixSD systems called 'guix deploy'.

Possible use-cases are:

1. managing on-premise "bare metal" servers

2. managing "stateful" virtual machines (on-premise cluster or a hosting provider such as AWS) where machines are updated in-place

3. managing "stateless" virtual machines where machines are replaced rather than updated (typically involves auto-scaling on a "cloud" hosting provider such as AWS; "blue-green" deployment)

Coming up with a core implementation that is flexible enough to handle all of these concrete use-cases will be an interesting challenge. Ideally, things will compose. For example, a user could simultaneously deploy updates to a bare metal database *and* an AWS auto-scaling group full of web frontend servers *and* a stateful VM used an SSH bastion.

Mentors: David Thompson, Chris Webber

Booting via network

Some administrators have a lot of machines and would like to install GuixSD via the network on an "empty" machine without walking to each one.

A possible way to do that is to make the "empty" machine request an IP address to use (via DHCP), load the files required for booting (via TFTP) and then boot Linux (make the CPU jump there) and have the booted Linux mount the root (via NFS).

The "empty" computer has to support the first part. Fortunately, a lot of computers do support this. Network cards and many laptops support booting in this manner - and many fall back to it if nothing else works. The technology is called "PXE". There's also the U-Boot pxe module (part of U-Boot) which allows you to boot ARMv7 machines from the network in this manner (however, it's not clear whether these machines have a fixed MAC address or other unique ID).

This work would entail:

1. Checking the feasibility by manually configuring dnsmasq, tftpd and NFS to serve files - and testing it on an "empty" computer.

2. Extending the boot mechanism of GuixSD to mount root from NFS in case the system has been booted via PXE.

3. Working out the actual design.

4. Creating a new Guix service "dnsmasq-service" to provide a DHCP server and to also provide PXE-specific DHCP options. See also https://www.tecmint.com/install-pxe-network-boot-server-in-centos-7/

5. Creating a new Guix service "tftpd-service" which allows users to use the service extension facility so that other services can provide files - like bootloader files, installation files, other misc files etc. GuixSD's inetd already contains tftpd (a file server using the TFTP protocol - which is customary to use) - however, tftpd's shared directory cannot managed by GuixSD yet. Dnsmasq also has its own tftp server, so this would have to be investigated. It might be useful to serve the contents of GuixSD system disk images, see https://wellsie.net/p/286/ .

6. Providing said files also via NFS. It might be useful to provide loop mounts to be able to serve the contents of GuixSD system disk images.

7. Creating a new Guix service "guixsd-installation-service" which extends or requires other services (like tftpd, dhcpd) and allows the user to specify a system to serve to the user.

8. Working out and documenting the security implications.

Future work could be to extend this to supply different systems to different users, however this work wouldn't require it.

Mentors: Danny Milosavljevic

GNUnet integration

Relying on a central server to download substitutes (aka. compiled softwares) is not efficient and not resilient since the server providing substitutes is single point of failure (SPOF).

In this task we propose to extend guix to support *publishing* and *downloading* substitutes over the GNUnet peer-to-peer File System.

This work would entail:

1. Improving the GNUnet guile bindings a) to avoid any memory leaks using guile's set-pointer-finalizer! and b) use disjoint types for pointers using guile's define-wrapped-pointer-type

2. Extend package definitions in guix to include an optional gnunet uri

3. Create a sheperd service for gnunet master (there is package defintion for gnunet master in gnunet-guile2 bindings).

4. Extend guix download to support downloading files over gnunet

5. Extend guix publish to publish substitutes over gnunet

Mentors:  ???

Project ideas for the GNU Shepherd

Syntax and semantics of systemd units in the Shepherd

The GNU Shepherd has a Scheme interface to define services, most importantly their dependencies and actions associated with them. The goal of this project is twofold. The first part consists in matching the semantics of systemd's .service unit files, more precisely a subset thereof (for instance, the DBus-related part may be omitted.) As part of this work, the Shepherd should be extended with cgroup support on systems that support it. The Shepherd should also be extended with a procedure to load a .service file and return a <service> object.

The second part will consist in implementing other types of units, in particular socket activation and .device.

Mentors: Ludovic

Add an extensible event loop

As of version 0.3, the Shepherd uses a trivial event loop where it waits for connections from clients such as the 'herd' command and also, via Guile "system asyncs", waits for SIGCHLD signals corresopnding to the death of child processes. Thus, we assume that clients will not block, and we also expect the 'start' and 'stop' methods of each service (and in fact, each service action) to be non-blocking. Consequently, the Shepherd can only serve one client request at a time, and services are retricted in what they can do. For instance, it's currently impossible to have a REPL server in the Shepherd, unless resorting to multiple threads, which is not an option.

The goal of this project is to make the Shepherd rely on an extensible event loop. Services should be able to add file descriptors to watch or delays to wait for to the event loop. The event loop will avoid the "callback hell" by using either a monad (a monad is essentially "continuation-passing style", CPS, which is essentially "callbacks"), or by using delimited continuations, for instance via Fibers or maybe 8sync.

Mentors: Ludovic + Chris Webber(, +...?)

Project ideas for Cuirass

Cuirass is a build automation server using Guix.

Adding modules in support of continuous integration

Currently, the NixOS project kindly offers to run continuous integration (CI) jobs at hydra.nixos.org for GNU packages, as explained here.

The goal would be to provide Guile modules that would make it possible to have similar CI jobs written using Cuirass. Currently, useful support is limited to dist-package and (guix build gnu-dist), which facilitate the creation of jobs that run 'make distcheck'.

Mentors: Ludovic

Adding a web interface similar to the Hydra web interface

Unlike Hydra, Cuirass does not have a pretty web interface. It also doesn't implement the full API that Hydra provides. This makes it difficult to schedule evaluations of feature branches, and to monitor the current state of an evaluation.

The goal here is to implement more of the Hydra API in Cuirass and to provide a web interface that allows a user to view build failures, view evaluations, see the overall build progress, etc.

Mentors: Ricardo