Yet Another SysAdmin Website - Blog

A difficult 23.11 nixos upgrade story

2024-02-06

Introduction

Back in December I upgraded my nixos servers from the 23.05 release to 23.11. I had to debug a strange issue where my servers were no longer reachable after rebooting the new version.

The problem

I am using LUKS encryption for the root filesystem, and am used to the comfort of unlocking the partition thanks to an SSH server embedded in the initrd. This setup has the security flaw that the initrd could be replaced by a malicious party, but this is not something I am concerned about for personal stuff so please ignore it.

The following configuration made it work on nixos 23.05:

{ config, pkgs, ... }:
{
 boot.initrd.network = {
 enable = true;
 ssh = {
 enable = true;
 port = 22;
 authorizedKeys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AABCDLOJV3913FRYgCVA2plFB8W8sF9LfbzXZOrxqaOrrwco" ];
 hostKeys = [ "/etc/ssh/ssh_host_rsa_key" "/etc/ssh/ssh_host_ed25519_key" ];
 };
 };
}

What happened

Being a good sysadmin I read the release notes and caught:

The boot.initrd.network.udhcp.enable option allows control over DHCP during Stage 1 regardless of what networking.useDHCP is set to.

I thought nothing of it… But I should have!

Behind this message is the fact that if you did not set networking.useDHCP = true; globally, your initrd in nixos 23.11 will no longer do a DHCP lookup. This is a behavioral change I find baffling because it worked perfectly in 23.05! My configuration used DHCP but set explicitly on the interfaces that need it, not globally. As a networking engineer I loathe useless traffic on my networks, this includes DHCP requests for devices that do not need it.

Nixos 23.11 needs a boot.initrd.network.udhcpc.enable = true; in order to boot correctly again. Finding this new setting was not too hard - a few minutes of head scratching and intuition did the trick - but as usual I am on the lookout for a learning opportunity.

Configuration diffs

The first thing I looked for is a way to diff between two nixos configurations. I ended up disappointed because I did not find a way to do it neither easily nor exhaustively! There are quite advanced things for nix itself, but for nixos it is quite terse.

The most advanced thing I managed is to have a diff between configurations that were activated on the same machine: diff on just the build server does not work, this needs to happen on the machine where the configuration is deployed live.

The nixos diffs I managed are limited to installed packages or installed files and their size changes, nothing seems to allow me to dive into what is inside the initrd.

nix --extra-experimental-features nix-command profile diff-closures --profile /nix/var/nix/profiles/system

Conclusion

This upgrade experience did not inspire a lot of confidence in me. Nixos is a great project and I wholeheartedly thank all its contributors for their efforts and dedication, but as a sysadmin this is not the kind of defaults that I ever want to see change silently.

I still think nixos has great potential and deserves more recognition.

Testing opentofu

2024-01-31

Introduction

This January, the opentofu project announced the general availability of their terraform fork. Not much changes for now between terraform and opentofu (and that is a good thing!), as far as I can tell the announcement was mostly about the new provider registry and of course the truly open source license.

Registry change

The opentofu registry already has all the providers you are accustomed to, but your state will need to be migrated with:

tofu init -upgrade`

For some providers you might encounter the following warning:

- Installed cloudflare/cloudflare v4.23.0. Signature validation was skipped due to the registry not containing GPG keys for this provider

This is harmless and will resolve itself when the providers’ developers provide the public GPG key used to sign their releases to the opentofu registry. The process is very simple thanks to their GitHub workflow automation.

Little improvements

tofu init seems significantly faster than terraform init.
You never could interrupt a terraform plan with C-C. I am so very glad to see that it is not a problem with opentofu! This really needs more advertising: proper Unix signal handling is like a superpower that is too often ignored by modern software.
tofu test can be used to assert things about your state and your configuration. I did not play with it yet but it opens a whole new realm of possibilities!
tofu import can use expressions referencing other values or resources attributes, this is a big deal when handling massive imports!

Eventline terraform provider

I did the required pull requests on the opentofu registry to have my Eventline provider all fixed up and ready to rock!

Conclusion

I hope opentofu really takes off, the little improvements they made already feel like a breath of fresh air. Terraform could be so much more!

How to resize the persistent volumes of a kubernetes statefulset

2024-01-15

Introduction

Kubernetes statefulsets are great but they come with their share of limitations. One of those limitations is that you cannot edit or patch many important keys of the YAML spec of an object after it has been created, in particular the requested volume size of the volumeClaimTemplates.

How to

The work around consists of deleting the statefulset while leaving the objects created from it intact. In my example, I am resizing the persistent disks for a redis cluster created with the chart from bitnami, from 1GB to 2GB. It lives on a cluster named myth in the namespace redis. The statefulset is named redis-node and spawns three pods and three pvcs.

Storage class

First of all you need to ensure the storage class of the persistent volumes supports volume expansion. Most CSI drivers do, but the storage class do not necessarily have it enabled.

To get the storage class to look for you can use (k is my shell alias to the kubectl command):

k --context myth -n redis get pvc redis-data-redis-node-0 -o jsonpath='{.spec.storageClassName}'

Let’s say that the storage class is named standard, one of the builtin ones when installing a kubernetes cluster on gcp. Let’s inspect it:

k --context myth get storageclass standard -o jsonpath='{.allowVolumeExpansion}'

If you get false or an empty output then your storage class is missing a allowVolumeExpansion: true. If that is the case, you need to patch your storage class with:

k --context myth patch storageclass standard --patch '{"allowVolumeExpansion": true}'

Note that this object is not namespaced, you are changing this for your whole cluster.

Resizing the persistent volumes

Resize the pvcs:

k --context myth -n redis patch pvc redis-data-redis-node-0 --patch '{"spec": {"resources": {"requests": {"storage": "2Gi"}}}}'
k --context myth -n redis patch pvc redis-data-redis-node-1 --patch '{"spec": {"resources": {"requests": {"storage": "2Gi"}}}}'
k --context myth -n redis patch pvc redis-data-redis-node-2 --patch '{"spec": {"resources": {"requests": {"storage": "2Gi"}}}}'

Recreate the statefulset

Get the statefulset:

k --context myth -n redis get statefulset redis-node -o YAML > redis-statefulset.yaml

Edit this yaml file to change the size in the volumeClaimTemplates, remove the status keys (and their values) in the file.

With this yaml file ready, we can remove the statefulset without deleting the other kubernetes objects it spawned:

k --context myth -n redis delete statefulset redis-node --cascade=orphan

Recreate the statefulset from the modified yaml:

k --context myth -n redis apply -f redis-statefulset.yaml

Beware that this last action will restart the pods.

Conclusion

Kubernetes is a convoluted beast, not everything makes sense. Hopefully this work around will be useful to you until the day the developers decide it should be reasonable to be able to resize persistent volumes of statefulsets directly.

Migrating miniflux to nixos

2024-01-07

Introduction

I am migrating several services from a k3s kubernetes cluster to a nixos server. Here is how I performed the operation with my miniflux rss reader.

Miniflux with nixos

Miniflux is packaged on nixos, but I am used to the container image so I am sticking with it for now.

Here is the module I wrote to deploy a miniflux container, configure postgresql and borg backups:

{ config, lib, pkgs, ... }:
{
 imports = [
 ../../lib/borg-client.nix
 ../../lib/postgresql.nix
 ../../lib/nginx.nix
 ];
 environment.etc."borg-miniflux-db.key" = {
 mode = "0400";
 source = ./borg-db.key;
 };
 services = {
 borgbackup.jobs = let defaults = {
 compression = "auto,zstd";
 doInit = true;
 encryption.mode = "none";
 prune.keep = {
 daily = 14;
 weekly = 4;
 monthly = 3;
 };
 startAt = "daily";
 }; in {
 "miniflux-db" = defaults // {
 environment.BORG_RSH = "ssh -i /etc/borg-miniflux-db.key";
 paths = "/tmp/miniflux.sql";
 postHook = "rm -f /tmp/miniflux.sql";
 preHook = ''rm -f /tmp/miniflux.sql; /run/current-system/sw/bin/pg_dump -h localhost -U miniflux -d miniflux > /tmp/miniflux.sql'';
 repo = "ssh://borg@gcp.adyxax.org/srv/borg/miniflux-db";
 };
 };
 nginx.virtualHosts."miniflux.adyxax.org" = {
 forceSSL = true;
 locations = {
 "/" = {
 proxyPass = "http://127.0.0.1:8084";
 };
 };
 sslCertificate = "/etc/nginx/adyxax.org.crt";
 sslCertificateKey = "/etc/nginx/adyxax.org.key";
 };
 postgresql = {
 ensureUsers = [{
 name = "miniflux";
 ensurePermissions = { "DATABASE \"miniflux\"" = "ALL PRIVILEGES"; };
 }];
 ensureDatabases = ["miniflux"];
 };
 };
 virtualisation.oci-containers.containers = {
 miniflux = {
 environment = {
 ADMIN_PASSWORD = lib.removeSuffix "\n" (builtins.readFile ./admin-password.key);
 ADMIN_USERNAME = "admin";
 DATABASE_URL = "postgres://miniflux:" + (lib.removeSuffix "\n" (builtins.readFile ./database-password.key)) + "@10.88.0.1/miniflux?sslmode=disable";
 RUN_MIGRATIONS = "1";
 };
 image = "miniflux/miniflux:2.0.50";
 ports = ["127.0.0.1:8084:8080"];
 };
 };
}

Dependencies

The dependencies are mostly the same as in my article about vaultwarden migration.

Migration process

The first step is obviously to deploy this new configuration to the server, then I need to login and manually restore the backups.

make run host=dalinar.adyxax.org

The container will be failing because no password is set on the database user yet, so I stop it:

systemctl stop podman-miniflux

There is only one backup job for miniflux and it holds a dump of the database:

export BORG_RSH="ssh -i /etc/borg-miniflux-db.key"
borg list ssh://borg@gcp.adyxax.org/srv/borg/miniflux-db
borg extract ssh://borg@gcp.adyxax.org/srv/borg/miniflux-db::dalinar-miniflux-db-2023-11-20T00:00:01
psql -h localhost -U postgres -d miniflux

Restoring the data itself is done with the psql shell:

ALTER USER miniflux WITH PASSWORD 'XXXXXX';
\i tmp/miniflux.sql

Afterwards I clean up the database dump and restart miniflux:

rm -rf tmp/
systemctl start podman-miniflux

To wrap this up I migrate the DNS records to the new host, update my monitoring system and clean up the namespace on the k3s server.

Conclusion

I did all this in november, I have quite the backlog of articles to write!

Migrating vaultwarden to nixos

2023-12-20

Introduction

I am migrating several services from a k3s kubernetes cluster to a nixos server. Here is how I performed the operation with my vaultwarden password manager.

Vaultwarden with nixos

Vaultwarden is packaged on nixos, but I am used to the hosting the container image and upgrading it at my own pace so I am sticking with it for now.

Here is the module I wrote to deploy a vaultwarden container, configure postgresql and borg backups in apps/vaultwarden/app.nix:

{ config, lib, pkgs, ... }:
{
 imports = [
 ../../lib/nginx.nix
 ../../lib/postgresql.nix
 ];
 environment.etc = {
 "borg-vaultwarden-db.key" = {
 mode = "0400";
 source = ./borg-db.key;
 };
 "borg-vaultwarden-storage.key" = {
 mode = "0400";
 source = ./borg-storage.key;
 };
 };
 services = {
 borgbackup.jobs = let defaults = {
 compression = "auto,zstd";
 doInit = true;
 encryption.mode = "none";
 prune.keep = {
 daily = 14;
 weekly = 4;
 monthly = 3;
 };
 startAt = "daily";
 }; in {
 "vaultwarden-db" = defaults // {
 environment.BORG_RSH = "ssh -i /etc/borg-vaultwarden-db.key";
 paths = "/tmp/vaultwarden.sql";
 postHook = "rm -f /tmp/vaultwarden.sql";
 preHook = ''rm -f /tmp/vaultwarden.sql; /run/current-system/sw/bin/pg_dump -h localhost -U vaultwarden -d vaultwarden > /tmp/vaultwarden.sql'';
 repo = "ssh://borg@gcp.adyxax.org/srv/borg/vaultwarden-db";
 };
 "vaultwarden-storage" = defaults // {
 environment.BORG_RSH = "ssh -i /etc/borg-vaultwarden-storage.key";
 paths = "/srv/vaultwarden";
 repo = "ssh://borg@gcp.adyxax.org/srv/borg/vaultwarden-storage";
 };
 };
 nginx.virtualHosts = let commons = {
 forceSSL = true;
 locations = {
 "/" = {
 proxyPass = "http://127.0.0.1:8083";
 };
 };
 }; in {
 "pass.adyxax.org" = commons // {
 sslCertificate = "/etc/nginx/adyxax.org.crt";
 sslCertificateKey = "/etc/nginx/adyxax.org.key";
 };
 };
 postgresql = {
 ensureUsers = [{
 name = "vaultwarden";
 ensureDBOwnership = true;
 }];
 ensureDatabases = ["vaultwarden"];
 };
 };
 virtualisation.oci-containers.containers = {
 vaultwarden = {
 environment = {
 ADMIN_TOKEN = builtins.readFile ./argon-token.key;
 DATABASE_MAX_CONNS = "2";
 DATABASE_URL = "postgres://vaultwarden:" + (lib.removeSuffix "\n" (builtins.readFile ./database-password.key)) + "@10.88.0.1/vaultwarden?sslmode=disable";
 };
 image = "vaultwarden/server:1.30.1";
 ports = ["127.0.0.1:8083:80"];
 volumes = [ "/srv/vaultwarden/:/data" ];
 };
 };
}

Dependencies

Borg

Borg needs to be running on another server with the following configuration stored in my apps/vaultwarden/borg.nix file:

{ config, pkgs, ... }:
{
 imports = [
 ../../lib/borg.nix
 ];
 users.users.borg.openssh.authorizedKeys.keys = [
 ("command=\"borg serve --restrict-to-path /srv/borg/vaultwarden-db\",restrict " + (builtins.readFile ./borg-db.key.pub))
 ("command=\"borg serve --restrict-to-path /srv/borg/vaultwarden-storage\",restrict " + (builtins.readFile ./borg-storage.key.pub))
 ];
}

PostgreSQL

My postgreSQL module defines the following global configuration:

{ config, lib, pkgs, ... }:
{
 networking.firewall.interfaces."podman0".allowedTCPPorts = [ 5432 ];
 services.postgresql = {
 enable = true;
 enableTCPIP = true;
 package = pkgs.postgresql_15;
 authentication = pkgs.lib.mkOverride 10 ''
 #type database DBuser auth-method
 local all all trust
 # podman
 host all all 10.88.0.0/16 scram-sha-256
 '';
 };
}

Since for now I am running nothing outside of containers on this server, I am trusting the unix socket connections. Depending on what you are doing you might want a stronger auth-method there.

Nginx

My nginx module defines the following global configuration:

{ config, lib, pkgs, ... }:
{
 environment.etc = let permissions = { mode = "0400"; uid= config.ids.uids.nginx; }; in {
 "nginx/adyxax.org.crt" = permissions // { source = ../../01-legacy/adyxax.org.crt; };
 "nginx/adyxax.org.key" = permissions // { source = ../../01-legacy/adyxax.org.key; };
 };
 networking.firewall.allowedTCPPorts = [ 80 443 ];
 services.nginx = {
 clientMaxBodySize = "40M";
 enable = true;
 enableReload = true;
 recommendedGzipSettings = true;
 recommendedOptimisation = true;
 recommendedProxySettings = true;
 };
}

Secrets

There are several secrets referenced in the configuration, these are all git-crypted files:

argon-token.key
borg-db.key
borg-storage.key
database-password.key

Migration process

The first step is obviously to deploy this new configuration to the server, then I need to login and manually restore the backups.

make run host=myth.adyxax.org

The container will be failing because no password is set on the database user yet, so I stop it:

systemctl stop podman-vaultwarden

There are two backup jobs for vaultwarden: one for its storage and the second one for the database.

export BORG_RSH="ssh -i /etc/borg-vaultwarden-storage.key"
borg list ssh://borg@gcp.adyxax.org/srv/borg/vaultwarden-storage
borg extract ssh://borg@gcp.adyxax.org/srv/borg/vaultwarden-storage::dalinar-vaultwarden-storage-2023-11-19T00:00:01
mv srv/vaultwarden /srv/

export BORG_RSH="ssh -i /etc/borg-vaultwarden-db.key"
borg list ssh://borg@gcp.adyxax.org/srv/borg/vaultwarden-db
borg extract ssh://borg@gcp.adyxax.org/srv/borg/vaultwarden-db::dalinar-vaultwarden-db-2023-11-19T00:00:01
psql -h localhost -U postgres -d vaultwarden

Restoring the data itself is done with the psql shell:

ALTER USER vaultwarden WITH PASSWORD 'XXXXX';
\i tmp/vaultwarden.sql

Afterwards I clean up the database dump and restart vaultwarden:

rm -rf tmp/
systemctl start podman-vaultwarden

To wrap this up I migrate the DNS records to the new host, update my monitoring system and clean up the namespace on the k3s server.

Conclusion

Automating things with nixos is satisfying, but it does not abstract all the sysadmin’s work away.

I am not quite satisfied with my borg configuration entries. I should be able to write this more elegantly when I find the time, but it works.

Memory difficulties with nixos

2023-12-14

Introduction

I encountered my first difficulties with nixos which required some ingenuity outside of the natural learning curve.

On memory and lightweight software

The VPS hosts I am using are not really beefy. Three of these only have 1GB of ram which is not a lot by today’s standards, but quite sufficient for many usages. The services I self host are quite lightweight so I never had problems when running Alpine Linux, Debian, FreeBSD or OpenBSD on these small machines. Of course k3s was reserved for my beefier 2GB hosts, but nixos seemed it could fit. Like any operating system, it consumes little memory at rest.

The one big memory constraint coming from nixos might not be obvious: it is when rebuilding the configurations! For an almost empty host, very simple configuration and no services besides dhcp, ssh, journal and cron, a nixos configuration build could take about 500MB of ram. That is not negligible but it fit.

With some services like an irc server, eventline, privatebin and gotosocial, the configuration got more complex and nixos more demanding, consuming about 700MB for a build.

Building nixos remotely

I hit a wall when I started using a second channel to pull more recent packages. I wanted bleeding edge packages for things like Emacs, but stable ones for all the other parts of the system… and I could no longer build nixos locally! 1GB is not enough to have the packages sources and resolve dependencies when building the configuration.

Therefore I started building nixos configurations remotely. My workstation does the heavy lifting of building the configuration then copying all the derivations (target configurations, packages and files) to the hosts.

Activating the configuration still involves a spike of memory consumption on the hosts of about 500MB, but it is less than the 1.2GB it takes to build the configurations. Despite this, I experienced a few painful out of memory when deploying a new configuration. Now I shutdown the most demanding services before deploying, like gotosocial which can sometimes consume 200MB of ram by itself.

Upgrading to 23.11

I had a bad experience upgrading from 23.05 to the recent 23.11 release. I do not know how the diffs between configurations are calculated by nix, but I could not deploy on my 1GB hosts!

I worked around this by using dd to copy the hard drive images and start them in virtual machines locally. This allowed me to upgrade then copy the images the other way. Still, that is a painful process. The back and forth copying involves a similar process than I described to remount partitions as read-only in a previous article.

Conclusion

Beware if you intend on using nixos on small machines! I will continue experimenting with nix because it still seems worthwhile and I want to continue learning it, but if I end up switching back to another operating system (be it Alpine, Debian or a BSD) it will be because the configuration build process became too painful to bear.

Finishing advent of code 2022 in Haskell

2023-12-05

Introduction

I wrote about doing the advent of code 2022 in zig, but I did not complete the year. I stopped on using zig on day 15 when I hit a bug when using hashmaps that I could not solve in time and continued in JavaScript until day 22. On day 22 part 2, you need to fold a cube and move on it keeping track of your orientation… It was hard!

Last week I wanted to warm up for the current advent of code and therefore took it up again… it was (almost) easy with Haskell!

Day 22 - Monkey Map

You get an input that looks like this:

 ...#
.#..
#...
....
...#.......#
........#...
..#....#....
..........#.
...#....
.....#..
.#......
......#.
10R5L5R10L4R5L5

The . are floor tiles, the # are impassable walls. You have a cursor starting on the leftmost tile on the first line. The cursor moves and the empty spaces do not exist: if you step out you wrap around: easy enough… until part 2!

Here is how I parse the input:

type Line = V.Vector Char
type Map = V.Vector Line
data Instruction = Move Int | L | R deriving Show
data Input = Input Map [Instruction] deriving Show
type Parser = Parsec Void String

parseMapLine :: Parser Line
parseMapLine = do
 line <- some (char '.' <|> char ' ' <|> char '#') <* eol
 return $ V.generate (length line) (line !!)

parseMap :: Parser Map
parseMap = do
 lines <- some parseMapLine <* eol
 return $ V.generate (length lines) (lines !!)

parseInstruction :: Parser Instruction
parseInstruction = (Move . read <$> some digitChar)
 <|> (char 'L' $> L)
 <|> (char 'R' $> R)

parseInput' :: Parser Input
parseInput' = Input <$> parseMap
 <*> some parseInstruction <* eol <* eof

In part 2 you learn that your input pattern is in fact 6 squares that can be folded to form a cube. Now instead of simply wrapping the empty spaces, when stepping out you need to find out were you end up on the cube and with which orientation.

Here is a visualization I made in excalidraw to understand how folding the cube based on my input would work (this does not match the example above but matched the players’ input):

The whole code is available on my git server but here is the core of my solver for this puzzle:

stepOutside :: Map -> Int -> Int -> Int -> Heading -> Int -> Cursor
stepOutside m s x y h i | (t, h) == (a, N) = proceed fw (fn + rx) E
 | (t, h) == (a, W) = proceed dw (ds - ry) E
 | (t, h) == (b, N) = proceed (fw + rx) fs N
 | (t, h) == (b, E) = proceed ee (es - ry) W
 | (t, h) == (b, S) = proceed ce (cn + rx) W
 | (t, h) == (c, W) = proceed (dw + ry) dn S
 | (t, h) == (c, E) = proceed (bw + ry) bs N
 | (t, h) == (d, N) = proceed cw (cn + rx) E
 | (t, h) == (d, W) = proceed aw (as - ry) E
 | (t, h) == (e, E) = proceed be (bs - ry) W
 | (t, h) == (e, S) = proceed fe (fn + rx) W
 | (t, h) == (f, W) = proceed (aw + ry) an S
 | (t, h) == (f, S) = proceed (bw + rx) bn S
 | (t, h) == (f, E) = proceed (ew + ry) es N
 where
 (tx, rx) = x `divMod` s
 (ty, ry) = y `divMod` s
 t = (tx, ty)
 proceed :: Int -> Int -> Heading -> Cursor
 proceed x' y' h' = case m V.! y' V.! x' of
 '.' -> step m s (Cursor x' y' h') (Move $ i - 1)
 '#' -> Cursor x y h
 (ax, ay) = (1, 0)
 (bx, by) = (2, 0)
 (cx, cy) = (1, 1)
 (dx, dy) = (0, 2)
 (ex, ey) = (1, 2)
 (fx, fy) = (0, 3)
 a = (ax, ay)
 b = (bx, by)
 c = (cx, cy)
 d = (dx, dy)
 e = (ex, ey)
 f = (fx, fy)
 (an, as, aw, ae) = (ay * s, (ay+1)*s-1, ax *s, (ax+1)*s-1)
 (bn, bs, bw, be) = (by * s, (by+1)*s-1, bx *s, (bx+1)*s-1)
 (cn, cs, cw, ce) = (cy * s, (cy+1)*s-1, cx *s, (cx+1)*s-1)
 (dn, ds, dw, de) = (dy * s, (dy+1)*s-1, dx *s, (dx+1)*s-1)
 (en, es, ew, ee) = (ey * s, (ey+1)*s-1, ex *s, (ex+1)*s-1)
 (fn, fs, fw, fe) = (fy * s, (fy+1)*s-1, fx *s, (fx+1)*s-1)

This stepOutside function takes in argument the map, its size, the cursor’s (x, y) position and heading h, while i is the number of steps to perform. I first compute on which face the cursor is, and based on its heading where it should end up. I then use the faces coordinates to compute the final position, being careful to follow on the schematic how the transition is performed.

Conclusion

The next days where quite a lot easier than this one. Haskell is really a great language for puzzle solving thanks to its excellent parsing capabilities and its incredible type system.

A great thing that should speak of Haskell’s qualities is that it is the second year of advent of code that I completed all 25 days: both times it was thanks to Haskell! I think I should revisit the years 2021 that I did with Go next: I stopped on day 19 because it involved a three dimensional puzzle that was quite difficult.

Managing multiple nixos hosts, remotely

2023-11-28

Introduction

There seems to be almost too many tools to manage nix configurations with too many different approaches, each with their quirks and learning curve. Googling this issue was more troubling than it should be!

Therefore I tried to keep things simple and converged on a code organization that I find flexible enough for my current nixos needs without anything more than the standard nix tools.

Repository layout

Here are the directories inside my nixos repository:

├── apps
│ ├── eventline
│ ├── files
│ ├── gotosocial
│ ├── miniflux
│ ├── privatebin
│ └── vaultwarden
├── hosts
│ ├── dalinar.adyxax.org
│ ├── gcp.adyxax.org
│ └── myth.adyxax.org
└── lib
└── common

apps

The apps directory contains files and configurations about each application I manage. Here is what an app folder looks like:

└── apps
└── eventline
├── app.nix
├── borg-db.key
├── borg-db.key.pub
├── borg.nix
├── eventline-entrypoint
└── eventline.yaml

Each of the app directories has an app.nix file detailing the nix configuration to deploy the app that will be included by the host running it, and a borg.nix with the configurations for the host that will be the borg backups target. In my setup each app has its own set of ssh keys (which are encrypted with git-crypt) for its borg jobs.

The remaining files are specific to the app. In this example there is a configuration file and a custom entrypoint for a container image.

hosts

The hosts directory contains the specific configurations and files for each host running nixos. Here is what it looks like:

hosts/dalinar.adyxax.org/
├── configuration.nix
├── hardware-configuration.nix
└── wg0.key

The confiuration.nix currently looks like:

{ config, pkgs, ... }:
{
 imports = [
 ./hardware-configuration.nix
 ../../apps/eventline/app.nix
 ../../apps/gotosocial/app.nix
 ../../apps/ngircd.nix
 ../../apps/privatebin/app.nix
 ../../apps/teamspeak.nix
 ../../lib/boot-uefi.nix
 ../../lib/common.nix
 ];
 environment.etc."wireguard/wg0.key".source = ./wg0.key;
 networking = {
 hostName = "dalinar";
 wireguard.interfaces."wg0" = {
 ips = [ "10.1.2.11/32" ];
 listenPort = 342;
 peers = [
 { publicKey = "7mij2whbm0qMx/D12zdMS5i9lt3ZSI3quNomTI+BSgk=";
 allowedIPs = [ "10.1.2.14/32" ];
 endpoint = "lumapps-jde.adyxax.org:342"; }
 ];
 };
 };
 systemd.network.networks.wan = {
 address = [ "2603:c022:c002:8500:e2a4:f02e:43b0:c1d8/128" ];
 matchConfig.Name = "eth0";
 networkConfig = { DHCP = "ipv4"; IPv6AcceptRA = true; };
 };
}

The hardware-configuration.nix is taken directly from the host machine after its installation.

The content of wg0.key is encrypted with git-crypt too and generated with:

wg genkey

lib

The contents of the lib directory are used either directly from the hosts configurations, or from the apps configurations:

lib
├── boot-bios.nix
├── boot-uefi.nix
├── common
│ ├── borg-client.nix
│ ├── check-mk-agent.nix
│ ├── dns.nix
│ ├── mosh.nix
│ ├── network.nix
│ ├── nix.nix
│ ├── openssh.nix
│ ├── tmux.conf
│ ├── tmux.nix
│ └── wireguard.nix
├── common.nix
├── julien.nix
├── luks.nix
├── nginx.nix
└── postgresql.nix

All the files in lib/common/ are included in lib/common.nix. These are split in self contained logical parts.

Deploying to a remote host

I use the following GNUmakefile to deploy from my workstation or from my eventline server to my hosts:

SHELL := bash
.SHELLFLAGS := -eu -o pipefail -c
.ONESHELL:
.DEFAULT_GOAL := help
.DELETE_ON_ERROR:
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules

##### TASKS ####################################################################
.PHONY: run
run: mandatory-host-param ## make run host=<hostname>
 nixos-rebuild switch --target-host root@$(host) -I nixos-config=hosts/$(host)/configuration.nix

.PHONY: update
update: ## make update
 nix-channel --update

##### UTILS ####################################################################
.PHONY: help
help:
 @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

.PHONY: mandatory-host-param
mandatory-host-param:
ifndef host
 @echo "Error: host parameter is not set"; exit 1
else
ifeq ($(wildcard hosts/$(host)), )
 @echo "Error: host has no configuration in ./hosts/$(host)"; exit 1
endif
endif

This way I can make run host=dalinar.adyxax.org to build locally dalinar’s configuration and deploy it remotely.

Conclusion

I am quite happy with the simplicity of this system for now. Everything works smoothly and tinkering with the configurations does not involve any magic.

The one thing I really want to improve is the wireguard peers management which is a lot more involved than it needs to be. I will also explore using custom variables in order to simplify the hosts configurations.

In the next articles I will detail the code behind some of these apps and lib files.

Recovering a nixos installation from a Linux rescue image

2023-11-13

Introduction

This article explains how to chroot into a nixos system from a Linux rescue image. I recently had to do this while installing a nixos at ovh: I used an UEFI base image I prepared for oracle cloud instead of a legacy BIOS image. I could have just started the copy again using the right image, but it was an opportunity for learning and I took it.

Chrooting into a nixos system

This works from any Linux system given you adjust the device paths. It will mount your nixos and chroot into it:

mount /dev/sdb2 /mnt/
cd /mnt
mount -R /dev dev
mount -R /proc proc
mount -R /sys sys
mount /dev/sdb1 boot
chroot ./ /nix/var/nix/profiles/system/activate
chroot ./ /run/current-system/sw/bin/bash

A nixos system needs to have some runtime things populated under /run in order for it to work correctly, that is the reason for the profile activation step.

Generating a new hardware-configuration.nix

Upon installation, a /etc/nixos/hardware-configuration.nix file is automatically created with specifics of your system. If you need to update it, know that its contents comes from the following command:

nixos-generate-config --show-hardware-config

Building a new configuration

Nixos has a configuration build sandbox that will not work from the chroot. To disable it I had to temporarily set the following in /etc/nix/nix.conf:

sandbox = false

Do not forget to reactivate it later!

Next you will need to have a working DNS to make any meaningful change to a nixos configuration, because it will almost certainly need to download some new derivation. Since the resolv.conf is a symlink, you need to remove it before writing into it:

rm /etc/resolv.conf
echo 'nameserver 1.1.1.1' > /etc/resolv.conf

You should now be able to rebuild your system to apply your configuration fix:

nixos-rebuild --install-bootloader boot

Conclusion

Nixos will not break often, and when it does you should be able to simply rollback from your boot loader menu. But if anything worse happens or if you are migrating a nixos installation to another chassis, or salving a hard drive… now you know how to proceed!

Deploying a web application to nixos

2023-10-06

Introduction

Gotosocial is a service that was running on one of my FreeBSD servers. Being a simple web application it is a good candidate to showcase what I like most about nixos and its declarative configurations!

A bit about the nix language

I recommend you read the official documentation, but here is the minimal to get you started:

every statement ends with a semicolon.
The basic block structures are in fact Sets, meaning lists of key-value pairs where the keys are unique.
The {...}: { } that structure the whole file is a module definition. In the first curly braces are arguments.
The let ...; in { } construct is a way to define local variables for usage in the block following the in.
You can write strings with double quotes or double single quotes. This makes it so that you almost never need to escape characters! The double single quotes also allow to write multi line strings that will smartly strip the starting white spaces.
file system paths are not strings!
list elements are separated by white spaces.
You can merge the keys in two sets with //, often used in conjunction with let local variables.
imports work by merging sets and appending lists.

Statements can be grouped but nothing is mandatory. For example the following are completely equivalent:

environment = {
 etc."gotosocial.yaml" = {
 mode = "0444";
 source = ./gotosocial.yaml;
 };
 systemPackages = [ pkgs.sqlite ];
};

environment.etc."gotosocial.yaml" = {
 mode = "0444";
 source = ./gotosocial.yaml;
};
environment.systemPackages = [ pkgs.sqlite ];

environment.etc."gotosocial.yaml".mode = "0444";
environment.etc."gotosocial.yaml".source = ./gotosocial.yaml;
environment.systemPackages = [ pkgs.sqlite ];

Configuration

The following configuration does in order:

Imports the Nginx.nix module defined in the next section.
Deploys Gotosocial’s YAML configuration file.
Installs sqlite, necessary for our database backup preHook.
Defines two Borg backup jobs: one for the SQLite database and one for the local storage.
Configures an Nginx virtual host.
Deploys the Gotosocial container.

{ config, pkgs, ... }:
{
 imports = [
 ../lib/nginx.nix
 ];
 environment = {
 etc."gotosocial.yaml" = {
 mode = "0444";
 source = ./gotosocial.yaml;
 };
 systemPackages = [ pkgs.sqlite ];
 };
 services = {
 borgbackup.jobs = let defaults = {
 compression = "auto,zstd";
 encryption.mode = "none";
 environment.BORG_RSH = "ssh -i /etc/borg.key";
 prune.keep = {
 daily = 14;
 weekly = 4;
 monthly = 3;
 };
 repo = "ssh://borg@kaladin.adyxax.org/srv/borg/dalinar.adyxax.org";
 startAt = "daily";
 }; in {
 "gotosocial-db" = defaults // {
 paths = "/tmp/gotosocial-sqlite.db";
 postHook = "rm -f /tmp/gotosocial-sqlite.db";
 preHook = ''
 rm -f /tmp/gotosocial-sqlite.db
 echo 'VACUUM INTO "/tmp/gotosocial-sqlite.db"' | \
 /run/current-system/sw/bin/sqlite3 /srv/gotosocial/sqlite.db
 '';
 };
 "gotosocial-storage" = defaults // { paths = "/srv/gotosocial/storage"; };
 };
 nginx.virtualHosts."fedi.adyxax.org" = {
 forceSSL = true;
 locations = {
 "/" = {
 proxyPass = "http://127.0.0.1:8082";
 proxyWebsockets = true;
 };
 };
 sslCertificate = "/etc/nginx/adyxax.org.crt";
 sslCertificateKey = "/etc/nginx/adyxax.org.key";
 };
 };
 virtualisation.oci-containers.containers.gotosocial = {
 cmd = [ "--config-path" "/gotosocial.yaml" ];
 image = "superseriousbusiness/gotosocial:0.11.1";
 ports = ["127.0.0.1:8082:8080"];
 volumes = [
 "/etc/gotosocial.yaml:/gotosocial.yaml:ro"
 "/srv/gotosocial/:/gotosocial/storage/"
 ];
 };
}

Nginx

I will go into details in a next article about imports and how I organize my configurations, just know that in this case imports work intuitively. Here is the lib/nginx.nix file defining common configuration for Nginx:

{ config, pkgs, ... }:
{
 environment.etc = let permissions = { mode = "0400"; uid= config.ids.uids.nginx; }; in {
 "nginx/adyxax.org.crt" = permissions // { source = ../../01-legacy/adyxax.org.crt; };
 "nginx/adyxax.org.key" = permissions // { source = ../../01-legacy/adyxax.org.key; };
 };
 networking.firewall.allowedTCPPorts = [ 80 443 ];
 services.nginx = {
 clientMaxBodySize = "40M";
 enable = true;
 enableReload = true;
 recommendedGzipSettings = true;
 recommendedOptimisation = true;
 recommendedProxySettings = true;
 };
}

Deploying

Being an existing service for me, I transferred gotosocial’s storage data and database using rsync. With that done, bringing the service back up was only a matter of migrating the DNS and running the now familiar:

nixos-rebuild switch

Conclusion

I hope you find this way of declaratively configuring a whole operating system as elegant as I do. The nix configuration language is a bit rough, but I find it is not so hard to wrap your head around the basics. When it all clicks it is nice to know that you can reproduce this deployment anywhere just from this configuration!

Installing nixos on a vps

2023-10-04

Introduction

Not many providers consider nixos as a first class citizen, you need a little know how to be able to set it up in a not so friendly environment. Nixos’s wiki has several procedures to achieve this but I found those either too complicated or not up to date. This article presents my prefered way to install an operating system somewhere it is not supported to do so, and it works for anything.

Installation

Prepare a virtual machine

If you followed my last article, you should have a nixos virtual machine ready to go. You just need to upload it somewhere. I chose kaladin.adyxax.org, another one of my machines, and to serve the machine over ssh. Alternatively you could use a web server or even socat/netcat if it strikes your fancy.

Bootstrap your vps or compute instance

Install your vps or compute instance normally using a Linux distribution (or any of the BSD) that is supported by your provider. Connect to it as root.

Remount disk partitions as read only

We are going to remount the partitions as the running OS as read only. In order to do that, we are going to shutdown nearly everything! If at some point you lose access to your system, just force reboot it and try again. Our goal is for those commands to run without an error:

swapoff -a
mount -o remount,ro /boot
mount -o remount,ro /

If there are other disk partitions mounted, those must be remounted read only as well. Check cat /proc/mounts if you do not know what to look for.

Be aware that selinux could block you. If that is the case, deactivate it, reboot and start over.

On most Linux you can list running services using systemctl|grep running and begin running systemctl stop commands on almost anything, just remember to keep what your running session depends on:

init
session-XX
user@0 (root) and any user@XX where XX is the uid you connected with

Everything else should be fair game, what you are looking for are processus that keep files opened for writing. Those can be identified with:

lsof / | awk '$4 ~ /[0-9].*w/'
fuser -v -m /
ps aux
systemctl|grep running

Here is a list of what I shutdown on an oracle cloud compute before I could remount / read only:

systemctl stop smartd
systemctl stop rpcbind
systemctl stop rpcbind.socket
systemctl stop systemd-journald-dev-log.socket
systemctl stop systemd-journald.socket
systemctl stop systemd-udevd-control.socket
systemctl stop systemd-udevd-kernel.socket
systemctl stop tuned.service
systemctl stop user@1000.service
systemctl stop user@989.service
systemctl stop rsyslog
systemctl stop oswatcher.service
systemctl stop oracle-cloud-agent.service
systemctl stop oracle-cloud-agent-updater.service
systemctl stop gssproxy.service
systemctl stop crond.service
systemctl stop chronyd.service
systemctl stop auditd.service
systemctl stop atd.service
systemctl stop auditd.service
systemctl stop sssd
systemctl stop sssd_bd
systemctl stop firewalld
systemctl stop auditd
systemctl stop iscsid
systemctl stop iscsid.socket
systemctl stop dbus.socket
systemctl stop dbus
systemctl stop systemd-udevd
systemctl stop sshd
systemctl stop libstoragemgmt.service
systemctl stop irqbalance.service
systemctl stop getty@tty1.service
systemctl stop serial-getty@ttyS0.service

Remember, your success condition is to be able to run this without errors:

swapoff -a
mount -o remount,ro /boot
mount -o remount,ro /

As soon as this is done and you only have ro in cat /proc/mounts for your disk partitions you can stop shutting down services.

Copying the virtual machine you prepared

When successful at remounting your partitions read only, then retrieve your virtual machine image. You will need to copy it directly to the disk, here is how I do it using ssh:

ssh root@kaladin.adyxax.org "dd if=/nixos-uefi.raw" | dd of=/dev/sda

Reboot and test

Once the copy is complete, you will have to force reboot your machine. After a minute you should be able to ssh to it and get a nixos shell!

You will need a virtual console or KVM of some sort to debug your image if something went wrong. All providers have this capability, you just have to find it in their webui.

Conclusion

I used this procedure successfully on ovh, hetzner, google cloud and on oracle cloud and I believe it should work anywhere. I used it for nixos, but also to install some Gentoo, OpenBSD or FreeBSD where those were not supported either.

Getting started with nixos

2023-09-30

Introduction

After discovering nix I quickly jumped into nixos, the Linux distribution based on nix. It has been a few months now and I very much like nixos’s stability and reproducibility. Upgrades went smoothly each time and I migrated quite a few services to a nixos server.

Installation

Virtual machine bootstrap

Installing nixos is really not hard, you quickly get to a basic setup you can completely understand thanks to its declarative nature. When I began tinkering with nixos, my goal was to install it on a vps for which I needed UEFI support, here is how I bootstrapped a virtual machine locally:

qemu-img create -f raw nixos.raw 4G
qemu-system-x86_64 -drive file=nixos.raw,format=raw,cache=writeback \
 -cdrom Downloads/nixos-minimal-23.05.1994.af8279f65fe-x86_64-linux.iso \
 -boot d -machine type=q35,accel=kvm -cpu host -smp 2 -m 1024 -vnc :0 \
 -device virtio-net,netdev=vmnic -netdev user,id=vmnic,hostfwd=tcp::10022-:22 \
 -bios /usr/share/edk2-ovmf/OVMF_CODE.fd

Partitioning

From there, I performed the following simple partitioning (just one big root partition):

parted /dev/sda -- mklabel gpt
parted /dev/sda -- mkpart ESP fat32 1MB 512MB
parted /dev/sda -- set 1 esp on
parted /dev/sda -- mkpart primary 512MB 100%
mkfs.fat -F 32 -n boot /dev/sda1
mkfs.ext4 -L nixos /dev/sda2
mount /dev/disk/by-label/nixos /mnt
mkdir -p /mnt/boot
mount /dev/disk/by-label/boot /mnt/boot

Initial configuration

The initial configuration is generated with:

nixos-generate-config --root /mnt

This will generate a /mnt/etc/nixos/hardware-configuration.nix with the specifics of your machine along with a basic /mnt/etc/nixos/configuration.nix that I replaced with:

{ config, pkgs, ... }:
{
 imports = [
 ./hardware-configuration.nix
 ];
 boot.kernelParams = [
 "console=ttyS0"
 "console=tty1"
 "libiscsi.debug_libiscsi_eh=1"
 "nvme.shutdown_timeout=10"
 ];
 boot.loader = {
 efi.canTouchEfiVariables = true;
 systemd-boot.enable = true;
 };
 environment.systemPackages = with pkgs; [
 curl
 tmux
 vim
 ];
 networking = {
 dhcpcd.enable = false;
 hostname = "dalinar";
 nameservers = [ "1.1.1.1" "9.9.9.9" ];
 firewall = {
 allowedTCPPorts = [ 22 ];
 logRefusedConnections = false;
 logRefusedPackets = false;
 };
 usePredictableInterfaceNames = false;
 };
 nix = {
 settings.auto-optimise-store = true;
 extraOptions = ''
 min-free = ${toString (1024 * 1024 * 1024)}
 max-free = ${toString (2048 * 1024 * 1024)}
 '';
 gc = {
 automatic = true;
 dates = "weekly";
 options = "--delete-older-than 30d";
 };
 };
 security = {
 doas.enable = true;
 sudo.enable = false;
 };
 services = {
 openssh = {
 enable = true;
 settings.KbdInteractiveAuthentication = false;
 settings.PasswordAuthentication = false;
 };
 resolved.enable = false;
 };
 systemd.network.enable = true;
 time.timeZone = "Europe/Paris";
 users.users = {
 adyxax = {
 description = "Julien Dessaux";
 extraGroups = [ "wheel" ];
 hashedPassword = "$y$j9T$Nne7Ad1nxNmluCKBzBG3//$h93j8xxfBUD98f/7nGQqXPeM3QdZatMbzZ0p/G2P/l1";
 home = "/home/julien";
 isNormalUser = true;
 openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILOJV391WFRYgCVA2plFB8W8sF9LfbzXZOrxqaOrrwco adyxax@yen" ];
 };
 root = {
 hashedPassword = "$y$j8F$ummLlZmPdS1KGxSnwH8CY.$bjvADB9IdfwzO6/2if5Sl9DeCmCRdasknq4IJEAuxyA";
 openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILOJV391WFRYgCVA2plFB8W8sF9LfbzXZOrxqaOrrwco adyxax@yen" ];
 };
 };
 # This value determines the NixOS release from which the default
 # settings for stateful data, like file locations and database versions
 # on your system were taken. It's perfectly fine and recommended to leave
 # this value at the release version of the first install of this system.
 # Before changing this value read the documentation for this option
 # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
 system.stateVersion = "23.05";
 # Copy the NixOS configuration file and link it from the resulting system
 # (/run/current-system/configuration.nix). This is useful in case you
 # accidentally delete configuration.nix.
 system.copySystemConfiguration = true;
}

This will setup a system that in particular will use the systemd-bootd boot loader in lieu of grub and systemd-networkd instead of NetworkManager. Not much else is going on. The nix section slows builds a bit but greatly reduced disk space consumption.

Installation

nixos-install --no-root-passwd

Rebooting

In order to boot on the newlly installed system and not the installer, the virtual machine command needs to be changed, so shutdown your system with:

halt -p

And start it with:

qemu-system-x86_64 -drive file=nixos.raw,format=raw,cache=writeback \
 -boot c -machine type=q35,accel=kvm -cpu host -smp 2 -m 1024 -vnc :0 \
 -device virtio-net,netdev=vmnic -netdev user,id=vmnic,hostfwd=tcp::10022-:22 \
 -bios /usr/share/edk2-ovmf/OVMF_CODE.fd

Updating the configuration

If you change the configuration, you need to rebuild the system with:

nixos-rebuild switch

Upgrading

You can rebuild your system with the latest nixos packages using:

nix-channel --update
nixos-rebuild switch

Conclusion

Installing and tinkering with nixos is quite fun! In the next articles I will explain how I organized my configurations to manage multiple servers, how to use a luks encrypted system and remotely unlock them after rebooting, and how to run the builds for small servers from a much more powerful machine.

Getting started with nix

2023-09-09

Introduction

I have been using nix for a few months now. It is a modern package manager that focuses on reproducible builds and was a first step before using nixos, a linux distribution based around nix and its capabilities that I find intriguing. Being able to have a fully reproducible system from a declarative configuration is something I find enticing.

Getting started

You can get started using nix on any linux distribution, even on macos or windows! You do not need to reinstall anything or boot another operating system: you can install nix and start taking advantage of it anytime anywhere.

The official documentation (which you should refer to) mentions two alternatives: one which runs a daemon to allow for multiple users to use nix on the same system, and a simpler one without a running daemon which I chose to follow.

I recommend you audit the installation script, it is always a good idea to do so (and in this case it is quite simple to read what it does), but here are the three installation steps:

doas mkdir /nix
doas chown adyxax /nix
sh <(curl -L https://nixos.org/nix/install) --no-daemon

If this completes without error, you now have nix installed and just need to activate it in your shell with:

source ~/.nix-profile/etc/profile.d/nix.sh

To make this persistent add it where relevant for your shell and distribution, it could be in ~/.bashrc, ~/.profile, ~/.zshrc, etc:

if [ -e "${HOME}/.nix-profile/etc/profile.d/nix.sh" ]; then
 source "${HOME}/.nix-profile/etc/profile.d/nix.sh"
fi

Using nix

Nix channels

By default, your nix installation should use the unstable profile. That just means bleeding edge packages, but I like to be explicit when using bleeding edge stuff therefore I did:

nix-channel --remove nixpkgs
nix-channel --add https://nixos.org/channels/nixos-23.05 nixpkgs
nix-channel --add https://nixos.org/channels/nixos-unstable nixpkgs-unstable
nix-channel --update

23.05 is the current stable release channel at the time of this writing. Please check the current one at the time of your reading and use that.

Be careful not to change this version number mindlessly as it can affect anything stateful you install with nix. The most common problem you will encounter is about file locations that change with major database versions (for example postgresql14 and 15). Changing this 23.05 version would not migrate your data, so be careful that you can migrate or have migrated all the state from your nix packages which is affected by this kind of version changes. I will write a blog article about this when it happens to me.

Searching packages

The easiest and fastest way is through nixos’s website: https://search.nixos.org/packages?channel=23.05

If you want to do it from the cli beware that it is a bit slow, particularly on the first run (maybe it is building some cache):

$ nix-env -qaP firefox # short for: nix-env --query --available --attr-path firefox
nixpkgs.firefox-esr-102 firefox-102.15.0esr
nixpkgs-unstable.firefox-esr-102 firefox-102.15.0esr
nixpkgs.firefox-esr firefox-115.2.0esr
nixpkgs.firefox-esr-wayland firefox-115.2.0esr
nixpkgs-unstable.firefox-esr firefox-115.2.0esr
nixpkgs-unstable.firefox-esr-wayland firefox-115.2.0esr
nixpkgs.firefox firefox-117.0
nixpkgs.firefox-wayland firefox-117.0
nixpkgs-unstable.firefox firefox-117.0
nixpkgs-unstable.firefox-mobile firefox-117.0
nixpkgs-unstable.firefox-wayland firefox-117.0
nixpkgs.firefox-beta firefox-117.0b9
nixpkgs.firefox-devedition firefox-117.0b9
nixpkgs-unstable.firefox-beta firefox-117.0b9
nixpkgs-unstable.firefox-devedition firefox-117.0b9

As you can see, the nixpkgs stable channels does not lag behind unstable for most day to day things you would need updated, but it will for more system things or experimental software.

$ nix-env -qaP gotosocial
nixpkgs-unstable.gotosocial gotosocial-0.11.0

Installing packages

nix-env -iA nixpkgs.emacs29 # short for: nix-env --install --attr nixpkgs.emacs29

Listing installed packages

$ nix-env -qs # short for: nix-env --query --status
IPS emacs-29.1

Note that the installed package name changed completely and no longer reference nixpkgs or nixpkgs-unstable! That comes from the notion of nix derivations which we will not get into in this article.

Upgrading packages

nix-channel --update
nix-env --upgrade

Uninstalling packages

nix-env --uninstall emacs-29.1

Maintaining nix itself

Updating nix

nix-channel --update
nix-env --install --attr nixpkgs.nix nixpkgs.cacert

Uninstalling nix

If at some point you want to stop using nix and uninstall it, simply run:

rm -rf "${HOME}/.nix-profile"
doas rm -rf /nix

Conclusion

This article is a first overview of nix that can get you started, we did not get into the best parts yet: profile management, rolling back to a previous packages state, packaging software, building container images and of course nixos itself. So much material for future articles!

I have been a happy Gentoo user for close to twenty years now and do not plan to switch anytime soon for many reasons, but it is nice to have another packages repository to play with.

Writing a terraform provider for eventline

2023-08-04

Introduction

I have been using terraform to manage infrastructure both personnaly and at work for several years now and I know this tool quite well. I have been searching for an excuse to write a terraform provider for quite some time in order to dive deeper into terraform and I finally realised that I had just such excuse!

I started using eventline when it was released a year ago and have been very happy with it. Turns out I could benefit from a terraform provider to provision identities or jobs when deploying new hosts, so here I go!

Writing a terraform provider

Where to start

The recommended way is to fork the terraform provider scaffolding framework repository from Hashicorp. This is what I did, but it came with some frustration. Hashicorp recently deprecated another way of developing terraform providers called SDKv2, thereform the big downside is that almost all the examples, blog posts or existing providers you would like to take inspiration from are all using the old sdk!

Without good examples, you are left with reading the documentation (which I found a bit lacking) and reading the sources of hashicorp’s framework and libraries (which thanks to go’s “boringness” is surprisingly possible, even enjoyable).

The project name

I did not find it explicitely documented so here it is for you: you MUST name your provider’s repository terraform-provider-something, otherwise the builtin CI from the framework repository will not work with some very cryptic errors!

Terraform types wrapping

One thing that puzzled me a bit was how to make terraform’s schema types work with go types. When writing your datasources and resources, you define your types like this simple:

type ProjectResourceModel struct {
 Id types.String `tfsdk:"id"`
 Name types.String `tfsdk:"name"`
}

This go type is associated with a schema function that will look like:

func (r *ProjectResource) Schema(ctx context.Context, req resource.SchemaRequest, resp *resource.SchemaResponse) {
 resp.Schema = schema.Schema{
 Attributes: map[string]schema.Attribute{
 "id": schema.StringAttribute{
 Computed: true,
 MarkdownDescription: "Project Id",
 PlanModifiers: []planmodifier.String{
 stringplanmodifier.UseStateForUnknown(),
 },
 },
 "name": schema.StringAttribute{
 MarkdownDescription: "Project name",
 Required: true,
 },
 },
 MarkdownDescription: "Eventline project resource",
 }
}

To use this resource, the user of this terraform provider wlil provide a name and will get back an id. To use the name in your code, you will need to do:

var data *ProjectResourceModel
resp.Diagnostics.Append(req.Plan.Get(ctx, &data)...)
if resp.Diagnostics.HasError() {
 return
}
name := data.Name.ValueString() //get the go string out of the terraform resource schema

To provision the Id:

 data.Id = types.StringValue(id) // wraps the go string into the right type for terraform resource schema
 resp.Diagnostics.Append(resp.State.Set(ctx, &data)...)

Schema with nested list attributes

The examples form hashicorp all reference list with simple types. If you want to better describe your resources and datasources, you will need to write your lists in this manner:

func (d *ProjectsDataSource) Schema(ctx context.Context, req datasource.SchemaRequest, resp *datasource.SchemaResponse) {
 resp.Schema = schema.Schema{
 Attributes: map[string]schema.Attribute{
 "elements": schema.ListNestedAttribute{
 Computed: true,
 MarkdownDescription: "The list of projects.",
 NestedObject: schema.NestedAttributeObject{
 Attributes: map[string]schema.Attribute{
 "id": schema.StringAttribute{
 Computed: true,
 MarkdownDescription: "The identifier of the project.",
 },
 "name": schema.StringAttribute{
 Computed: true,
 MarkdownDescription: "The name of the project.",
 },
 },
 },
 },
 },
 MarkdownDescription: "Use this data source to retrieve information about existing eventline projects.",
 }
}

Testing your work with a provider override

In order to develop comfortably your provider, you will need a ~/.terraformrc file that looks like the following:

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
disable_checkpoint = true

provider_installation {
 dev_overrides {
 "adyxax/eventline" = "/home/julien/.go/bin/"
 }

 # For all other providers, install them directly from their origin provider
 # registries as normal. If you omit this, Terraform will _only_ use
 # the dev_overrides block, and so no other providers will be available.
 direct {}
}

Use the binary subfolder of your $GOPATH and this will work. When you go install your provider, the resulting binary will get copied there and be picked up by terraform on each plan or apply. Yes: the neat thing is that you do not need to run init constantly!

Provider documentation

The provider’s documentation can be generated with go generate. It will use the MarddownDescription attributes you defined in your schema descriptions so make those good entries. As the name suggest, you can use multiline markdown so go crazy with it!

Another piece to know about is the examples folder in your repository. If you give it a structure like:

examples/
├── data-sources
│   ├── eventline_identities
│   │   └── data-source.tf
│   ├── eventline_jobs
│   │   └── data-source.tf
│   ├── eventline_project
│   │   └── data-source.tf
│   └── eventline_projects
│   └── data-source.tf
├── provider
│   └── provider.tf
├── README.md
└── resources
└── eventline_project
├── import.sh
└── resource.tf

Then your objects documentation will get augmented with useful examples for the users of your provider.

Conclusion

Writing a terraform provider is a lot of fun, I recommend it! If you have a piece of software that you wish had a terraform provider, know that it is not that hard to make it a reality.

Here is the repository of my eventline provider for reference and here is the terraform provider’s page.

Space Traders

2023-07-08

Introduction

A few weeks ago, a friend stumbled upon Space Traders. He shared the link along with his enthusiasm knowing very well I would not resist its appeal.

The game

SpaceTraders is an API-based game where you acquire and manage a fleet of ships to explore, trade, and one day fight your way across the galaxy. It is not finished and very much in alpha state. There have been a few bugs but nothing major so far.

You can use any programming language you want to query the API and control your ships, query market prices or shipyards stocks, explore systems, mine or survey asteroids. You run your code wherever you like, however you like.

One of the challenges is that you are rate limited to 2 requests per seconds, with a 10 requests burst over 10 seconds. Because of that, any competitive agent will need to be efficient in the commands it sends and the strategy it chooses!

Getting started

My recent experiences with Haskell made me itch to get started in this language, but I finally decided against it. I was at a level of proficiency where I know it would have been too ambitious a task. I would have just ended up tinkering with data types and abstractions instead of learning the API and experimenting with the game.

Therefore I went with (vanilla) JavaScript. It is quite a nice language for prototyping despite its many pitfalls, and I quickly got an agent working its way through the first faction contract. This first contract is like a tutorial for the game and the documentation guides you through it. I refined my agent along the way and am proud to have something that can mine the requested good (selling anything else), then navigate and deliver goods. It loops like that until the contract is fulfilled.

It might be premature optimisation but I am caching a maximum of information in an SQLite database in order to reduce the amount of API calls my code needs to make. I am taking advantage of SQLite’s JSON support to store the JSON data from the API calls, which is a lot easier than expressing all the information in SQL tables, columns and references. I add the necessary index on the JSON fields I query against.

The network requests are all handled by a queue processor which relies on a priority queue. When the agent needs to make an API call, it places it along with a promise into the priority queue, choosing the right priority depending on the action needed. For example ships actions that will gain credits will take priority over exploration tasks, or market refresh tasks. Centralizing the network requests in this manner allows me to strictly respect the rate limits and not hammer needlessly the game’s servers.

Going further

I started adding more complex behaviors to my ships. For example, a navigation request will check if the ship is docker or not, and undock it if that is the case. Upon arrival it will attempt to refuel. Another example is a navigation request which will check the ship’s position for asteroids. If it is not a mining location, the ship will automatically navigate to where it can mine, and refuel if needed.

With all this implemented, I should begin tackling exploration. My navigation code currently only works in a single system and I need to handle FTL jumps or warp drives depending on the destination.

I also want to implement automatic ship purchasing depending on the current agent’s goals, but I feel limited by JavaScript’s dynamic nature when iterating on the code. I am tired of fighting with runtime error and exceptions, therefore I just started rewriting my agent in Haskell.

Conclusion

I learned a lot about async in JavaScript with this project! I encourage anyone with a bit of free time to give it a try, be it to learn a new language or improve in one you already know. My code is available on my git server if you want to have a look. Do not hesitate to reach me on mastodon @adyxax@adyxax.org if you want to discuss space traders!

Advent of code 2020 in haskell

2023-06-22

Introduction

I did the advent of code 2020 in haskell, I had a great time! I did it following advent of code 2022 in zig, while reading Haskell Programming From First Principles a few months ago.

Haskell for puzzles

Parsing

I used megaparsec extensively, it felt like a cheat code to be able to process the input so easily! This holds especially true for day 4 where you need to parse something like:

ecl:gry pid:860033327 eyr:2020 hcl:#fffffd
byr:1937 iyr:2017 cid:147 hgt:183cm
iyr:2013 ecl:amb cid:350 eyr:2023 pid:028048884
hcl:#cfa07d byr:1929
hcl:#ae17e1 iyr:2013
eyr:2024
ecl:brn pid:760753108 byr:1931
hgt:179cm
hcl:#cfa07d eyr:2025 pid:166559648
iyr:2011 ecl:brn hgt:59in

The keys can be in any order so you need to account for permutations. Furthermore, entries each have their own set of rules in order to be valid. For example a height needs to have a unit in cm on inches and be in a certain range, while colors need to start with a hash sign and be composed of 6 hexadecimal digits.

All this could be done at parsing time, haskell made this almost easy: I kid you not!

The type system

I used and abused the type system in order to have straightforward algorithms where if it compile then it works. A very notable example comes from day 25 where I used the Data.Mod library to have modulus integers enforced by the type system. That’s right, in haskell that is possible!

Performance

Only one puzzle had me reach for optimizations in order to run in less than a second. All the others ran successfully with a simple runghc <solution>.hs! For this slow one, I sped it up by reaching for:

ghc --make -O3 first.hs && time ./first

Memory

I had no memory problems and laziness was not an issue either. Haskell really is a fantastic language.

Solution Templates

Simple parsing

Not all days called for advanced parsing. Some just made me look for a concise way of doing things. Here is (spoiler alert) my solution for the first part of day 6 as an example:

-- requires cabal install --lib split Unique
module Main (main) where
import Control.Monad (void, when)
import Data.List.Split (splitOn)
import Data.List.Unique (sortUniq)
import Data.Monoid (mconcat)
import System.Exit (die)

exampleExpectedOutput = 11

parseInput :: String -> IO [String]
parseInput filename = do
 input <- readFile filename
 return $ map (sortUniq . mconcat . lines) $ splitOn "\n\n" input

compute :: [String] -> Int
compute = sum . map length

main :: IO ()
main = do
 example <- parseInput "example"
 let exampleOutput = compute example
 when (exampleOutput /= exampleExpectedOutput) (die $ "example failed: got " ++ show exampleOutput ++ " instead of " ++ show exampleExpectedOutput)
 input <- parseInput "input"
 print $ compute input

Advanced parsing

Here is (spoiler alert) my solution for the first part of day 24 as an example:

-- requires cabal install --lib megaparsec parser-combinators
module Main (main) where
import Control.Monad (void, when)
import Data.List qualified as L
import Data.Map qualified as M
import Data.Maybe (fromJust)
import Data.Set qualified as S
import Data.Void (Void)
import Text.Megaparsec
import Text.Megaparsec.Char
import System.Exit (die)

exampleExpectedOutput = 10

data Direction = E | W | NE | NW | SE | SW
type Directions = [Direction]
type Coordinates = (Int, Int, Int)
type Floor = M.Map Coordinates Bool
type Input = [Directions]
type Parser = Parsec Void String

parseDirection :: Parser Direction
parseDirection = (string "se" *> return SE)
 <|> (string "sw" *> return SW)
 <|> (string "ne" *> return NE)
 <|> (string "nw" *> return NW)
 <|> (char 'e' *> return E)
 <|> (char 'w' *> return W)

parseInput' :: Parser Input
parseInput' = some (some parseDirection <* optional (char '\n')) <* eof

parseInput :: String -> IO Input
parseInput filename = do
 input <- readFile filename
 case runParser parseInput' filename input of
 Left bundle -> die $ errorBundlePretty bundle
 Right input' -> return input'

compute :: Input -> Int
compute input = M.size . M.filter id $ L.foldl' compute' M.empty input
 where
 compute' :: Floor -> Directions -> Floor
 compute' floor directions = case M.lookup destination floor of
 Just f -> M.insert destination (not f) floor
 Nothing -> M.insert destination True floor
 where
 destination :: Coordinates
 destination = L.foldl' run (0, 0, 0) directions
 run :: Coordinates -> Direction -> Coordinates
 run (x, y, z) E = (x+1,y-1,z)
 run (x, y, z) W = (x-1,y+1,z)
 run (x, y, z) NE = (x+1,y,z-1)
 run (x, y, z) SW = (x-1,y,z+1)
 run (x, y, z) NW = (x,y+1,z-1)
 run (x, y, z) SE = (x,y-1,z+1)

main :: IO ()
main = do
 example <- parseInput "example"
 let exampleOutput = compute example
 when (exampleOutput /= exampleExpectedOutput) (die $ "example failed: got " ++ show exampleOutput ++ " instead of " ++ show exampleExpectedOutput)
 input <- parseInput "input"
 print $ compute input

Conclusion

Learning haskell is worthwhile, it is really a great language with so many qualities. Puzzle solving is a use case where it shines so bright, thanks to its excellent parsing capabilities and its incredible type system.

A great thing that should speak of haskell’s qualities is that it is the first year of advent of code that I completed all 25 days. I should revisit the years 2021 and 2022 that I did with golang and zig respectively and maybe finish those!