How (Not) to Use CoreOS

Introduction

TL;DR: don't do this if you just want a home server

The main idea here is not about CoreOS, but rather about Bootable Container Images. The basic idea of bootc that it uses container tooling to create custom OS images. Which means that you write a Containerfile like you would for a docker image and use dnf to install packages and build, etc., with the resulting image having a kernel that can be directly booted on bare metal or virtual machine. CoreOS is now also an OCI container image that is compatible with bootc, which means you can use CoreOS as the base image.

Difference between bootc and CoreOS

The most important differences between bootc and CoreOS are that CoreOS supports a first boot installer and configuration tool called ignition.

bootc is versioned after the Fedora base, while CoreOS has a rolling upgrade model.

Also, CoreOS does not ship with python, which means bootc-image-builder will fail building an image that does not have python, see this issue.

Ignition

Ignition provide a powerful configuration mechanism that uses a JSON file as the single source of truth to describe the state of the machine.

Admittedly it somewhat conflicts with bootc's model where you use a Containerfile to describe the state of the system, but Ignition performs a series of initilization during initramfs, particularly manipulating disks.

Ignition will only run on the first boot of the system, this is achieved thanks to various systemd components such as systemd-repart.

You use Butane which is a YAML to JSON conversion tool to write ignition.

Yes...but why?

So, this is all great and new and exciting for DevOPs and ITs, but is there any reason you want to use it for your little home server? While you lose much of the benefit when you only have one machine to play with, there are a few things that might interest you:

• Immutability: by design, CoreOS and bootc have the sysroot being readonly, the only writable locations are /etc and /var. This allows atomic upgrade and rollback possible. When you upgrade the system to a new version of image, you have the ability to rollback to the previous version safely. This is the biggest (perhaps the only) selling point of Fedora Atomic Desktop series, because users can never mess around in critical part of the system and causes irreversible damage. This is known as hermetic /usr. This is similar to using non-root package manager like homebrew to keep the system untouched. NixOS and SteamOS also uses a similar approach. As a side note, Fedora is trying for the "reproducible build" NixOS provides as well, which when done means that the whole system would be completely reproducible.
• You can set up automatic update because of the safe rollback mechanism.
• It forces you to gather your setup into one central location, after which you can easily reproduce it on any machine.

How to do it

• First, you want to build a container image that overlays the additional packages you want to use, particulaly those that require a kernel module, such as zfs, VirtualBox, or nvidia drivers. Other things that are missing from CoreOS you might want are firewalld, wireguard, nfs, and samba. There are examples at layering-examples for building kernel modules and installing them.
• Then, you can write a butane config that summarizes the state of the system, such as users, storage, and container files.
• Finally, you do not want to install the image you built directly if you built over CoreOS. Instead, you install CoreOS with the ignition file, but include a systemd unit that rebases to your system image on first boot. For example:

variant: fcos
version: 1.4.0
systemd:
  units:
    # Our custom unit for rebasing
    - name: rebase-custom.service
      enabled: true
      contents: |
        [Unit]
        Description=Fetch and deploy target image
        # Only run on the firstboot
        ConditionFirstBoot=true
        After=network-online.target
        [Service]
        # This ordering is important
        After=ignition-firstboot-complete.service
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=rpm-ostree rebase --reboot ostree-unverified-registry:quay.io/example/example-derived:latest
        [Install]
        WantedBy=multi-user.target

This is necessary because... I have no idea why. Bootc does not want to support ignition directly as it is out of scope, and coreos-installer won't apply ignition correctly on a raw disk image produced with bootc-image-builder. So the workflow podman build -> bootc-image-builder --type raw -> coreos-installer does not work.

Actually, there are projects that are building CoreOS based images already, notably ublue-os/ucore.

Problems, yes, we have some already

The first thing is there are secrets that you might not want to include in the Containerfile or ignition and upload them to a public GitHub repo. For example private keys in your container systemd units, your personal domain, or even your SSH public keys, which is technically safe but can be used to track your identity. As a solution, you could use systemd credential to manage secrets. Currently, there are some problems with rootless containers if you try to use this setup with systemd version less than 258, see this issue and this Reddit post.

The projects are designed such that you host your image on a remote registry. In a large organization this makes sense, but this makes a single machine bootstraping itself difficult. You could of course host your image on a free container registry such as building and hosting on Github CI, but then you cannot mitigate the above mentioned problem. Basically, it is hard to install the system without the assist of a second server.

You could, still, install all subsequent updates to the system locally, by just building the new image locally, and rebase using the local container storage.

Conclusion

Unless you are extremely paranoid about server maintaience, this might an over kill. However, it does force you to review the security of all the service components on your server and I learned a lot during the process.