Hey guys, this is my third blog post this month (quite the new record!)

I’ve been sitting around this holiday season using all my PTO that won’t roll over, and I forgot what it was like to have so much time and nothing to do. Recently I’ve been getting more antsy, and decided it was time to mess around with my computer again, and hack up some weird stuff.

Since I’m on an immutable system, a fact that I always manage to bring up in every blog post, I wanted to focus on my developer pain-points. My biggest gripe with my system is that VSCode only can reasonably be ran in a Toolbx or Distrobox container. Running VSCode in a flatpak is unfortunately not really sexy.

Image of VSCode running in a flatpak

It still looks pretty sexy though! (Catppuccin my beloved)

I don’t think this is something that can be fundamentally solved, because of the way Flatpak is structured. The /usr directory is for the runtime, the /app directory is for the app, and some trickery occurs to make both those directories usable for binaries and libraries. The host’s /usr is never used, however you can expose it using the --filesystem=host, or the more recently added --filesystem=host-root (you’re welcome). When exposing it like this, you are actually mounting it to /run/host/bin or /run/host/root/usr/bin, so it’s not super helpful anyways. The lack of host binaries is actually a great thing for most apps, and even IDEs can thrive without the host’s /usr. I do believe though that the dev experience without a mutable /usr is really rough.

Developers are installing countless language servers, and libraries needed to develop their software. Oftentimes, in the immutable world, we just use containers, and I think devcontainers are just okay (I could rant about devcontainers too, and how they’re the only useful way to interact with containers through VSCode and it shouldn’t be that way, but I’ll spare anyone reading this). The alternative to devcontainers is running VSCode from within the container itself, but I find myself yearning for a more integrated experience than something like Distrobox/Toolbx. Flatpak is really well integrated with the host system, BUT suffers from a lack of mutability (which isn’t a total bad thing, I’d rather not accidentally remove all of the flatpak runtime from within the container with a misplaced rm -rf…)

I’m no stranger to cursed flatpak experiments at this point…

Last time I fiddled with flatpak, I tried to put dnf in a flatpak and later settled on homebrew instead, as putting a package manager in a flatpak pretty much requires /usr to be mutable. You can find that blog post here.

I recently did some work on my old project Flatrun in a branch. This project lets you run flatpaks without explicitly installing them to a system or user installation. I think this stuff could be integrated into upstream flatpak, but I didn’t see the point at the time with messing with C, as Rust is much nicer to use.

Today though, I was feeling like adding something to Flatpak itself, which is why I thought of the fact that I wasn’t able to do a few things in Flatpak that I’d like to do:

  • I can’t mount folders from the host system into any arbitrary location in the container. For example, in podman, you can use --volume=/home/rbrue/Documents:/var/lib/host-documents:ro to mount my Documents folder read-only to /var/lib/host-documents in the container.
  • I can’t make /usr or /app mutable (it’s debatable whether this is actually useful, but I wanted to experiment)

I realized that the way to make both of these things possible would be to allow flatpak run to take in additional bubblewrap options to pass to bwrap.

NOTE: Quick overview on Bubblewrap, it’s a userspace sandboxing program that Flatpak relies on for constructing the sandbox for applications.

Obviously, this kind of permission should only be used for debugging purposes, as providing access to this kind of thing would make static flatpak permissions obsolete, and also open up applications to request permissions that look much more confusing to an end user.

Making the change in Flatpak actually wasn’t that hard, it just involved making a --bwrap-arg argument that takes the list of what’s added to it and appends it to the end of the bwrap invocation.

You can find my PR to Flatpak to add this here.

Now that this is implemented, let’s try to make /usr mutable.

Making /usr mutable

Here are the steps I took to make /usr mutable using an overlayfs. Thanks to the person who implemented overlay functionality in bubblewrap (Oh look, it’s another Ryan, kudos!)

NOTE: For OverlayFS, there are 3 main directories, the src (the read-only base, in this case our runtime), the rwsrc (where changes are saved to), and the workdir (required for overlayfs to process files).

  • Create a rwsrc and workdir directory. It doesn’t matter where, but they should be on the same filesystem, due to how OverlayFS works.

  • Find the directory that is usually mounted to /usr in the flatpak. For the Freedesktop Platform, which I tested on, I found it at /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/24.08/5365bb0e377aa4dc0ebb3ddbc5af846eb1cfa9222146754997d615b6c8c63f0b/files, but this path will vary by commit and version.

  • Copy the directory structure of the runtime into rwsrc. This is necessary because OverlayFS doesn’t let you create files in directories that only exist in the read-only src and not the rwsrc.

    rsync -a --include="*/" --exclude="*" /var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/24.08/5365bb0e377aa4dc0ebb3ddbc5af846eb1cfa9222146754997d615b6c8c63f0b/files/ ./rwsrc/

  • Using a Flatpak binary built the patch I posted above, run your desired Flatpak app like so:

    flatpak run --command="bash" --bwrap-arg=--overlay-src --bwrap-arg=/var/lib/flatpak/runtime/org.freedesktop.Platform/x86_64/24.08/5365bb0e377aa4dc0ebb3ddbc5af846eb1cfa9222146754997d615b6c8c63f0b/files --bwrap-arg=--overlay --bwrap-arg=$(pwd)/rwsrc --bwrap-arg=$(pwd)/workdir --bwrap-arg=/usr org.freedesktop.Platform

    (Boy that’s a mouthful)

The way I implemented the --bwrap-arg argument was to just collect them into an array of strings based on each --bwrap-arg provided. This means that you cannot space-separate arguments and expect that to be passed in correctly to bwrap, you have to use a new --bwrap-arg for each part of the argument, which is why that command has several --bwrap-arg uses.

Since the custom bwrap args are appended at the end, it overrides the mounting of the runtime RO into /usr, and the OverlayFS we define is used instead.

Here’s a video of it in action (Mastodon Post):

I don’t really know if this is super useful or not, but I really love messing around with Flatpak in crazy ways. I think Flatpak is one of the best things to happen to the Linux desktop, and I like poking at it to try and solve pain points for the <5% of graphical apps that require special care under a Flatpak sandbox.

That’s all for now :)