Note: This is yet another post made with the Fresh editor, over SSH, into my server ;)

Hey everyone, it’s long overdue for me to be talking about COSMIC again!

This blog post will be an overview of how I maintain the Fedora COSMIC spin, and the workflows I have built up over the course of maintaining an entire DE on a distro.

For those of you who don’t know, I maintain all the COSMIC packages on the Fedora distro – I handle updates and patches for all the packages whenever a new tag comes out. Fedora is one of the most complex distros when it comes to tooling, so hopefully this post will give you an idea of the workflow that I’ve made for myself over the last 2 years of working on Fedora + COSMIC!

1. The cosmic-packaging repository

The cosmic-packaging repository contains about 90% of the build tooling I use when managing package upgrades and the nightly repository (I’ll talk more about that later). It’s also where I did the first hacking on COSMIC packages back in the day, before Fedora COSMIC was a spin!

The important bits of the modern repo structure for the maintenance workflow are the patches directory, where patches exist that will be upstreamed in the next tagged release, and the scripts directory, which is the meat of the repo, where all the scripts that are used to build and update are located.

- patches:
  - cosmic-app-library:
    - 0001-fix-memory-leak.patch (Just an example)
    - ...
  - ...
- scripts: The meat of the repo, where all the scripts that are used to build and update are located.
  - bootstrap.sh
  - cosmic-packaging-bootstrap.py
  - ...

2. COPR builds

I maintain both a Nightly and Tagged COPR for COSMIC packages. Both of them use the same scripts, just with different command line arguments pertaining to a -git release and a release targeting a tag.

Each copr package runs a copy of the bootstrap script, which clones the cosmic-packaging repo and the upstream RPM spec repo (from src.fedoraproject.org/rpms/{package}), then runs the cosmic-packaging-bootstrap python script. This script will handle vendoring rust dependencies, applying patches from the patches directory, and more.

Once the directory is set up, Mock will create an SRPM, then build it, all online in the COPR. This part we don’t have to worry about, as long as we have everything set up properly.

If the package fails to build, I inspect the build logs and try to find the issue. Oftentimes, issues will require a new patch, so I will go in, add patches to cosmic-packaging, push, and rebuild.

If the package succeeds, voila! We have a built COPR package that users can download. For the nightly repos, I have a GitHub Action that will rebuild the packages twice a day, if the latest commit of the COSMIC repo doesn’t match the latest built. This check is important to keep the COPR gods happy – I don’t want to be rebuilding packages that already have the latest version! For the most part, I don’t have to check on the nightly builds, as I will be fixing any issues with them implicitly when I fix the tagged releases. From my understanding, a lot of people still use the nightly COPR, which is what it’s for. I’m glad people want the latest and greatest, and it doesn’t require a lot of maintenance on my part. The time-consuming part is the actual upstreaming of the packages to Fedora…

3. Upstreaming to Fedora

NOTE: I am not covering what it takes to get a package accepted into Fedora in this blog post, maybe something for a future post, but it took months of work on my part and the Packaging Guidelines are comprehensive and strict. Be warned if you want to package your favorite apps in Fedora – it’s not easy!

NOTE #2: I only package tagged releases once every other week due to life being busy :p

When a new COSMIC update gets tagged, for example, epoch-1.0.8, this is my packaging workflow:

1. I trigger a full rebuild of all the packages on the cosmic-epoch-tagged COPR. (The nightly COPR gets updated independently and continuously.)
2. I monitor the builds over the next day or two (or even three sometimes, COPR can be pretty slow!).
3. I make sure that all the builds have succeeded on all of the architectures Fedora supports (x86_64, aarch64, s390x, ppc64le). This can be painful, as s390x and ppc64le frequently fail. I’d say there’s an 80% chance every update for those to fail for whatever reason.
4. I make changes as needed if builds fail. Rebuild the failed builds, and repeat steps 2 and 3.
5. Once all the builds are succeeded, I run the cosmic-packaging-new-release.py script from the cosmic-packaging repo. This script downloads the latest succeeded build from the tagged COPR, imports them into the corresponding Fedora repo (on src.fedoraproject.org), and queues a build for every branch we support (I try to support basically everything that isn’t EOL).
6. Monitor all the builds in Fedora Koji
7. If any of the builds fail, or the initial setup for the build fails (which happens often), rebuild. We know at this point that builds can succeed, thus we don’t need to check for patches. Most of the time it’s a one-off build problem with the exotic arches mentioned earlier.
8. Once all the builds succeed, create the update in Fedora Bodhi.
9. Post the update links in the Fedora COSMIC SIG chat, for review by the users.

This workflow takes anywhere from 2 days to 1 week depending on whether I’m lucky with things “just working(tm)”.

Compared to just building nightly packages in COPR, the tagged releases take plenty of time to work out. I have a decent system, and it can’t be fully automated in part due to Fedora’s guidelines on releases, so this is about the best I can do.

4. “How can I help with Fedora COSMIC?”

So glad you asked! ;)

The biggest bottleneck these days when doing package maintenance is finding out what is wrong with failed builds in the COPR. If you have packaging experience, rust experience, or COSMIC experience, any help would be appreciated when it comes time to package a new tag! The Fedora COSMIC people hang out in the Fedora COSMIC SIG chat so feel free to join even if you just want to lurk.

Since I am a lot busier than when I started the Fedora COSMIC spin (I was a graduate student at the time and that was surprisingly less busy than the real world), I greatly appreciate those in the SIG chat who find issues ahead of me getting around to researching problems.

If you are a seasoned packager, or a COSMIC fan with some dev experience, also consider looking at the COSMIC spec files at src.fedoraproject.org/rpms/{package_name}, and suggesting fixes where relevant.

If none of those avenues to help apply to you, that’s ok! Bug reports are helpful, here at Fedora or upstream.