If you weren’t already aware, the Linux community is currently up in a kerfluffle about Red Hat’s latest licensing changes. To summarize:

  • Red Hat makes an operating system Red Hat Enterprise Linux (RHEL), and until a week or two ago, also published the source code publicly in the spirit of free and open source software.
  • Red Hat makes money by selling RHEL subscriptions and support.
  • Throughout the years, other organizations have republished RHEL for free (CentOS, Oracle Linux, Alma Linux, Rocky Linux, etc.).
  • In an effort to kill the republished copies of RHEL, Red Hat will no longer publish their source code publicly in a way that’s easy to rebuild. Now they publish the source to CentOS Stream, a “testing OS” that is very similar, but not quite compatible with software built for RHEL.

Anyhow, the community is upset, because it means RHEL as a free and open-source product will likely no longer be available. Yes, you can still pay for RHEL. Yes, you can download 16 free subscriptions that you’re not allowed to use for anything that could possibly earn you money. Yes, you can still get a different free product with CentOS Stream. Yes, Red Hat’s changes might even be legal, even if they no longer follow the spirit of the GPL license that makes open-source software possible.

But this sucks.⌗

If you are a sysadmin, developer, or just anyone who works with computers professionally, you’re going to learn a lot of “stuff” over the course of your career. I’ll speak personally from my experience as a sysadmin here (call it “devops”, “sre”, “platform engineer”, or whatever the job title of the month is, it’s the same job). This “stuff” you need to learn includes operating systems (like RHEL), programming languages, infrastructure-as-code tools, and other more esoteric stuff like Kubernetes, cloud providers, databases, and more.

This is a lot of stuff to learn, but over the course of your career, you’ll build a “stack” of tools you’re intimately familiar with and can take with you between jobs. It’s a very empowering feeling, and all of these skills are basically your career and job mobility. You can take these skills anywhere to do to anything a computer can possibly do - for free!

As a sysadmin, pretty much the first step of building this stack is picking an operating system and becoming intimately familiar with it. Right now there are sort of 3 major Linux OS ecosystems that people choose from for work:

  • Red Hat land: RHEL, Fedora, and the RHEL clones (Alma Linux, Rocky Linux, Oracle Linux, CentOS Stream).
  • Debian and friends: Debian, Ubuntu, and Pop!_OS
  • SUSE: OpenSUSE Tumbleweed, OpenSUSE Leap, and SLES

Linux world map

There are a couple common elements here:

  • Each landmass (er, ecosystem?) has a fast moving, high-quality, desktop OS you can use on your laptop that’s similar to the ones you’ll use on servers. This isn’t an issue here so I’ll gloss over this.
  • Each OS “ecosystem” has one OS with long-term support (2 years or more). This is important for businesses because once you have a lot of servers, it’s just impractical to be upgrading the OS all the time. So long-term support is essential.
  • No licensing restrictions. All of these ecosystems have the ability for you to deploy as many copies of an OS as you want, for whatever use case you want, for free.

The last one is important, and is what’s at stake here in this case for RHEL and its related tooling. We need to be honest here - no one wants to pay for Linux itself. If we wanted to piss away hundreds of dollars a year for each server we’d all be using Windows.

Red Hat’s goal here is to convert “freeloaders” like myself (and bascially all the previous companies I’ve worked at) into paying customers by taking away the ability to use their ecosystem for free. Red Hat has tried to do damage control on all this and the response by their “core systems VP” just makes things look even worse:

[…] we have determined that there isn’t value in having a downstream rebuilder.

The generally accepted position that these free rebuilds are just funnels churning out RHEL experts and turning into sales just isn’t reality. I wish we lived in that world, but it’s not how it actually plays out. Instead, we’ve found a group of users, many of whom belong to large or very large IT organizations, that want the stability, lifecycle and hardware ecosystem of RHEL without having to actually support the maintainers, engineers, writers, and many more roles that create it. These users also have decided not to use one of the many other Linux distributions.

Red Hat believes that the existence of Alma Linux and Rocky Linux is cannibalizing sales of RHEL subscriptions. “Every Alma Linux and Rocky Linux install is a lost sale! Maybe if we destroyed all of the rebuilds, all of the people using them would buy RHEL instead? The community will be so eager to reward Red Hat by buying subscriptions to our products now that the alternatives don’t exist anymore, right?”

Wait - “buy subscriptions”?

“We’re pirates, we don’t even know what that means!”

Red Hat has completely missed why people use their software. No one cares about the support subscriptions. No one cares about “Red Hat Enterprise Linux” or “Red Hat” at all.

Red Hat’s actual product is its community.⌗

  • As a software developer (either free or paid), RHEL-based OSes are an attractive target to build for because it has a large number of users and businesses using it (read: potential customers). You could build for Ubuntu and RHEL and that basically covered 95% of Linux-based businesses in North America with money.
  • As an IT professional, the dominance of RHEL-based OSes meant that learning RHEL was a good investment of your time: the more businesses that used it, the more attractive you’d be to employers and you’d be able to start contributing to a company that used RHEL or a RHEL-clone on day one (instead of needing to learn a new OS every time).
  • As a business, selecting a RHEL-based OS was attractive: there was a large community which meant that most bugs got reported and fixed by community members before you ran into them. There was lots of free+paid software available with guides written by the community, so you didn’t have to install any other OSes just to install some weird piece of software you needed. RHEL skills were also relatively common, so you’d be able to hire knowledgeable people and spend less time training people from scratch. RHEL and its clones were sponsored by a company that appeared to be healthy and profitable so you knew the OS wasn’t going to suddenly implode and you’d have to spend effort jumping ship.
  • As for Red Hat and IBM itself, more users of free RHEL clones meant that there’d be more chances to sell Red Hat and IBM’s other, much more valuable software offerings. For instance, when I worked in supercomputing with Queen’s University and Compute Canada, we were all rabid CentOS users that saw absolutely zero value in RHEL, but we were more than happy to shell out hundreds of thousands of dollars each year for GPFS (now IBM Spectrum Scale), Tivoli Storage Manager (now IBM Spectrum Protect), and IBM+Lenovos’s servers and hardware support. IBM made so much money off of us as CentOS users they bought my boss and I a free vacation to Vegas one year to go to their conference and do lines with our account manager (that last part is a joke, my old boss and I aren’t cool enough to get invited to those kind of parties).

All of this community value from using RHEL is based on sheer numbers. The more users there are, the more developers will write software for it. The more software there is for RHEL, the more users there will be. The more stable this community appears, the more likely businesses and professionals will invest in it and stay long-term. The larger the community, the more chance for Red Hat (and IBM) to sell whatever products they had. All of the other work with RHEL that Red Hat claims is just so valuable is just a bonus. Ubuntu, SUSE, Debian, and Amazon (yes, Amazon) tick all the same checkboxes that RHEL does- the biggest factor keeping customers in the RHEL ecosystem is the community that’s sprung up around it.

The “community as the product” is particularly evident with another Red Hat product: Ansible. Ansible is an automation tool that’s commonly used to automatically configure servers and perform common operations tasks. Though the Ansible software itself is nifty and you can do a mind-blowing amount of shit with it, the real value of using Ansible is actually its community, specifically community-generated “Ansible roles”. For the unintiated, Ansible roles are neat little self-contained bundles of Ansible code that setup a server to do advanced things without you actually needing to know the specifics of how to do these things yourself.

By using Ansible, you get to use all this work these people are giving back to Red Hat and the community for free. If this community disappeared, the entire point of using Ansible (and Ansible’s commercial value as a product) would disappear overnight. And that’s what’s happening right now.

For better or worse, many FOSS software ecosystems have a cult of personality that revolves around a single, ultra-productive community member. For instance Linus Torvalds basically was Linux incarnate for several decades. The R programming language revolves around Hadley Wickham. He wrote so many amazing R packages it created an entire data science ecosystem called the “Hadleyverse” (he asked to change the name to “tidyverse” because he wanted to be modest). Ansible has one of these people too: Jeff Geerling - also known as “geerlingguy”.

geerlingguy is Ansible. I would conservatively estimate that >50% of the good (as in, you’d actually want to use these instead of writing your own) Ansible roles on Ansible Galaxy are written by him directly. He literally wrote the book on Ansible. I’ve even used “Do you know who geerlingguy is?” as an interview question - if someone doesn’t know who he is, it’s obvious they’ve never spent any serious time with Ansible (this question also absolutely fucks with people trying to use ChatGPT and read off the screen to fake it during job interviews. Yes… we’re on to you.).

Not only have Red Hat’s latest moves alienated their largest contributor, he’s gone scorched-earth and begun actively removing support for RHEL from all of his Ansible roles. Jeff Geerling is now advertising on Twitter about just how easily you can use Red Hat’s own Ansible product to migrate off of RHEL using the cross-platform tooling he’s written (I’d link to the tweet, but Twitter is offline). RHEL’s vendor lock-in isn’t an issue when you have automated tools like Ansible to reproduce your RHEL servers on another distribution like Debian in a matter of minutes.

This is a disaster for Red Hat. And its not an isolated incident: EPEL maintainers are leaving in droves. The last comment is particularly telling in regards to how most contributors to Red Hat’s products are feeling right now:

Why should we keep contributing to EPEL? To be forced to use 16 free RHEL instances maximum? What is the advantage for us volunteer contributors? I mean, we did not do it for personal advantage, we did it to help us each other within the Enterprise Linux distros community, but this Red Hat move will kill the Enterprise Linux distros community, leaving only with RHEL, which is mostly a paid subscription distribution, let’s call things with their proper name

These free EPEL contributors are essential to Red Hat’s business model. Without them, Red Hat would lose a majority of the software it needs to compete with Ubuntu and Debian’s massive software catalog. Despite being a multi-billion dollar corporation, Red Hat has never had the resources to maintain all of this software by itself. In an unrelated incident, RHEL also failed to retain it’s LibreOffice maintainer, and will stop shipping LibreOffice as a result. This leaves RHEL somewhat perplexingly as the only “enterprise” Linux distribution without an enterprise office suite. Lennart Poettering, another hyper-productive developer (responsible for PulseAudio and systemd) actually left Red Hat last year to go work at Microsoft. Red Hat even fired the Fedora Program Manager who manages the upstream Fedora distribution that Red Hat repackages and rebrands as RHEL itself. Red Hat’s most valuable staff and community contributors are either being fired or jumping ship.

Intentionally or not, Red Hat seems to be doing everything it can to destroy the community that makes RHEL a product you’d want to consider purchasing in the first place.

A failure to monetize, and having your skillset put behind a paywall⌗

Let’s pretend that Red Hat is somehow successful in killing all of its downstream rebuilders (Alma Linux, Oracle Linux, Rocky Linux, etc.). RHEL has an extremely customer-hostile monetization scheme:

  • You can use it for free if you’re learning it.
  • As soon as you want to use it for anything commercial, Red Hat wants an unjustifiably high licensing fee. Unlike with Ubuntu Pro, where you can selectively choose to buy support for key systems, without a free RHEL-clone available you’d need to pay a subscription fee for every single production system.
  • As a software vendor, if your stuff only runs on RHEL, then your customers are forced to pay an extra fee for the OS as well, which makes your product less competitive.
  • Even if you can stomach the licensing fees, there is no way to convert from a free install (CentOS Stream, Alma Linux, Rocky Linux, etc.) to a paid install without reinstalling the OS. So not only do you have to pay Red Hat tons of money, you also get the joy of reinstalling the operating system on every single machine you have.
  • If you switch companies and the new company doesn’t want to use RHEL, you are out of the ecosystem permanently. There won’t be a way to onboard your new company into the RHEL ecosystem for free anymore and using RHEL itself will be a very hard sell (see above).
  • RHEL professionals will “auto-convert” from using RHEL to something else at an extremely high rate because having your career be held hostage to a yearly subscription just isn’t a very empowering feeling.

Speaking for myself, the last factor is the most significant. The people responsible for advocating buying Red Hat and IBM’s RHEL-based products in the first place are being alienated because it feels like their skills with those products are getting put behind a paywall. One of the attractive things about building a career with RHEL-based OSes until now has been that you can pick up and take your skills anywhere, for free. The latest moves to kill off the RHEL-downstream OSes makes it feel like an important part of your skillset is getting put behind a paywall.

Let’s sum things up:⌗

The main reason to use RHEL-based Linux these days is because of the really great community. Most of RHEL’s software and useful tooling comes from free labor by the community. The latest licensing changes are designed to monetize every cent they can, driving away all of this “community value” that makes RHEL an attractive product in the first place.

“The more you tighten your grip, the more subscriptions will slip through your fingers”

Like many other companies before it, Red Hat seems to have entered the “enshittification” death-spiral:

Here is how platforms die: First, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die.

Red Hat hasn’t yet successfully killed its RHEL-clone downstreams, but the writing seems to be on the wall. There is a bad actor at the very core of the Red Hat ecosystem: Red Hat itself. There doesn’t seem to be a long-term future for the Red Hat community and RHEL now that the “enshittification” process is in full swing (we are just starting the “abuse the business customers” stage- Red Hat can’t put the squeeze on them if there are easy alternatives). It seems like the future will just be many years of slowly increasing RHEL license fees while people leave and the product gets worse and worse.

Why stay?⌗

When I originally wrote this article, I was really irritated by Red Hat’s decision to try to kill CentOS a second time. It was important to take a step back, “touch grass” as they say, and think about why I felt this way: it’s just an operating system… why am I so upset about this that I would type all of this out? (I don’t even depend on RHEL or RHEL-clones for work anymore.) I think why I was so upset is that the best part about Linux is that it’s just a giant community of people who help each other and try to make the world a better place (or at least the world of computing)- for free! Do we have to monetize this to death? Does everything have to end in a profit-seeking death spiral?

Anyhow, I guess this article is basically just a really roundabout way of saying I’m dropping official support for RHEL in the software I write. Any continued support is just a happy coincidence of the fact that SUSE and Fedora share the same RPM build toolchains. I won’t pretend that I’m an important community member or my contributions are so valuable that Red Hat will go under without me, but it’s just not worth putting in free labor to support a yet another company who is doing everything possible to use everyone else’s work and give nothing back.

“This effort is no longer profitable!”