alt text

Hey everyone,

This is another update about the development of LXC 3.0.

As of yesterday the cgmanager and cgfs cgroup drivers have been removed from the codebase. In the good long tradition of all LXC projects to try our hardest to never regress our users and to clearly communicate invasive changes I wanted to take the time to explain why these two cgroup drivers have been removed in a little more detail.

CGManager

The CGManager cgroup driver relies on the upstream CGManager project which was created and written by fellow LXC and LXD maintainer Serge Hallyn back in late 2013.

The need for CGManager has been fading over the years as its main features can now be achieved in more standard and efficient ways:

  • Allowing nested containers to control their own cgroups.
  • Enabling cgroup management for unprivileged users running unprivileged containers, i.e. containers employing user namespaces and idmappings.

A first effort to deprecate CGManager happened with the inclusion of the new cgfsng cgroup driver in LXC combined with LXCFS support for creating a per-container cgroup view in userspace.

The LXCFS approach had the benefit of working with all existing software that would normally interact with cgroups through the filesystem and was also more efficient (multi-threaded) compared to the single-threaded DBUS API that CGManager was offering.

The later inclusion of the cgroup namespace in the mainline kernel finally moved all of this into the kernel, completely removing the need for a userspace solution to the problem.

CGManager itself is currently considered as deprecated and will not see any further release and so there is little point in LXC keeping support for it.

cgfs

The cgfs driver dates back from the origins of the cgroup subsystem and early integration in Linux distributions.

At that point in time, it was somewhat common for cgroup controllers to all be co-mounted or be co-mounted in big chunks, often under /dev/cgroup or /cgroup.

LXC therefore needed a lot of logic to figure out exactly what cgroup controller could be found and where. It also had to enable a number of different flags to have the then widely different controllers behave in a similar way.

Nowadays, all Linux distributions that setup cgroups will mount a split layout, typically with one controller per directory under /sys/fs/cgroup. LXC can rely on this and so knows exactly where to find all cgroup controllers without having to do complex mount table parsing and guesses.

That’s what the cgfsng driver, introduced in LXC 2.0, does with the old cgfs driver only there as a fallback. We’ve very rarely witnessed that LXC fell back to the cgfs driver and if it did it usually resulted in another failure.

As the cgfs driver is old, complex and hard to maintain and doesn’t seem to actually be handling any real world use cases, it will similarly be dropped.

More General Reasons

These are some arguments that apply to both cgroup drivers and to coding in general:

  • code blindness

    This is a general phenomenon that comes in two forms. Most developers will know what I’m talking about. It either means one has stared at a codepath for too long to really see problems anymore or it means that one has literally forgotten that a codepath actually exists. The latter is what I fear would happen with the cgfs driver. One day, I’d be looking at that code muttering “I have no memory of this place.”.

  • forgotten logic

    Reasoning about fallback/legacy codepaths is not a thing developers will have to do often. This makes it less likely for them to figure out bugs quickly. This is frustrating to users but also to developers since it increases maintenance costs significantly.

  • a well of bugs

    Legacy codepaths are a good source for bugs. This is especially true for the cgroup codepaths because the rise of the unified cgroup hierarchy changes things significantly. A lot of assumptions about cgroup management are changing and updating all three cgroup drivers would be a massive undertaking and actually pointless.

Adding New cgroup Drivers

This is a note to developers more than to users. The current cgfsng cgroup driver is not the last word. It has been adapted to be compatible with legacy cgroup hierarchies and the unified cgroup hierarchies and actually also supports hybrid cgroup layouts where some controllers are mounted into separate legacy cgroup hierarchies and others are present in the unified cgroup hierarchy. But if there is a legitimate need for someone to come up with a different cgroup driver they should be aware that the way the LXC cgroup drivers are written is modular. In essence it is close to what some languages would call an “interface”. New cgroup drivers just need to implement it. :)

Removing code that has been around and was maintained for a long time is of course hard since a lot of effort and thought has gone into it and one needs to be especially careful to not regress users that still rely on such codepaths quite heavily. But mostly it is a sign of a healthy project: within a week LXC got rid of 4,479 lines of code.

At the end, I want to say thank you to my fellow maintainers Stéphane Graber (@stgraber) and Serge Hallyn (@sehh) for all the reviews, merges, ideas, and constructive criticism over all those months. And of course, thanks to all the various contributors be it from companies like Huawei, Nvidia, Red Hat or individual contributors sending fixes all over the place. We greatly appreciate this! Keep the patches coming.

Christian & Stéphane