On The Way To LXC 3.0: Removal of cgmanager And cgfs cgroup Drivers
This is another update about the development of
As of yesterday the
cgfs cgroup drivers have been
removed from the codebase. In the good long tradition of all
projects to try our hardest to never regress our users and to clearly
communicate invasive changes I wanted to take the time to explain why
these two cgroup drivers have been removed in a little more detail.
CGManager cgroup driver relies on the upstream
CGManager project which
was created and written by fellow
LXD maintainer Serge
Hallyn back in late 2013.
The need for
CGManager has been fading over the years as its main
features can now be achieved in more standard and efficient ways:
- Allowing nested containers to control their own cgroups.
- Enabling cgroup management for unprivileged users running unprivileged containers, i.e. containers employing user namespaces and idmappings.
A first effort to deprecate CGManager happened with the inclusion of the
cgfsng cgroup driver in LXC combined with
LXCFS support for
creating a per-container cgroup view in userspace.
LXCFS approach had the benefit of working with all existing software
that would normally interact with cgroups through the filesystem and was
also more efficient (multi-threaded) compared to the single-threaded
DBUS API that
CGManager was offering.
The later inclusion of the cgroup namespace in the mainline kernel finally moved all of this into the kernel, completely removing the need for a userspace solution to the problem.
CGManager itself is currently considered as deprecated and will not see
any further release and so there is little point in
LXC keeping support
cgfs driver dates back from the origins of the cgroup subsystem
and early integration in Linux distributions.
At that point in time, it was somewhat common for cgroup controllers to
all be co-mounted or be co-mounted in big chunks, often under
LXC therefore needed a lot of logic to figure out exactly what cgroup
controller could be found and where. It also had to enable a number of
different flags to have the then widely different controllers behave in
a similar way.
Nowadays, all Linux distributions that setup cgroups will mount a split
layout, typically with one controller per directory under
LXC can rely on this and so knows exactly where to find
all cgroup controllers without having to do complex mount table parsing
That’s what the
cgfsng driver, introduced in
LXC 2.0, does with the old
cgfs driver only there as a fallback. We’ve very rarely witnessed that
LXC fell back to the
cgfs driver and if it did it usually resulted in
cgfs driver is old, complex and hard to maintain and doesn’t
seem to actually be handling any real world use cases, it will similarly
More General Reasons
These are some arguments that apply to both cgroup drivers and to coding in general:
This is a general phenomenon that comes in two forms. Most developers will know what I’m talking about. It either means one has stared at a codepath for too long to really see problems anymore or it means that one has literally forgotten that a codepath actually exists. The latter is what I fear would happen with the
cgfsdriver. One day, I’d be looking at that code muttering “I have no memory of this place.”.
Reasoning about fallback/legacy codepaths is not a thing developers will have to do often. This makes it less likely for them to figure out bugs quickly. This is frustrating to users but also to developers since it increases maintenance costs significantly.
a well of bugs
Legacy codepaths are a good source for bugs. This is especially true for the cgroup codepaths because the rise of the unified cgroup hierarchy changes things significantly. A lot of assumptions about cgroup management are changing and updating all three cgroup drivers would be a massive undertaking and actually pointless.
Adding New cgroup Drivers
This is a note to developers more than to users. The current
driver is not
the last word. It has been adapted to be compatible with legacy cgroup
hierarchies and the unified cgroup hierarchies and actually also supports
hybrid cgroup layouts where some controllers are mounted into separate legacy
cgroup hierarchies and others are present in the unified cgroup hierarchy. But
if there is a legitimate need for someone to come up with a different cgroup
driver they should be aware that the way the
LXC cgroup drivers are written
is modular. In essence it is close to what some languages would call an
“interface”. New cgroup drivers just need to implement it. :)
Removing code that has been around and was maintained for a long time is
of course hard since a lot of effort and thought has gone into it and
one needs to be especially careful to not regress users that still rely
on such codepaths quite heavily. But mostly it is a sign of a healthy
project: within a week
LXC got rid of
4,479 lines of code.
At the end, I want to say thank you to my fellow maintainers Stéphane Graber (@stgraber) and Serge Hallyn (@sehh) for all the reviews, merges, ideas, and constructive criticism over all those months. And of course, thanks to all the various contributors be it from companies like Huawei, Nvidia, Red Hat or individual contributors sending fixes all over the place. We greatly appreciate this! Keep the patches coming.
Christian & Stéphane