| Message ID | 20191111083348.3964650-1-christian@paral.in |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [1/1] package/systemd: hybrid cgroupfs hierarchy for docker compatibility | expand |
On Mon, Nov 11, 2019 at 1:33 AM Christian Stewart <christian@paral.in> wrote: > > Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243. > According to the systemd documentation: > > systemd now defaults to the "unified" cgroup hierarchy setup during > build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default. > Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream > production distributions might want to continue to use > -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately > the popular container managers have not caught up with the kernel API changes. > > Changing this option to "hybrid" or "legacy" fixes the Docker startup. > > Reference: https://github.com/opencontainers/runc/issues/654 > > Signed-off-by: Christian Stewart <christian@paral.in> > --- > package/systemd/systemd.mk | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk > index 94d5f703cd..92490eb86b 100644 > --- a/package/systemd/systemd.mk > +++ b/package/systemd/systemd.mk > @@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \ > -Dima=false \ > -Dldconfig=false \ > -Ddefault-dnssec=no \ > + -Ddefault-hierarchy=hybrid \ This should probably be conditionally enabled only when packages that don't support the unified cgroup hierarchy are enabled. > -Dtests=false \ > -Dsplit-bin=true \ > -Dsplit-usr=false \ > -- > 2.24.0 >
Keeping a list like that is going to be tricky... Let's just hope the pressure on docker is strong enough and in a version or two we can move to unified also: Tested-by: Jérémy Rosen <jeremy.rosen@smile.fr> Le lun. 11 nov. 2019 à 10:05, James Hilliard <james.hilliard1@gmail.com> a écrit : > On Mon, Nov 11, 2019 at 1:33 AM Christian Stewart <christian@paral.in> > wrote: > > > > Docker fails to start with "Devices cgroup isn't mounted" as of systemd > 243. > > According to the systemd documentation: > > > > systemd now defaults to the "unified" cgroup hierarchy setup during > > build-time, i.e. -Ddefault-hierarchy=unified is now the build-time > default. > > Previously, -Ddefault-hierarchy=hybrid was the default. [...] > Downstream > > production distributions might want to continue to use > > -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as > unfortunately > > the popular container managers have not caught up with the kernel API > changes. > > > > Changing this option to "hybrid" or "legacy" fixes the Docker startup. > > > > Reference: https://github.com/opencontainers/runc/issues/654 > > > > Signed-off-by: Christian Stewart <christian@paral.in> > > --- > > package/systemd/systemd.mk | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk > > index 94d5f703cd..92490eb86b 100644 > > --- a/package/systemd/systemd.mk > > +++ b/package/systemd/systemd.mk > > @@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \ > > -Dima=false \ > > -Dldconfig=false \ > > -Ddefault-dnssec=no \ > > + -Ddefault-hierarchy=hybrid \ > This should probably be conditionally enabled only when packages that > don't support the unified cgroup hierarchy are enabled. > > -Dtests=false \ > > -Dsplit-bin=true \ > > -Dsplit-usr=false \ > > -- > > 2.24.0 > > >
Hello,
On Mon, 11 Nov 2019 11:09:27 +0100
Jérémy ROSEN <jeremy.rosen@smile.fr> wrote:
> Keeping a list like that is going to be tricky...
You don't really have to keep a list. You can have a blind option:
config BR2_PACKAGE_SYSTEMD_NEEDS_HYBRID_HIERARCHY
bool
systemd.mk would observe this boolean to enable (or not) the hybrid
hierarchy, and packages that need it would select this boolean.
That being said, is there any drawback in unconditionally using the
hybrid hierarchy? James, since you suggested to make it conditional, do
you have some arguments/feedback?
Thanks,
Thomas
It's about what implementation of cgroup is used by systemd to maintain the cgroup-tree of processes * legacy is v1 * hybrid is both v1 and v2 * unified is v2 only (note that only the default is changed, this can be changed via the kernel command-line at boot time) v1 is deprecated on the kernel side because it was... a mess. The systemd people are migrating stuff from v1 to v2 as new controllers are added to v2, but at this point there is little left that is v1 only thus the change of the default value Very few user-space projects care about cgroups : mainly systemd and container-managers. Container-manager need to interact with systemd as described in https://systemd.io/CGROUP_DELEGATION.html which allows to drop v1 and to correctly spread the responsibility between systemd and the container managers without stepping on each other's toes. Also note that Fedora31 will use the unified cgroup hierarchy. A good summary of adoption status : https://medium.com/nttlabs/cgroup-v2-596d035be4d7 At that point, I think staying in hybrid mode is the safest bet : everything in user-space hasn't migrated yet. Let Fedora take the fire and force everybody to migrate, and let's revisit the subject once (at least) Docker has migrated... Le lun. 11 nov. 2019 à 15:39, Thomas Petazzoni <thomas.petazzoni@bootlin.com> a écrit : > Hello, > > On Mon, 11 Nov 2019 11:09:27 +0100 > Jérémy ROSEN <jeremy.rosen@smile.fr> wrote: > > > Keeping a list like that is going to be tricky... > > You don't really have to keep a list. You can have a blind option: > > config BR2_PACKAGE_SYSTEMD_NEEDS_HYBRID_HIERARCHY > bool > > systemd.mk would observe this boolean to enable (or not) the hybrid > hierarchy, and packages that need it would select this boolean. > > That being said, is there any drawback in unconditionally using the > hybrid hierarchy? James, since you suggested to make it conditional, do > you have some arguments/feedback? > > Thanks, > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com >
Hello, (If possible, you should bottom post instead of top post. Thanks!) On Mon, 11 Nov 2019 15:56:04 +0100 Jérémy ROSEN <jeremy.rosen@smile.fr> wrote: > It's about what implementation of cgroup is used by systemd to maintain the > cgroup-tree of processes > > * legacy is v1 > * hybrid is both v1 and v2 > * unified is v2 only > (note that only the default is changed, this can be changed via the kernel > command-line at boot time) > > v1 is deprecated on the kernel side because it was... a mess. > The systemd people are migrating stuff from v1 to v2 as new controllers are > added to v2, but at this point there is little left that is v1 only thus > the change of the default value > > Very few user-space projects care about cgroups : mainly systemd and > container-managers. Container-manager need to interact with systemd as > described in > https://systemd.io/CGROUP_DELEGATION.html which allows to drop v1 and to > correctly spread the responsibility between systemd and the container > managers without > stepping on each other's toes. > Also note that Fedora31 will use the unified cgroup hierarchy. > A good summary of adoption status : > https://medium.com/nttlabs/cgroup-v2-596d035be4d7 > > At that point, I think staying in hybrid mode is the safest bet : > everything in user-space hasn't migrated yet. > Let Fedora take the fire and force everybody to migrate, and let's revisit > the subject once (at least) Docker has migrated... OK, thanks a lot for this explanation, much clearer! Thomas
On Mon, 11 Nov 2019 00:33:48 -0800 Christian Stewart <christian@paral.in> wrote: > Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243. > According to the systemd documentation: > > systemd now defaults to the "unified" cgroup hierarchy setup during > build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default. > Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream > production distributions might want to continue to use > -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately > the popular container managers have not caught up with the kernel API changes. > > Changing this option to "hybrid" or "legacy" fixes the Docker startup. > > Reference: https://github.com/opencontainers/runc/issues/654 > > Signed-off-by: Christian Stewart <christian@paral.in> > --- > package/systemd/systemd.mk | 1 + > 1 file changed, 1 insertion(+) Applied to master, thanks. It would be nice to have some runtime tests around Docker, to verify that it is still working properly at runtime, so that we can catch this kind of issues. Thanks! Thomas
Hi all, (re-sending this from the correct email address) Jeremy ROSEN <jeremy.rosen@smile.fr> writes: > v1 is deprecated on the kernel side because it was... a mess. > The systemd people are migrating stuff from v1 to v2 as new controllers a= re > added to v2, but at this point there is little left that is v1 only thus > the change of the default value Thomas Petazzoni writes: >> systemd.mk would observe this boolean to enable (or not) the hybrid >> hierarchy, and packages that need it would select this boolean. >> >> That being said, is there any drawback in unconditionally using the >> hybrid hierarchy? James, since you suggested to make it conditional, do >> you have some arguments/feedback? I originally planned to submit this patch with a 3-state option for which mode is desired (legacy, hybrid, merged) and then having Docker and other packages depend on hybrid or legacy. However, I realized that this would be quite a lot of work to then trace down all of the packages that potentially need the hybrid configuration, and that in the name of "not breaking userspace" it makes sense for Buildroot to follow the direction in the systemd NEWS and just default to "hybrid" for now. If someone wants to do the above work to make the 3-state option, we could merge that first, with "hybrid" as the default, and later add dependencies from Docker -> systemd for that option. > At that point, I think staying in hybrid mode is the safest bet : > everything in user-space hasn't migrated yet. > Let Fedora take the fire and force everybody to migrate, and let's revisit > the subject once (at least) Docker has migrated... Agreed, we can probably wait until Docker has finished their migration to begin seriously considering moving off of "hybrid." Best regards, Christian On Mon, Nov 11, 2019 at 7:29 AM Thomas Petazzoni < thomas.petazzoni@bootlin.com> wrote: > On Mon, 11 Nov 2019 00:33:48 -0800 > Christian Stewart <christian@paral.in> wrote: > > > Docker fails to start with "Devices cgroup isn't mounted" as of systemd > 243. > > According to the systemd documentation: > > > > systemd now defaults to the "unified" cgroup hierarchy setup during > > build-time, i.e. -Ddefault-hierarchy=unified is now the build-time > default. > > Previously, -Ddefault-hierarchy=hybrid was the default. [...] > Downstream > > production distributions might want to continue to use > > -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as > unfortunately > > the popular container managers have not caught up with the kernel API > changes. > > > > Changing this option to "hybrid" or "legacy" fixes the Docker startup. > > > > Reference: https://github.com/opencontainers/runc/issues/654 > > > > Signed-off-by: Christian Stewart <christian@paral.in> > > --- > > package/systemd/systemd.mk | 1 + > > 1 file changed, 1 insertion(+) > > Applied to master, thanks. It would be nice to have some runtime tests > around Docker, to verify that it is still working properly at runtime, > so that we can catch this kind of issues. > > Thanks! > > Thomas > -- > Thomas Petazzoni, CTO, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com >
diff --git a/package/systemd/systemd.mk b/package/systemd/systemd.mk index 94d5f703cd..92490eb86b 100644 --- a/package/systemd/systemd.mk +++ b/package/systemd/systemd.mk @@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \ -Dima=false \ -Dldconfig=false \ -Ddefault-dnssec=no \ + -Ddefault-hierarchy=hybrid \ -Dtests=false \ -Dsplit-bin=true \ -Dsplit-usr=false \
Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243. According to the systemd documentation: systemd now defaults to the "unified" cgroup hierarchy setup during build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default. Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream production distributions might want to continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately the popular container managers have not caught up with the kernel API changes. Changing this option to "hybrid" or "legacy" fixes the Docker startup. Reference: https://github.com/opencontainers/runc/issues/654 Signed-off-by: Christian Stewart <christian@paral.in> --- package/systemd/systemd.mk | 1 + 1 file changed, 1 insertion(+)