package/glibc: allow runing on kernel older than used for the headers

From: "Yann E. MORIN" <yann.morin@orange.com>

Currently, we configure glibc to not add compatibility support for
kernel older than the one used for the headers. This is on the
expectation that the system will never run on a kernel that is older
than the one used for the headers and, when Buildroot builds the kernel,
on another, older kernel.

However, in some situations, it is possible to build for a generic
system, where the kernel may be a different version. This can be the
case, for example, when Building an image that is to be used in a
container that can run on a range of machines each with different kernel
versions. In such a case, it is interesting to build glibc in a way as
to take better advantage of the newer kernels, and thus using newer
kernel headers, while still allowing running onn older kernels, and thus
carrying more compatibility code.

We add an option to glibc to be allow the user to provide the oldest
kernel version they expect to use, and use that if specified; if not
specified (the default), use the version of the kernel headers as was
done previously.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Frederic GARDES <frederic.gardes@orange.com>
---
 package/glibc/Config.in | 12 ++++++++++++
 package/glibc/glibc.mk  |  5 ++++-
 2 files changed, 16 insertions(+), 1 deletion(-)

Hi Yann,

> Currently, we configure glibc to not add compatibility support for
> kernel older than the one used for the headers. This is on the
> expectation that the system will never run on a kernel that is older
> than the one used for the headers and, when Buildroot builds the kernel,
> on another, older kernel.
> 
> However, in some situations, it is possible to build for a generic
> system, where the kernel may be a different version. This can be the
> case, for example, when Building an image that is to be used in a
> container that can run on a range of machines each with different kernel
> versions. In such a case, it is interesting to build glibc in a way as
> to take better advantage of the newer kernels, and thus using newer
> kernel headers, while still allowing running onn older kernels, and thus
> carrying more compatibility code.
> 
> We add an option to glibc to be allow the user to provide the oldest
> kernel version they expect to use, and use that if specified; if not
> specified (the default), use the version of the kernel headers as was
> done previously.

That's a very welcome one as it bites us all the time we try to run
more recent rootfs with some older kernels.

> Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
> Cc: Frederic GARDES <frederic.gardes@orange.com>

> +config BR2_PACKAGE_GLIBC_KERNEL_COMPAT
> +       string "Oldest kernel version supported"
> +       help
> +         If you plan on running on various kernel versions, enter the
> +         oldest version you expect to run on here.
> +
> +         Note that the older the version, the more backward compatibility
> +         code is added, and the slower the code may get.
> +
> +         Leave it empty (the default) to use the same version as used for
> +         the kernel headers.

I'd say there's the third option and frankly I like this one the most - default
oldest version assumed by the glibc itself. The beauty of it is it's really the
oldest version of the kernel which is supported by a given glibc version for
a particular architecture.

>  config BR2_PACKAGE_GLIBC_UTILS
>          bool "Install glibc utilities"
>          help
> diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
> index 5c26b0e6df..43015417c2 100644
> --- a/package/glibc/glibc.mk
> +++ b/package/glibc/glibc.mk
> @@ -98,6 +98,9 @@ endif
>  GLIBC_MAKE = $(BR2_MAKE)
>  GLIBC_CONF_ENV += ac_cv_prog_MAKE="$(BR2_MAKE)"
>  
> +GLIBC_KERNEL_VERSION = $(or $(call qstrip,$(BR2_PACKAGE_GLIBC_KERNEL_COMPAT)), \
> +                           $(call qstrip,$(BR2_TOOLCHAIN_HEADERS_AT_LEAST)))
> +
>  # Even though we use the autotools-package infrastructure, we have to
>  # override the default configure commands for several reasons:
>  #
> @@ -128,7 +131,7 @@ define GLIBC_CONFIGURE_CMDS
>                  --disable-profile \
>                  --disable-werror \
>                  --without-gd \
> -               --enable-kernel=$(call qstrip,$(BR2_TOOLCHAIN_HEADERS_AT_LEAST)) \
> +               --enable-kernel=$(GLIBC_KERNEL_VERSION) \

See [1] and how "arch_minimum_kernel" is used in [2].
I.e. for that third option to work we need to skip setup of
"--enable-kernel" completely.

[1] https://sourceware.org/git/?p=glibc.git;a=blob;f=configure;h=8e5bee775a651fcbaaa96ede8039ae1f049e296e;hb=HEAD#l3529
[2] https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/configure.ac;h=197b7e66c8fe6eb0a1136c992478b9a2272713e7;hb=HEAD#l30

-Alexey

Alexey, All,

On 2022-02-01 14:23 +0000, Alexey Brodkin spake thusly:
[--SNIP--]
> > +config BR2_PACKAGE_GLIBC_KERNEL_COMPAT
> > +       string "Oldest kernel version supported"
> > +       help
> > +         If you plan on running on various kernel versions, enter the
> > +         oldest version you expect to run on here.
> > +
> > +         Note that the older the version, the more backward compatibility
> > +         code is added, and the slower the code may get.
> > +
> > +         Leave it empty (the default) to use the same version as used for
> > +         the kernel headers.
> I'd say there's the third option and frankly I like this one the most - default
> oldest version assumed by the glibc itself. The beauty of it is it's really the
> oldest version of the kernel which is supported by a given glibc version for
> a particular architecture.

I also considered that, but I believe that a user should explicitly
request the oldest kernel they want to support (yes, they'd have to do
their homework to see what that version can be for their architecture).
Letting the system decide on itself is prone to providing some
surprises... However, letting the user provide an explicit version is
also prone to surprises, because if that version is too old, glibc
resets it to the oldest it actually supports, and just merely emits a
warning message, which does not prevent the build to succeed (it still
fails at runtime in the usual way then)...

So, in either case, meh...

So, we'd have to differentiate between the three cases:
  - use same as headers (should be the default to keep current
    behaviour)
  - use oldest supported by glibc
  - use explcitly specified version

So, two options there:

  - recognise a magical value in BR2_PACKAGE_GLIBC_KERNEL_COMPAT, like
    'oldest' (bikeshed, go) to mean the oldest glibc can support, empty
    to mean "same as headers", or an actual value.

  - add a boolean "Specify oldest kernel supported", and hide
    BR2_PACKAGE_GLIBC_KERNEL_COMPAT behind that boolean; then if the
    boolean is not set, use same as headers (as today); if it is set,
    then use BR2_PACKAGE_GLIBC_KERNEL_COMPAT is set, or let glibc decide
    if not set.

Thoughts?

Needless to say, I don't have much sympathy for the first option,
magical values are bad... But I am not too fond of the second option
either, but I can't see a better way to provide for the three different
cases...

> [1] https://sourceware.org/git/?p=glibc.git;a=blob;f=configure;h=8e5bee775a651fcbaaa96ede8039ae1f049e296e;hb=HEAD#l3529
> [2] https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/configure.ac;h=197b7e66c8fe6eb0a1136c992478b9a2272713e7;hb=HEAD#l30

Yeah, I already had a look at those for various archs.

Thanks!

Regards,
Yann E. MORIN.

Hi Yann,

> > > +config BR2_PACKAGE_GLIBC_KERNEL_COMPAT
> > > +       string "Oldest kernel version supported"
> > > +       help
> > > +         If you plan on running on various kernel versions, enter the
> > > +         oldest version you expect to run on here.
> > > +
> > > +         Note that the older the version, the more backward compatibility
> > > +         code is added, and the slower the code may get.
> > > +
> > > +         Leave it empty (the default) to use the same version as used for
> > > +         the kernel headers.
> > I'd say there's the third option and frankly I like this one the most - default
> > oldest version assumed by the glibc itself. The beauty of it is it's really the
> > oldest version of the kernel which is supported by a given glibc version for
> > a particular architecture.
> 
> I also considered that, but I believe that a user should explicitly
> request the oldest kernel they want to support (yes, they'd have to do
> their homework to see what that version can be for their architecture).

Yup, let's provide our user with enough rope ;)

> Letting the system decide on itself is prone to providing some
> surprises... However, letting the user provide an explicit version is
> also prone to surprises, because if that version is too old, glibc
> resets it to the oldest it actually supports, and just merely emits a
> warning message, which does not prevent the build to succeed (it still
> fails at runtime in the usual way then)...
> 
> So, in either case, meh...

Agree.

> So, we'd have to differentiate between the three cases:
>   - use same as headers (should be the default to keep current
>     behaviour)
>   - use oldest supported by glibc
>   - use explcitly specified version
> 
> So, two options there:
> 
>   - recognise a magical value in BR2_PACKAGE_GLIBC_KERNEL_COMPAT, like
>     'oldest' (bikeshed, go) to mean the oldest glibc can support, empty
>     to mean "same as headers", or an actual value.

This option looks a bit more clean to me as we don't introduce yet another
hidden option (as in the one below). Though I would think not that many
people is really worried by the toolchain internals, so maybe you introduce
something, it quickly gets discussed and merged? :)

Alternatively we may explicitly set per-arch minimal version in their
corresponding "arch/Config.in.xxx" so at least this is more visible
as not each and every user is willing to dig into glibc's internals.

Thinking out loud further along the lines above, maybe even in the same way
explicitly set kernel version that matches used headers (if we automatically
extract that info from some existing Buildroot variable)? Ir it's really
something insane?

>   - add a boolean "Specify oldest kernel supported", and hide
>     BR2_PACKAGE_GLIBC_KERNEL_COMPAT behind that boolean; then if the
>     boolean is not set, use same as headers (as today); if it is set,
>     then use BR2_PACKAGE_GLIBC_KERNEL_COMPAT is set, or let glibc decide
>     if not set.
> 

-Alexey

On 01/02/2022 16:09, yann.morin@orange.com wrote:
> Alexey, All,
> 
> On 2022-02-01 14:23 +0000, Alexey Brodkin spake thusly:
> [--SNIP--]
>>> +config BR2_PACKAGE_GLIBC_KERNEL_COMPAT
>>> +       string "Oldest kernel version supported"
>>> +       help
>>> +         If you plan on running on various kernel versions, enter the
>>> +         oldest version you expect to run on here.
>>> +
>>> +         Note that the older the version, the more backward compatibility
>>> +         code is added, and the slower the code may get.
>>> +
>>> +         Leave it empty (the default) to use the same version as used for
>>> +         the kernel headers.
>> I'd say there's the third option and frankly I like this one the most - default
>> oldest version assumed by the glibc itself. The beauty of it is it's really the
>> oldest version of the kernel which is supported by a given glibc version for
>> a particular architecture.
> 
> I also considered that, but I believe that a user should explicitly
> request the oldest kernel they want to support (yes, they'd have to do
> their homework to see what that version can be for their architecture).

  I'm doubting if the added value of this is sufficient to warrant the complexity.

  I could be swayed if it would give a build-time error if we know that it's not 
going to run on such an old kernel. But as mentioned below, there's just a warning.

  In addition, if you build a userspace with recent headers, this is going to 
allow packages that depend on TOOLCHAIN_HEADERS_AT_LEAST_... Which may cause 
runtime failures when running those packages on an older kernel. So the whole 
concept of allowing older kernels is already a bit shaky.


  So I think that a simple boolean config (still called 
BR2_PACKAGE_GLIBC_KERNEL_COMPAT) would be sufficient. Doesn't solve the 
TOOLCHAIN_HEADERS_AT_LEAST problem, but at least init will be running (well, 
except if it's systemd and you try to run on 3.4...)


  For context, this is the commit that originally introduced --enable-kernel:

commit fd5bcd0eda8fb21f639c34a09b212e6f9b066a04
Author: Sam bobroff <sam.bobroff@au1.ibm.com>
Date:   Thu Jan 28 04:51:23 2016

     package/glibc: set --enable-kernel to match kernel

     Glibc is currently configured without any "--enable-kernel" option.
     This causes it to use the oldest possible kernel API, slowing it down
     and preventing it from using any kernel features from later versions.

     Since we are likely building a kernel and matching glibc together,
     backwards compatability is probably unnecessary so this patch
     unconditionally configures glibc with --enable-kernel set to
     BR2_TOOLCHAIN_HEADERS_AT_LEAST.


  It's true that the slowdown will probably be smaller if you only have a 
partial fallback, but is that really going to be significant? You can convince 
me with numbers :-)


  Regards,
  Arnout



> Letting the system decide on itself is prone to providing some
> surprises... However, letting the user provide an explicit version is
> also prone to surprises, because if that version is too old, glibc
> resets it to the oldest it actually supports, and just merely emits a
> warning message, which does not prevent the build to succeed (it still
> fails at runtime in the usual way then)...
> 
> So, in either case, meh...
> 
> So, we'd have to differentiate between the three cases:
>    - use same as headers (should be the default to keep current
>      behaviour)
>    - use oldest supported by glibc
>    - use explcitly specified version
> 
> So, two options there:
> 
>    - recognise a magical value in BR2_PACKAGE_GLIBC_KERNEL_COMPAT, like
>      'oldest' (bikeshed, go) to mean the oldest glibc can support, empty
>      to mean "same as headers", or an actual value.
> 
>    - add a boolean "Specify oldest kernel supported", and hide
>      BR2_PACKAGE_GLIBC_KERNEL_COMPAT behind that boolean; then if the
>      boolean is not set, use same as headers (as today); if it is set,
>      then use BR2_PACKAGE_GLIBC_KERNEL_COMPAT is set, or let glibc decide
>      if not set.
> 
> Thoughts?
> 
> Needless to say, I don't have much sympathy for the first option,
> magical values are bad... But I am not too fond of the second option
> either, but I can't see a better way to provide for the three different
> cases...
> 
>> [1] https://sourceware.org/git/?p=glibc.git;a=blob;f=configure;h=8e5bee775a651fcbaaa96ede8039ae1f049e296e;hb=HEAD#l3529
>> [2] https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/configure.ac;h=197b7e66c8fe6eb0a1136c992478b9a2272713e7;hb=HEAD#l30
> 
> Yeah, I already had a look at those for various archs.
> 
> Thanks!
> 
> Regards,
> Yann E. MORIN.
>