Allowing user to run ldconfig in post-build script

Hi!

Since commit 9c40723, handling of ldconfig in the main Makefile as been dropped,
and if /etc/ld.so.conf is found in the target root filesystem, the build fails.

Not being able to run ldconfig and generate the ld.so cache has two drawbacks:

 - it prevents the user from installing some libraries in other locations than
   /lib and /usr/lib (e.g. /opt/foo/lib). This can be solved with symlinks,
   though.
 - when running a program, ld.so has to explore all the library paths to find
   the correct location of the required libraries. This results in zillions of
   failed calls to open(), i.e. wasted time.

So, it would be better to add an entry in the system configuration menu so that
the user can explicitly disable the sanity check introduced by 9c40723. Hence
he/she will be able to run the cross-compiled version of ldconfig (found in
${STAGING_DIR}/sbin/ldconfig) via the version of QEMU matching the target.

Attached are:

 - a patch to add the aforementioned configuration entry, named
   BR2_TARGET_LDCONFIG.
 - an example of post-build script to run ldconfig via QEMU.

However, instead of using a post-build script, the operations could be performed
in a Buildroot Makefile. Which one should be updated? The main Makefile or the
Makefile of the skeleton package? In any cases, a dependency to host-qemu will
be introduced.

Regards,

--
ELB

Hello,

On Thu, 16 Jun 2016 15:12:06 +0200 (CEST), Eric Le Bihan wrote:

> Since commit 9c40723, handling of ldconfig in the main Makefile as been dropped,
> and if /etc/ld.so.conf is found in the target root filesystem, the build fails.
> 
> Not being able to run ldconfig and generate the ld.so cache has two drawbacks:
> 
>  - it prevents the user from installing some libraries in other locations than
>    /lib and /usr/lib (e.g. /opt/foo/lib). This can be solved with symlinks,
>    though.

Does the dynamic linker uses /etc/ld.so.conf at runtime to find other
libraries, even if there is no /etc/ld.so.cache? If that's the case,
then our check for ld.so.conf being absent is somewhat wrong, as it
would be valid to have, independently of whether ldconfig has created
ld.so.cache or not.

>  - when running a program, ld.so has to explore all the library paths to find
>    the correct location of the required libraries. This results in zillions of
>    failed calls to open(), i.e. wasted time.

Right.

> So, it would be better to add an entry in the system configuration menu so that
> the user can explicitly disable the sanity check introduced by 9c40723. Hence
> he/she will be able to run the cross-compiled version of ldconfig (found in
> ${STAGING_DIR}/sbin/ldconfig) via the version of QEMU matching the target.
> 
> Attached are:
> 
>  - a patch to add the aforementioned configuration entry, named
>    BR2_TARGET_LDCONFIG.
>  - an example of post-build script to run ldconfig via QEMU.
> 
> However, instead of using a post-build script, the operations could be performed
> in a Buildroot Makefile. Which one should be updated? The main Makefile or the
> Makefile of the skeleton package? In any cases, a dependency to host-qemu will
> be introduced.

With the proposal from Yann to have different skeletons for systemd and
traditional init, I am not sure having this logic in the skeleton
package is the most appropriate. Indeed, this logic is useful
regardless of the init system being used.

However, in order to merge something like this, I'd like to have a
solution that covers glibc, uClibc and musl, or at least takes those
different cases into account by making it available only for the C
libraries that support it (but that mean investigating how uClibc and
musl support ld.so.conf/ld.so.cache).

Another thing that bothers me is why it is not possible to have a
cross-compilation aware ldconfig. This would really be much, much nicer
than running ldconfig under qemu.

Thanks,

Thomas

Le Thu, 16 Jun 2016 15:37:10 +0200,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a écrit :

> > Since commit 9c40723, handling of ldconfig in the main Makefile as
> > been dropped, and if /etc/ld.so.conf is found in the target root
> > filesystem, the build fails.
> > 
> > Not being able to run ldconfig and generate the ld.so cache has two
> > drawbacks:
> > 
> >  - it prevents the user from installing some libraries in other
> > locations than /lib and /usr/lib (e.g. /opt/foo/lib). This can be
> > solved with symlinks, though.  
> 
> Does the dynamic linker uses /etc/ld.so.conf at runtime to find other
> libraries, even if there is no /etc/ld.so.cache? If that's the case,
> then our check for ld.so.conf being absent is somewhat wrong, as it
> would be valid to have, independently of whether ldconfig has created
> ld.so.cache or not.

Inspecting the glibc source code shows that:

 - /etc/ld.so.conf is only read by ldconfig (see elf/ldconfig.c). 
 - /etc/ld.so.cache is only read by the dynamic loader (see elf/dl-cache.c).

This is confirmed when using strace. So, IMHO, this sanity check is of
no use.

> However, in order to merge something like this, I'd like to have a
> solution that covers glibc, uClibc and musl, or at least takes those
> different cases into account by making it available only for the C
> libraries that support it (but that mean investigating how uClibc and
> musl support ld.so.conf/ld.so.cache).

From the INSTALL file in musl source code, we learn that if dynamic
linking is enabled, the dynamic linker will be
/lib/ld-musl-$ARCH.so.1. It will look for libraries in the paths
listed in /etc/ld-musl-$ARCH.path.

Debian provides ld-musl-config, which aggregates the contents of the
files found in /etc/ld-musl-$ARCH.d/ to create /etc/ld-musl-$ARCH.path.

However musl is traditionally used for static linking.

uclibc-ng also provides ldconfig, which behaves like the glibc version
(see ldso/man/ldconfig.8 in the source tree).

> Another thing that bothers me is why it is not possible to have a
> cross-compilation aware ldconfig. This would really be much, much
> nicer than running ldconfig under qemu.

Yocto provides a recipe named ldconfig-native_2.12.1.bb [1], which
contains a verbatim copy of the ldconfig source code from glibc, and a
truck load of patches. It looks a bit hairy...

[1] http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb

Regards,

Thomas, Éric, All,

On 2016-06-16 15:37 +0200, Thomas Petazzoni spake thusly:
> On Thu, 16 Jun 2016 15:12:06 +0200 (CEST), Eric Le Bihan wrote:
> 
> > Since commit 9c40723, handling of ldconfig in the main Makefile as been dropped,
> > and if /etc/ld.so.conf is found in the target root filesystem, the build fails.
> > 
> > Not being able to run ldconfig and generate the ld.so cache has two drawbacks:
> > 
> >  - it prevents the user from installing some libraries in other locations than
> >    /lib and /usr/lib (e.g. /opt/foo/lib). This can be solved with symlinks,
> >    though.
> 
> Does the dynamic linker uses /etc/ld.so.conf at runtime to find other
> libraries, even if there is no /etc/ld.so.cache? If that's the case,
> then our check for ld.so.conf being absent is somewhat wrong, as it
> would be valid to have, independently of whether ldconfig has created
> ld.so.cache or not.

What I understand is that, to find a library, the linker will:

 1) if there is a cache, see if it knows about that library in the cache;

 2) if no cache, or if not known in the cache, look for ld.so.conf and
    look for that library in all paths listed in there;

 3) if still not found, look in the "well-known" locations, usually
    /usr/lib then /lib  (or their variants, depending on the
    mutlilib/multiarch uglyness)

So, we can well have any combination of existing/non-existing ld.so.cache
or ld.so.conf.

> >  - when running a program, ld.so has to explore all the library paths to find
> >    the correct location of the required libraries. This results in zillions of
> >    failed calls to open(), i.e. wasted time.

Have you actually measured this overhaed?

I'd like we avoid "solving" this if the gain is not even measurable...

> > So, it would be better to add an entry in the system configuration menu so that
> > the user can explicitly disable the sanity check introduced by 9c40723. Hence
> > he/she will be able to run the cross-compiled version of ldconfig (found in
> > ${STAGING_DIR}/sbin/ldconfig) via the version of QEMU matching the target.
> > 
> > Attached are:
> > 
> >  - a patch to add the aforementioned configuration entry, named
> >    BR2_TARGET_LDCONFIG.
> >  - an example of post-build script to run ldconfig via QEMU.
> > 
> > However, instead of using a post-build script, the operations could be performed
> > in a Buildroot Makefile. Which one should be updated? The main Makefile or the
> > Makefile of the skeleton package? In any cases, a dependency to host-qemu will
> > be introduced.
> 
> With the proposal from Yann to have different skeletons for systemd and
> traditional init, I am not sure having this logic in the skeleton
> package is the most appropriate. Indeed, this logic is useful
> regardless of the init system being used.

And even without the one-skeleton-per-init-system, I doubt it belongs
to the skeleton. The cache should be constructed after we have all the
libraries, so it can only really be in target-finalize )or be a hook
thereof).

Now, where to put the actual code? In its own location, no need to
fatten the main Makefile. We can always add .mk fragments, e.g. in
support/mk/ or somesuch.

> However, in order to merge something like this, I'd like to have a
> solution that covers glibc, uClibc and musl, or at least takes those
> different cases into account by making it available only for the C
> libraries that support it (but that mean investigating how uClibc and
> musl support ld.so.conf/ld.so.cache).
> 
> Another thing that bothers me is why it is not possible to have a
> cross-compilation aware ldconfig. This would really be much, much nicer
> than running ldconfig under qemu.

Well, host-qemu: please, no. It is really ugly... :-/

I've started looking at the format for ld.so.cachei (for glibc!), and it
does not seem to be overly complex. It is not trivial, but the bulk of
it is a concatenation of:

    [cache-in-old-format]
    [cache-in-new-format]

The old format is for libc5/glibc2 so we don't care about it at all. It
looks like it is not even required to be present.

The new format is:

    magic
    nb_libs
    libs[0]
    ...
    libs[nb_libs-1]
    string
    string
    ...

Now, I'm not sure what "lib[n]" is (some bitfields with some flags!),
but the stings are pretty obvious: they are a pair "SONAME\0PATH\0",
sorted (alphabetically, I guess). It has to be sorted, because ld.so
expects this: it uses a binary search to find the corresponding library.

I'm still looking at what the "libs[n]" are made of, and how they
correlate to the strings section...

So, I think it would be pretty straightforward to create a cache.
Brought to you under the brand "Famous Last Words (TM)."

Regards,
Yann E. MORIN.

Hello,

On Thu, 16 Jun 2016 21:45:22 +0200, Eric Le Bihan wrote:

> > > Not being able to run ldconfig and generate the ld.so cache has two
> > > drawbacks:
> > > 
> > >  - it prevents the user from installing some libraries in other
> > > locations than /lib and /usr/lib (e.g. /opt/foo/lib). This can be
> > > solved with symlinks, though.    
> > 
> > Does the dynamic linker uses /etc/ld.so.conf at runtime to find other
> > libraries, even if there is no /etc/ld.so.cache? If that's the case,
> > then our check for ld.so.conf being absent is somewhat wrong, as it
> > would be valid to have, independently of whether ldconfig has created
> > ld.so.cache or not.  
> 
> Inspecting the glibc source code shows that:
> 
>  - /etc/ld.so.conf is only read by ldconfig (see elf/ldconfig.c). 
>  - /etc/ld.so.cache is only read by the dynamic loader (see elf/dl-cache.c).

OK.

> This is confirmed when using strace. So, IMHO, this sanity check is of
> no use.

If what you say above is true, then your conclusion is completely
wrong: the sanity check is *absolutely* mandatory.

Since Buildroot does *NOT* generate ld.so.cache, and ld.so.conf is not
read by the dynamic loader, having a ld.so.conf in a Buildroot
filesystem is a mistake, as the user might thing that the dynamic
linker will search in the directories listed in ld.so.conf, but it is
not true: the dynamic linker only searches in the libraries referenced
in ld.so.cache, and ignores ld.so.conf.

> > However, in order to merge something like this, I'd like to have a
> > solution that covers glibc, uClibc and musl, or at least takes those
> > different cases into account by making it available only for the C
> > libraries that support it (but that mean investigating how uClibc and
> > musl support ld.so.conf/ld.so.cache).  
> 
> From the INSTALL file in musl source code, we learn that if dynamic
> linking is enabled, the dynamic linker will be
> /lib/ld-musl-$ARCH.so.1. It will look for libraries in the paths
> listed in /etc/ld-musl-$ARCH.path.
> 
> Debian provides ld-musl-config, which aggregates the contents of the
> files found in /etc/ld-musl-$ARCH.d/ to create /etc/ld-musl-$ARCH.path.

So it uses configuration files that are incompatible with the ones used
by glibc.

> However musl is traditionally used for static linking.

I don't think that's a good assumption, at least in the context of
Buildroot.

> uclibc-ng also provides ldconfig, which behaves like the glibc version
> (see ldso/man/ldconfig.8 in the source tree).

OK. But IIRC, the ld.so.cache functionality in uClibc-ng is optional,
and I'm not sure we have it enabled by default.

The uClibc-ng ld.so.cache format would be fully compatible with
the one used in glibc ?

> > Another thing that bothers me is why it is not possible to have a
> > cross-compilation aware ldconfig. This would really be much, much
> > nicer than running ldconfig under qemu.  
> 
> Yocto provides a recipe named ldconfig-native_2.12.1.bb [1], which
> contains a verbatim copy of the ldconfig source code from glibc, and a
> truck load of patches. It looks a bit hairy...
> 
> [1] http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb

Yes, I know they are doing this with a shitload of patches. Maybe we
should host a project on github for a cross-compilation capable
ldconfig, and then offer the Yocto people to use it?

Best regards,

Thomas

Thomas, Éric, All,

On 2016-06-16 22:28 +0200, Thomas Petazzoni spake thusly:
> On Thu, 16 Jun 2016 21:45:22 +0200, Eric Le Bihan wrote:
> > > > Not being able to run ldconfig and generate the ld.so cache has two
> > > > drawbacks:
[--SNIP--]
> > uclibc-ng also provides ldconfig, which behaves like the glibc version
> > (see ldso/man/ldconfig.8 in the source tree).
> 
> OK. But IIRC, the ld.so.cache functionality in uClibc-ng is optional,
> and I'm not sure we have it enabled by default.
> 
> The uClibc-ng ld.so.cache format would be fully compatible with
> the one used in glibc ?

AFAICS, yes. Their ldocnfig code looks suspiciously like the glibc one.

> > > Another thing that bothers me is why it is not possible to have a
> > > cross-compilation aware ldconfig. This would really be much, much
> > > nicer than running ldconfig under qemu.  
> > 
> > Yocto provides a recipe named ldconfig-native_2.12.1.bb [1], which
> > contains a verbatim copy of the ldconfig source code from glibc, and a
> > truck load of patches. It looks a bit hairy...
> > 
> > [1] http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-core/glibc/ldconfig-native_2.12.1.bb
> 
> Yes, I know they are doing this with a shitload of patches. Maybe we
> should host a project on github for a cross-compilation capable
> ldconfig, and then offer the Yocto people to use it?

Meh... ;-]

Regards,
Yann E. MORIN.

Hello,

On Thu, 16 Jun 2016 21:53:10 +0200, Yann E. MORIN wrote:

> > Does the dynamic linker uses /etc/ld.so.conf at runtime to find other
> > libraries, even if there is no /etc/ld.so.cache? If that's the case,
> > then our check for ld.so.conf being absent is somewhat wrong, as it
> > would be valid to have, independently of whether ldconfig has created
> > ld.so.cache or not.  
> 
> What I understand is that, to find a library, the linker will:
> 
>  1) if there is a cache, see if it knows about that library in the cache;
> 
>  2) if no cache, or if not known in the cache, look for ld.so.conf and
>     look for that library in all paths listed in there;
> 
>  3) if still not found, look in the "well-known" locations, usually
>     /usr/lib then /lib  (or their variants, depending on the
>     mutlilib/multiarch uglyness)

This quite different than what Eric said. If indeed ld.so.conf is
indeed read if there is no cache, or the library has not been found
through the cache, then we could potentially remove the check on
ld.so.conf in the main Makefile.

*However* there is still the problem that having a ld.so.conf may work
for glibc, but not necessarily for uClibc/musl. So if a package adds
some contents to ld.so.conf or ld.so.conf.d, thinking it will "just
work", it might be OK with glibc, but not with musl/uClibc. It's also
for this reason that I added this check in the first place.

> > With the proposal from Yann to have different skeletons for systemd and
> > traditional init, I am not sure having this logic in the skeleton
> > package is the most appropriate. Indeed, this logic is useful
> > regardless of the init system being used.  
> 
> And even without the one-skeleton-per-init-system, I doubt it belongs
> to the skeleton. The cache should be constructed after we have all the
> libraries, so it can only really be in target-finalize )or be a hook
> thereof).
> 
> Now, where to put the actual code? In its own location, no need to
> fatten the main Makefile. We can always add .mk fragments, e.g. in
> support/mk/ or somesuch.

Agreed. I'd prefer to avoid cluttering the main Makefile with more and
more stuff. We need to find some good places to move certain parts of
the main Makefile that don't belong to any package, but that also don't
really belong to the main Makefile.

Thomas

Le Thu, 16 Jun 2016 22:28:54 +0200,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a écrit :

> On Thu, 16 Jun 2016 21:45:22 +0200, Eric Le Bihan wrote:
> 
> > > > Not being able to run ldconfig and generate the ld.so cache has
> > > > two drawbacks:
> > > > 
> > > >  - it prevents the user from installing some libraries in other
> > > > locations than /lib and /usr/lib (e.g. /opt/foo/lib). This can
> > > > be solved with symlinks, though.      
> > > 
> > > Does the dynamic linker uses /etc/ld.so.conf at runtime to find
> > > other libraries, even if there is no /etc/ld.so.cache? If that's
> > > the case, then our check for ld.so.conf being absent is somewhat
> > > wrong, as it would be valid to have, independently of whether
> > > ldconfig has created ld.so.cache or not.    
> > 
> > Inspecting the glibc source code shows that:
> > 
> >  - /etc/ld.so.conf is only read by ldconfig (see elf/ldconfig.c). 
> >  - /etc/ld.so.cache is only read by the dynamic loader (see
> > elf/dl-cache.c).  
> 
> OK.
> 
> > This is confirmed when using strace. So, IMHO, this sanity check is
> > of no use.  
> 
> If what you say above is true, then your conclusion is completely
> wrong: the sanity check is *absolutely* mandatory.
> 
> Since Buildroot does *NOT* generate ld.so.cache, and ld.so.conf is not
> read by the dynamic loader, having a ld.so.conf in a Buildroot
> filesystem is a mistake, as the user might thing that the dynamic
> linker will search in the directories listed in ld.so.conf, but it is
> not true: the dynamic linker only searches in the libraries referenced
> in ld.so.cache, and ignores ld.so.conf.

OK. So the check is needed to get a clean target root file system,
devoid of files which may mislead the user. However, it makes sense to
provide a configuration entry to disable it, for the ld.so-savvy users
who want to create the cache on their own, no?

Hi!

Le Thu, 16 Jun 2016 21:53:10 +0200,
"Yann E. MORIN" <yann.morin.1998@free.fr> a écrit :

> On 2016-06-16 15:37 +0200, Thomas Petazzoni spake thusly:
> > On Thu, 16 Jun 2016 15:12:06 +0200 (CEST), Eric Le Bihan wrote:
> >   
> > > Since commit 9c40723, handling of ldconfig in the main Makefile
> > > as been dropped, and if /etc/ld.so.conf is found in the target
> > > root filesystem, the build fails.
> > > 
> > > Not being able to run ldconfig and generate the ld.so cache has
> > > two drawbacks:
> > > 
> > >  - it prevents the user from installing some libraries in other
> > > locations than /lib and /usr/lib (e.g. /opt/foo/lib). This can be
> > > solved with symlinks, though.  
> > 
> > Does the dynamic linker uses /etc/ld.so.conf at runtime to find
> > other libraries, even if there is no /etc/ld.so.cache? If that's
> > the case, then our check for ld.so.conf being absent is somewhat
> > wrong, as it would be valid to have, independently of whether
> > ldconfig has created ld.so.cache or not.  
> 
> What I understand is that, to find a library, the linker will:
> 
>  1) if there is a cache, see if it knows about that library in the
> cache;
> 
>  2) if no cache, or if not known in the cache, look for ld.so.conf and
>     look for that library in all paths listed in there;
> 
>  3) if still not found, look in the "well-known" locations, usually
>     /usr/lib then /lib  (or their variants, depending on the
>     mutlilib/multiarch uglyness)

The MAN page for ld.so [1] only refers to /etc/ld.so.cache when ld.so
searches for a library.

> > >  - when running a program, ld.so has to explore all the library
> > > paths to find the correct location of the required libraries.
> > > This results in zillions of failed calls to open(), i.e. wasted
> > > time.  
> 
> Have you actually measured this overhaed?

No. I'll try to get some numbers. I only used strace.

[1] http://man7.org/linux/man-pages/man8/ld.so.8.html

Regards,

On 16-06-16 23:15, Eric Le Bihan wrote:
> Le Thu, 16 Jun 2016 22:28:54 +0200,
> Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a écrit :
> 
>> On Thu, 16 Jun 2016 21:45:22 +0200, Eric Le Bihan wrote:
>>
>>>>> Not being able to run ldconfig and generate the ld.so cache has
>>>>> two drawbacks:
>>>>>
>>>>>  - it prevents the user from installing some libraries in other
>>>>> locations than /lib and /usr/lib (e.g. /opt/foo/lib). This can
>>>>> be solved with symlinks, though.      
>>>>
>>>> Does the dynamic linker uses /etc/ld.so.conf at runtime to find
>>>> other libraries, even if there is no /etc/ld.so.cache? If that's
>>>> the case, then our check for ld.so.conf being absent is somewhat
>>>> wrong, as it would be valid to have, independently of whether
>>>> ldconfig has created ld.so.cache or not.    
>>>
>>> Inspecting the glibc source code shows that:
>>>
>>>  - /etc/ld.so.conf is only read by ldconfig (see elf/ldconfig.c). 
>>>  - /etc/ld.so.cache is only read by the dynamic loader (see
>>> elf/dl-cache.c).  
>>
>> OK.
>>
>>> This is confirmed when using strace. So, IMHO, this sanity check is
>>> of no use.  
>>
>> If what you say above is true, then your conclusion is completely
>> wrong: the sanity check is *absolutely* mandatory.
>>
>> Since Buildroot does *NOT* generate ld.so.cache, and ld.so.conf is not
>> read by the dynamic loader, having a ld.so.conf in a Buildroot
>> filesystem is a mistake, as the user might thing that the dynamic
>> linker will search in the directories listed in ld.so.conf, but it is
>> not true: the dynamic linker only searches in the libraries referenced
>> in ld.so.cache, and ignores ld.so.conf.
> 
> OK. So the check is needed to get a clean target root file system,
> devoid of files which may mislead the user. However, it makes sense to
> provide a configuration entry to disable it, for the ld.so-savvy users
> who want to create the cache on their own, no?

 No, the check is there to be able to detect packages that create an ld.so.conf
(and run ldconfig) because they install libraries in non-default paths. This
will cause runtime errors, and we want to detect them at build time.

 Regards,
 Arnout