store debug information in $HOST_DIR/lib/debug

Before striping the binaries in $TARGET_DIR, the debug information is saved to
$HOST_DIR/lib/debug using the following command as described in [1]:

  $ objcopy --only-keep-debug --compress-debug-sections $file

This allows remote debugging as described in section 8.12.2 with striped
binaries in the final image. Therefore the build ID is included during the build
process, if debug information is enabled via BR2_ENABLE_DEBUG.

[1] https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

Signed-off-by: Gerhard Heift <Gerhard@Heift.Name>
---
 Makefile                   |  6 ++---
 package/Makefile.in        | 12 ++++++---
 support/scripts/copy-debug | 52 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 63 insertions(+), 7 deletions(-)
 create mode 100755 support/scripts/copy-debug

El mié., 23 ene. 2019 a las 14:22, Gerhard Heift
(<gerhard@heift.name>) escribió:
>
> Before striping the binaries in $TARGET_DIR, the debug information is saved to
> $HOST_DIR/lib/debug using the following command as described in [1]:
>
>   $ objcopy --only-keep-debug --compress-debug-sections $file
>
> This allows remote debugging as described in section 8.12.2 with striped
> binaries in the final image. Therefore the build ID is included during the build
> process, if debug information is enabled via BR2_ENABLE_DEBUG.
>
> [1] https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
>
> Signed-off-by: Gerhard Heift <Gerhard@Heift.Name>
> ---
>  Makefile                   |  6 ++---
>  package/Makefile.in        | 12 ++++++---
>  support/scripts/copy-debug | 52 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 63 insertions(+), 7 deletions(-)
>  create mode 100755 support/scripts/copy-debug
>

If I understand this correctly then the sysroot you'd specify in gdb
would be this new directory output/host/lib/debug, right?
Previously this gdb sysroot was set as output/staging, see
toolchain/helpers.mk. I believe that path should be adapted in your
patch.

It's unclear to me why you selected 'output/host/lib/debug' as path:
this is not about host stuff but target stuff. Moreover, it's also not
only about 'lib' but also binaries.

Do you have some numbers regarding size impact on the buildroot
directory as a whole for a typical build?

Best regards,
Thomas

Am Mi., 30. Jan. 2019 um 21:29 Uhr schrieb Thomas De Schampheleire
<patrickdepinguin@gmail.com>:
>
> El mié., 23 ene. 2019 a las 14:22, Gerhard Heift
> (<gerhard@heift.name>) escribió:
> >
> > Before striping the binaries in $TARGET_DIR, the debug information is saved to
> > $HOST_DIR/lib/debug using the following command as described in [1]:
> >
> >   $ objcopy --only-keep-debug --compress-debug-sections $file
> >
> > This allows remote debugging as described in section 8.12.2 with striped
> > binaries in the final image. Therefore the build ID is included during the build
> > process, if debug information is enabled via BR2_ENABLE_DEBUG.
> >
> > [1] https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
> >
> > Signed-off-by: Gerhard Heift <Gerhard@Heift.Name>
> > ---
> >  Makefile                   |  6 ++---
> >  package/Makefile.in        | 12 ++++++---
> >  support/scripts/copy-debug | 52 ++++++++++++++++++++++++++++++++++++++
> >  3 files changed, 63 insertions(+), 7 deletions(-)
> >  create mode 100755 support/scripts/copy-debug
> >
>
> If I understand this correctly then the sysroot you'd specify in gdb
> would be this new directory output/host/lib/debug, right?
> Previously this gdb sysroot was set as output/staging, see
> toolchain/helpers.mk. I believe that path should be adapted in your
> patch.
>
> It's unclear to me why you selected 'output/host/lib/debug' as path:
> this is not about host stuff but target stuff. Moreover, it's also not
> only about 'lib' but also binaries.

I just started the cross debugger as mentioned in [1] with
  $ host/bin/<tuple>-gdb -x staging/usr/share/buildroot/gdbinit
target/usr/bin/ls
and let gdb show me, where it searchs for the debug symbols with (see [2]):
  (gdb) show debug-file-directory
which outputs the following:

  > The directory where separate debug symbols are searched for is
"$HOST_DIR/lib/debug".

This is the reason, why I selected this path. I don't need to change
anything else beside adding the build-id and copying the symbols to
this path with the provided wrapper.

If you want to store the debug information in another location, either
the gdbinit generation has to be adjusted in toolchain/helpers.mk to
contain something like
  set debug-file-directory $DEBUGDIR
or the directory has to be provided to the configuration script of gdb
in packages/gdb/gdb.mk like this
  HOST_GDB_CONF_OPTS += --with-separate-debug-dir=$DEBUGDIR

The default directory is derived from the prefix and libdir [3], where
in this case prefix=$HOST_DIR and libdir=$prefix/lib and therefore
DEBUGDIR=$HOST_DIR/lib/debug
which is the directory I used with this patch.

If another location is disered, I am happy to provied an extended
patch, either by extending the gdbinit generation or by adding the
configuration option for gdb.

If I understand the directory hierarchy correct, then the debug
symbols should be located under
DEBUGDIR=$STAGING_DIR/usr/lib/debug
Is this correct?

> Do you have some numbers regarding size impact on the buildroot
> directory as a whole for a typical build?

I just did a test with my personal build and it looks like the
build-id, which is added to identify the debugging symbols, increses
each binary or library by 56 to 60 bytes.

> Best regards,
> Thomas

Best regards,
  Gerhard

[1] https://buildroot.org/downloads/manual/manual.html#_using_literal_gdb_literal_in_buildroot
[2] https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
[3] https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=gdb/configure.ac;h=d4133ea71e2207c459cab0cadf51a810fe200141;hb=f99d9b9f436dce02aa06839395c67d400b2a0de0#l120

Hello Gerhard,

Thanks for this contribution. I'm adding in Cc a few other people to
hopefully get their attention.

On Wed, 23 Jan 2019 14:15:39 +0100
Gerhard Heift <gerhard@heift.name> wrote:

> Before striping the binaries in $TARGET_DIR, the debug information is saved to
> $HOST_DIR/lib/debug using the following command as described in [1]:
> 
>   $ objcopy --only-keep-debug --compress-debug-sections $file
> 
> This allows remote debugging as described in section 8.12.2 with striped
> binaries in the final image. Therefore the build ID is included during the build
> process, if debug information is enabled via BR2_ENABLE_DEBUG.
> 
> [1] https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
> 
> Signed-off-by: Gerhard Heift <Gerhard@Heift.Name>

When you build with BR2_ENABLE_DEBUG=y and BR2_STRIP_strip=y, all the
libraries in $(STAGING_DIR) are installed with debugging symbols, and
they are not stripped.

As explained in our manual section 8.12.2, we provide a gdbinit that
tells gdb to look in $(STAGING_DIR) for debugging symbols.

Your patch has the effect of duplicating the debugging symbols: they
are already in $(STAGING_DIR), and they will now be duplicated in
$(HOST_DIR)/lib/debug. The only benefit that I can see is that the
gdbinit file is no longer needed because gdb will automatically look in
$(HOST_DIR)/lib/debug. But that can probably be resolved by making
$(HOST_DIR)/lib/debug a symlink to $(STAGING_DIR). Could you try this
instead ?

Thanks a lot,

Thomas

El lun., 4 feb. 2019 a las 17:58, Thomas Petazzoni
(<thomas.petazzoni@bootlin.com>) escribió:
>
> Hello Gerhard,
>
> Thanks for this contribution. I'm adding in Cc a few other people to
> hopefully get their attention.
>
> On Wed, 23 Jan 2019 14:15:39 +0100
> Gerhard Heift <gerhard@heift.name> wrote:
>
> > Before striping the binaries in $TARGET_DIR, the debug information is saved to
> > $HOST_DIR/lib/debug using the following command as described in [1]:
> >
> >   $ objcopy --only-keep-debug --compress-debug-sections $file
> >
> > This allows remote debugging as described in section 8.12.2 with striped
> > binaries in the final image. Therefore the build ID is included during the build
> > process, if debug information is enabled via BR2_ENABLE_DEBUG.
> >
> > [1] https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
> >
> > Signed-off-by: Gerhard Heift <Gerhard@Heift.Name>
>
> When you build with BR2_ENABLE_DEBUG=y and BR2_STRIP_strip=y, all the
> libraries in $(STAGING_DIR) are installed with debugging symbols, and
> they are not stripped.
>
> As explained in our manual section 8.12.2, we provide a gdbinit that
> tells gdb to look in $(STAGING_DIR) for debugging symbols.
>
> Your patch has the effect of duplicating the debugging symbols: they
> are already in $(STAGING_DIR), and they will now be duplicated in
> $(HOST_DIR)/lib/debug. The only benefit that I can see is that the
> gdbinit file is no longer needed because gdb will automatically look in
> $(HOST_DIR)/lib/debug. But that can probably be resolved by making
> $(HOST_DIR)/lib/debug a symlink to $(STAGING_DIR). Could you try this
> instead ?

The difference that this patch brings is that now also binaries can be
debugged based from the target directory, i.e. you don't need to find
the binary in output/build/foo-version/somewhere/.

Today, the staging directory only contains packages that explicitly
set FOO_INSTALL_STAGING = YES, which is typically only done for
libraries.

Best regards,
Thomas

On Mon, 4 Feb 2019 19:05:26 +0100
Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> wrote:

> The difference that this patch brings is that now also binaries can be
> debugged based from the target directory, i.e. you don't need to find
> the binary in output/build/foo-version/somewhere/.
> 
> Today, the staging directory only contains packages that explicitly
> set FOO_INSTALL_STAGING = YES, which is typically only done for
> libraries.

Yes, but:

1) Typically the binary being debugged has to be passed explicitly as
   gdb argument, it's not automatically "inferred" by gdb. So having it
   in output/build/foo-<version>/somewhere/ or elsewhere doesn't make a
   huge difference.

2) If for convenience reasons we want to make all binaries with
   debugging symbols easily available, then the long term solution we
   have been discussing for a while is to install all packages to
   STAGING_DIR. So basically I'm more for extending STAGING_DIR to
   contain binaries rather than to create a partial copy of STAGING_DIR
   in HOST_DIR/lib/debug.

Thomas

All,

On 2019-02-04 19:19 +0100, Thomas Petazzoni spake thusly:
> On Mon, 4 Feb 2019 19:05:26 +0100
> Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> wrote:
> 
> > The difference that this patch brings is that now also binaries can be
> > debugged based from the target directory, i.e. you don't need to find
> > the binary in output/build/foo-version/somewhere/.
> > 
> > Today, the staging directory only contains packages that explicitly
> > set FOO_INSTALL_STAGING = YES, which is typically only done for
> > libraries.
> 
> Yes, but:
> 
> 1) Typically the binary being debugged has to be passed explicitly as
>    gdb argument, it's not automatically "inferred" by gdb. So having it
>    in output/build/foo-<version>/somewhere/ or elsewhere doesn't make a
>    huge difference.
> 
> 2) If for convenience reasons we want to make all binaries with
>    debugging symbols easily available, then the long term solution we
>    have been discussing for a while is to install all packages to
>    STAGING_DIR. So basically I'm more for extending STAGING_DIR to
>    contain binaries rather than to create a partial copy of STAGING_DIR
>    in HOST_DIR/lib/debug.

I side with Thomas P. on that one.

We've had this discussion a few times already, and the conslusion was to
try and install everything in staging.

Then, target can be created at the end, by copying staging/ to target/
and do the target-finalize step on that copy.

Or so that is the main idea, and it can be refined to account for subtle
details, such as the on-going per-package directory and top-level
parallel build...

Regards,
Yann E. MORIN.