Message ID | 20190924192435.19413-1-gabriel@inconstante.net.br |
---|---|
State | New |
Headers | show |
Series | Add helper script for glibc debugging | expand |
On 9/24/19 3:24 PM, Gabriel F. T. Gomes wrote: > From: "Gabriel F. T. Gomes" <gabrielftg@linux.ibm.com> > > At Cauldron, while Florian showed his development workflow to us, we had > a brief discussion about debugging glibc with GDB. After that, I spent > some time with Arjun and we shared our debugging scripts and tricks, > then we came to the conclusion that this could be helpful to other > people, maybe even more so to newcomers. So, I got my off-tree script > and converted it into this auto-generated version. > > Still at Cauldron, Carlos mentioned that it wasn't trivial to let GDB > know about the symbols within glibc. This script doesn't solve this for > every glibc debugging needs (I suppose it doesn't help at all when > debugging the initial steps of program loading, for example), but it > makes it a little easier when debugging test cases, as it automatically > loads the symbols from them. > > Maybe there are more trivial ways to do it, but that's what I have been > using for some time. I like this script! I don't immediately see how this solves the required gdb setup though? https://sourceware.org/glibc/wiki/Testing/Builds#Required_gdb_setup Did I miss something? It looks like the script is *almost* there in terms of functionality to make it possible to debug single-threaded and multi-threaded programs. > -- 8< -- > This patch adds a new make rule that generates a helper script for > debugging glibc test cases. The new script, debugglibc.sh, is similar > to testrun.sh, in the sense that it allows the execution of the > specified test case, however, it opens the test case in GDB, setting the > library path the same way that testrun.sh does. The commands are based > on the instruction on the wiki page for glibc debugging [1]. > > By default, the script tells GDB to load the test case for symbol > information, so that, when a breakpoint is hit, the call stack is > displayed correctly (instead of printing lots of '??'s). For instance, > after running 'make' and 'make check', one could do the following: > > ./debugglibc.sh -t $PWD/stdlib/tst-strfrom -B strfromf -B strfromd > > Reading symbols from /home/gabriel/build/x86_64/glibc/elf/ld.so... > Breakpoint 1 at 0x1098 > add symbol table from file "/home/gabriel/build/x86_64/glibc/stdlib/tst-strfrom" > > Breakpoint 1, 0x00007ffff7fd5098 in _dl_start_user () from /home/gabriel/build/x86_64/glibc/elf/ld.so > Breakpoint 2 at 0x7ffff7e4dba0: file strfrom-skeleton.c, line 41. > Breakpoint 3 at 0x7ffff7e4ddd0: file strfrom-skeleton.c, line 41. > > Notice that the script will always start GDB with the program running > and halted at _dl_start_user. So, in order to reach the actual > breakpoint of interest, one should hit 'c', not 'r': > > >>> c > Continuing. > Testing in locale: C > > Breakpoint 2, strfromf (dest=dest@entry=0x7fffffffe02b "\377", size=5, format=0x405010 "%g", f=12345.3447) at strfrom-skeleton.c:41 > 41 sfile.f._sbf._f._lock = NULL; > > An inspect the call stack with 'bt', as usual, and see symbols from both > the test case and from the libraries themselves: > > >>> bt > #0 strfromf (dest=dest@entry=0x7fffffffe02b "\377", size=5, format=0x405010 "%g", f=12345.3447) at strfrom-skeleton.c:41 > #1 0x000000000040246a in test_f () at tst-strfrom.c:67 > #2 test_locale (locale=locale@entry=0x405046 "C") at tst-strfrom.c:77 > #3 0x000000000040358b in do_test () at tst-strfrom.c:84 > #4 legacy_test_function (argc=<optimized out>, argv=<optimized out>) at ../test-skeleton.c:56 > #5 0x0000000000403b25 in support_test_main (argc=1, argc@entry=2, argv=0x7fffffffe258, argv@entry=0x7fffffffe250, config=config@entry=0x7fffffffe120) at support_test_main.c:350 > #6 0x0000000000402315 in main (argc=argc@entry=2, argv=argv@entry=0x7fffffffe250) at ../support/test-driver.c:168 > #7 0x00007ffff7e35deb in __libc_start_main (main=0x4022e0 <main>, argc=2, argv=0x7fffffffe250, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe248) at ../csu/libc-start.c:308 > #8 0x000000000040234a in _start () at ../sysdeps/x86_64/start.S:120 > > Tested for x86_64. > > [1] https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging > --- > Makefile | 159 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 158 insertions(+), 1 deletion(-) > > diff --git a/Makefile b/Makefile > index 67ddd01bfe..d62f5a348c 100644 > --- a/Makefile > +++ b/Makefile > @@ -187,7 +187,164 @@ $(common-objpfx)testrun.sh: $(common-objpfx)config.make \ > mv -f $@T $@ > postclean-generated += testrun.sh > > -others: $(common-objpfx)testrun.sh > +define debugglibc > +#!/bin/bash > + > +# Defaults > +SOURCE_DIR="$$HOME/src/glibc" > +BUILD_DIR="$$PWD" > +DIRECT=true > +SYMBOLSFILE=true > + > +unset TESTCASE > +unset CMD_FILE > +unset BREAKPOINTS > + > +usage() > +{ > +cat << EOF > +Usage: $$0 [OPTION] > + > + -h, --help > + Prints this message and leaves. > + > + The following options require one argument: > + > + -b, --build, --build-dir > + Absolute path to glibc build directory > + Defaults to \$$PWD > + -s, --source, --source-dir > + Absolute path to glibc source files > + Defaults to \$$HOME/src/glibc > + -t, --testcase > + Absolute path to program to be tested > + Defaults to \$$PWD/stdlib/tst-strfrom > + -B, --breakpoint > + Breakpoints to set at the beginning of the execution > + (each breakpoint demands its own -B option, e.g. -B foo -B bar) > + > + The following options do not take arguments: > + > + -I, --no-direct > + Selects whether to pass the flag --direct to gdb. > + Required for glibc test cases and not allowed for non-glibc tests. > + Default behaviour is to pass the flag --direct to gdb. > + -S, --no-symbols-file > + Do not tell GDB to load debug symbols from the testcase. > +EOF > +} > + > +# Parse input options > +while [[ $$# > 0 ]] > +do > + key="$$1" > + case $$key in > + -h|--help) > + usage > + exit 0 > + ;; > + -b|--build|--build-dir) > + BUILD_DIR=$$2 > + shift > + ;; > + -s|--source|--source-dir) > + SOURCE_DIR=$$2 > + shift > + ;; > + -t|--testcase) > + TESTCASE=$$2 > + shift > + ;; > + -B|--breakpoint) > + BREAKPOINTS="$$BREAKPOINTS\n break $$2" > + shift > + ;; > + -I|--no-direct) > + DIRECT=false > + ;; > + -S|--no-symbols-file) > + SYMBOLSFILE=false > + ;; > + *) > + usage > + exit 1 > + ;; > + esac > + shift > +done > + > +# Update arguments > +if [ ! -v TESTCASE ] > +then > + TESTCASE="$$BUILD_DIR/stdlib/tst-strfrom" > +fi > +if [ ! -v CMD_FILE ] > +then > + CMD_FILE="$$BUILD_DIR/debugglibc.gdb" > +fi > + > +# Expand direct argument > +if [ "$$DIRECT" == true ] > +then > + DIRECT="--direct" > +else > + DIRECT="" > +fi > + > +# Expand symbols loading command > +if [ "$$SYMBOLSFILE" == true ] > +then > + SYMBOLSFILE="add-symbol-file $${TESTCASE}" > +else > + SYMBOLSFILE="" > +fi > + > +# GDB commands template > +template () > +{ > +cat <<EOF > +set environment C -E -x c-header > +break _dl_start_user > +__SYMBOLSFILE__ > +run --library-path $(rtld-prefix) \ > +__TESTCASE__ __DIRECT__ > +__BREAKPOINTS__ > +EOF > +} > + > +# Generate the commands file for gdb initialization > +template | sed \ > + -e "s|__SYMBOLSFILE__|$$SYMBOLSFILE|" \ > + -e "s|__TESTCASE__|$$TESTCASE|" \ > + -e "s|__DIRECT__|$$DIRECT|" \ > + -e "s|__BREAKPOINTS__|$$BREAKPOINTS|" \ > + > $$CMD_FILE > + > +echo > +echo "Debugging glibc..." > +echo "Build directory : $$BUILD_DIR" > +echo "Source directory : $$SOURCE_DIR" > +echo "GLIBC Testcase : $$TESTCASE" > +echo "GDB Commands : $$CMD_FILE" > +echo > + > +# We need to make sure that gdb is linked against the standalone glibc > +# so that it picks up the correct nptl_db/libthread_db.so. So that means > +# invoking gdb using the standalone glibc's linker. > +$(test-via-rtld-prefix) \ > +/usr/bin/gdb -q -x $${CMD_FILE} -d $${SOURCE_DIR} $${BUILD_DIR}/elf/ld.so > +endef > + > +# This is another handy script for debugging dynamically linked program > +# against the current libc build for testing. > +$(common-objpfx)debugglibc.sh: $(common-objpfx)config.make \ > + $(..)Makeconfig $(..)Makefile > + $(file >$@T,$(debugglibc)) > + chmod a+x $@T > + mv -f $@T $@ > +postclean-generated += debugglibc.sh debugglibc.gdb > + > +others: $(common-objpfx)testrun.sh $(common-objpfx)debugglibc.sh > > # Makerules creates a file `stubs' in each subdirectory, which > # contains `#define __stub_FUNCTION' for each function defined in that >
Hi, Carlos, On Tue, 24 Sep 2019, Carlos O'Donell wrote: >I don't immediately see how this solves the required gdb setup though? > >https://sourceware.org/glibc/wiki/Testing/Builds#Required_gdb_setup > >Did I miss something? I think you found a problem with the conversion from my off-tree script to this auto-generated version. In the off-tree script, the argument to --library-path was hardcoded and it included the path to nptl_db, as mentioned in the wiki page I used [1] (look for occurrences of "${GLIBC}/nptl_db:"). In the auto-generated version, I used the makefile expansion of "$(rtld-prefix)", which I though gave the same path, but I failed to notice that nptl_db was missing (nptl is there, but not nptl_db). It should be easy to complement the argument to --library-path. I'll check if that actually solves the problem (as the wiki [1] seems to suggest), then send another version of this patch. Thanks for your careful review! :) [1] https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging
On Tue, Sep 24, 2019 at 04:24:35PM -0300, Gabriel F. T. Gomes wrote: [...] > +$(test-via-rtld-prefix) \ > +/usr/bin/gdb -q -x $${CMD_FILE} -d $${SOURCE_DIR} $${BUILD_DIR}/elf/ld.so Please do not hardcode the full path to gdb.
On Tue, 24 Sep 2019, Gabriel F. T. Gomes wrote: > +# Defaults > +SOURCE_DIR="$$HOME/src/glibc" I think this should default to the actual source directory used when configuring glibc. > + -t, --testcase > + Absolute path to program to be tested > + Defaults to \$$PWD/stdlib/tst-strfrom I don't think there should be a default for this at all. (I find use of --enable-hardcoded-path-in-tests convenient for debugging to avoid the complications this patch deals with.)
On Sep 24 2019, "Gabriel F. T. Gomes" <gabriel@inconstante.net.br> wrote: > Maybe there are more trivial ways to do it, but that's what I have been > using for some time. The easiest way is to configure with --enable-hardcoded-path-in-tests or setting build-hardcoded-path-in-tests, though that only works for test programs. If you need to debug one of the included programs there needs to be a way to inject `-Wl,-dynamic-linker=$(elf-objpfx)$(rtld-installed-name) -Wl,-rpath=$(rpath-link)$(patsubst %,:%,$(sysdep-library-path))' into the link command (setting config-LDFLAGS used to accomplish this, but that variable no longer exists). Andreas.
diff --git a/Makefile b/Makefile index 67ddd01bfe..d62f5a348c 100644 --- a/Makefile +++ b/Makefile @@ -187,7 +187,164 @@ $(common-objpfx)testrun.sh: $(common-objpfx)config.make \ mv -f $@T $@ postclean-generated += testrun.sh -others: $(common-objpfx)testrun.sh +define debugglibc +#!/bin/bash + +# Defaults +SOURCE_DIR="$$HOME/src/glibc" +BUILD_DIR="$$PWD" +DIRECT=true +SYMBOLSFILE=true + +unset TESTCASE +unset CMD_FILE +unset BREAKPOINTS + +usage() +{ +cat << EOF +Usage: $$0 [OPTION] + + -h, --help + Prints this message and leaves. + + The following options require one argument: + + -b, --build, --build-dir + Absolute path to glibc build directory + Defaults to \$$PWD + -s, --source, --source-dir + Absolute path to glibc source files + Defaults to \$$HOME/src/glibc + -t, --testcase + Absolute path to program to be tested + Defaults to \$$PWD/stdlib/tst-strfrom + -B, --breakpoint + Breakpoints to set at the beginning of the execution + (each breakpoint demands its own -B option, e.g. -B foo -B bar) + + The following options do not take arguments: + + -I, --no-direct + Selects whether to pass the flag --direct to gdb. + Required for glibc test cases and not allowed for non-glibc tests. + Default behaviour is to pass the flag --direct to gdb. + -S, --no-symbols-file + Do not tell GDB to load debug symbols from the testcase. +EOF +} + +# Parse input options +while [[ $$# > 0 ]] +do + key="$$1" + case $$key in + -h|--help) + usage + exit 0 + ;; + -b|--build|--build-dir) + BUILD_DIR=$$2 + shift + ;; + -s|--source|--source-dir) + SOURCE_DIR=$$2 + shift + ;; + -t|--testcase) + TESTCASE=$$2 + shift + ;; + -B|--breakpoint) + BREAKPOINTS="$$BREAKPOINTS\n break $$2" + shift + ;; + -I|--no-direct) + DIRECT=false + ;; + -S|--no-symbols-file) + SYMBOLSFILE=false + ;; + *) + usage + exit 1 + ;; + esac + shift +done + +# Update arguments +if [ ! -v TESTCASE ] +then + TESTCASE="$$BUILD_DIR/stdlib/tst-strfrom" +fi +if [ ! -v CMD_FILE ] +then + CMD_FILE="$$BUILD_DIR/debugglibc.gdb" +fi + +# Expand direct argument +if [ "$$DIRECT" == true ] +then + DIRECT="--direct" +else + DIRECT="" +fi + +# Expand symbols loading command +if [ "$$SYMBOLSFILE" == true ] +then + SYMBOLSFILE="add-symbol-file $${TESTCASE}" +else + SYMBOLSFILE="" +fi + +# GDB commands template +template () +{ +cat <<EOF +set environment C -E -x c-header +break _dl_start_user +__SYMBOLSFILE__ +run --library-path $(rtld-prefix) \ +__TESTCASE__ __DIRECT__ +__BREAKPOINTS__ +EOF +} + +# Generate the commands file for gdb initialization +template | sed \ + -e "s|__SYMBOLSFILE__|$$SYMBOLSFILE|" \ + -e "s|__TESTCASE__|$$TESTCASE|" \ + -e "s|__DIRECT__|$$DIRECT|" \ + -e "s|__BREAKPOINTS__|$$BREAKPOINTS|" \ + > $$CMD_FILE + +echo +echo "Debugging glibc..." +echo "Build directory : $$BUILD_DIR" +echo "Source directory : $$SOURCE_DIR" +echo "GLIBC Testcase : $$TESTCASE" +echo "GDB Commands : $$CMD_FILE" +echo + +# We need to make sure that gdb is linked against the standalone glibc +# so that it picks up the correct nptl_db/libthread_db.so. So that means +# invoking gdb using the standalone glibc's linker. +$(test-via-rtld-prefix) \ +/usr/bin/gdb -q -x $${CMD_FILE} -d $${SOURCE_DIR} $${BUILD_DIR}/elf/ld.so +endef + +# This is another handy script for debugging dynamically linked program +# against the current libc build for testing. +$(common-objpfx)debugglibc.sh: $(common-objpfx)config.make \ + $(..)Makeconfig $(..)Makefile + $(file >$@T,$(debugglibc)) + chmod a+x $@T + mv -f $@T $@ +postclean-generated += debugglibc.sh debugglibc.gdb + +others: $(common-objpfx)testrun.sh $(common-objpfx)debugglibc.sh # Makerules creates a file `stubs' in each subdirectory, which # contains `#define __stub_FUNCTION' for each function defined in that