Message ID | 87tuo24kxi.fsf@oldenburg.str.redhat.com |
---|---|
State | New |
Headers | show |
Series | [v3] nptl_db: Support different libpthread/ld.so load orders (bug 27744) | expand |
* Florian Weimer via Libc-alpha: > libthread_db is loaded once GDB encounters libpthread, and at this > point, ld.so may not have been processed by GDB yet. As a result, > _rtld_global cannot be accessed by regular means from libthread_db. > To make this work until GDB can be fixed, acess _rtld_global through > a pointer stored in libpthread. > > The new test does not reproduce bug 27744 with > --disable-hardcoded-path-in-tests, but is still a valid smoke test. > With --enable-hardcoded-path-in-tests, it is necessary to avoid > add-symbol-file because this can tickle a GDB bug. > > Fixes commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move > stack list variables into _rtld_global"). > > --- > v3: Mark test as UNSUPPORTED if launching gdb fails with ENOENT. > This is the version I'd like to commit soon. I've pushed this version given that Pedro acknowledged v2. Thanks, Florian
On Wed, Apr 21, 2021 at 6:04 AM Florian Weimer via Libc-alpha <libc-alpha@sourceware.org> wrote: > > * Florian Weimer via Libc-alpha: > > > libthread_db is loaded once GDB encounters libpthread, and at this > > point, ld.so may not have been processed by GDB yet. As a result, > > _rtld_global cannot be accessed by regular means from libthread_db. > > To make this work until GDB can be fixed, acess _rtld_global through > > a pointer stored in libpthread. > > > > The new test does not reproduce bug 27744 with > > --disable-hardcoded-path-in-tests, but is still a valid smoke test. > > With --enable-hardcoded-path-in-tests, it is necessary to avoid > > add-symbol-file because this can tickle a GDB bug. > > > > Fixes commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move > > stack list variables into _rtld_global"). > > > > --- > > v3: Mark test as UNSUPPORTED if launching gdb fails with ENOENT. > > This is the version I'd like to commit soon. > > I've pushed this version given that Pedro acknowledged v2. > I got FAIL: nptl/tst-pthread-gdb-attach FAIL: nptl/tst-pthread-gdb-attach-static when testing i686 build on Fedora 33: [New LWP 1329590] Trying host libthread_db library: /export/build/gnu/tools-build/glibc-32bit-cet-gitlab/build-i686-linux/nptl_db/libthread_db.so.1. dlopen failed: /export/build/gnu/tools-build/glibc-32bit-cet-gitlab/build-i686-linux/nptl_db/libthread_db.so.1: wrong ELF class: ELFCLASS32. thread_db_load_search returning 0 Is it possible to skip this when gdb and libthread_db.so.1 have different ELC classes?
The 04/21/2021 13:47, Florian Weimer via Libc-alpha wrote: > * Florian Weimer via Libc-alpha: > > > libthread_db is loaded once GDB encounters libpthread, and at this > > point, ld.so may not have been processed by GDB yet. As a result, > > _rtld_global cannot be accessed by regular means from libthread_db. > > To make this work until GDB can be fixed, acess _rtld_global through > > a pointer stored in libpthread. > > > > The new test does not reproduce bug 27744 with > > --disable-hardcoded-path-in-tests, but is still a valid smoke test. > > With --enable-hardcoded-path-in-tests, it is necessary to avoid > > add-symbol-file because this can tickle a GDB bug. > > > > Fixes commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move > > stack list variables into _rtld_global"). > > > > --- > > v3: Mark test as UNSUPPORTED if launching gdb fails with ENOENT. > > This is the version I'd like to commit soon. > > I've pushed this version given that Pedro acknowledged v2. on the aarch64 buildbot i see FAIL: nptl/tst-pthread-gdb-attach $ cat nptl/tst-pthread-gdb-attach.out +set debug libthread-db 1 +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db +attach 461328 [New LWP 461329] Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. td_ta_new failed: application not linked with libthread thread_db_load_search returning 0 Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. [Thread debugging using libthread_db enabled] Using host libthread_db library "/work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1". thread_db_load_search returning 1 0x0000ffff96d3bb18 in __futex_abstimed_wait_common64 (futex_word=0xffff96cca210, expected=461329, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, +break debugger_inspection_point Breakpoint 1 at 0x1ea8: file tst-pthread-gdb-attach.c, line 76. +continue Timed out: killed the child process Termination time: 2021-04-21T18:44:39.686118409 Last write to standard output: 2021-04-21T18:44:19.867449485 the test nptl/tst-pthread-gdb-attach-static passes: $ cat nptl/tst-pthread-gdb-attach-static.out +set debug libthread-db 1 +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db +attach 461347 [New LWP 461348] Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. [Thread debugging using libthread_db enabled] Using host libthread_db library "/work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1". thread_db_load_search returning 1 0x0000000000419c40 in __futex_abstimed_wait_common64 (futex_word=0xffff94b08590, expected=461348, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, +break debugger_inspection_point Breakpoint 1 at 0x400720: file tst-pthread-gdb-attach.c, line 76. +continue [Switching to Thread 0xffff94b084c0 (LWP 461348)] Thread 2 "tst-pthread-gdb" hit Breakpoint 1, debugger_inspection_point () at tst-pthread-gdb-attach.c:76 76 } +thread 1 [Switching to thread 1 (Thread 0x828d000 (LWP 461347))] #0 0x0000000000419c40 in __futex_abstimed_wait_common64 (futex_word=0xffff94b08590, expected=461348, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, +print altered_by_debugger $1 = 0 +print altered_by_debugger = 1 $2 = 1 +thread 2 [Switching to thread 2 (Thread 0xffff94b084c0 (LWP 461348))] #0 debugger_inspection_point () at tst-pthread-gdb-attach.c:76 76 } +print altered_by_debugger $3 = 0 +print altered_by_debugger = 2 $4 = 2 +continue [Thread 0xffff94b084c0 (LWP 461348) exited] [Inferior 1 (process 461347) exited normally]
* Szabolcs Nagy: > The 04/21/2021 13:47, Florian Weimer via Libc-alpha wrote: >> * Florian Weimer via Libc-alpha: >> >> > libthread_db is loaded once GDB encounters libpthread, and at this >> > point, ld.so may not have been processed by GDB yet. As a result, >> > _rtld_global cannot be accessed by regular means from libthread_db. >> > To make this work until GDB can be fixed, acess _rtld_global through >> > a pointer stored in libpthread. >> > >> > The new test does not reproduce bug 27744 with >> > --disable-hardcoded-path-in-tests, but is still a valid smoke test. >> > With --enable-hardcoded-path-in-tests, it is necessary to avoid >> > add-symbol-file because this can tickle a GDB bug. >> > >> > Fixes commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move >> > stack list variables into _rtld_global"). >> > >> > --- >> > v3: Mark test as UNSUPPORTED if launching gdb fails with ENOENT. >> > This is the version I'd like to commit soon. >> >> I've pushed this version given that Pedro acknowledged v2. > > on the aarch64 buildbot i see > > FAIL: nptl/tst-pthread-gdb-attach > > $ cat nptl/tst-pthread-gdb-attach.out > +set debug libthread-db 1 > +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach > add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" > +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db > +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db > +attach 461328 > [New LWP 461329] > Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. > td_ta_new failed: application not linked with libthread > thread_db_load_search returning 0 Unfortunately I cannot reproduce this with binutils 2.30, GDB 8.2 and GCC 8.4.1. Has your environment newer or older versions? Thanks, Florian
The 04/22/2021 10:33, Florian Weimer wrote: > * Szabolcs Nagy: > > The 04/21/2021 13:47, Florian Weimer via Libc-alpha wrote: > >> * Florian Weimer via Libc-alpha: > >> > >> > libthread_db is loaded once GDB encounters libpthread, and at this > >> > point, ld.so may not have been processed by GDB yet. As a result, > >> > _rtld_global cannot be accessed by regular means from libthread_db. > >> > To make this work until GDB can be fixed, acess _rtld_global through > >> > a pointer stored in libpthread. > >> > > >> > The new test does not reproduce bug 27744 with > >> > --disable-hardcoded-path-in-tests, but is still a valid smoke test. > >> > With --enable-hardcoded-path-in-tests, it is necessary to avoid > >> > add-symbol-file because this can tickle a GDB bug. > >> > > >> > Fixes commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move > >> > stack list variables into _rtld_global"). > >> > > >> > --- > >> > v3: Mark test as UNSUPPORTED if launching gdb fails with ENOENT. > >> > This is the version I'd like to commit soon. > >> > >> I've pushed this version given that Pedro acknowledged v2. > > > > on the aarch64 buildbot i see > > > > FAIL: nptl/tst-pthread-gdb-attach > > > > $ cat nptl/tst-pthread-gdb-attach.out > > +set debug libthread-db 1 > > +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach > > add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" > > +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db > > +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db > > +attach 461328 > > [New LWP 461329] > > Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. > > td_ta_new failed: application not linked with libthread > > thread_db_load_search returning 0 > > Unfortunately I cannot reproduce this with binutils 2.30, GDB 8.2 and > GCC 8.4.1. Has your environment newer or older versions? ubuntu flavour of gcc 8.2.0 binutils 2.31.1 gdb 8.2 happens on armhf and aarch64 too.
* H. J. Lu: > On Wed, Apr 21, 2021 at 6:04 AM Florian Weimer via Libc-alpha > <libc-alpha@sourceware.org> wrote: >> >> * Florian Weimer via Libc-alpha: >> >> > libthread_db is loaded once GDB encounters libpthread, and at this >> > point, ld.so may not have been processed by GDB yet. As a result, >> > _rtld_global cannot be accessed by regular means from libthread_db. >> > To make this work until GDB can be fixed, acess _rtld_global through >> > a pointer stored in libpthread. >> > >> > The new test does not reproduce bug 27744 with >> > --disable-hardcoded-path-in-tests, but is still a valid smoke test. >> > With --enable-hardcoded-path-in-tests, it is necessary to avoid >> > add-symbol-file because this can tickle a GDB bug. >> > >> > Fixes commit 1daccf403b1bd86370eb94edca794dc106d02039 ("nptl: Move >> > stack list variables into _rtld_global"). >> > >> > --- >> > v3: Mark test as UNSUPPORTED if launching gdb fails with ENOENT. >> > This is the version I'd like to commit soon. >> >> I've pushed this version given that Pedro acknowledged v2. >> > > I got > > FAIL: nptl/tst-pthread-gdb-attach > FAIL: nptl/tst-pthread-gdb-attach-static > > when testing i686 build on Fedora 33: > > [New LWP 1329590] > Trying host libthread_db library: > /export/build/gnu/tools-build/glibc-32bit-cet-gitlab/build-i686-linux/nptl_db/libthread_db.so.1. > dlopen failed: /export/build/gnu/tools-build/glibc-32bit-cet-gitlab/build-i686-linux/nptl_db/libthread_db.so.1: > wrong ELF class: ELFCLASS32. > thread_db_load_search returning 0 > > Is it possible to skip this when gdb and libthread_db.so.1 have different > ELC classes? Right, that seems a good idea. I've put together a hackish patch: nptl: Check for compatible GDB in nptl/tst-pthread-gdb-attach <https://sourceware.org/pipermail/libc-alpha/2021-April/125301.html> Thanks, Florian
The 04/22/2021 10:05, Szabolcs Nagy via Gdb-patches wrote: > The 04/22/2021 10:33, Florian Weimer wrote: > > * Szabolcs Nagy: > > > on the aarch64 buildbot i see > > > > > > FAIL: nptl/tst-pthread-gdb-attach > > > > > > $ cat nptl/tst-pthread-gdb-attach.out > > > +set debug libthread-db 1 > > > +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach > > > add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" > > > +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db > > > +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db > > > +attach 461328 > > > [New LWP 461329] > > > Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. > > > td_ta_new failed: application not linked with libthread > > > thread_db_load_search returning 0 > > > > Unfortunately I cannot reproduce this with binutils 2.30, GDB 8.2 and > > GCC 8.4.1. Has your environment newer or older versions? > > ubuntu flavour of > > gcc 8.2.0 > binutils 2.31.1 > gdb 8.2 > > happens on armhf and aarch64 too. happens with gcc-10.2.0, gdb-9.2, binutils-2.34 too (still ubuntu)
* Szabolcs Nagy: > The 04/22/2021 10:05, Szabolcs Nagy via Gdb-patches wrote: >> The 04/22/2021 10:33, Florian Weimer wrote: >> > * Szabolcs Nagy: >> > > on the aarch64 buildbot i see >> > > >> > > FAIL: nptl/tst-pthread-gdb-attach >> > > >> > > $ cat nptl/tst-pthread-gdb-attach.out >> > > +set debug libthread-db 1 >> > > +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach >> > > add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" >> > > +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db >> > > +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db >> > > +attach 461328 >> > > [New LWP 461329] >> > > Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. >> > > td_ta_new failed: application not linked with libthread >> > > thread_db_load_search returning 0 >> > >> > Unfortunately I cannot reproduce this with binutils 2.30, GDB 8.2 and >> > GCC 8.4.1. Has your environment newer or older versions? >> >> ubuntu flavour of >> >> gcc 8.2.0 >> binutils 2.31.1 >> gdb 8.2 >> >> happens on armhf and aarch64 too. > > happens with gcc-10.2.0, gdb-9.2, binutils-2.34 too > (still ubuntu) Still no luck with binutils 2.35.1, a GCC 11 snapshot, and GDB 10.1. Do you run this test as a regular user? Could it be caused by YAMA ptrace policies? Thanks, Florian
The 04/22/2021 13:55, Florian Weimer wrote: > * Szabolcs Nagy: > > The 04/22/2021 10:05, Szabolcs Nagy via Gdb-patches wrote: > >> The 04/22/2021 10:33, Florian Weimer wrote: > >> > * Szabolcs Nagy: > >> > > on the aarch64 buildbot i see > >> > > > >> > > FAIL: nptl/tst-pthread-gdb-attach > >> > > > >> > > $ cat nptl/tst-pthread-gdb-attach.out > >> > > +set debug libthread-db 1 > >> > > +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach > >> > > add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" > >> > > +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db > >> > > +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db > >> > > +attach 461328 > >> > > [New LWP 461329] > >> > > Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. > >> > > td_ta_new failed: application not linked with libthread > >> > > thread_db_load_search returning 0 > >> > > >> > Unfortunately I cannot reproduce this with binutils 2.30, GDB 8.2 and > >> > GCC 8.4.1. Has your environment newer or older versions? > >> > >> ubuntu flavour of > >> > >> gcc 8.2.0 > >> binutils 2.31.1 > >> gdb 8.2 > >> > >> happens on armhf and aarch64 too. > > > > happens with gcc-10.2.0, gdb-9.2, binutils-2.34 too > > (still ubuntu) > > Still no luck with binutils 2.35.1, a GCC 11 snapshot, and GDB 10.1. > > Do you run this test as a regular user? Could it be caused by YAMA > ptrace policies? $ cat /proc/sys/kernel/yama/ptrace_scope 0 and the -static test passes so there gdb works. i can reproduce it on all ubuntu setups i have.
The 04/22/2021 13:44, Szabolcs Nagy via Gdb-patches wrote: > The 04/22/2021 13:55, Florian Weimer wrote: > > * Szabolcs Nagy: > > > The 04/22/2021 10:05, Szabolcs Nagy via Gdb-patches wrote: > > >> The 04/22/2021 10:33, Florian Weimer wrote: > > >> > * Szabolcs Nagy: > > >> > > on the aarch64 buildbot i see > > >> > > > > >> > > FAIL: nptl/tst-pthread-gdb-attach > > >> > > > > >> > > $ cat nptl/tst-pthread-gdb-attach.out > > >> > > +set debug libthread-db 1 > > >> > > +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach > > >> > > add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" > > >> > > +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db > > >> > > +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db > > >> > > +attach 461328 > > >> > > [New LWP 461329] > > >> > > Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. > > >> > > td_ta_new failed: application not linked with libthread > > >> > > thread_db_load_search returning 0 > > >> > > > >> > Unfortunately I cannot reproduce this with binutils 2.30, GDB 8.2 and > > >> > GCC 8.4.1. Has your environment newer or older versions? > > >> > > >> ubuntu flavour of > > >> > > >> gcc 8.2.0 > > >> binutils 2.31.1 > > >> gdb 8.2 > > >> > > >> happens on armhf and aarch64 too. > > > > > > happens with gcc-10.2.0, gdb-9.2, binutils-2.34 too > > > (still ubuntu) > > > > Still no luck with binutils 2.35.1, a GCC 11 snapshot, and GDB 10.1. > > > > Do you run this test as a regular user? Could it be caused by YAMA > > ptrace policies? > > $ cat /proc/sys/kernel/yama/ptrace_scope > 0 > > and the -static test passes so there gdb works. > i can reproduce it on all ubuntu setups i have. if i rerun the link command of the test exe but with -no-pie instead of -pie then the test passes with that binary. i suspect gdb places the breakpoint at the wrong place in pie for some reason. can be ubuntu tooling specific. see the breakpoint location (the exe base offset is missing): +attach 1254516 [New LWP 1254517] Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. td_ta_new failed: application not linked with libthread thread_db_load_search returning 0 Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. [Thread debugging using libthread_db enabled] Using host libthread_db library "/home/szabolcs/try/build/nptl_db/libthread_db.so.1". thread_db_load_search returning 1 0x0000ffff9d3d89c4 in __futex_abstimed_wait_common64 (futex_word=0xffff9d363210, expected=1254517, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, +break debugger_inspection_point Breakpoint 1 at 0x20c0: file tst-pthread-gdb-attach.c, line 123. +continue
* Szabolcs Nagy: > if i rerun the link command of the test exe but with -no-pie > instead of -pie then the test passes with that binary. > > i suspect gdb places the breakpoint at the wrong place in pie > for some reason. can be ubuntu tooling specific. see the > breakpoint location (the exe base offset is missing): Thanks for investigating. > +attach 1254516 > [New LWP 1254517] > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > td_ta_new failed: application not linked with libthread > thread_db_load_search returning 0 > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/home/szabolcs/try/build/nptl_db/libthread_db.so.1". > thread_db_load_search returning 1 > 0x0000ffff9d3d89c4 in __futex_abstimed_wait_common64 (futex_word=0xffff9d363210, expected=1254517, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 > 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, > +break debugger_inspection_point > Breakpoint 1 at 0x20c0: file tst-pthread-gdb-attach.c, line 123. > +continue Would you please check if the issue goes away if you replace "add-symbol-file %1$s/nptl/tst-pthread-gdb-attach\n" with "file %1$s/nptl/tst-pthread-gdb-attach\n" ? (I assume this happens without --enable-hardcoded-path-in-tests.) Thanks, Florian
On Thu, Apr 22, 2021 at 7:13 AM Szabolcs Nagy via Libc-alpha <libc-alpha@sourceware.org> wrote: > > The 04/22/2021 13:55, Florian Weimer wrote: > > * Szabolcs Nagy: > > > The 04/22/2021 10:05, Szabolcs Nagy via Gdb-patches wrote: > > >> The 04/22/2021 10:33, Florian Weimer wrote: > > >> > * Szabolcs Nagy: > > >> > > on the aarch64 buildbot i see > > >> > > > > >> > > FAIL: nptl/tst-pthread-gdb-attach > > >> > > > > >> > > $ cat nptl/tst-pthread-gdb-attach.out > > >> > > +set debug libthread-db 1 > > >> > > +add-symbol-file /work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach > > >> > > add symbol table from file "/work/glibc-aarch64-linux/build/build/nptl/tst-pthread-gdb-attach" > > >> > > +set auto-load safe-path /work/glibc-aarch64-linux/build/build/nptl_db > > >> > > +set libthread-db-search-path /work/glibc-aarch64-linux/build/build/nptl_db > > >> > > +attach 461328 > > >> > > [New LWP 461329] > > >> > > Trying host libthread_db library: /work/glibc-aarch64-linux/build/build/nptl_db/libthread_db.so.1. > > >> > > td_ta_new failed: application not linked with libthread > > >> > > thread_db_load_search returning 0 > > >> > > > >> > Unfortunately I cannot reproduce this with binutils 2.30, GDB 8.2 and > > >> > GCC 8.4.1. Has your environment newer or older versions? > > >> > > >> ubuntu flavour of > > >> > > >> gcc 8.2.0 > > >> binutils 2.31.1 > > >> gdb 8.2 > > >> > > >> happens on armhf and aarch64 too. > > > > > > happens with gcc-10.2.0, gdb-9.2, binutils-2.34 too > > > (still ubuntu) > > > > Still no luck with binutils 2.35.1, a GCC 11 snapshot, and GDB 10.1. > > > > Do you run this test as a regular user? Could it be caused by YAMA > > ptrace policies? > > $ cat /proc/sys/kernel/yama/ptrace_scope > 0 > > and the -static test passes so there gdb works. > i can reproduce it on all ubuntu setups i have. Ubuntu has different directory layouts. Could it be the issue?
The 04/22/2021 16:25, Florian Weimer wrote: > * Szabolcs Nagy: > > > if i rerun the link command of the test exe but with -no-pie > > instead of -pie then the test passes with that binary. > > > > i suspect gdb places the breakpoint at the wrong place in pie > > for some reason. can be ubuntu tooling specific. see the > > breakpoint location (the exe base offset is missing): > > Thanks for investigating. > > > +attach 1254516 > > [New LWP 1254517] > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > > td_ta_new failed: application not linked with libthread > > thread_db_load_search returning 0 > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > > [Thread debugging using libthread_db enabled] > > Using host libthread_db library "/home/szabolcs/try/build/nptl_db/libthread_db.so.1". > > thread_db_load_search returning 1 > > 0x0000ffff9d3d89c4 in __futex_abstimed_wait_common64 (futex_word=0xffff9d363210, expected=1254517, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 > > 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, > > +break debugger_inspection_point > > Breakpoint 1 at 0x20c0: file tst-pthread-gdb-attach.c, line 123. > > +continue > > Would you please check if the issue goes away if you replace > > "add-symbol-file %1$s/nptl/tst-pthread-gdb-attach\n" > > with > "file %1$s/nptl/tst-pthread-gdb-attach\n" > > ? > > (I assume this happens without --enable-hardcoded-path-in-tests.) yes, it seems gdb does not work with ld.so ./exe if exe is pie. i could not get it to work wit file either. add-symbol-file fails even without ld.so, just pie exe fails. you have to use -o offset argument to add-symbol-file and manually fish out the base offset from info proc map, i don't think that can be easily scripted.
* Szabolcs Nagy: >> Would you please check if the issue goes away if you replace >> >> "add-symbol-file %1$s/nptl/tst-pthread-gdb-attach\n" >> >> with >> "file %1$s/nptl/tst-pthread-gdb-attach\n" >> >> ? >> >> (I assume this happens without --enable-hardcoded-path-in-tests.) > > yes, it seems gdb does not work with > > ld.so ./exe > > if exe is pie. i could not get it to work wit file either. > > add-symbol-file fails even without ld.so, just pie exe fails. But “file” works? > you have to use -o offset argument to add-symbol-file and > manually fish out the base offset from info proc map, i don't > think that can be easily scripted. Oh, it's essentially a self-attach (after fork), so we can just read the base address from the running process. If there's an easy way to do that. In theory, it's just looking at a symbol at the main program and subtracting its virtual offset, but I don't know how to get that. Thanks, Florian
On Thu, Apr 22, 2021 at 10:16 AM Szabolcs Nagy via Gdb-patches <gdb-patches@sourceware.org> wrote: > > The 04/22/2021 16:25, Florian Weimer wrote: > > * Szabolcs Nagy: > > > > > if i rerun the link command of the test exe but with -no-pie > > > instead of -pie then the test passes with that binary. > > > > > > i suspect gdb places the breakpoint at the wrong place in pie > > > for some reason. can be ubuntu tooling specific. see the > > > breakpoint location (the exe base offset is missing): > > > > Thanks for investigating. > > > > > +attach 1254516 > > > [New LWP 1254517] > > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > > > td_ta_new failed: application not linked with libthread > > > thread_db_load_search returning 0 > > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > > > [Thread debugging using libthread_db enabled] > > > Using host libthread_db library "/home/szabolcs/try/build/nptl_db/libthread_db.so.1". > > > thread_db_load_search returning 1 > > > 0x0000ffff9d3d89c4 in __futex_abstimed_wait_common64 (futex_word=0xffff9d363210, expected=1254517, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 > > > 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, > > > +break debugger_inspection_point > > > Breakpoint 1 at 0x20c0: file tst-pthread-gdb-attach.c, line 123. > > > +continue > > > > Would you please check if the issue goes away if you replace > > > > "add-symbol-file %1$s/nptl/tst-pthread-gdb-attach\n" > > > > with > > "file %1$s/nptl/tst-pthread-gdb-attach\n" > > > > ? > > > > (I assume this happens without --enable-hardcoded-path-in-tests.) > > yes, it seems gdb does not work with > > ld.so ./exe > > if exe is pie. i could not get it to work wit file either. Just pass -no-pie to build exe. > add-symbol-file fails even without ld.so, just pie exe fails. > > you have to use -o offset argument to add-symbol-file and > manually fish out the base offset from info proc map, i don't > think that can be easily scripted. >
* H. J. Lu: > On Thu, Apr 22, 2021 at 10:16 AM Szabolcs Nagy via Gdb-patches > <gdb-patches@sourceware.org> wrote: >> >> The 04/22/2021 16:25, Florian Weimer wrote: >> > * Szabolcs Nagy: >> > >> > > if i rerun the link command of the test exe but with -no-pie >> > > instead of -pie then the test passes with that binary. >> > > >> > > i suspect gdb places the breakpoint at the wrong place in pie >> > > for some reason. can be ubuntu tooling specific. see the >> > > breakpoint location (the exe base offset is missing): >> > >> > Thanks for investigating. >> > >> > > +attach 1254516 >> > > [New LWP 1254517] >> > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. >> > > td_ta_new failed: application not linked with libthread >> > > thread_db_load_search returning 0 >> > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. >> > > [Thread debugging using libthread_db enabled] >> > > Using host libthread_db library "/home/szabolcs/try/build/nptl_db/libthread_db.so.1". >> > > thread_db_load_search returning 1 >> > > 0x0000ffff9d3d89c4 in __futex_abstimed_wait_common64 (futex_word=0xffff9d363210, expected=1254517, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 >> > > 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, >> > > +break debugger_inspection_point >> > > Breakpoint 1 at 0x20c0: file tst-pthread-gdb-attach.c, line 123. >> > > +continue >> > >> > Would you please check if the issue goes away if you replace >> > >> > "add-symbol-file %1$s/nptl/tst-pthread-gdb-attach\n" >> > >> > with >> > "file %1$s/nptl/tst-pthread-gdb-attach\n" >> > >> > ? >> > >> > (I assume this happens without --enable-hardcoded-path-in-tests.) >> >> yes, it seems gdb does not work with >> >> ld.so ./exe >> >> if exe is pie. i could not get it to work wit file either. > > Just pass -no-pie to build exe. Ah right, the penny finally dropped. I've seen this failure before during regular glibc debugging. Fedora apparently has a downstream-only patch. 8-/ This is what I came up with: nptl: Do not build nptl/tst-pthread-gdb-attach as PIE diff --git a/nptl/Makefile b/nptl/Makefile index a3d1ef8d66..294bb2faa4 100644 --- a/nptl/Makefile +++ b/nptl/Makefile @@ -377,6 +377,9 @@ endif CFLAGS-tst-pthread-gdb-attach-static.c := $(CFLAGS-printers-tests) CPPFLAGS-tst-pthread-gdb-attach-static.c := \ $(CFLAGS-printers-tests) -DDO_ADD_SYMBOL_FILE=0 +# As of version 9.2, GDB cannot attach properly to PIE programs that +# were launched with an explicit ld.so invocation. +tst-pthread-gdb-attach-no-pie = yes ifeq ($(build-shared),yes) tests-printers-libs := $(shared-thread-library) It's not needed for the static test even with --enable-static-pie. Thanks, Florian
On Thu, Apr 22, 2021 at 10:55 AM Florian Weimer <fweimer@redhat.com> wrote: > > * H. J. Lu: > > > On Thu, Apr 22, 2021 at 10:16 AM Szabolcs Nagy via Gdb-patches > > <gdb-patches@sourceware.org> wrote: > >> > >> The 04/22/2021 16:25, Florian Weimer wrote: > >> > * Szabolcs Nagy: > >> > > >> > > if i rerun the link command of the test exe but with -no-pie > >> > > instead of -pie then the test passes with that binary. > >> > > > >> > > i suspect gdb places the breakpoint at the wrong place in pie > >> > > for some reason. can be ubuntu tooling specific. see the > >> > > breakpoint location (the exe base offset is missing): > >> > > >> > Thanks for investigating. > >> > > >> > > +attach 1254516 > >> > > [New LWP 1254517] > >> > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > >> > > td_ta_new failed: application not linked with libthread > >> > > thread_db_load_search returning 0 > >> > > Trying host libthread_db library: /home/szabolcs/try/build/nptl_db/libthread_db.so.1. > >> > > [Thread debugging using libthread_db enabled] > >> > > Using host libthread_db library "/home/szabolcs/try/build/nptl_db/libthread_db.so.1". > >> > > thread_db_load_search returning 1 > >> > > 0x0000ffff9d3d89c4 in __futex_abstimed_wait_common64 (futex_word=0xffff9d363210, expected=1254517, clockid=<optimized out>, abstime=0x0, private=<optimized out>, cancel=cancel@entry=true) at futex-internal.c:74 > >> > > 74 err = INTERNAL_SYSCALL_CANCEL (futex_time64, futex_word, op, expected, > >> > > +break debugger_inspection_point > >> > > Breakpoint 1 at 0x20c0: file tst-pthread-gdb-attach.c, line 123. > >> > > +continue > >> > > >> > Would you please check if the issue goes away if you replace > >> > > >> > "add-symbol-file %1$s/nptl/tst-pthread-gdb-attach\n" > >> > > >> > with > >> > "file %1$s/nptl/tst-pthread-gdb-attach\n" > >> > > >> > ? > >> > > >> > (I assume this happens without --enable-hardcoded-path-in-tests.) > >> > >> yes, it seems gdb does not work with > >> > >> ld.so ./exe > >> > >> if exe is pie. i could not get it to work wit file either. > > > > Just pass -no-pie to build exe. > > Ah right, the penny finally dropped. I've seen this failure before > during regular glibc debugging. Fedora apparently has a downstream-only > patch. 8-/ > > This is what I came up with: > > nptl: Do not build nptl/tst-pthread-gdb-attach as PIE > > diff --git a/nptl/Makefile b/nptl/Makefile > index a3d1ef8d66..294bb2faa4 100644 > --- a/nptl/Makefile > +++ b/nptl/Makefile > @@ -377,6 +377,9 @@ endif > CFLAGS-tst-pthread-gdb-attach-static.c := $(CFLAGS-printers-tests) > CPPFLAGS-tst-pthread-gdb-attach-static.c := \ > $(CFLAGS-printers-tests) -DDO_ADD_SYMBOL_FILE=0 > +# As of version 9.2, GDB cannot attach properly to PIE programs that > +# were launched with an explicit ld.so invocation. > +tst-pthread-gdb-attach-no-pie = yes LGTM. Thanks. > ifeq ($(build-shared),yes) > tests-printers-libs := $(shared-thread-library) > > It's not needed for the static test even with --enable-static-pie. > > Thanks, > Florian >
diff --git a/nptl/Makefile b/nptl/Makefile index 8fe92d43fa..e665d37e52 100644 --- a/nptl/Makefile +++ b/nptl/Makefile @@ -313,7 +313,8 @@ tests = tst-attr2 tst-attr3 tst-default-attr \ tst-thread-affinity-sched \ tst-pthread-defaultattr-free \ tst-pthread-attr-sigmask \ - tst-pthread-timedlock-lockloop + tst-pthread-timedlock-lockloop \ + tst-pthread-gdb-attach tst-pthread-gdb-attach-static tests-container = tst-pthread-getattr @@ -359,6 +360,19 @@ CPPFLAGS-test-cond-printers.c := $(CFLAGS-printers-tests) CPPFLAGS-test-rwlockattr-printers.c := $(CFLAGS-printers-tests) CPPFLAGS-test-rwlock-printers.c := $(CFLAGS-printers-tests) +# Reuse the CFLAGS setting for the GDB attaching test. It needs +# debugging information. +CFLAGS-tst-pthread-gdb-attach.c := $(CFLAGS-printers-tests) +CPPFLAGS-tst-pthread-gdb-attach.c := $(CFLAGS-printers-tests) +ifeq ($(build-shared)$(build-hardcoded-path-in-tests),yesno) +CPPFLAGS-tst-pthread-gdb-attach.c += -DDO_ADD_SYMBOL_FILE=1 +else +CPPFLAGS-tst-pthread-gdb-attach.c += -DDO_ADD_SYMBOL_FILE=0 +endif +CFLAGS-tst-pthread-gdb-attach-static.c := $(CFLAGS-printers-tests) +CPPFLAGS-tst-pthread-gdb-attach-static.c := \ + $(CFLAGS-printers-tests) -DDO_ADD_SYMBOL_FILE=0 + ifeq ($(build-shared),yes) tests-printers-libs := $(shared-thread-library) else @@ -430,7 +444,8 @@ link-libc-static := $(common-objpfx)libc.a $(static-gnulib) \ tests-static += tst-stackguard1-static \ tst-cancel24-static \ tst-mutex8-static tst-mutexpi8-static tst-sem11-static \ - tst-sem12-static tst-cond11-static + tst-sem12-static tst-cond11-static \ + tst-pthread-gdb-attach-static tests += tst-cancel24-static diff --git a/nptl/pthread_create.c b/nptl/pthread_create.c index 6c645aff48..f13d8e44a4 100644 --- a/nptl/pthread_create.c +++ b/nptl/pthread_create.c @@ -51,6 +51,14 @@ static td_thr_events_t __nptl_threads_events __attribute_used__; /* Pointer to descriptor with the last event. */ static struct pthread *__nptl_last_event __attribute_used__; +#ifdef SHARED +/* This variable is used to access _rtld_global from libthread_db. If + GDB loads libpthread before ld.so, it is not possible to resolve + _rtld_global directly during libpthread initialization. */ +static struct rtld_global *__nptl_rtld_global __attribute_used__ + = &_rtld_global; +#endif + /* Number of threads running. */ unsigned int __nptl_nthreads = 1; diff --git a/nptl/tst-pthread-gdb-attach-static.c b/nptl/tst-pthread-gdb-attach-static.c new file mode 100644 index 0000000000..e159632cac --- /dev/null +++ b/nptl/tst-pthread-gdb-attach-static.c @@ -0,0 +1 @@ +#include "tst-pthread-gdb-attach.c" diff --git a/nptl/tst-pthread-gdb-attach.c b/nptl/tst-pthread-gdb-attach.c new file mode 100644 index 0000000000..0603ad844d --- /dev/null +++ b/nptl/tst-pthread-gdb-attach.c @@ -0,0 +1,143 @@ +/* Smoke testing GDB process attach with thread-local variable access. + Copyright (C) 2021 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + <https://www.gnu.org/licenses/>. */ + +/* This test runs GDB against a forked copy of itself, to check + whether libthread_db can be loaded, and that access to thread-local + variables works. */ + +#include <errno.h> +#include <stdlib.h> +#include <support/check.h> +#include <support/support.h> +#include <support/temp_file.h> +#include <support/test-driver.h> +#include <support/xstdio.h> +#include <support/xthread.h> +#include <support/xunistd.h> +#include <unistd.h> + +/* Starts out as zero, changed to 1 or 2 by the debugger, depending on + the thread. */ +__thread volatile int altered_by_debugger; + +/* Writes the GDB script to run the test to PATH. */ +static void +write_gdbscript (const char *path, int tested_pid) +{ + FILE *fp = xfopen (path, "w"); + fprintf (fp, + "set trace-commands on\n" + "set debug libthread-db 1\n" +#if DO_ADD_SYMBOL_FILE + /* Do not do this unconditionally to work around a GDB + assertion failure: ../../gdb/symtab.c:6404: + internal-error: CORE_ADDR get_msymbol_address(objfile*, + const minimal_symbol*): Assertion `(objf->flags & + OBJF_MAINLINE) == 0' failed. */ + "add-symbol-file %1$s/nptl/tst-pthread-gdb-attach\n" +#endif + "set auto-load safe-path %1$s/nptl_db\n" + "set libthread-db-search-path %1$s/nptl_db\n" + "attach %2$d\n", + support_objdir_root, tested_pid); + fputs ("break debugger_inspection_point\n" + "continue\n" + "thread 1\n" + "print altered_by_debugger\n" + "print altered_by_debugger = 1\n" + "thread 2\n" + "print altered_by_debugger\n" + "print altered_by_debugger = 2\n" + "continue\n", + fp); + xfclose (fp); +} + +/* The test sets a breakpoint on this function and alters the + altered_by_debugger thread-local variable. */ +void __attribute__ ((weak)) +debugger_inspection_point (void) +{ +} + +/* Thread function for the test thread in the subprocess. */ +static void * +subprocess_thread (void *closure) +{ + /* Wait until altered_by_debugger changes the value away from 0. */ + while (altered_by_debugger == 0) + { + usleep (100 * 1000); + debugger_inspection_point (); + } + + TEST_COMPARE (altered_by_debugger, 2); + return NULL; +} + +/* This function implements the subprocess under test. It creates a + second thread, waiting for its value to change to 2, and checks + that the main thread also changed its value to 1. */ +static void +in_subprocess (void) +{ + pthread_t thr = xpthread_create (NULL, subprocess_thread, NULL); + TEST_VERIFY (xpthread_join (thr) == NULL); + TEST_COMPARE (altered_by_debugger, 1); + _exit (0); +} + +static int +do_test (void) +{ + pid_t tested_pid = xfork (); + if (tested_pid == 0) + in_subprocess (); + char *tested_pid_string = xasprintf ("%d", tested_pid); + + char *gdbscript; + xclose (create_temp_file ("tst-pthread-gdb-attach-", &gdbscript)); + write_gdbscript (gdbscript, tested_pid); + + pid_t gdb_pid = xfork (); + if (gdb_pid == 0) + { + clearenv (); + xdup2 (STDOUT_FILENO, STDERR_FILENO); + execlp ("gdb", "gdb", "-nx", "-batch", "-x", gdbscript, NULL); + if (errno == ENOENT) + _exit (EXIT_UNSUPPORTED); + else + _exit (1); + } + + int status; + TEST_COMPARE (xwaitpid (gdb_pid, &status, 0), gdb_pid); + if (WIFEXITED (status) && WEXITSTATUS (status) == EXIT_UNSUPPORTED) + /* gdb is not installed. */ + return EXIT_UNSUPPORTED; + TEST_COMPARE (status, 0); + TEST_COMPARE (xwaitpid (tested_pid, &status, 0), tested_pid); + TEST_COMPARE (status, 0); + + free (tested_pid_string); + free (gdbscript); + return 0; +} + +#include <support/test-driver.c> diff --git a/nptl_db/structs.def b/nptl_db/structs.def index 999a9fc35a..8a613dd2f5 100644 --- a/nptl_db/structs.def +++ b/nptl_db/structs.def @@ -100,8 +100,7 @@ DB_STRUCT_FIELD (pthread, dtvp) #endif #if !(IS_IN (libpthread) && !defined SHARED) -DB_STRUCT (rtld_global) -DB_RTLD_VARIABLE (_rtld_global) +DB_VARIABLE (__nptl_rtld_global) #endif DB_RTLD_GLOBAL_FIELD (dl_tls_dtv_slotinfo_list) DB_RTLD_GLOBAL_FIELD (dl_stack_user) diff --git a/nptl_db/td_init.c b/nptl_db/td_init.c index 1d15681228..06b5adc5c2 100644 --- a/nptl_db/td_init.c +++ b/nptl_db/td_init.c @@ -33,13 +33,14 @@ td_init (void) bool __td_ta_rtld_global (td_thragent_t *ta) { - if (ta->ta_addr__rtld_global == 0 - && td_mod_lookup (ta->ph, LD_SO, SYM__rtld_global, - &ta->ta_addr__rtld_global) != PS_OK) + if (ta->ta_addr__rtld_global == 0) { - ta->ta_addr__rtld_global = (void*)-1; - return false; + psaddr_t rtldglobalp; + if (DB_GET_VALUE (rtldglobalp, ta, __nptl_rtld_global, 0) == TD_OK) + ta->ta_addr__rtld_global = rtldglobalp; + else + ta->ta_addr__rtld_global = (void *) -1; } - else - return ta->ta_addr__rtld_global != (void*)-1; + + return ta->ta_addr__rtld_global != (void *)-1; } diff --git a/nptl_db/thread_dbP.h b/nptl_db/thread_dbP.h index 580a70c471..712fa3aeb6 100644 --- a/nptl_db/thread_dbP.h +++ b/nptl_db/thread_dbP.h @@ -108,6 +108,8 @@ struct td_thragent # undef DB_SYMBOL # undef DB_VARIABLE + psaddr_t ta_addr__rtld_global; + /* The method of locating a thread's th_unique value. */ enum {