diff mbox

libbacktrace patch committed: Trace through shared libraries

Message ID mcrpq4ra4n3.fsf@google.com
State New
Headers show

Commit Message

Ian Lance Taylor Oct. 9, 2012, 6:20 p.m. UTC
This patch to libbacktrace adds support for tracing through shared
libraries.  The libraries are found by calling dl_iterate_phdr, when it
is available.

This patch has some preliminary support for tracing through libaries
opened via dlopen, but there is no code for actually finding such
libraries.  It would require keeping track of the modules that have been
read, and, when some PC is found with no definition, calling
dl_iterate_phdr again to see if any libraries have been dlopen'ed.

I do not know how to support dlclose.

Patch bootstrapped and ran libbacktrace and Go testsuites on
x86_64-unknown-linux-gnu.  Committed to mainline.

Ian


2012-10-09  Ian Lance Taylor  <iant@google.com>

	Add support for tracing through shared libraries.
	* configure.ac: Check for link.h and dl_iterate_phdr.
	* elf.c: #include <link.h> if system has dl_iterate_phdr.  #undef
	ELF macros before #defining them.
	(dl_phdr_info, dl_iterate_phdr): Define if system does not have
	dl_iterate_phdr.
	(struct elf_syminfo_data): Add next field.
	(elf_initialize_syminfo): Initialize next field.
	(elf_add_syminfo_data): New static function.
	(elf_add): New static function, broken out of
	backtrace_initialize.  Call backtrace_dwarf_add instead of
	backtrace_dwarf_initialize.
	(struct phdr_data): Define.
	(phdr_callback): New static function.
	(backtrace_initialize): Call elf_add.
	* dwarf.c (struct dwarf_data): Add next and base_address fields.
	(add_unit_addr): Add base_address parameter.  Change all callers.
	(add_unit_ranges, build_address_map): Likewise.
	(add_line): Add ddata parameter.  Change all callers.
	(read_line_program, add_function_range): Likewise.
	(dwarf_lookup_pc): New static function, broken out of
	dwarf_fileline.
	(dwarf_fileline): Call dwarf_lookup_pc.
	(build_dwarf_data): New static function.
	(backtrace_dwarf_add): New function.
	(backtrace_dwarf_initialize): Remove.
	* internal.h (backtrace_dwarf_initialize): Don't declare.
	(backtrace_dwarf_add): Declare.
	* configure, config.h.in: Rebuild.

Comments

Basile Starynkevitch Oct. 9, 2012, 6:32 p.m. UTC | #1
On Tue, Oct 09, 2012 at 11:20:48AM -0700, Ian Lance Taylor wrote:
> This patch to libbacktrace adds support for tracing through shared
> libraries.  The libraries are found by calling dl_iterate_phdr, when it
> is available.

This functionality is definitely useful for meta-plugins like MELT 
(since melt.so is dlopen-ing things)

> 
> This patch has some preliminary support for tracing through libaries
> opened via dlopen, but there is no code for actually finding such
> libraries.  


Perhaps you might use dladdr, on the few systems (notably GNU/Libc on Linux) which have it?
From man dlsym output:

       int dladdr(void *addr, Dl_info *info);

       The  function  dladdr()  takes  a function pointer and tries to resolve
       name and file where it  is  located.   Information  is  stored  in  the
       Dl_info structure:

           typedef struct {
               const char *dli_fname;  /* Pathname of shared object that
                                          contains address */
               void       *dli_fbase;  /* Address at which shared object
                                          is loaded */
               const char *dli_sname;  /* Name of nearest symbol with address
                                          lower than addr */
               void       *dli_saddr;  /* Exact address of symbol named
                                          in dli_sname */
           } Dl_info;

       If no symbol matching addr could be found, then dli_sname and dli_saddr
       are set to NULL.

       dladdr() returns 0 on error, and nonzero on success.

Or maybe I misunderstood libbacktrace and my comment is irrelevant. If it is the case, sorry for the noise.

You could provide an extra API to register dlopen & dlclose to libbacktrace, if that helps you
(of course, I would prefer avoiding that)

Cheers
Ian Lance Taylor Oct. 9, 2012, 8:43 p.m. UTC | #2
On Tue, Oct 9, 2012 at 11:32 AM, Basile Starynkevitch
<basile@starynkevitch.net> wrote:
>
> You could provide an extra API to register dlopen & dlclose to libbacktrace, if that helps you
> (of course, I would prefer avoiding that)

I would prefer avoiding that as well.

Calling dl_iterate_phdr can tell libbacktrace reliably the set of
shared libraries that are currently loaded.  The trick is knowing when
to call it.  It may simply be OK to call it every time we look up a PC
value, in which case we can look at the dlpi_adds and dlpi_subs field.
 I'm not sure.

Ian
Jakub Jelinek Oct. 9, 2012, 8:54 p.m. UTC | #3
On Tue, Oct 09, 2012 at 01:43:06PM -0700, Ian Lance Taylor wrote:
> On Tue, Oct 9, 2012 at 11:32 AM, Basile Starynkevitch
> <basile@starynkevitch.net> wrote:
> >
> > You could provide an extra API to register dlopen & dlclose to libbacktrace, if that helps you
> > (of course, I would prefer avoiding that)
> 
> I would prefer avoiding that as well.
> 
> Calling dl_iterate_phdr can tell libbacktrace reliably the set of
> shared libraries that are currently loaded.  The trick is knowing when
> to call it.  It may simply be OK to call it every time we look up a PC
> value, in which case we can look at the dlpi_adds and dlpi_subs field.
>  I'm not sure.

See unwind-dw2-fde-dip.c how it uses it.  I think for all PC queries from
one backtrace you should get away with using a cache of last looked up
library even without calling dl_iterate_phdr.  PCs in the backtrace
can't go away from under you in a properly written application (but only
those, not others).  In between different backtrace calls you need to call
dl_iterate_phdr (primarily to lock the ld.so lock and prevent dlopen/dlclose
from other threads), but can use a cache there and use dlpi_adds/dlpi_subs
to find out when to invalidate the cache.

	Jakub
Hans-Peter Nilsson Oct. 10, 2012, 4:28 a.m. UTC | #4
On Tue, 9 Oct 2012, Ian Lance Taylor wrote:
> Patch bootstrapped and ran libbacktrace and Go testsuites on
> x86_64-unknown-linux-gnu.  Committed to mainline.

Trying x86_64 host gcc 4.1.2 (--target=rl78-elf but not the issue) at
r192285:

/bin/sh ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H
-I. -I/tmp/rl78b/gcc/libbacktrace  -I /tmp/rl78b/gcc/li\
bbacktrace/../include -I /tmp/rl78b/gcc/libbacktrace/../libgcc
-I ../libgcc -I ../gcc/include -I ../../gcc/include  -fu\
nwind-tables -W -Wall -Wwrite-strings -Wstrict-prototypes
-Wmissing-prototypes -Wold-style-definition -Wmissing-format-\
attribute -Wcast-qual -Werror  -g -O2 -c -o dwarf.lo
/tmp/rl78b/gcc/libbacktrace/dwarf.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I.
-I/tmp/rl78b/gcc/libbacktrace -I
/tmp/rl78b/gcc/libbacktrace/../include -I /\
tmp/rl78b/gcc/libbacktrace/../libgcc -I ../libgcc -I
../gcc/include -I ../../gcc/include -funwind-tables -W -Wall
-Wwri\
te-strings -Wstrict-prototypes -Wmissing-prototypes
-Wold-style-definition -Wmissing-format-attribute -Wcast-qual
-Werr\
or -g -O2 -c /tmp/rl78b/gcc/libbacktrace/dwarf.c -o dwarf.o
cc1: warnings being treated as errors
/tmp/rl78b/gcc/libbacktrace/dwarf.c: In function
'dwarf_fileline':
/tmp/rl78b/gcc/libbacktrace/dwarf.c:2766: warning: dereferencing
type-punned pointer will break strict-aliasing rules
/tmp/rl78b/gcc/libbacktrace/dwarf.c: In function
'backtrace_dwarf_add':
/tmp/rl78b/gcc/libbacktrace/dwarf.c:2887: warning: dereferencing
type-punned pointer will break strict-aliasing rules
/tmp/rl78b/gcc/libbacktrace/dwarf.c:2899: warning: dereferencing
type-punned pointer will break strict-aliasing rules
make[3]: *** [dwarf.lo] Error 1


brgds, H-P
Ian Lance Taylor Oct. 10, 2012, 4:37 a.m. UTC | #5
On Tue, Oct 9, 2012 at 9:28 PM, Hans-Peter Nilsson <hp@bitrange.com> wrote:
> On Tue, 9 Oct 2012, Ian Lance Taylor wrote:
>> Patch bootstrapped and ran libbacktrace and Go testsuites on
>> x86_64-unknown-linux-gnu.  Committed to mainline.
>
> Trying x86_64 host gcc 4.1.2 (--target=rl78-elf but not the issue) at
> r192285:
>
> /bin/sh ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H
> -I. -I/tmp/rl78b/gcc/libbacktrace  -I /tmp/rl78b/gcc/li\
> bbacktrace/../include -I /tmp/rl78b/gcc/libbacktrace/../libgcc
> -I ../libgcc -I ../gcc/include -I ../../gcc/include  -fu\
> nwind-tables -W -Wall -Wwrite-strings -Wstrict-prototypes
> -Wmissing-prototypes -Wold-style-definition -Wmissing-format-\
> attribute -Wcast-qual -Werror  -g -O2 -c -o dwarf.lo
> /tmp/rl78b/gcc/libbacktrace/dwarf.c
> libtool: compile:  gcc -DHAVE_CONFIG_H -I.
> -I/tmp/rl78b/gcc/libbacktrace -I
> /tmp/rl78b/gcc/libbacktrace/../include -I /\
> tmp/rl78b/gcc/libbacktrace/../libgcc -I ../libgcc -I
> ../gcc/include -I ../../gcc/include -funwind-tables -W -Wall
> -Wwri\
> te-strings -Wstrict-prototypes -Wmissing-prototypes
> -Wold-style-definition -Wmissing-format-attribute -Wcast-qual
> -Werr\
> or -g -O2 -c /tmp/rl78b/gcc/libbacktrace/dwarf.c -o dwarf.o
> cc1: warnings being treated as errors
> /tmp/rl78b/gcc/libbacktrace/dwarf.c: In function
> 'dwarf_fileline':
> /tmp/rl78b/gcc/libbacktrace/dwarf.c:2766: warning: dereferencing
> type-punned pointer will break strict-aliasing rules
> /tmp/rl78b/gcc/libbacktrace/dwarf.c: In function
> 'backtrace_dwarf_add':
> /tmp/rl78b/gcc/libbacktrace/dwarf.c:2887: warning: dereferencing
> type-punned pointer will break strict-aliasing rules
> /tmp/rl78b/gcc/libbacktrace/dwarf.c:2899: warning: dereferencing
> type-punned pointer will break strict-aliasing rules
> make[3]: *** [dwarf.lo] Error 1

It's an incorrect warning from an old version of GCC.  Fixed like so.
Bootstrapped and ran libbacktrace testsuite on
x86_64-unknown-linux-gnu.  Committed to mainline.

Ian


2012-10-09  Ian Lance Taylor  <iant@google.com>

	* dwarf.c (dwarf_fileline): Add cast to avoid warning.
	(backtrace_dwarf_add): Likewise.
Rainer Orth Oct. 10, 2012, 2:13 p.m. UTC | #6
Ian Lance Taylor <iant@google.com> writes:

> This patch to libbacktrace adds support for tracing through shared
> libraries.  The libraries are found by calling dl_iterate_phdr, when it
> is available.

Unfortunately, this breaks all Go link tests on Solaris 10:

output is:
Undefined			first referenced
 symbol  			    in file
dl_iterate_phdr                     /var/gcc/gcc-4.8.0-20121010/10-gcc-gas/i386-pc-solaris2.10/./libgo/.libs/libgo.so
ld: fatal: symbol referencing errors. No output written to /var/gcc/gcc-4.8.0-20121010/10-gcc-gas/gcc/testsuite/go/array-1.x
collect2: error: ld returned 1 exit status

FAIL: go.go-torture/execute/array-1.go compilation,  -O0 

Solaris 10 Update 10 or sufficiently recent linker patches introduced
dl_iterate_phdr on S10 as a backport from Solaris 11, but unlike S11, it
lives in libdl.so only.  The current dl_iterate_phdr check misses that,
and given that it's only available in the most recent update, it's
probably best to avoid d_i_p on Solaris 10 altogether.

	Rainer
Ian Lance Taylor Oct. 10, 2012, 2:47 p.m. UTC | #7
On Wed, Oct 10, 2012 at 7:13 AM, Rainer Orth
<ro@cebitec.uni-bielefeld.de> wrote:
>
> Solaris 10 Update 10 or sufficiently recent linker patches introduced
> dl_iterate_phdr on S10 as a backport from Solaris 11, but unlike S11, it
> lives in libdl.so only.  The current dl_iterate_phdr check misses that,
> and given that it's only available in the most recent update, it's
> probably best to avoid d_i_p on Solaris 10 altogether.

Does this patch fix the problem?

Ian
Rainer Orth Oct. 11, 2012, 8:44 a.m. UTC | #8
Ian Lance Taylor <iant@google.com> writes:

> On Wed, Oct 10, 2012 at 7:13 AM, Rainer Orth
> <ro@cebitec.uni-bielefeld.de> wrote:
>>
>> Solaris 10 Update 10 or sufficiently recent linker patches introduced
>> dl_iterate_phdr on S10 as a backport from Solaris 11, but unlike S11, it
>> lives in libdl.so only.  The current dl_iterate_phdr check misses that,
>> and given that it's only available in the most recent update, it's
>> probably best to avoid d_i_p on Solaris 10 altogether.
>
> Does this patch fix the problem?

Yes, works like a charm: Solaris 10/x86 results (both go and libgo) are
back to normal.  I'm only seeing two failures:

The 32-bit net/http test tails (may be transient):

--- FAIL: http_test.TestTransportPersistConnLeak (3.53 seconds)
:0: 	goroutine growth: 106 -> 181 -> 125 (delta: 19)
:0: 	too many new goroutines
		FAIL
FAIL: net/http

FAIL: runtime/pprof

Already reported as PR go/54873.

Thanks.
	Rainer
Ian Lance Taylor Oct. 11, 2012, 4:43 p.m. UTC | #9
On Thu, Oct 11, 2012 at 1:44 AM, Rainer Orth
<ro@cebitec.uni-bielefeld.de> wrote:
> Ian Lance Taylor <iant@google.com> writes:
>
>> On Wed, Oct 10, 2012 at 7:13 AM, Rainer Orth
>> <ro@cebitec.uni-bielefeld.de> wrote:
>>>
>>> Solaris 10 Update 10 or sufficiently recent linker patches introduced
>>> dl_iterate_phdr on S10 as a backport from Solaris 11, but unlike S11, it
>>> lives in libdl.so only.  The current dl_iterate_phdr check misses that,
>>> and given that it's only available in the most recent update, it's
>>> probably best to avoid d_i_p on Solaris 10 altogether.
>>
>> Does this patch fix the problem?
>
> Yes, works like a charm: Solaris 10/x86 results (both go and libgo) are
> back to normal.

Thanks, committed with this ChangeLog entry.

Ian


2012-10-11  Ian Lance Taylor  <iant@google.com>

	* configure.ac: Do not use dl_iterate_phdr on Solaris 10.
	* configure: Rebuild.
diff mbox

Patch

Index: dwarf.c
===================================================================
--- dwarf.c	(revision 192266)
+++ dwarf.c	(working copy)
@@ -333,6 +333,10 @@  struct unit_addrs_vector
 
 struct dwarf_data
 {
+  /* The data for the next file we know about.  */
+  struct dwarf_data *next;
+  /* The base address for this file.  */
+  uintptr_t base_address;
   /* A sorted list of address ranges.  */
   struct unit_addrs *addrs;
   /* Number of address ranges in list.  */
@@ -831,12 +835,18 @@  function_addrs_search (const void *vkey,
    success, 0 on failure.  */
 
 static int
-add_unit_addr (struct backtrace_state *state, struct unit_addrs addrs,
+add_unit_addr (struct backtrace_state *state, uintptr_t base_address,
+	       struct unit_addrs addrs,
 	       backtrace_error_callback error_callback, void *data,
 	       struct unit_addrs_vector *vec)
 {
   struct unit_addrs *p;
 
+  /* Add in the base address of the module here, so that we can look
+     up the PC directly.  */
+  addrs.low += base_address;
+  addrs.high += base_address;
+
   /* Try to merge with the last entry.  */
   if (vec->count > 0)
     {
@@ -1156,9 +1166,10 @@  lookup_abbrev (struct abbrevs *abbrevs, 
    1 on success, 0 on failure.  */
 
 static int
-add_unit_ranges (struct backtrace_state *state, struct unit *u,
-		 uint64_t ranges, uint64_t base, int is_bigendian,
-		 const unsigned char *dwarf_ranges, size_t dwarf_ranges_size,
+add_unit_ranges (struct backtrace_state *state, uintptr_t base_address,
+		 struct unit *u, uint64_t ranges, uint64_t base,
+		 int is_bigendian, const unsigned char *dwarf_ranges,
+		 size_t dwarf_ranges_size,
 		 backtrace_error_callback error_callback, void *data,
 		 struct unit_addrs_vector *addrs)
 {
@@ -1202,7 +1213,8 @@  add_unit_ranges (struct backtrace_state 
 	  a.low = low + base;
 	  a.high = high + base;
 	  a.u = u;
-	  if (!add_unit_addr (state, a, error_callback, data, addrs))
+	  if (!add_unit_addr (state, base_address, a, error_callback, data,
+			      addrs))
 	    return 0;
 	}
     }
@@ -1218,7 +1230,7 @@  add_unit_ranges (struct backtrace_state 
    on success, 0 on failure.  */
 
 static int
-build_address_map (struct backtrace_state *state,
+build_address_map (struct backtrace_state *state, uintptr_t base_address,
 		   const unsigned char *dwarf_info, size_t dwarf_info_size,
 		   const unsigned char *dwarf_abbrev, size_t dwarf_abbrev_size,
 		   const unsigned char *dwarf_ranges, size_t dwarf_ranges_size,
@@ -1417,9 +1429,10 @@  build_address_map (struct backtrace_stat
 
 	  if (have_ranges)
 	    {
-	      if (!add_unit_ranges (state, u, ranges, lowpc, is_bigendian,
-				    dwarf_ranges, dwarf_ranges_size,
-				    error_callback, data, addrs))
+	      if (!add_unit_ranges (state, base_address, u, ranges, lowpc,
+				    is_bigendian, dwarf_ranges,
+				    dwarf_ranges_size, error_callback, data,
+				    addrs))
 		{
 		  free_abbrevs (state, &u->abbrevs, error_callback, data);
 		  backtrace_free (state, u, sizeof *u, error_callback, data);
@@ -1434,7 +1447,8 @@  build_address_map (struct backtrace_stat
 	      a.high = highpc;
 	      a.u = u;
 
-	      if (!add_unit_addr (state, a, error_callback, data, addrs))
+	      if (!add_unit_addr (state, base_address, a, error_callback, data,
+				  addrs))
 		{
 		  free_abbrevs (state, &u->abbrevs, error_callback, data);
 		  backtrace_free (state, u, sizeof *u, error_callback, data);
@@ -1463,8 +1477,9 @@  build_address_map (struct backtrace_stat
    building.  Returns 1 on success, 0 on failure.  */
 
 static int
-add_line (struct backtrace_state *state, uintptr_t pc, const char *filename,
-	  int lineno, backtrace_error_callback error_callback, void *data,
+add_line (struct backtrace_state *state, struct dwarf_data *ddata,
+	  uintptr_t pc, const char *filename, int lineno,
+	  backtrace_error_callback error_callback, void *data,
 	  struct line_vector *vec)
 {
   struct line *ln;
@@ -1484,7 +1499,10 @@  add_line (struct backtrace_state *state,
   if (ln == NULL)
     return 0;
 
-  ln->pc = pc;
+  /* Add in the base address here, so that we can look up the PC
+     directly.  */
+  ln->pc = pc + ddata->base_address;
+
   ln->filename = filename;
   ln->lineno = lineno;
 
@@ -1672,9 +1690,9 @@  read_line_header (struct backtrace_state
    success, 0 on failure.  */
 
 static int
-read_line_program (struct backtrace_state *state, struct unit *u,
-		   const struct line_header *hdr, struct dwarf_buf *line_buf,
-		   struct line_vector *vec)
+read_line_program (struct backtrace_state *state, struct dwarf_data *ddata,
+		   struct unit *u, const struct line_header *hdr,
+		   struct dwarf_buf *line_buf, struct line_vector *vec)
 {
   uint64_t address;
   unsigned int op_index;
@@ -1706,8 +1724,8 @@  read_line_program (struct backtrace_stat
 		      / hdr->max_ops_per_insn);
 	  op_index = (op_index + advance) % hdr->max_ops_per_insn;
 	  lineno += hdr->line_base + (int) (op % hdr->line_range);
-	  add_line (state, address, filename, lineno, line_buf->error_callback,
-		    line_buf->data, vec);
+	  add_line (state, ddata, address, filename, lineno,
+		    line_buf->error_callback, line_buf->data, vec);
 	}
       else if (op == DW_LNS_extended_op)
 	{
@@ -1795,7 +1813,7 @@  read_line_program (struct backtrace_stat
 	  switch (op)
 	    {
 	    case DW_LNS_copy:
-	      add_line (state, address, filename, lineno,
+	      add_line (state, ddata, address, filename, lineno,
 			line_buf->error_callback, line_buf->data, vec);
 	      break;
 	    case DW_LNS_advance_pc:
@@ -1923,7 +1941,7 @@  read_line_info (struct backtrace_state *
   if (!read_line_header (state, u, is_dwarf64, &line_buf, hdr))
     goto fail;
 
-  if (!read_line_program (state, u, hdr, &line_buf, &vec))
+  if (!read_line_program (state, ddata, u, hdr, &line_buf, &vec))
     goto fail;
 
   if (line_buf.reported_underflow)
@@ -2076,13 +2094,18 @@  read_referenced_name (struct dwarf_data 
    success, 0 on error.  */
 
 static int
-add_function_range (struct backtrace_state *state, struct function *function,
-		    uint64_t lowpc, uint64_t highpc,
+add_function_range (struct backtrace_state *state, struct dwarf_data *ddata,
+		    struct function *function, uint64_t lowpc, uint64_t highpc,
 		    backtrace_error_callback error_callback,
 		    void *data, struct function_vector *vec)
 {
   struct function_addrs *p;
 
+  /* Add in the base address here, so that we can look up the PC
+     directly.  */
+  lowpc += ddata->base_address;
+  highpc += ddata->base_address;
+
   if (vec->count > 0)
     {
       p = (struct function_addrs *) vec->vec.base + vec->count - 1;
@@ -2153,8 +2176,8 @@  add_function_ranges (struct backtrace_st
 	base = high;
       else
 	{
-	  if (!add_function_range (state, function, low + base, high + base,
-				   error_callback, data, vec))
+	  if (!add_function_range (state, ddata, function, low + base,
+				   high + base, error_callback, data, vec))
 	    return 0;
 	}
     }
@@ -2364,7 +2387,7 @@  read_function_entry (struct backtrace_st
 	    {
 	      if (highpc_is_relative)
 		highpc += lowpc;
-	      if (!add_function_range (state, function, lowpc, highpc,
+	      if (!add_function_range (state, ddata, function, lowpc, highpc,
 				       error_callback, data, vec))
 		return 0;
 	    }
@@ -2522,15 +2545,17 @@  report_inlined_functions (uintptr_t pc, 
   return 0;
 }
 
-/* Return the file/line information for a PC using the DWARF mapping
-   we built earlier.  */
+/* Look for a PC in the DWARF mapping for one module.  On success,
+   call CALLBACK and return whatever it returns.  On error, call
+   ERROR_CALLBACK and return 0.  Sets *FOUND to 1 if the PC is found,
+   0 if not.  */
 
 static int
-dwarf_fileline (struct backtrace_state *state, uintptr_t pc,
-		backtrace_full_callback callback,
-		backtrace_error_callback error_callback, void *data)
+dwarf_lookup_pc (struct backtrace_state *state, struct dwarf_data *ddata,
+		 uintptr_t pc, backtrace_full_callback callback,
+		 backtrace_error_callback error_callback, void *data,
+		 int *found)
 {
-  struct dwarf_data *ddata;
   struct unit_addrs *entry;
   struct unit *u;
   int new_data;
@@ -2542,14 +2567,17 @@  dwarf_fileline (struct backtrace_state *
   int lineno;
   int ret;
 
-  ddata = (struct dwarf_data *) state->fileline_data;
+  *found = 1;
 
   /* Find an address range that includes PC.  */
   entry = bsearch (&pc, ddata->addrs, ddata->addrs_count,
 		   sizeof (struct unit_addrs), unit_addrs_search);
 
   if (entry == NULL)
-    return callback (data, pc, NULL, 0, NULL);
+    {
+      *found = 0;
+      return 0;
+    }
 
   /* If there are multiple ranges that contain PC, use the last one,
      in order to produce predictable results.  If we assume that all
@@ -2656,7 +2684,8 @@  dwarf_fileline (struct backtrace_state *
 	 try again to see if there is a better compilation unit for
 	 this PC.  */
       if (new_data)
-	dwarf_fileline (state, pc, callback, error_callback, data);
+	return dwarf_lookup_pc (state, ddata, pc, callback, error_callback,
+				data, found);
       return callback (data, pc, NULL, 0, NULL);
     }
 
@@ -2705,39 +2734,93 @@  dwarf_fileline (struct backtrace_state *
   return callback (data, pc, filename, lineno, function->name);
 }
 
-/* Build our data structures from the .debug_info and .debug_line
-   sections.  Set *FILELINE_FN and *FILELINE_DATA.  Return 1 on
-   success, 0 on failure.  */
 
-int
-backtrace_dwarf_initialize (struct backtrace_state *state,
-			    const unsigned char *dwarf_info,
-			    size_t dwarf_info_size,
-			    const unsigned char *dwarf_line,
-			    size_t dwarf_line_size,
-			    const unsigned char *dwarf_abbrev,
-			    size_t dwarf_abbrev_size,
-			    const unsigned char *dwarf_ranges,
-			    size_t dwarf_ranges_size,
-			    const unsigned char *dwarf_str,
-			    size_t dwarf_str_size,
-			    int is_bigendian,
-			    backtrace_error_callback error_callback,
-			    void *data, fileline *fileline_fn)
+/* Return the file/line information for a PC using the DWARF mapping
+   we built earlier.  */
+
+static int
+dwarf_fileline (struct backtrace_state *state, uintptr_t pc,
+		backtrace_full_callback callback,
+		backtrace_error_callback error_callback, void *data)
+{
+  struct dwarf_data *ddata;
+  int found;
+  int ret;
+
+  if (!state->threaded)
+    {
+      for (ddata = (struct dwarf_data *) state->fileline_data;
+	   ddata != NULL;
+	   ddata = ddata->next)
+	{
+	  ret = dwarf_lookup_pc (state, ddata, pc, callback, error_callback,
+				 data, &found);
+	  if (ret != 0 || found)
+	    return ret;
+	}
+    }
+  else
+    {
+      struct dwarf_data **pp;
+
+      pp = (struct dwarf_data **) &state->fileline_data;
+      while (1)
+	{
+	  ddata = *pp;
+	  /* Atomic load.  */
+	  while (!__sync_bool_compare_and_swap (pp, ddata, ddata))
+	    ddata = *pp;
+
+	  if (ddata == NULL)
+	    break;
+
+	  ret = dwarf_lookup_pc (state, ddata, pc, callback, error_callback,
+				 data, &found);
+	  if (ret != 0 || found)
+	    return ret;
+
+	  pp = &ddata->next;
+	}
+    }
+
+  /* FIXME: See if any libraries have been dlopen'ed.  */
+
+  return callback (data, pc, NULL, 0, NULL);
+}
+
+/* Initialize our data structures from the DWARF debug info for a
+   file.  Return NULL on failure.  */
+
+static struct dwarf_data *
+build_dwarf_data (struct backtrace_state *state,
+		  uintptr_t base_address,
+		  const unsigned char *dwarf_info,
+		  size_t dwarf_info_size,
+		  const unsigned char *dwarf_line,
+		  size_t dwarf_line_size,
+		  const unsigned char *dwarf_abbrev,
+		  size_t dwarf_abbrev_size,
+		  const unsigned char *dwarf_ranges,
+		  size_t dwarf_ranges_size,
+		  const unsigned char *dwarf_str,
+		  size_t dwarf_str_size,
+		  int is_bigendian,
+		  backtrace_error_callback error_callback,
+		  void *data)
 {
   struct unit_addrs_vector addrs_vec;
   struct unit_addrs *addrs;
   size_t addrs_count;
   struct dwarf_data *fdata;
 
-  if (!build_address_map (state, dwarf_info, dwarf_info_size, dwarf_abbrev,
-			  dwarf_abbrev_size, dwarf_ranges, dwarf_ranges_size,
-			  dwarf_str, dwarf_str_size, is_bigendian,
-			  error_callback, data, &addrs_vec))
-    return 0;
+  if (!build_address_map (state, base_address, dwarf_info, dwarf_info_size,
+			  dwarf_abbrev, dwarf_abbrev_size, dwarf_ranges,
+			  dwarf_ranges_size, dwarf_str, dwarf_str_size,
+			  is_bigendian, error_callback, data, &addrs_vec))
+    return NULL;
 
   if (!backtrace_vector_release (state, &addrs_vec.vec, error_callback, data))
-    return 0;
+    return NULL;
   addrs = (struct unit_addrs *) addrs_vec.vec.base;
   addrs_count = addrs_vec.count;
   qsort (addrs, addrs_count, sizeof (struct unit_addrs), unit_addrs_compare);
@@ -2746,8 +2829,10 @@  backtrace_dwarf_initialize (struct backt
 	   backtrace_alloc (state, sizeof (struct dwarf_data),
 			    error_callback, data));
   if (fdata == NULL)
-    return 0;
+    return NULL;
 
+  fdata->next = NULL;
+  fdata->base_address = base_address;
   fdata->addrs = addrs;
   fdata->addrs_count = addrs_count;
   fdata->dwarf_info = dwarf_info;
@@ -2761,7 +2846,77 @@  backtrace_dwarf_initialize (struct backt
   fdata->is_bigendian = is_bigendian;
   memset (&fdata->fvec, 0, sizeof fdata->fvec);
 
-  state->fileline_data = fdata;
+  return fdata;
+}
+
+/* Build our data structures from the DWARF sections for a module.
+   Set FILELINE_FN and STATE->FILELINE_DATA.  Return 1 on success, 0
+   on failure.  */
+
+int
+backtrace_dwarf_add (struct backtrace_state *state,
+		     uintptr_t base_address,
+		     const unsigned char *dwarf_info,
+		     size_t dwarf_info_size,
+		     const unsigned char *dwarf_line,
+		     size_t dwarf_line_size,
+		     const unsigned char *dwarf_abbrev,
+		     size_t dwarf_abbrev_size,
+		     const unsigned char *dwarf_ranges,
+		     size_t dwarf_ranges_size,
+		     const unsigned char *dwarf_str,
+		     size_t dwarf_str_size,
+		     int is_bigendian,
+		     backtrace_error_callback error_callback,
+		     void *data, fileline *fileline_fn)
+{
+  struct dwarf_data *fdata;
+
+  fdata = build_dwarf_data (state, base_address, dwarf_info, dwarf_info_size,
+			    dwarf_line, dwarf_line_size, dwarf_abbrev,
+			    dwarf_abbrev_size, dwarf_ranges, dwarf_ranges_size,
+			    dwarf_str, dwarf_str_size, is_bigendian,
+			    error_callback, data);
+  if (fdata == NULL)
+    return 0;
+
+  if (!state->threaded)
+    {
+      struct dwarf_data **pp;
+
+      for (pp = (struct dwarf_data **) &state->fileline_data;
+	   *pp != NULL;
+	   pp = &(*pp)->next)
+	;
+      *pp = fdata;
+    }
+  else
+    {
+      while (1)
+	{
+	  struct dwarf_data **pp;
+
+	  pp = (struct dwarf_data **) &state->fileline_data;
+
+	  while (1)
+	    {
+	      struct dwarf_data *p;
+
+	      /* Atomic load.  */
+	      p = *pp;
+	      while (!__sync_bool_compare_and_swap (pp, p, p))
+		p = *pp;
+
+	      if (p == NULL)
+		break;
+
+	      pp = &p->next;
+	    }
+
+	  if (__sync_bool_compare_and_swap (pp, NULL, fdata))
+	    break;
+	}
+    }
 
   *fileline_fn = dwarf_fileline;
 
Index: elf.c
===================================================================
--- elf.c	(revision 192266)
+++ elf.c	(working copy)
@@ -36,9 +36,36 @@  POSSIBILITY OF SUCH DAMAGE.  */
 #include <string.h>
 #include <sys/types.h>
 
+#ifdef HAVE_DL_ITERATE_PHDR
+#include <link.h>
+#endif
+
 #include "backtrace.h"
 #include "internal.h"
 
+#ifndef HAVE_DL_ITERATE_PHDR
+
+/* Dummy version of dl_iterate_phdr for systems that don't have it.  */
+
+#define dl_phdr_info x_dl_phdr_info
+#define dl_iterate_phdr x_dl_iterate_phdr
+
+struct dl_phdr_info
+{
+  uintptr_t dlpi_addr;
+  const char *dlpi_name;
+};
+
+static int
+dl_iterate_phdr (int (*callback) (struct dl_phdr_info *,
+				  size_t, void *) ATTRIBUTE_UNUSED,
+		 void *data ATTRIBUTE_UNUSED)
+{
+  return 0;
+}
+
+#endif /* ! defined (HAVE_DL_ITERATE_PHDR) */
+
 /* The configure script must tell us whether we are 32-bit or 64-bit
    ELF.  We could make this code test and support either possibility,
    but there is no point.  This code only works for the currently
@@ -49,6 +76,33 @@  POSSIBILITY OF SUCH DAMAGE.  */
 #error "Unknown BACKTRACE_ELF_SIZE"
 #endif
 
+/* <link.h> might #include <elf.h> which might define our constants
+   with slightly different values.  Undefine them to be safe.  */
+
+#undef EI_NIDENT
+#undef EI_MAG0
+#undef EI_MAG1
+#undef EI_MAG2
+#undef EI_MAG3
+#undef EI_CLASS
+#undef EI_DATA
+#undef EI_VERSION
+#undef ELF_MAG0
+#undef ELF_MAG1
+#undef ELF_MAG2
+#undef ELF_MAG3
+#undef ELFCLASS32
+#undef ELFCLASS64
+#undef ELFDATA2LSB
+#undef ELFDATA2MSB
+#undef EV_CURRENT
+#undef SHN_LORESERVE
+#undef SHN_XINDEX
+#undef SHT_SYMTAB
+#undef SHT_STRTAB
+#undef SHT_DYNSYM
+#undef STT_FUNC
+
 /* Basic types.  */
 
 typedef uint16_t Elf_Half;
@@ -214,6 +268,8 @@  struct elf_symbol
 
 struct elf_syminfo_data
 {
+  /* Symbols for the next module.  */
+  struct elf_syminfo_data *next;
   /* The ELF symbols, sorted by address.  */
   struct elf_symbol *symbols;
   /* The number of symbols.  */
@@ -337,12 +393,58 @@  elf_initialize_syminfo (struct backtrace
   qsort (elf_symbols, elf_symbol_count, sizeof (struct elf_symbol),
 	 elf_symbol_compare);
 
+  sdata->next = NULL;
   sdata->symbols = elf_symbols;
   sdata->count = elf_symbol_count;
 
   return 1;
 }
 
+/* Add EDATA to the list in STATE.  */
+
+static void
+elf_add_syminfo_data (struct backtrace_state *state,
+		      struct elf_syminfo_data *edata)
+{
+  if (!state->threaded)
+    {
+      struct elf_syminfo_data **pp;
+
+      for (pp = (struct elf_syminfo_data **) &state->syminfo_data;
+	   *pp != NULL;
+	   pp = &(*pp)->next)
+	;
+      *pp = edata;
+    }
+  else
+    {
+      while (1)
+	{
+	  struct elf_syminfo_data **pp;
+
+	  pp = (struct elf_syminfo_data **) &state->syminfo_data;
+
+	  while (1)
+	    {
+	      struct elf_syminfo_data *p;
+
+	      /* Atomic load.  */
+	      p = *pp;
+	      while (!__sync_bool_compare_and_swap (pp, p, p))
+		p = *pp;
+
+	      if (p == NULL)
+		break;
+
+	      pp = &p->next;
+	    }
+
+	  if (__sync_bool_compare_and_swap (pp, NULL, edata))
+	    break;
+	}
+    }
+}
+
 /* Return the symbol name and value for a PC.  */
 
 static void
@@ -364,14 +466,12 @@  elf_syminfo (struct backtrace_state *sta
     callback (data, pc, sym->name, sym->address);
 }
 
-/* Initialize the backtrace data we need from an ELF executable.  At
-   the ELF level, all we need to do is find the debug info
-   sections.  */
+/* Add the backtrace data for one ELF file.  */
 
-int
-backtrace_initialize (struct backtrace_state *state, int descriptor,
-		      backtrace_error_callback error_callback,
-		      void *data, fileline *fileline_fn)
+static int
+elf_add (struct backtrace_state *state, int descriptor, uintptr_t base_address,
+	 backtrace_error_callback error_callback, void *data,
+	 fileline *fileline_fn, int *found_sym, int *found_dwarf)
 {
   struct backtrace_view ehdr_view;
   Elf_Ehdr ehdr;
@@ -400,6 +500,9 @@  backtrace_initialize (struct backtrace_s
   struct backtrace_view debug_view;
   int debug_view_valid;
 
+  *found_sym = 0;
+  *found_dwarf = 0;
+
   shdrs_view_valid = 0;
   names_view_valid = 0;
   symtab_view_valid = 0;
@@ -516,6 +619,8 @@  backtrace_initialize (struct backtrace_s
   dynsym_shndx = 0;
 
   memset (sections, 0, sizeof sections);
+
+  /* Look for the symbol table.  */
   for (i = 1; i < shnum; ++i)
     {
       const Elf_Shdr *shdr;
@@ -552,12 +657,7 @@  backtrace_initialize (struct backtrace_s
 
   if (symtab_shndx == 0)
     symtab_shndx = dynsym_shndx;
-  if (symtab_shndx == 0)
-    {
-      state->syminfo_fn = elf_nosyms;
-      state->syminfo_data = NULL;
-    }
-  else
+  if (symtab_shndx != 0)
     {
       const Elf_Shdr *symtab_shdr;
       unsigned int strtab_shndx;
@@ -604,8 +704,9 @@  backtrace_initialize (struct backtrace_s
 	 string table permanently.  */
       backtrace_release_view (state, &symtab_view, error_callback, data);
 
-      state->syminfo_fn = elf_syminfo;
-      state->syminfo_data = sdata;
+      *found_sym = 1;
+
+      elf_add_syminfo_data (state, sdata);
     }
 
   /* FIXME: Need to handle compressed debug sections.  */
@@ -635,7 +736,6 @@  backtrace_initialize (struct backtrace_s
       if (!backtrace_close (descriptor, error_callback, data))
 	goto fail;
       *fileline_fn = elf_nodebug;
-      state->fileline_data = NULL;
       return 1;
     }
 
@@ -654,21 +754,23 @@  backtrace_initialize (struct backtrace_s
     sections[i].data = ((const unsigned char *) debug_view.data
 			+ (sections[i].offset - min_offset));
 
-  if (!backtrace_dwarf_initialize (state,
-				   sections[DEBUG_INFO].data,
-				   sections[DEBUG_INFO].size,
-				   sections[DEBUG_LINE].data,
-				   sections[DEBUG_LINE].size,
-				   sections[DEBUG_ABBREV].data,
-				   sections[DEBUG_ABBREV].size,
-				   sections[DEBUG_RANGES].data,
-				   sections[DEBUG_RANGES].size,
-				   sections[DEBUG_STR].data,
-				   sections[DEBUG_STR].size,
-				   ehdr.e_ident[EI_DATA] == ELFDATA2MSB,
-				   error_callback, data, fileline_fn))
+  if (!backtrace_dwarf_add (state, base_address,
+			    sections[DEBUG_INFO].data,
+			    sections[DEBUG_INFO].size,
+			    sections[DEBUG_LINE].data,
+			    sections[DEBUG_LINE].size,
+			    sections[DEBUG_ABBREV].data,
+			    sections[DEBUG_ABBREV].size,
+			    sections[DEBUG_RANGES].data,
+			    sections[DEBUG_RANGES].size,
+			    sections[DEBUG_STR].data,
+			    sections[DEBUG_STR].size,
+			    ehdr.e_ident[EI_DATA] == ELFDATA2MSB,
+			    error_callback, data, fileline_fn))
     goto fail;
 
+  *found_dwarf = 1;
+
   return 1;
 
  fail:
@@ -686,3 +788,115 @@  backtrace_initialize (struct backtrace_s
     backtrace_close (descriptor, error_callback, data);
   return 0;
 }
+
+/* Data passed to phdr_callback.  */
+
+struct phdr_data
+{
+  struct backtrace_state *state;
+  backtrace_error_callback error_callback;
+  void *data;
+  fileline *fileline_fn;
+  int *found_sym;
+  int *found_dwarf;
+};
+
+/* Callback passed to dl_iterate_phdr.  Load debug info from shared
+   libraries.  */
+
+static int
+phdr_callback (struct dl_phdr_info *info, size_t size ATTRIBUTE_UNUSED,
+	       void *pdata)
+{
+  struct phdr_data *pd = (struct phdr_data *) pdata;
+  int descriptor;
+  fileline elf_fileline_fn;
+  int found_dwarf;
+
+  /* There is not much we can do if we don't have the module name.  If
+     the base address is 0, this is probably the executable, which we
+     already loaded.  */
+  if (info->dlpi_name == NULL
+      || info->dlpi_name[0] == '\0'
+      || info->dlpi_addr == 0)
+    return 0;
+
+  descriptor = backtrace_open (info->dlpi_name, pd->error_callback, pd->data);
+  if (descriptor < 0)
+    return 0;
+
+  if (elf_add (pd->state, descriptor, info->dlpi_addr, pd->error_callback,
+	       pd->data, &elf_fileline_fn, pd->found_sym, &found_dwarf))
+    {
+      if (found_dwarf)
+	{
+	  *pd->found_dwarf = 1;
+	  *pd->fileline_fn = elf_fileline_fn;
+	}
+    }
+
+  return 0;
+}
+
+/* Initialize the backtrace data we need from an ELF executable.  At
+   the ELF level, all we need to do is find the debug info
+   sections.  */
+
+int
+backtrace_initialize (struct backtrace_state *state, int descriptor,
+		      backtrace_error_callback error_callback,
+		      void *data, fileline *fileline_fn)
+{
+  int found_sym;
+  int found_dwarf;
+  syminfo elf_syminfo_fn;
+  fileline elf_fileline_fn;
+  struct phdr_data pd;
+
+  if (!elf_add (state, descriptor, 0, error_callback, data, &elf_fileline_fn,
+		&found_sym, &found_dwarf))
+    return 0;
+
+  pd.state = state;
+  pd.error_callback = error_callback;
+  pd.data = data;
+  pd.fileline_fn = fileline_fn;
+  pd.found_sym = &found_sym;
+  pd.found_dwarf = &found_dwarf;
+
+  dl_iterate_phdr (phdr_callback, (void *) &pd);
+
+  elf_syminfo_fn = found_sym ? elf_syminfo : elf_nosyms;
+  if (!state->threaded)
+    {
+      if (state->syminfo_fn == NULL || found_sym)
+	state->syminfo_fn = elf_syminfo_fn;
+    }
+  else
+    {
+      __sync_bool_compare_and_swap (&state->syminfo_fn, NULL, elf_syminfo_fn);
+      if (found_sym)
+	__sync_bool_compare_and_swap (&state->syminfo_fn, elf_nosyms,
+				      elf_syminfo_fn);
+    }
+
+  if (!state->threaded)
+    {
+      if (state->fileline_fn == NULL || state->fileline_fn == elf_nodebug)
+	*fileline_fn = elf_fileline_fn;
+    }
+  else
+    {
+      fileline current_fn;
+
+      /* Atomic load.  */
+      current_fn = state->fileline_fn;
+      while (!__sync_bool_compare_and_swap (&state->fileline_fn, current_fn,
+					    current_fn))
+	current_fn = state->fileline_fn;
+      if (current_fn == NULL || current_fn == elf_nodebug)
+	*fileline_fn = elf_fileline_fn;
+    }
+
+  return 1;
+}
Index: internal.h
===================================================================
--- internal.h	(revision 192266)
+++ internal.h	(working copy)
@@ -215,21 +215,22 @@  extern int backtrace_initialize (struct 
 				 void *data,
 				 fileline *fileline_fn);
 
-/* Prepare to read file/line information from DWARF debug data.  */
+/* Add file/line information for a DWARF module.  */
 
-extern int backtrace_dwarf_initialize (struct backtrace_state *state,
-				       const unsigned char* dwarf_info,
-				       size_t dwarf_info_size,
-				       const unsigned char *dwarf_line,
-				       size_t dwarf_line_size,
-				       const unsigned char *dwarf_abbrev,
-				       size_t dwarf_abbrev_size,
-				       const unsigned char *dwarf_ranges,
-				       size_t dwarf_range_size,
-				       const unsigned char *dwarf_str,
-				       size_t dwarf_str_size,
-				       int is_bigendian,
-				       backtrace_error_callback error_callback,
-				       void *data, fileline *fileline_fn);
+extern int backtrace_dwarf_add (struct backtrace_state *state,
+				uintptr_t base_address,
+				const unsigned char* dwarf_info,
+				size_t dwarf_info_size,
+				const unsigned char *dwarf_line,
+				size_t dwarf_line_size,
+				const unsigned char *dwarf_abbrev,
+				size_t dwarf_abbrev_size,
+				const unsigned char *dwarf_ranges,
+				size_t dwarf_range_size,
+				const unsigned char *dwarf_str,
+				size_t dwarf_str_size,
+				int is_bigendian,
+				backtrace_error_callback error_callback,
+				void *data, fileline *fileline_fn);
 
 #endif
Index: configure.ac
===================================================================
--- configure.ac	(revision 192266)
+++ configure.ac	(working copy)
@@ -226,6 +226,24 @@  if test "$ALLOC_FILE" = "alloc.lo"; then
 fi
 AC_SUBST(BACKTRACE_USES_MALLOC)
 
+# Check for dl_iterate_phdr.
+AC_CHECK_HEADERS(link.h)
+if test "$ac_cv_header_link_h" = "no"; then
+  have_dl_iterate_phdr=no
+else
+  if test -n "${with_target_subdir}"; then
+    # When built as a GCC target library, we can't do a link test.
+    AC_EGREP_HEADER([dl_iterate_phdr], [link.h], [have_dl_iterate_phdr=yes],
+		    [have_dl_iterate_phdr=no])
+  else
+    AC_CHECK_FUNC([dl_iterate_phdr], [have_dl_iterate_phdr=yes],
+		  [have_dl_iterate_phdr=no])
+  fi
+fi
+if test "$have_dl_iterate_phdr" = "yes"; then
+  AC_DEFINE(HAVE_DL_ITERATE_PHDR, 1, [Define if dl_iterate_phdr is available.])
+fi
+
 # Check for the fcntl function.
 if test -n "${with_target_subdir}"; then
    case "${host}" in