Patchwork libsanitizer merge from upstream r175042

login
register
mail settings
Submitter Jakub Jelinek
Date Feb. 13, 2013, 9:51 a.m.
Message ID <20130213095107.GM4385@tucnak.redhat.com>
Download mbox | patch
Permalink /patch/220103/
State New
Headers show

Comments

Jakub Jelinek - Feb. 13, 2013, 9:51 a.m.
On Wed, Feb 13, 2013 at 01:19:47PM +0400, Konstantin Serebryany wrote:
> The attached patch is the libsanitizer merge from upstream r175042.
> 
> Lots of changes. Among other things:
>  - x86_64 linux: change the shadow offset to 0x7fff8000 (~5% speedup)
>  - the new asan allocator is enabled on Mac (was enabled on Linux before).
>  - tsan finds races between atomic and plain accesses
>  - better scanf interceptor, enabled by default
>  - don't include linux/futex.h (fixes PR56128)
>  - simple tests seem to work (again?) on PowerPC64 with 44-bit address
> space (46 AS not tested)
> 
> Patch for libsanitizer is automatically generated by libsanitizer/merge.sh
> Tested with
> rm -rf */{*/,}libsanitizer \
>   && make -j 50 \
>   && make -C gcc check-g{cc,++}
> RUNTESTFLAGS='--target_board=unix\{-m32,-m64\} asan.exp'
> 
> Our internal LLVM bots (Linux, Mac and Android) are green.
> 
> Ok to commit?


+	* sanitizer.def: rename __asan_init to __asan_init_v1.

Likewise, "Rename".

+	* testsuite/c-c++-common/asan/strncpy-overflow-1.c: update the test
+	to match the fresh asan run-time.

"Update".  Also, these two go into gcc/testsuite/ChangeLog, without
testsuite/ prefix in the pathnames.

+	* testsuite/c-c++-common/asan/rlimit-mmap-test-1.c: Ditto.
+

	Jakub
Konstantin Serebryany - Feb. 13, 2013, 10:28 a.m.
On Wed, Feb 13, 2013 at 1:51 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Wed, Feb 13, 2013 at 01:19:47PM +0400, Konstantin Serebryany wrote:
>> The attached patch is the libsanitizer merge from upstream r175042.
>>
>> Lots of changes. Among other things:
>>  - x86_64 linux: change the shadow offset to 0x7fff8000 (~5% speedup)
>>  - the new asan allocator is enabled on Mac (was enabled on Linux before).
>>  - tsan finds races between atomic and plain accesses
>>  - better scanf interceptor, enabled by default
>>  - don't include linux/futex.h (fixes PR56128)
>>  - simple tests seem to work (again?) on PowerPC64 with 44-bit address
>> space (46 AS not tested)
>>
>> Patch for libsanitizer is automatically generated by libsanitizer/merge.sh
>> Tested with
>> rm -rf */{*/,}libsanitizer \
>>   && make -j 50 \
>>   && make -C gcc check-g{cc,++}
>> RUNTESTFLAGS='--target_board=unix\{-m32,-m64\} asan.exp'
>>
>> Our internal LLVM bots (Linux, Mac and Android) are green.
>>
>> Ok to commit?
>
> --- libsanitizer/asan/asan_mapping.h    (revision 195997)
> +++ libsanitizer/asan/asan_mapping.h    (working copy)
> @@ -34,27 +34,16 @@
>  #   if defined(__powerpc64__)
>  #    define SHADOW_OFFSET (1ULL << 41)
>  #   else
> -#    define SHADOW_OFFSET (1ULL << 44)
> +#    define SHADOW_OFFSET 0x7fff8000ULL
>  #   endif
>  #  endif
>  # endif
>  #endif  // ASAN_FLEXIBLE_MAPPING_AND_OFFSET
>
> This is inconsistent with the i386.c change.  You said the 0x7fff8000ULL
> shadow offset doesn't work on Darwin, so either the above should be
>
> +#    if ASAN_MAC
> +#     define SHADOW_OFFSET (1ULL << 44)
> +#    else
> +#     define SHADOW_OFFSET 0x7fff8000ULL
> +#    endif

Right. In LLVM we test only with ASAN_FLEXIBLE_MAPPING_AND_OFFSET==1,
so this came unnoticed.
Fixed in r175049.

>
> or i386.c should use 0x7fff8000 even for TARGET_MACHO && TARGET_LP64.
>
> --- gcc/config/i386/i386.c      (revision 195997)
> +++ gcc/config/i386/i386.c      (working copy)
> @@ -5436,7 +5436,9 @@
>  static unsigned HOST_WIDE_INT
>  ix86_asan_shadow_offset (void)
>  {
> -  return (unsigned HOST_WIDE_INT) 1 << (TARGET_LP64 ? 44 : 29);
> +  return TARGET_LP64 ? (TARGET_MACHO ? (HOST_WIDE_INT_1 << 44)
> +                     : HOST_WIDE_INT_C (0x7fff8000))
> +             : (HOST_WIDE_INT_1 << 29);
>
> Please use tabs instead of 8 spaces, and indent it properly (second line
> : below the second ?, third line : below first ?).
>
> +2013-02-13  Kostya Serebryany  <kcc@google.com>
> +
> +       * config/i386/i386.c: use 0x7fff8000 as asan_shadow_offset on x86_64
> +       linux.
>
> Start sentence, so "Use" instead of "use".

done

>
> +       * sanitizer.def: rename __asan_init to __asan_init_v1.
>
> Likewise, "Rename".

done

>
> +       * testsuite/c-c++-common/asan/strncpy-overflow-1.c: update the test
> +       to match the fresh asan run-time.
>
> "Update".  Also, these two go into gcc/testsuite/ChangeLog, without
> testsuite/ prefix in the pathnames.

done.

--kcc
>
> +       * testsuite/c-c++-common/asan/rlimit-mmap-test-1.c: Ditto.
> +
>
>         Jakub
Jakub Jelinek - Feb. 13, 2013, 10:32 a.m.
On Wed, Feb 13, 2013 at 02:28:25PM +0400, Konstantin Serebryany wrote:
> Right. In LLVM we test only with ASAN_FLEXIBLE_MAPPING_AND_OFFSET==1,
> so this came unnoticed.
> Fixed in r175049.
...

This is ok, thanks.

	Jakub
Jakub Jelinek - Feb. 13, 2013, 11:59 a.m.
On Wed, Feb 13, 2013 at 11:32:00AM +0100, Jakub Jelinek wrote:
> On Wed, Feb 13, 2013 at 02:28:25PM +0400, Konstantin Serebryany wrote:
> > Right. In LLVM we test only with ASAN_FLEXIBLE_MAPPING_AND_OFFSET==1,
> > so this came unnoticed.
> > Fixed in r175049.
> ...
> 
> This is ok, thanks.

Unfortunately, it seems everything fails with that change :( on Linux.
The problem is that the default prelink library range for x86_64 is
0x3000000000LL to 0x4000000000LL, and that unfortunately overlaps
with the 0x7fff8000LL to 0x10007fff8000LL range that asan wants to use
for the shadow mapping.  And the reason for that prelink default range is
that earlier (see e.g.
http://lwn.net/Articles/106177/
) Linux on x86_64 used much smaller virtual address space than it does now.
Not sure if there are still systems running pre-2.6.9 kernels or whenever the PML4
change made it into Linux kernel on x86-64 and whether people use prelink on
them.  But in any case, even if I change the prelink range now (perhaps
conditionally on the size of address space detected by prelink), it will
still cause issues.

So, either we need to revert that i386.c and asan_mapping.h (SHADOW_OFFSET)
change, or support non-contiguous shadow memory for the Linux x86-64 case.
What could work is if we had:

0x000000000000  -	0x00007fff8000	low memory
0x00007fff8000	-	0x00008fff7000	shadow mem for low memory
0x00008fff7000	-	0x00067fff8000	protected
0x00067fff8000	-	0x00087fff8000	shadow mem for mid memory
0x00087fff8000	-	0x003000000000	protected
0x003000000000  -	0x004000000000	mid memory
0x004000000000	-	0x02008fff7000	protected
0x02008fff7000	-	0x10007fff8000	shadow mem for high memory
0x10007fff8000	-	0x7fffffffffff	high memory

asan_mapping.h then would need to introduce AddrIsInMidMem and
AddrIsInMidShadow inlines (perhaps defined to false for configurations
that don't need 3 part memory), use those in AddrIsInMem
and AddrIsInShadow, tweak AddrIsInShadowGap (as it has now more gaps)
for this configuration and tweak the mapping code.

	Jakub
Konstantin Serebryany - Feb. 13, 2013, 12:32 p.m.
On Wed, Feb 13, 2013 at 3:59 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Wed, Feb 13, 2013 at 11:32:00AM +0100, Jakub Jelinek wrote:
>> On Wed, Feb 13, 2013 at 02:28:25PM +0400, Konstantin Serebryany wrote:
>> > Right. In LLVM we test only with ASAN_FLEXIBLE_MAPPING_AND_OFFSET==1,
>> > so this came unnoticed.
>> > Fixed in r175049.
>> ...
>>
>> This is ok, thanks.
>
> Unfortunately, it seems everything fails with that change :( on Linux.
> The problem is that the default prelink library range for x86_64 is
> 0x3000000000LL to 0x4000000000LL, and that unfortunately overlaps

Forgive my ignorance, what is the  default prelink library range?

> with the 0x7fff8000LL to 0x10007fff8000LL range that asan wants to use
> for the shadow mapping.  And the reason for that prelink default range is
> that earlier (see e.g.
> http://lwn.net/Articles/106177/
> ) Linux on x86_64 used much smaller virtual address space than it does now.
> Not sure if there are still systems running pre-2.6.9 kernels or whenever the PML4
> change made it into Linux kernel on x86-64 and whether people use prelink on
> them.  But in any case, even if I change the prelink range now (perhaps
> conditionally on the size of address space detected by prelink), it will
> still cause issues.
>
> So, either we need to revert that i386.c and asan_mapping.h (SHADOW_OFFSET)
> change, or support non-contiguous shadow memory for the Linux x86-64 case.

I suggest to either revert or (better) to support flexible mapping and
revert the offset only in the gcc compiler module
(leaving asan-rt unchanged).

non-contiguous shadow memory sounds too scary and costly to support,
not worth the benefit.

> What could work is if we had:
>
> 0x000000000000  -       0x00007fff8000  low memory
> 0x00007fff8000  -       0x00008fff7000  shadow mem for low memory
> 0x00008fff7000  -       0x00067fff8000  protected
> 0x00067fff8000  -       0x00087fff8000  shadow mem for mid memory
> 0x00087fff8000  -       0x003000000000  protected
> 0x003000000000  -       0x004000000000  mid memory
> 0x004000000000  -       0x02008fff7000  protected
> 0x02008fff7000  -       0x10007fff8000  shadow mem for high memory
> 0x10007fff8000  -       0x7fffffffffff  high memory
>
> asan_mapping.h then would need to introduce AddrIsInMidMem and
> AddrIsInMidShadow inlines (perhaps defined to false for configurations
> that don't need 3 part memory), use those in AddrIsInMem
> and AddrIsInShadow, tweak AddrIsInShadowGap (as it has now more gaps)
> for this configuration and tweak the mapping code.
>
>         Jakub
Jakub Jelinek - Feb. 13, 2013, 12:48 p.m.
On Wed, Feb 13, 2013 at 04:32:33PM +0400, Konstantin Serebryany wrote:
> > Unfortunately, it seems everything fails with that change :( on Linux.
> > The problem is that the default prelink library range for x86_64 is
> > 0x3000000000LL to 0x4000000000LL, and that unfortunately overlaps
> 
> Forgive my ignorance, what is the  default prelink library range?

Prelink is a program of mine (see e.g.
http://people.redhat.com/jakub/prelink.pdf
) that speeds up dynamic linking of programs.
It is used by default on various Linux distributions.  prelinked shared
libraries (and dynamic linker) have their base addresses chosen by the
prelink program (and, by default in the default range of shared libraries
for the architecture, which is 0x3000000000LL to 0x4000000000LL for x86_64).
The addresses are usually (depending on prelink options) randomized upon
prelinking, so different hosts use different addresses and the same host
even after a while uses different addresses too, but by default for a few
weeks, if say glibc isn't upgraded in between, you get the same addresses
for the libraries all the time.
So, you get something like:
cat /proc/self/maps
00400000-0040b000 r-xp 00000000 08:02 1201920                            /usr/bin/cat
0060a000-0060b000 r--p 0000a000 08:02 1201920                            /usr/bin/cat
0060b000-0060c000 rw-p 0000b000 08:02 1201920                            /usr/bin/cat
01431000-01452000 rw-p 00000000 00:00 0                                  [heap]
3fe9400000-3fe9420000 r-xp 00000000 08:02 1179965                        /usr/lib64/ld-2.15.so
3fe961f000-3fe9620000 r--p 0001f000 08:02 1179965                        /usr/lib64/ld-2.15.so
3fe9620000-3fe9621000 rw-p 00020000 08:02 1179965                        /usr/lib64/ld-2.15.so
3fe9621000-3fe9622000 rw-p 00000000 00:00 0 
3fe9800000-3fe99ac000 r-xp 00000000 08:02 1180697                        /usr/lib64/libc-2.15.so
3fe99ac000-3fe9bac000 ---p 001ac000 08:02 1180697                        /usr/lib64/libc-2.15.so
3fe9bac000-3fe9bb0000 r--p 001ac000 08:02 1180697                        /usr/lib64/libc-2.15.so
3fe9bb0000-3fe9bb2000 rw-p 001b0000 08:02 1180697                        /usr/lib64/libc-2.15.so
3fe9bb2000-3fe9bb7000 rw-p 00000000 00:00 0 
7fae406f5000-7fae46b22000 r--p 00000000 08:02 1215605                    /usr/lib/locale/locale-archive
7fae46b22000-7fae46b25000 rw-p 00000000 00:00 0 
7fae46b42000-7fae46b43000 rw-p 00000000 00:00 0 
7fffe04f7000-7fffe0518000 rw-p 00000000 00:00 0                          [stack]
7fffe05e6000-7fffe05e7000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Perhaps Ubuntu doesn't enable prelink by default, but all the world isn't Ubuntu.

> I suggest to either revert or (better) to support flexible mapping and
> revert the offset only in the gcc compiler module
> (leaving asan-rt unchanged).

I don't see how could flexible mapping help in this case, it just can't work either.
The only exception are PIE binaries, which by design aren't prelinked and kernel
for them disregards the prelinking of the dynamic linker, so the dynamic linker
for PIEs isn't loaded at the prelink chosen address, but at some randomized address.

If you try -Wl,-Ttext* with flexible mapping and use either zero base, or say 0x7fff8000,
you run into the exactly same issues, wether with gcc or clang.

> non-contiguous shadow memory sounds too scary and costly to support,
> not worth the benefit.

Why do you think it should be too costly?

	Jakub
Konstantin Serebryany - Feb. 13, 2013, 12:57 p.m.
On Wed, Feb 13, 2013 at 4:48 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Wed, Feb 13, 2013 at 04:32:33PM +0400, Konstantin Serebryany wrote:
>> > Unfortunately, it seems everything fails with that change :( on Linux.
>> > The problem is that the default prelink library range for x86_64 is
>> > 0x3000000000LL to 0x4000000000LL, and that unfortunately overlaps
>>
>> Forgive my ignorance, what is the  default prelink library range?
>
> Prelink is a program of mine (see e.g.
> http://people.redhat.com/jakub/prelink.pdf
> ) that speeds up dynamic linking of programs.
> It is used by default on various Linux distributions.

Can it be disabled somehow (for asan)?

>  prelinked shared
> libraries (and dynamic linker) have their base addresses chosen by the
> prelink program (and, by default in the default range of shared libraries
> for the architecture, which is 0x3000000000LL to 0x4000000000LL for x86_64).
> The addresses are usually (depending on prelink options) randomized upon
> prelinking, so different hosts use different addresses and the same host
> even after a while uses different addresses too, but by default for a few
> weeks, if say glibc isn't upgraded in between, you get the same addresses
> for the libraries all the time.
> So, you get something like:
> cat /proc/self/maps
> 00400000-0040b000 r-xp 00000000 08:02 1201920                            /usr/bin/cat
> 0060a000-0060b000 r--p 0000a000 08:02 1201920                            /usr/bin/cat
> 0060b000-0060c000 rw-p 0000b000 08:02 1201920                            /usr/bin/cat
> 01431000-01452000 rw-p 00000000 00:00 0                                  [heap]
> 3fe9400000-3fe9420000 r-xp 00000000 08:02 1179965                        /usr/lib64/ld-2.15.so
> 3fe961f000-3fe9620000 r--p 0001f000 08:02 1179965                        /usr/lib64/ld-2.15.so
> 3fe9620000-3fe9621000 rw-p 00020000 08:02 1179965                        /usr/lib64/ld-2.15.so
> 3fe9621000-3fe9622000 rw-p 00000000 00:00 0
> 3fe9800000-3fe99ac000 r-xp 00000000 08:02 1180697                        /usr/lib64/libc-2.15.so
> 3fe99ac000-3fe9bac000 ---p 001ac000 08:02 1180697                        /usr/lib64/libc-2.15.so
> 3fe9bac000-3fe9bb0000 r--p 001ac000 08:02 1180697                        /usr/lib64/libc-2.15.so
> 3fe9bb0000-3fe9bb2000 rw-p 001b0000 08:02 1180697                        /usr/lib64/libc-2.15.so
> 3fe9bb2000-3fe9bb7000 rw-p 00000000 00:00 0
> 7fae406f5000-7fae46b22000 r--p 00000000 08:02 1215605                    /usr/lib/locale/locale-archive
> 7fae46b22000-7fae46b25000 rw-p 00000000 00:00 0
> 7fae46b42000-7fae46b43000 rw-p 00000000 00:00 0
> 7fffe04f7000-7fffe0518000 rw-p 00000000 00:00 0                          [stack]
> 7fffe05e6000-7fffe05e7000 r-xp 00000000 00:00 0                          [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
>
> Perhaps Ubuntu doesn't enable prelink by default, but all the world isn't Ubuntu.
>
>> I suggest to either revert or (better) to support flexible mapping and
>> revert the offset only in the gcc compiler module
>> (leaving asan-rt unchanged).
>
> I don't see how could flexible mapping help in this case, it just can't work either.

If we enabled flexible mapping in the gcc build (making it more
similar to the llvm build)
we will be able to use the old 2^44 offset w/o changing the asan-rt.

> The only exception are PIE binaries, which by design aren't prelinked and kernel
> for them disregards the prelinking of the dynamic linker, so the dynamic linker
> for PIEs isn't loaded at the prelink chosen address, but at some randomized address.
>
> If you try -Wl,-Ttext* with flexible mapping and use either zero base, or say 0x7fff8000,
> you run into the exactly same issues, wether with gcc or clang.
>
>> non-contiguous shadow memory sounds too scary and costly to support,
>> not worth the benefit.
>
> Why do you think it should be too costly?

That's yet another set of spaghetti ifdefs.
I'd rather revert the whole thing back to 2^44 than do that.

--kcc
Jakub Jelinek - Feb. 13, 2013, 1:07 p.m.
On Wed, Feb 13, 2013 at 04:57:30PM +0400, Konstantin Serebryany wrote:
> On Wed, Feb 13, 2013 at 4:48 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> > On Wed, Feb 13, 2013 at 04:32:33PM +0400, Konstantin Serebryany wrote:
> >> > Unfortunately, it seems everything fails with that change :( on Linux.
> >> > The problem is that the default prelink library range for x86_64 is
> >> > 0x3000000000LL to 0x4000000000LL, and that unfortunately overlaps
> >>
> >> Forgive my ignorance, what is the  default prelink library range?
> >
> > Prelink is a program of mine (see e.g.
> > http://people.redhat.com/jakub/prelink.pdf
> > ) that speeds up dynamic linking of programs.
> > It is used by default on various Linux distributions.
> 
> Can it be disabled somehow (for asan)?

No.  You can disable it for the whole system (prelink -ua), but that is not
a sane requirement to running sanitized programs.  There is
also LD_USE_LOAD_BIAS=0 env variable, but 1) that has to be set before
running the program, setting it from within libasan ctor is too late, and
more importantly 2) it still doesn't affect the mapping of the dynamic
linker.  So, with LD_USE_LOAD_BIAS=0 cat /proc/self/maps the
dynamic linker will be still mapped somewhere in the 0x3000000000 to
0x4000000000 range, just other shared libraries mapped in by the dynamic
linker, rather than kernel directly, will be placed at the randomized (and
usually high) addresses, typically 0x7fXXXXXXXXXX.

> If we enabled flexible mapping in the gcc build (making it more
> similar to the llvm build)
> we will be able to use the old 2^44 offset w/o changing the asan-rt.

Sure, but it will be then slower, I thought you are looking for ASAN speed
improvements.

> That's yet another set of spaghetti ifdefs.
> I'd rather revert the whole thing back to 2^44 than do that.

Please revert the two hunks now.  I'll try to implement it eventually and
try to convince you ;)

	Jakub
Richard Guenther - Feb. 13, 2013, 1:27 p.m.
On Wed, Feb 13, 2013 at 2:07 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Wed, Feb 13, 2013 at 04:57:30PM +0400, Konstantin Serebryany wrote:
>> On Wed, Feb 13, 2013 at 4:48 PM, Jakub Jelinek <jakub@redhat.com> wrote:
>> > On Wed, Feb 13, 2013 at 04:32:33PM +0400, Konstantin Serebryany wrote:
>> >> > Unfortunately, it seems everything fails with that change :( on Linux.
>> >> > The problem is that the default prelink library range for x86_64 is
>> >> > 0x3000000000LL to 0x4000000000LL, and that unfortunately overlaps
>> >>
>> >> Forgive my ignorance, what is the  default prelink library range?
>> >
>> > Prelink is a program of mine (see e.g.
>> > http://people.redhat.com/jakub/prelink.pdf
>> > ) that speeds up dynamic linking of programs.
>> > It is used by default on various Linux distributions.
>>
>> Can it be disabled somehow (for asan)?
>
> No.  You can disable it for the whole system (prelink -ua), but that is not
> a sane requirement to running sanitized programs.  There is
> also LD_USE_LOAD_BIAS=0 env variable, but 1) that has to be set before
> running the program, setting it from within libasan ctor is too late, and
> more importantly 2) it still doesn't affect the mapping of the dynamic
> linker.  So, with LD_USE_LOAD_BIAS=0 cat /proc/self/maps the
> dynamic linker will be still mapped somewhere in the 0x3000000000 to
> 0x4000000000 range, just other shared libraries mapped in by the dynamic
> linker, rather than kernel directly, will be placed at the randomized (and
> usually high) addresses, typically 0x7fXXXXXXXXXX.

ASAN could set an ELF flag on the executable to tell the kernel not
to use prelinked objects?  That is, similar to how we handle executable
stacks?

Richard.

>> If we enabled flexible mapping in the gcc build (making it more
>> similar to the llvm build)
>> we will be able to use the old 2^44 offset w/o changing the asan-rt.
>
> Sure, but it will be then slower, I thought you are looking for ASAN speed
> improvements.
>
>> That's yet another set of spaghetti ifdefs.
>> I'd rather revert the whole thing back to 2^44 than do that.
>
> Please revert the two hunks now.  I'll try to implement it eventually and
> try to convince you ;)
>
>         Jakub
Jakub Jelinek - Feb. 13, 2013, 1:33 p.m.
On Wed, Feb 13, 2013 at 02:27:56PM +0100, Richard Biener wrote:
> ASAN could set an ELF flag on the executable to tell the kernel not
> to use prelinked objects?  That is, similar to how we handle executable
> stacks?

But we don't have such a flag right now, and what should old kernels that
don't support it do with that flag?  Should it be some kind of flag that
causes the programs not to run at all on kernel < 3.9 or whenever the flag
would be handled (can be done with .note.ABI-tag section, plus some flag
somewhere), or something else?  In any case, that would mean changes all
through the toolchain (glibc, linkers, kernel), compared to changing
two files in libsanitizer/asan/ (asan_mapping.h and asan_rtl.cc I believe).

	Jakub
Konstantin Serebryany - Feb. 13, 2013, 1:39 p.m.
On Wed, Feb 13, 2013 at 5:07 PM, Jakub Jelinek <jakub@redhat.com> wrote:
> On Wed, Feb 13, 2013 at 04:57:30PM +0400, Konstantin Serebryany wrote:
>> On Wed, Feb 13, 2013 at 4:48 PM, Jakub Jelinek <jakub@redhat.com> wrote:
>> > On Wed, Feb 13, 2013 at 04:32:33PM +0400, Konstantin Serebryany wrote:
>> >> > Unfortunately, it seems everything fails with that change :( on Linux.
>> >> > The problem is that the default prelink library range for x86_64 is
>> >> > 0x3000000000LL to 0x4000000000LL, and that unfortunately overlaps
>> >>
>> >> Forgive my ignorance, what is the  default prelink library range?
>> >
>> > Prelink is a program of mine (see e.g.
>> > http://people.redhat.com/jakub/prelink.pdf
>> > ) that speeds up dynamic linking of programs.
>> > It is used by default on various Linux distributions.
>>
>> Can it be disabled somehow (for asan)?
>
> No.  You can disable it for the whole system (prelink -ua), but that is not
> a sane requirement to running sanitized programs.

Why not?
:)

>> There is
> also LD_USE_LOAD_BIAS=0 env variable, but 1) that has to be set before
> running the program, setting it from within libasan ctor is too late,

This we can deal with.
We already setenv+reexec on Mac to solve similar issue with Mac's
dynamic run-time.

> and
> more importantly 2) it still doesn't affect the mapping of the dynamic
> linker.  So, with LD_USE_LOAD_BIAS=0 cat /proc/self/maps the
> dynamic linker will be still mapped somewhere in the 0x3000000000 to
> 0x4000000000 range, just other shared libraries mapped in by the dynamic
> linker, rather than kernel directly, will be placed at the randomized (and
> usually high) addresses, typically 0x7fXXXXXXXXXX.
>
>> If we enabled flexible mapping in the gcc build (making it more
>> similar to the llvm build)
>> we will be able to use the old 2^44 offset w/o changing the asan-rt.
>
> Sure, but it will be then slower, I thought you are looking for ASAN speed
> improvements.

Yes, and we already achieved it on ubuntu :)

>
>> That's yet another set of spaghetti ifdefs.
>> I'd rather revert the whole thing back to 2^44 than do that.
>
> Please revert the two hunks now.

That's a bummer. We've already deployed Clang with the new setting and
will likely not want to revert it there.
If we revert the two hunks in gcc (in gcc/config/i386/i386.c and in
asan_mapping.h) we will have larger inconsistency between gcc and
clang variants.
But if we switch to flexible mapping in gcc (will require gcc to emit
the two globals) we will only need to revert gcc/config/i386/i386.c
and the run-times in clang and gcc will remain the same.

> I'll try to implement it eventually and
> try to convince you ;)

That's surely not hard to implement, but very hard to support.

--kcc

>
>         Jakub
Jack Howarth - Feb. 13, 2013, 2:46 p.m.
On Wed, Feb 13, 2013 at 11:32:00AM +0100, Jakub Jelinek wrote:
> On Wed, Feb 13, 2013 at 02:28:25PM +0400, Konstantin Serebryany wrote:
> > Right. In LLVM we test only with ASAN_FLEXIBLE_MAPPING_AND_OFFSET==1,
> > so this came unnoticed.
> > Fixed in r175049.
> ...
> 
> This is ok, thanks.
> 
> 	Jakub

FYI, asan.exp shows no regression at -m32/-m64 on x86_64-apple-darwin12 using
gcc svn at r196013. Thanks for the new allocator support on darwin.
        Jack

Patch

--- libsanitizer/asan/asan_mapping.h	(revision 195997)
+++ libsanitizer/asan/asan_mapping.h	(working copy)
@@ -34,27 +34,16 @@ 
 #   if defined(__powerpc64__)
 #    define SHADOW_OFFSET (1ULL << 41)
 #   else
-#    define SHADOW_OFFSET (1ULL << 44)
+#    define SHADOW_OFFSET 0x7fff8000ULL
 #   endif
 #  endif
 # endif
 #endif  // ASAN_FLEXIBLE_MAPPING_AND_OFFSET

This is inconsistent with the i386.c change.  You said the 0x7fff8000ULL
shadow offset doesn't work on Darwin, so either the above should be

+#    if ASAN_MAC
+#     define SHADOW_OFFSET (1ULL << 44)
+#    else
+#     define SHADOW_OFFSET 0x7fff8000ULL
+#    endif

or i386.c should use 0x7fff8000 even for TARGET_MACHO && TARGET_LP64.

--- gcc/config/i386/i386.c	(revision 195997)
+++ gcc/config/i386/i386.c	(working copy)
@@ -5436,7 +5436,9 @@ 
 static unsigned HOST_WIDE_INT
 ix86_asan_shadow_offset (void)
 {
-  return (unsigned HOST_WIDE_INT) 1 << (TARGET_LP64 ? 44 : 29);
+  return TARGET_LP64 ? (TARGET_MACHO ? (HOST_WIDE_INT_1 << 44)
+                     : HOST_WIDE_INT_C (0x7fff8000))
+             : (HOST_WIDE_INT_1 << 29);

Please use tabs instead of 8 spaces, and indent it properly (second line
: below the second ?, third line : below first ?).

+2013-02-13  Kostya Serebryany  <kcc@google.com>
+
+	* config/i386/i386.c: use 0x7fff8000 as asan_shadow_offset on x86_64
+	linux.

Start sentence, so "Use" instead of "use".