| Message ID | cover.1537542735.git.andreyknvl@google.com |
|---|---|
| Headers | show |
| Series | kasan: add software tag-based mode for arm64 | expand |
On Fri, Sep 21, 2018 at 5:13 PM, Andrey Konovalov <andreyknvl@google.com> wrote: > This patchset adds a new software tag-based mode to KASAN [1]. > (Initially this mode was called KHWASAN, but it got renamed, > see the naming rationale at the end of this section). Reviewed-by: Dmitry Vyukov <dvyukov@google.com> > The plan is to implement HWASan [2] for the kernel with the incentive, > that it's going to have comparable to KASAN performance, but in the same > time consume much less memory, trading that off for somewhat imprecise > bug detection and being supported only for arm64. > > The underlying ideas of the approach used by software tag-based KASAN are: > > 1. By using the Top Byte Ignore (TBI) arm64 CPU feature, we can store > pointer tags in the top byte of each kernel pointer. > > 2. Using shadow memory, we can store memory tags for each chunk of kernel > memory. > > 3. On each memory allocation, we can generate a random tag, embed it into > the returned pointer and set the memory tags that correspond to this > chunk of memory to the same value. > > 4. By using compiler instrumentation, before each memory access we can add > a check that the pointer tag matches the tag of the memory that is being > accessed. > > 5. On a tag mismatch we report an error. > > With this patchset the existing KASAN mode gets renamed to generic KASAN, > with the word "generic" meaning that the implementation can be supported > by any architecture as it is purely software. > > The new mode this patchset adds is called software tag-based KASAN. The > word "tag-based" refers to the fact that this mode uses tags embedded into > the top byte of kernel pointers and the TBI arm64 CPU feature that allows > to dereference such pointers. The word "software" here means that shadow > memory manipulation and tag checking on pointer dereference is done in > software. As it is the only tag-based implementation right now, "software > tag-based" KASAN is sometimes referred to as simply "tag-based" in this > patchset. > > A potential expansion of this mode is a hardware tag-based mode, which would > use hardware memory tagging support (announced by Arm [3]) instead of > compiler instrumentation and manual shadow memory manipulation. > > Same as generic KASAN, software tag-based KASAN is strictly a debugging > feature. > > [1] https://www.kernel.org/doc/html/latest/dev-tools/kasan.html > > [2] http://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html > > [3] https://community.arm.com/processors/b/blog/posts/arm-a-profile-architecture-2018-developments-armv85a > > > ====== Rationale > > On mobile devices generic KASAN's memory usage is significant problem. One > of the main reasons to have tag-based KASAN is to be able to perform a > similar set of checks as the generic one does, but with lower memory > requirements. > > Comment from Vishwath Mohan <vishwath@google.com>: > > I don't have data on-hand, but anecdotally both ASAN and KASAN have proven > problematic to enable for environments that don't tolerate the increased > memory pressure well. This includes, > (a) Low-memory form factors - Wear, TV, Things, lower-tier phones like Go, > (c) Connected components like Pixel's visual core [1]. > > These are both places I'd love to have a low(er) memory footprint option at > my disposal. > > Comment from Evgenii Stepanov <eugenis@google.com>: > > Looking at a live Android device under load, slab (according to > /proc/meminfo) + kernel stack take 8-10% available RAM (~350MB). KASAN's > overhead of 2x - 3x on top of it is not insignificant. > > Not having this overhead enables near-production use - ex. running > KASAN/KHWASAN kernel on a personal, daily-use device to catch bugs that do > not reproduce in test configuration. These are the ones that often cost > the most engineering time to track down. > > CPU overhead is bad, but generally tolerable. RAM is critical, in our > experience. Once it gets low enough, OOM-killer makes your life miserable. > > [1] https://www.blog.google/products/pixel/pixel-visual-core-image-processing-and-machine-learning-pixel-2/ > > > ====== Technical details > > Software tag-based KASAN mode is implemented in a very similar way to the > generic one. This patchset essentially does the following: > > 1. TCR_TBI1 is set to enable Top Byte Ignore. > > 2. Shadow memory is used (with a different scale, 1:16, so each shadow > byte corresponds to 16 bytes of kernel memory) to store memory tags. > > 3. All slab objects are aligned to shadow scale, which is 16 bytes. > > 4. All pointers returned from the slab allocator are tagged with a random > tag and the corresponding shadow memory is poisoned with the same value. > > 5. Compiler instrumentation is used to insert tag checks. Either by > calling callbacks or by inlining them (CONFIG_KASAN_OUTLINE and > CONFIG_KASAN_INLINE flags are reused). > > 6. When a tag mismatch is detected in callback instrumentation mode > KASAN simply prints a bug report. In case of inline instrumentation, > clang inserts a brk instruction, and KASAN has it's own brk handler, > which reports the bug. > > 7. The memory in between slab objects is marked with a reserved tag, and > acts as a redzone. > > 8. When a slab object is freed it's marked with a reserved tag. > > Bug detection is imprecise for two reasons: > > 1. We won't catch some small out-of-bounds accesses, that fall into the > same shadow cell, as the last byte of a slab object. > > 2. We only have 1 byte to store tags, which means we have a 1/256 > probability of a tag match for an incorrect access (actually even > slightly less due to reserved tag values). > > Despite that there's a particular type of bugs that tag-based KASAN can > detect compared to generic KASAN: use-after-free after the object has been > allocated by someone else. > > > ====== Testing > > Some kernel developers voiced a concern that changing the top byte of > kernel pointers may lead to subtle bugs that are difficult to discover. > To address this concern deliberate testing has been performed. > > It doesn't seem feasible to do some kind of static checking to find > potential issues with pointer tagging, so a dynamic approach was taken. > All pointer comparisons/subtractions have been instrumented in an LLVM > compiler pass and a kernel module that would print a bug report whenever > two pointers with different tags are being compared/subtracted (ignoring > comparisons with NULL pointers and with pointers obtained by casting an > error code to a pointer type) has been used. Then the kernel has been > booted in QEMU and on an Odroid C2 board and syzkaller has been run. > > This yielded the following results. > > The two places that look interesting are: > > is_vmalloc_addr in include/linux/mm.h > is_kernel_rodata in mm/util.c > > Here we compare a pointer with some fixed untagged values to make sure > that the pointer lies in a particular part of the kernel address space. > Since tag-based KASAN doesn't add tags to pointers that belong to rodata > or vmalloc regions, this should work as is. To make sure debug checks to > those two functions that check that the result doesn't change whether > we operate on pointers with or without untagging has been added. > > A few other cases that don't look that interesting: > > Comparing pointers to achieve unique sorting order of pointee objects > (e.g. sorting locks addresses before performing a double lock): > > tty_ldisc_lock_pair_timeout in drivers/tty/tty_ldisc.c > pipe_double_lock in fs/pipe.c > unix_state_double_lock in net/unix/af_unix.c > lock_two_nondirectories in fs/inode.c > mutex_lock_double in kernel/events/core.c > > ep_cmp_ffd in fs/eventpoll.c > fsnotify_compare_groups fs/notify/mark.c > > Nothing needs to be done here, since the tags embedded into pointers > don't change, so the sorting order would still be unique. > > Checks that a pointer belongs to some particular allocation: > > is_sibling_entry in lib/radix-tree.c > object_is_on_stack in include/linux/sched/task_stack.h > > Nothing needs to be done here either, since two pointers can only belong > to the same allocation if they have the same tag. > > Overall, since the kernel boots and works, there are no critical bugs. > As for the rest, the traditional kernel testing way (use until fails) is > the only one that looks feasible. > > Another point here is that tag-based KASAN is available under a separate > config option that needs to be deliberately enabled. Even though it might > be used in a "near-production" environment to find bugs that are not found > during fuzzing or running tests, it is still a debug tool. > > > ====== Benchmarks > > The following numbers were collected on Odroid C2 board. Both generic and > tag-based KASAN were used in inline instrumentation mode. > > Boot time [1]: > * ~1.7 sec for clean kernel > * ~5.0 sec for generic KASAN > * ~5.0 sec for tag-based KASAN > > Network performance [2]: > * 8.33 Gbits/sec for clean kernel > * 3.17 Gbits/sec for generic KASAN > * 2.85 Gbits/sec for tag-based KASAN > > Slab memory usage after boot [3]: > * ~40 kb for clean kernel > * ~105 kb (~260% overhead) for generic KASAN > * ~47 kb (~20% overhead) for tag-based KASAN > > KASAN memory overhead consists of three main parts: > 1. Increased slab memory usage due to redzones. > 2. Shadow memory (the whole reserved once during boot). > 3. Quaratine (grows gradually until some preset limit; the more the limit, > the more the chance to detect a use-after-free). > > Comparing tag-based vs generic KASAN for each of these points: > 1. 20% vs 260% overhead. > 2. 1/16th vs 1/8th of physical memory. > 3. Tag-based KASAN doesn't require quarantine. > > [1] Time before the ext4 driver is initialized. > [2] Measured as `iperf -s & iperf -c 127.0.0.1 -t 30`. > [3] Measured as `cat /proc/meminfo | grep Slab`. > > > ====== Some notes > > A few notes: > > 1. The patchset can be found here: > https://github.com/xairy/kasan-prototype/tree/khwasan > > 2. Building requires a recent Clang version (7.0.0 or later). > > 3. Stack instrumentation is not supported yet and will be added later. > > > ====== Changes > > Changes in v9: > - Fixed kasan_init_slab_obj() hook when KASAN is disabled. > - Added assign_tag() function that preassigns tags for caches with > constructors. > - Fixed KASAN_TAG_MASK redefinition in include/linux/mm.h vs > mm/kasan/kasan.h. > > Changes in v8: > - Rebased onto 7876320f (4.19-rc4). > - Renamed KHWASAN to software tag-based KASAN (see the top of the cover > letter for details). > - Explicitly called tag-based KASAN a debug tool. > - Reused kasan_init_slab_obj() callback to preassign tags to caches > without constructors, remove khwasan_preset_sl(u/a)b_tag(). > - Moved move obj_to_index to include/linux/slab_def.h from mm/slab.c. > - Moved cache->s_mem untagging to alloc_slabmgmt() for SLAB. > - Fixed check_memory_region() to correctly handle user memory accesses and > size == 0 case. > - Merged __no_sanitize_hwaddress into __no_sanitize_address. > - Defined KASAN_SET_TAG and KASAN_RESET_TAG macros for non KASAN builds to > avoid duplication of __kimg_to_phys, _virt_addr_is_linear and > page_to_virt macros. > - Fixed and simplified find_first_bad_addr for generic KASAN. > - Use non symbolized example KASAN report in documentation. > - Mention clang version requirements for both KASAN modes in the Kconfig > options and in the documentation. > - Various small fixes. > > Version v7 got accidentally skipped. > > Changes in v6: > - Rebased onto 050cdc6c (4.19-rc1+). > - Added notes regarding patchset testing into the cover letter. > > Changes in v5: > - Rebased onto 1ffaddd029 (4.18-rc8). > - Preassign tags for objects from caches with constructors and > SLAB_TYPESAFE_BY_RCU caches. > - Fix SLAB allocator support by untagging page->s_mem in > kasan_poison_slab(). > - Performed dynamic testing to find potential places where pointer tagging > might result in bugs [1]. > - Clarified and fixed memory usage benchmarks in the cover letter. > - Added a rationale for having KHWASAN to the cover letter. > > Changes in v4: > - Fixed SPDX comment style in mm/kasan/kasan.h. > - Fixed mm/kasan/kasan.h changes being included in a wrong patch. > - Swapped "khwasan, arm64: fix up fault handling logic" and "khwasan: add > tag related helper functions" patches order. > - Rebased onto 6f0d349d (4.18-rc2+). > > Changes in v3: > - Minor documentation fixes. > - Fixed CFLAGS variable name in KASAN makefile. > - Added a "SPDX-License-Identifier: GPL-2.0" line to all source files > under mm/kasan. > - Rebased onto 81e97f013 (4.18-rc1+). > > Changes in v2: > - Changed kmalloc_large_node_hook to return tagged pointer instead of > using an output argument. > - Fix checking whether -fsanitize=hwaddress is supported by the compiler. > - Removed duplication of -fno-builtin for KASAN and KHWASAN. > - Removed {} block for one line for_each_possible_cpu loop. > - Made set_track() static inline as it is used only in common.c. > - Moved optimal_redzone() to common.c. > - Fixed using tagged pointer for shadow calculation in > kasan_unpoison_shadow(). > - Restored setting cache->align in kasan_cache_create(), which was > accidentally lost. > - Simplified __kasan_slab_free(), kasan_alloc_pages() and kasan_kmalloc(). > - Removed tagging from kasan_kmalloc_large(). > - Added page_kasan_tag_reset() to kasan_poison_slab() and removed > !PageSlab() check from page_to_virt. > - Reset pointer tag in _virt_addr_is_linear. > - Set page tag for each page when multiple pages are allocated or freed. > - Added a comment as to why we ignore cma allocated pages. > > Changes in v1: > - Rebased onto 4.17-rc4. > - Updated benchmarking stats. > - Documented compiler version requirements, memory usage and slowdown. > - Dropped kvm patches, as clang + arm64 + kvm is completely broken [1]. > > Changes in RFC v3: > - Renamed CONFIG_KASAN_CLASSIC and CONFIG_KASAN_TAGS to > CONFIG_KASAN_GENERIC and CONFIG_KASAN_HW respectively. > - Switch to -fsanitize=kernel-hwaddress instead of -fsanitize=hwaddress. > - Removed unnecessary excessive shadow initialization. > - Removed khwasan_enabled flag (it’s not needed since KHWASAN is > initialized before any slab caches are used). > - Split out kasan_report.c and khwasan_report.c from report.c. > - Moved more common KASAN and KHWASAN functions to common.c. > - Added tagging to pagealloc. > - Rebased onto 4.17-rc1. > - Temporarily dropped patch that adds kvm support (arm64 + kvm + clang > combo is broken right now [2]). > > Changes in RFC v2: > - Removed explicit casts to u8 * for kasan_mem_to_shadow() calls. > - Introduced KASAN_TCR_FLAGS for setting the TCR_TBI1 flag. > - Added a comment regarding the non-atomic RMW sequence in > khwasan_random_tag(). > - Made all tag related functions accept const void *. > - Untagged pointers in __kimg_to_phys, which is used by virt_to_phys. > - Untagged pointers in show_ptr in fault handling logic. > - Untagged pointers passed to KVM. > - Added two reserved tag values: 0xFF and 0xFE. > - Used the reserved tag 0xFF to disable validity checking (to resolve the > issue with pointer tag being lost after page_address + kmap usage). > - Used the reserved tag 0xFE to mark redzones and freed objects. > - Added mnemonics for esr manipulation in KHWASAN brk handler. > - Added a comment about the -recover flag. > - Some minor cleanups and fixes. > - Rebased onto 3215b9d5 (4.16-rc6+). > - Tested on real hardware (Odroid C2 board). > - Added better benchmarks. > > [1] https://lkml.org/lkml/2018/7/18/765 > [2] https://lkml.org/lkml/2018/4/19/775 > > Andrey Konovalov (20): > kasan, mm: change hooks signatures > kasan: move common generic and tag-based code to common.c > kasan: rename source files to reflect the new naming scheme > kasan: add CONFIG_KASAN_GENERIC and CONFIG_KASAN_SW_TAGS > kasan, arm64: adjust shadow size for tag-based mode > kasan: initialize shadow to 0xff for tag-based mode > kasan, arm64: untag address in __kimg_to_phys and _virt_addr_is_linear > kasan: add tag related helper functions > kasan: preassign tags to objects with ctors or SLAB_TYPESAFE_BY_RCU > mm: move obj_to_index to include/linux/slab_def.h > kasan, arm64: fix up fault handling logic > kasan, arm64: enable top byte ignore for the kernel > kasan, mm: perform untagged pointers comparison in krealloc > kasan: split out generic_report.c from report.c > kasan: add bug reporting routines for tag-based mode > kasan: add hooks implementation for tag-based mode > kasan, arm64: add brk handler for inline instrumentation > kasan, mm, arm64: tag non slab memory allocated via pagealloc > kasan: update documentation > kasan: add SPDX-License-Identifier mark to source files > > Documentation/dev-tools/kasan.rst | 232 +++++---- > arch/arm64/Kconfig | 1 + > arch/arm64/Makefile | 2 +- > arch/arm64/include/asm/brk-imm.h | 2 + > arch/arm64/include/asm/memory.h | 36 +- > arch/arm64/include/asm/pgtable-hwdef.h | 1 + > arch/arm64/kernel/traps.c | 68 ++- > arch/arm64/mm/fault.c | 3 + > arch/arm64/mm/kasan_init.c | 18 +- > arch/arm64/mm/proc.S | 8 +- > include/linux/compiler-clang.h | 5 +- > include/linux/kasan.h | 83 +++- > include/linux/mm.h | 29 ++ > include/linux/page-flags-layout.h | 10 + > include/linux/slab_def.h | 13 + > lib/Kconfig.kasan | 87 +++- > mm/cma.c | 11 + > mm/kasan/Makefile | 15 +- > mm/kasan/{kasan.c => common.c} | 651 +++++++++---------------- > mm/kasan/generic.c | 344 +++++++++++++ > mm/kasan/generic_report.c | 153 ++++++ > mm/kasan/{kasan_init.c => init.c} | 1 + > mm/kasan/kasan.h | 83 +++- > mm/kasan/quarantine.c | 1 + > mm/kasan/report.c | 272 +++-------- > mm/kasan/tags.c | 161 ++++++ > mm/kasan/tags_report.c | 58 +++ > mm/page_alloc.c | 1 + > mm/slab.c | 29 +- > mm/slab.h | 2 +- > mm/slab_common.c | 6 +- > mm/slub.c | 41 +- > scripts/Makefile.kasan | 27 +- > 33 files changed, 1629 insertions(+), 825 deletions(-) > rename mm/kasan/{kasan.c => common.c} (59%) > create mode 100644 mm/kasan/generic.c > create mode 100644 mm/kasan/generic_report.c > rename mm/kasan/{kasan_init.c => init.c} (99%) > create mode 100644 mm/kasan/tags.c > create mode 100644 mm/kasan/tags_report.c > > -- > 2.19.0.444.g18242da7ef-goog >
On Fri, Sep 21, 2018 at 5:13 PM, Andrey Konovalov <andreyknvl@google.com> wrote: > This patch updates KASAN documentation to reflect the addition of the new > tag-based mode. > > Signed-off-by: Andrey Konovalov <andreyknvl@google.com> > --- > Documentation/dev-tools/kasan.rst | 232 ++++++++++++++++++------------ > 1 file changed, 138 insertions(+), 94 deletions(-) > > diff --git a/Documentation/dev-tools/kasan.rst b/Documentation/dev-tools/kasan.rst > index aabc8738b3d8..a407e18afd32 100644 > --- a/Documentation/dev-tools/kasan.rst > +++ b/Documentation/dev-tools/kasan.rst > @@ -4,15 +4,25 @@ The Kernel Address Sanitizer (KASAN) > Overview > -------- > > -KernelAddressSANitizer (KASAN) is a dynamic memory error detector. It provides > -a fast and comprehensive solution for finding use-after-free and out-of-bounds > -bugs. > +KernelAddressSANitizer (KASAN) is a dynamic memory error detector designed to > +find out-of-bound and use-after-free bugs. KASAN has two modes: generic KASAN > +(similar to userspace ASan) and software tag-based KASAN (similar to userspace > +HWASan). > > -KASAN uses compile-time instrumentation for checking every memory access, > -therefore you will need a GCC version 4.9.2 or later. GCC 5.0 or later is > -required for detection of out-of-bounds accesses to stack or global variables. > +KASAN uses compile-time instrumentation to insert validity checks before every > +memory access, and therefore requires a compiler version that supports that. > > -Currently KASAN is supported only for the x86_64 and arm64 architectures. > +Generic KASAN is supported in both GCC and Clang. With GCC it requires version > +4.9.2 or later for basic support and version 5.0 or later for detection of > +out-of-bounds accesses for stack and global variables and for inline > +instrumentation mode (see the Usage section). With Clang it requires version > +3.7.0 or later and it doesn't support detection of out-of-bounds accesses for Note: this should actually be 7.0.0 and not 3.7.0 (as we need rL329612). > +global variables yet. > + > +Tag-based KASAN is only supported in Clang and requires version 7.0.0 or later. > + > +Currently generic KASAN is supported for the x86_64, arm64 and xtensa > +architectures, and tag-based KASAN is supported only for arm64. > > Usage > ----- > @@ -21,12 +31,14 @@ To enable KASAN configure kernel with:: > > CONFIG_KASAN = y > > -and choose between CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE. Outline and > -inline are compiler instrumentation types. The former produces smaller binary > -the latter is 1.1 - 2 times faster. Inline instrumentation requires a GCC > -version 5.0 or later. > +and choose between CONFIG_KASAN_GENERIC (to enable generic KASAN) and > +CONFIG_KASAN_SW_TAGS (to enable software tag-based KASAN). > > -KASAN works with both SLUB and SLAB memory allocators. > +You also need to choose between CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE. > +Outline and inline are compiler instrumentation types. The former produces > +smaller binary while the latter is 1.1 - 2 times faster. > + > +Both KASAN modes work with both SLUB and SLAB memory allocators. > For better bug detection and nicer reporting, enable CONFIG_STACKTRACE. > > To disable instrumentation for specific files or directories, add a line > @@ -43,85 +55,85 @@ similar to the following to the respective kernel Makefile: > Error reports > ~~~~~~~~~~~~~ > > -A typical out of bounds access report looks like this:: > +A typical out-of-bounds access generic KASAN report looks like this:: > > ================================================================== > - BUG: AddressSanitizer: out of bounds access in kmalloc_oob_right+0x65/0x75 [test_kasan] at addr ffff8800693bc5d3 > - Write of size 1 by task modprobe/1689 > - ============================================================================= > - BUG kmalloc-128 (Not tainted): kasan error > - ----------------------------------------------------------------------------- > - > - Disabling lock debugging due to kernel taint > - INFO: Allocated in kmalloc_oob_right+0x3d/0x75 [test_kasan] age=0 cpu=0 pid=1689 > - __slab_alloc+0x4b4/0x4f0 > - kmem_cache_alloc_trace+0x10b/0x190 > - kmalloc_oob_right+0x3d/0x75 [test_kasan] > - init_module+0x9/0x47 [test_kasan] > - do_one_initcall+0x99/0x200 > - load_module+0x2cb3/0x3b20 > - SyS_finit_module+0x76/0x80 > - system_call_fastpath+0x12/0x17 > - INFO: Slab 0xffffea0001a4ef00 objects=17 used=7 fp=0xffff8800693bd728 flags=0x100000000004080 > - INFO: Object 0xffff8800693bc558 @offset=1368 fp=0xffff8800693bc720 > - > - Bytes b4 ffff8800693bc548: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ > - Object ffff8800693bc558: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > - Object ffff8800693bc568: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > - Object ffff8800693bc578: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > - Object ffff8800693bc588: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > - Object ffff8800693bc598: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > - Object ffff8800693bc5a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > - Object ffff8800693bc5b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk > - Object ffff8800693bc5c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk. > - Redzone ffff8800693bc5d8: cc cc cc cc cc cc cc cc ........ > - Padding ffff8800693bc718: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ > - CPU: 0 PID: 1689 Comm: modprobe Tainted: G B 3.18.0-rc1-mm1+ #98 > - Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014 > - ffff8800693bc000 0000000000000000 ffff8800693bc558 ffff88006923bb78 > - ffffffff81cc68ae 00000000000000f3 ffff88006d407600 ffff88006923bba8 > - ffffffff811fd848 ffff88006d407600 ffffea0001a4ef00 ffff8800693bc558 > + BUG: KASAN: slab-out-of-bounds in kmalloc_oob_right+0xa8/0xbc [test_kasan] > + Write of size 1 at addr ffff8801f44ec37b by task insmod/2760 > + > + CPU: 1 PID: 2760 Comm: insmod Not tainted 4.19.0-rc3+ #698 > + Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014 > Call Trace: > - [<ffffffff81cc68ae>] dump_stack+0x46/0x58 > - [<ffffffff811fd848>] print_trailer+0xf8/0x160 > - [<ffffffffa00026a7>] ? kmem_cache_oob+0xc3/0xc3 [test_kasan] > - [<ffffffff811ff0f5>] object_err+0x35/0x40 > - [<ffffffffa0002065>] ? kmalloc_oob_right+0x65/0x75 [test_kasan] > - [<ffffffff8120b9fa>] kasan_report_error+0x38a/0x3f0 > - [<ffffffff8120a79f>] ? kasan_poison_shadow+0x2f/0x40 > - [<ffffffff8120b344>] ? kasan_unpoison_shadow+0x14/0x40 > - [<ffffffff8120a79f>] ? kasan_poison_shadow+0x2f/0x40 > - [<ffffffffa00026a7>] ? kmem_cache_oob+0xc3/0xc3 [test_kasan] > - [<ffffffff8120a995>] __asan_store1+0x75/0xb0 > - [<ffffffffa0002601>] ? kmem_cache_oob+0x1d/0xc3 [test_kasan] > - [<ffffffffa0002065>] ? kmalloc_oob_right+0x65/0x75 [test_kasan] > - [<ffffffffa0002065>] kmalloc_oob_right+0x65/0x75 [test_kasan] > - [<ffffffffa00026b0>] init_module+0x9/0x47 [test_kasan] > - [<ffffffff810002d9>] do_one_initcall+0x99/0x200 > - [<ffffffff811e4e5c>] ? __vunmap+0xec/0x160 > - [<ffffffff81114f63>] load_module+0x2cb3/0x3b20 > - [<ffffffff8110fd70>] ? m_show+0x240/0x240 > - [<ffffffff81115f06>] SyS_finit_module+0x76/0x80 > - [<ffffffff81cd3129>] system_call_fastpath+0x12/0x17 > + dump_stack+0x94/0xd8 > + print_address_description+0x73/0x280 > + kasan_report+0x144/0x187 > + __asan_report_store1_noabort+0x17/0x20 > + kmalloc_oob_right+0xa8/0xbc [test_kasan] > + kmalloc_tests_init+0x16/0x700 [test_kasan] > + do_one_initcall+0xa5/0x3ae > + do_init_module+0x1b6/0x547 > + load_module+0x75df/0x8070 > + __do_sys_init_module+0x1c6/0x200 > + __x64_sys_init_module+0x6e/0xb0 > + do_syscall_64+0x9f/0x2c0 > + entry_SYSCALL_64_after_hwframe+0x44/0xa9 > + RIP: 0033:0x7f96443109da > + RSP: 002b:00007ffcf0b51b08 EFLAGS: 00000202 ORIG_RAX: 00000000000000af > + RAX: ffffffffffffffda RBX: 000055dc3ee521a0 RCX: 00007f96443109da > + RDX: 00007f96445cff88 RSI: 0000000000057a50 RDI: 00007f9644992000 > + RBP: 000055dc3ee510b0 R08: 0000000000000003 R09: 0000000000000000 > + R10: 00007f964430cd0a R11: 0000000000000202 R12: 00007f96445cff88 > + R13: 000055dc3ee51090 R14: 0000000000000000 R15: 0000000000000000 > + > + Allocated by task 2760: > + save_stack+0x43/0xd0 > + kasan_kmalloc+0xa7/0xd0 > + kmem_cache_alloc_trace+0xe1/0x1b0 > + kmalloc_oob_right+0x56/0xbc [test_kasan] > + kmalloc_tests_init+0x16/0x700 [test_kasan] > + do_one_initcall+0xa5/0x3ae > + do_init_module+0x1b6/0x547 > + load_module+0x75df/0x8070 > + __do_sys_init_module+0x1c6/0x200 > + __x64_sys_init_module+0x6e/0xb0 > + do_syscall_64+0x9f/0x2c0 > + entry_SYSCALL_64_after_hwframe+0x44/0xa9 > + > + Freed by task 815: > + save_stack+0x43/0xd0 > + __kasan_slab_free+0x135/0x190 > + kasan_slab_free+0xe/0x10 > + kfree+0x93/0x1a0 > + umh_complete+0x6a/0xa0 > + call_usermodehelper_exec_async+0x4c3/0x640 > + ret_from_fork+0x35/0x40 > + > + The buggy address belongs to the object at ffff8801f44ec300 > + which belongs to the cache kmalloc-128 of size 128 > + The buggy address is located 123 bytes inside of > + 128-byte region [ffff8801f44ec300, ffff8801f44ec380) > + The buggy address belongs to the page: > + page:ffffea0007d13b00 count:1 mapcount:0 mapping:ffff8801f7001640 index:0x0 > + flags: 0x200000000000100(slab) > + raw: 0200000000000100 ffffea0007d11dc0 0000001a0000001a ffff8801f7001640 > + raw: 0000000000000000 0000000080150015 00000001ffffffff 0000000000000000 > + page dumped because: kasan: bad access detected > + > Memory state around the buggy address: > - ffff8800693bc300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > - ffff8800693bc380: fc fc 00 00 00 00 00 00 00 00 00 00 00 00 00 fc > - ffff8800693bc400: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > - ffff8800693bc480: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > - ffff8800693bc500: fc fc fc fc fc fc fc fc fc fc fc 00 00 00 00 00 > - >ffff8800693bc580: 00 00 00 00 00 00 00 00 00 00 03 fc fc fc fc fc > - ^ > - ffff8800693bc600: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > - ffff8800693bc680: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc > - ffff8800693bc700: fc fc fc fc fb fb fb fb fb fb fb fb fb fb fb fb > - ffff8800693bc780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > - ffff8800693bc800: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb > + ffff8801f44ec200: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb > + ffff8801f44ec280: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc > + >ffff8801f44ec300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 > + ^ > + ffff8801f44ec380: fc fc fc fc fc fc fc fc fb fb fb fb fb fb fb fb > + ffff8801f44ec400: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc > ================================================================== > > -The header of the report discribe what kind of bug happened and what kind of > -access caused it. It's followed by the description of the accessed slub object > -(see 'SLUB Debug output' section in Documentation/vm/slub.rst for details) and > -the description of the accessed memory page. > +The header of the report provides a short summary of what kind of bug happened > +and what kind of access caused it. It's followed by a stack trace of the bad > +access, a stack trace of where the accessed memory was allocated (in case bad > +access happens on a slab object), and a stack trace of where the object was > +freed (in case of a use-after-free bug report). Next comes a description of > +the accessed slab object and information about the accessed memory page. > > In the last section the report shows memory state around the accessed address. > Reading this part requires some understanding of how KASAN works. > @@ -138,18 +150,24 @@ inaccessible memory like redzones or freed memory (see mm/kasan/kasan.h). > In the report above the arrows point to the shadow byte 03, which means that > the accessed address is partially accessible. > > +For tag-based KASAN this last report section shows the memory tags around the > +accessed address (see Implementation details section). > + > > Implementation details > ---------------------- > > +Generic KASAN > +~~~~~~~~~~~~~ > + > From a high level, our approach to memory error detection is similar to that > of kmemcheck: use shadow memory to record whether each byte of memory is safe > -to access, and use compile-time instrumentation to check shadow memory on each > -memory access. > +to access, and use compile-time instrumentation to insert checks of shadow > +memory on each memory access. > > -AddressSanitizer dedicates 1/8 of kernel memory to its shadow memory > -(e.g. 16TB to cover 128TB on x86_64) and uses direct mapping with a scale and > -offset to translate a memory address to its corresponding shadow address. > +Generic KASAN dedicates 1/8th of kernel memory to its shadow memory (e.g. 16TB > +to cover 128TB on x86_64) and uses direct mapping with a scale and offset to > +translate a memory address to its corresponding shadow address. > > Here is the function which translates an address to its corresponding shadow > address:: > @@ -162,12 +180,38 @@ address:: > > where ``KASAN_SHADOW_SCALE_SHIFT = 3``. > > -Compile-time instrumentation used for checking memory accesses. Compiler inserts > -function calls (__asan_load*(addr), __asan_store*(addr)) before each memory > -access of size 1, 2, 4, 8 or 16. These functions check whether memory access is > -valid or not by checking corresponding shadow memory. > +Compile-time instrumentation is used to insert memory access checks. Compiler > +inserts function calls (__asan_load*(addr), __asan_store*(addr)) before each > +memory access of size 1, 2, 4, 8 or 16. These functions check whether memory > +access is valid or not by checking corresponding shadow memory. > > GCC 5.0 has possibility to perform inline instrumentation. Instead of making > function calls GCC directly inserts the code to check the shadow memory. > This option significantly enlarges kernel but it gives x1.1-x2 performance > boost over outline instrumented kernel. > + > +Software tag-based KASAN > +~~~~~~~~~~~~~~~~~~~~~~~~ > + > +Tag-based KASAN uses the Top Byte Ignore (TBI) feature of modern arm64 CPUs to > +store a pointer tag in the top byte of kernel pointers. Like generic KASAN it > +uses shadow memory to store memory tags associated with each 16-byte memory > +cell (therefore it dedicates 1/16th of the kernel memory for shadow memory). > + > +On each memory allocation tag-based KASAN generates a random tag, tags the > +allocated memory with this tag, and embeds this tag into the returned pointer. > +Software tag-based KASAN uses compile-time instrumentation to insert checks > +before each memory access. These checks make sure that tag of the memory that > +is being accessed is equal to tag of the pointer that is used to access this > +memory. In case of a tag mismatch tag-based KASAN prints a bug report. > + > +Software tag-based KASAN also has two instrumentation modes (outline, that > +emits callbacks to check memory accesses; and inline, that performs the shadow > +memory checks inline). With outline instrumentation mode, a bug report is > +simply printed from the function that performs the access check. With inline > +instrumentation a brk instruction is emitted by the compiler, and a dedicated > +brk handler is used to print bug reports. > + > +A potential expansion of this mode is a hardware tag-based mode, which would > +use hardware memory tagging support instead of compiler instrumentation and > +manual shadow memory manipulation. > -- > 2.19.0.444.g18242da7ef-goog >
On 09/21/2018 06:13 PM, Andrey Konovalov wrote: > Tag-based KASAN changes the value of the top byte of pointers returned > from the kernel allocation functions (such as kmalloc). This patch updates > KASAN hooks signatures and their usage in SLAB and SLUB code to reflect > that. > > Signed-off-by: Andrey Konovalov <andreyknvl@google.com> > --- > include/linux/kasan.h | 43 +++++++++++++++++++++++++++++-------------- > mm/kasan/kasan.c | 30 ++++++++++++++++++------------ > mm/slab.c | 12 ++++++------ > mm/slab.h | 2 +- > mm/slab_common.c | 4 ++-- > mm/slub.c | 15 +++++++-------- > 6 files changed, 63 insertions(+), 43 deletions(-) > > diff --git a/include/linux/kasan.h b/include/linux/kasan.h > index 46aae129917c..52c86a568a4e 100644 > --- a/include/linux/kasan.h > +++ b/include/linux/kasan.h > @@ -51,16 +51,16 @@ void kasan_cache_shutdown(struct kmem_cache *cache); > void kasan_poison_slab(struct page *page); > void kasan_unpoison_object_data(struct kmem_cache *cache, void *object); > void kasan_poison_object_data(struct kmem_cache *cache, void *object); > -void kasan_init_slab_obj(struct kmem_cache *cache, const void *object); > +void *kasan_init_slab_obj(struct kmem_cache *cache, const void *object); > > -void kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags); > +void *kasan_kmalloc_large(const void *ptr, size_t size, gfp_t flags); > void kasan_kfree_large(void *ptr, unsigned long ip); > void kasan_poison_kfree(void *ptr, unsigned long ip); > -void kasan_kmalloc(struct kmem_cache *s, const void *object, size_t size, > +void *kasan_kmalloc(struct kmem_cache *s, const void *object, size_t size, > gfp_t flags); This patch missed couple call-sites, in kmem_cache_alloc_trace() and kmem_cache_alloc_node_trace() in include/linux/slab.h, the return value of kasan_kmalloc() is ignored. Probably worth adding __must_check to functions like this. Once that fixed you can add Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> here and to the rest of the patches. They look fine me.
This patchset adds a new software tag-based mode to KASAN [1]. (Initially this mode was called KHWASAN, but it got renamed, see the naming rationale at the end of this section). The plan is to implement HWASan [2] for the kernel with the incentive, that it's going to have comparable to KASAN performance, but in the same time consume much less memory, trading that off for somewhat imprecise bug detection and being supported only for arm64. The underlying ideas of the approach used by software tag-based KASAN are: 1. By using the Top Byte Ignore (TBI) arm64 CPU feature, we can store pointer tags in the top byte of each kernel pointer. 2. Using shadow memory, we can store memory tags for each chunk of kernel memory. 3. On each memory allocation, we can generate a random tag, embed it into the returned pointer and set the memory tags that correspond to this chunk of memory to the same value. 4. By using compiler instrumentation, before each memory access we can add a check that the pointer tag matches the tag of the memory that is being accessed. 5. On a tag mismatch we report an error. With this patchset the existing KASAN mode gets renamed to generic KASAN, with the word "generic" meaning that the implementation can be supported by any architecture as it is purely software. The new mode this patchset adds is called software tag-based KASAN. The word "tag-based" refers to the fact that this mode uses tags embedded into the top byte of kernel pointers and the TBI arm64 CPU feature that allows to dereference such pointers. The word "software" here means that shadow memory manipulation and tag checking on pointer dereference is done in software. As it is the only tag-based implementation right now, "software tag-based" KASAN is sometimes referred to as simply "tag-based" in this patchset. A potential expansion of this mode is a hardware tag-based mode, which would use hardware memory tagging support (announced by Arm [3]) instead of compiler instrumentation and manual shadow memory manipulation. Same as generic KASAN, software tag-based KASAN is strictly a debugging feature. [1] https://www.kernel.org/doc/html/latest/dev-tools/kasan.html [2] http://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html [3] https://community.arm.com/processors/b/blog/posts/arm-a-profile-architecture-2018-developments-armv85a ====== Rationale On mobile devices generic KASAN's memory usage is significant problem. One of the main reasons to have tag-based KASAN is to be able to perform a similar set of checks as the generic one does, but with lower memory requirements. Comment from Vishwath Mohan <vishwath@google.com>: I don't have data on-hand, but anecdotally both ASAN and KASAN have proven problematic to enable for environments that don't tolerate the increased memory pressure well. This includes, (a) Low-memory form factors - Wear, TV, Things, lower-tier phones like Go, (c) Connected components like Pixel's visual core [1]. These are both places I'd love to have a low(er) memory footprint option at my disposal. Comment from Evgenii Stepanov <eugenis@google.com>: Looking at a live Android device under load, slab (according to /proc/meminfo) + kernel stack take 8-10% available RAM (~350MB). KASAN's overhead of 2x - 3x on top of it is not insignificant. Not having this overhead enables near-production use - ex. running KASAN/KHWASAN kernel on a personal, daily-use device to catch bugs that do not reproduce in test configuration. These are the ones that often cost the most engineering time to track down. CPU overhead is bad, but generally tolerable. RAM is critical, in our experience. Once it gets low enough, OOM-killer makes your life miserable. [1] https://www.blog.google/products/pixel/pixel-visual-core-image-processing-and-machine-learning-pixel-2/ ====== Technical details Software tag-based KASAN mode is implemented in a very similar way to the generic one. This patchset essentially does the following: 1. TCR_TBI1 is set to enable Top Byte Ignore. 2. Shadow memory is used (with a different scale, 1:16, so each shadow byte corresponds to 16 bytes of kernel memory) to store memory tags. 3. All slab objects are aligned to shadow scale, which is 16 bytes. 4. All pointers returned from the slab allocator are tagged with a random tag and the corresponding shadow memory is poisoned with the same value. 5. Compiler instrumentation is used to insert tag checks. Either by calling callbacks or by inlining them (CONFIG_KASAN_OUTLINE and CONFIG_KASAN_INLINE flags are reused). 6. When a tag mismatch is detected in callback instrumentation mode KASAN simply prints a bug report. In case of inline instrumentation, clang inserts a brk instruction, and KASAN has it's own brk handler, which reports the bug. 7. The memory in between slab objects is marked with a reserved tag, and acts as a redzone. 8. When a slab object is freed it's marked with a reserved tag. Bug detection is imprecise for two reasons: 1. We won't catch some small out-of-bounds accesses, that fall into the same shadow cell, as the last byte of a slab object. 2. We only have 1 byte to store tags, which means we have a 1/256 probability of a tag match for an incorrect access (actually even slightly less due to reserved tag values). Despite that there's a particular type of bugs that tag-based KASAN can detect compared to generic KASAN: use-after-free after the object has been allocated by someone else. ====== Testing Some kernel developers voiced a concern that changing the top byte of kernel pointers may lead to subtle bugs that are difficult to discover. To address this concern deliberate testing has been performed. It doesn't seem feasible to do some kind of static checking to find potential issues with pointer tagging, so a dynamic approach was taken. All pointer comparisons/subtractions have been instrumented in an LLVM compiler pass and a kernel module that would print a bug report whenever two pointers with different tags are being compared/subtracted (ignoring comparisons with NULL pointers and with pointers obtained by casting an error code to a pointer type) has been used. Then the kernel has been booted in QEMU and on an Odroid C2 board and syzkaller has been run. This yielded the following results. The two places that look interesting are: is_vmalloc_addr in include/linux/mm.h is_kernel_rodata in mm/util.c Here we compare a pointer with some fixed untagged values to make sure that the pointer lies in a particular part of the kernel address space. Since tag-based KASAN doesn't add tags to pointers that belong to rodata or vmalloc regions, this should work as is. To make sure debug checks to those two functions that check that the result doesn't change whether we operate on pointers with or without untagging has been added. A few other cases that don't look that interesting: Comparing pointers to achieve unique sorting order of pointee objects (e.g. sorting locks addresses before performing a double lock): tty_ldisc_lock_pair_timeout in drivers/tty/tty_ldisc.c pipe_double_lock in fs/pipe.c unix_state_double_lock in net/unix/af_unix.c lock_two_nondirectories in fs/inode.c mutex_lock_double in kernel/events/core.c ep_cmp_ffd in fs/eventpoll.c fsnotify_compare_groups fs/notify/mark.c Nothing needs to be done here, since the tags embedded into pointers don't change, so the sorting order would still be unique. Checks that a pointer belongs to some particular allocation: is_sibling_entry in lib/radix-tree.c object_is_on_stack in include/linux/sched/task_stack.h Nothing needs to be done here either, since two pointers can only belong to the same allocation if they have the same tag. Overall, since the kernel boots and works, there are no critical bugs. As for the rest, the traditional kernel testing way (use until fails) is the only one that looks feasible. Another point here is that tag-based KASAN is available under a separate config option that needs to be deliberately enabled. Even though it might be used in a "near-production" environment to find bugs that are not found during fuzzing or running tests, it is still a debug tool. ====== Benchmarks The following numbers were collected on Odroid C2 board. Both generic and tag-based KASAN were used in inline instrumentation mode. Boot time [1]: * ~1.7 sec for clean kernel * ~5.0 sec for generic KASAN * ~5.0 sec for tag-based KASAN Network performance [2]: * 8.33 Gbits/sec for clean kernel * 3.17 Gbits/sec for generic KASAN * 2.85 Gbits/sec for tag-based KASAN Slab memory usage after boot [3]: * ~40 kb for clean kernel * ~105 kb (~260% overhead) for generic KASAN * ~47 kb (~20% overhead) for tag-based KASAN KASAN memory overhead consists of three main parts: 1. Increased slab memory usage due to redzones. 2. Shadow memory (the whole reserved once during boot). 3. Quaratine (grows gradually until some preset limit; the more the limit, the more the chance to detect a use-after-free). Comparing tag-based vs generic KASAN for each of these points: 1. 20% vs 260% overhead. 2. 1/16th vs 1/8th of physical memory. 3. Tag-based KASAN doesn't require quarantine. [1] Time before the ext4 driver is initialized. [2] Measured as `iperf -s & iperf -c 127.0.0.1 -t 30`. [3] Measured as `cat /proc/meminfo | grep Slab`. ====== Some notes A few notes: 1. The patchset can be found here: https://github.com/xairy/kasan-prototype/tree/khwasan 2. Building requires a recent Clang version (7.0.0 or later). 3. Stack instrumentation is not supported yet and will be added later. ====== Changes Changes in v9: - Fixed kasan_init_slab_obj() hook when KASAN is disabled. - Added assign_tag() function that preassigns tags for caches with constructors. - Fixed KASAN_TAG_MASK redefinition in include/linux/mm.h vs mm/kasan/kasan.h. Changes in v8: - Rebased onto 7876320f (4.19-rc4). - Renamed KHWASAN to software tag-based KASAN (see the top of the cover letter for details). - Explicitly called tag-based KASAN a debug tool. - Reused kasan_init_slab_obj() callback to preassign tags to caches without constructors, remove khwasan_preset_sl(u/a)b_tag(). - Moved move obj_to_index to include/linux/slab_def.h from mm/slab.c. - Moved cache->s_mem untagging to alloc_slabmgmt() for SLAB. - Fixed check_memory_region() to correctly handle user memory accesses and size == 0 case. - Merged __no_sanitize_hwaddress into __no_sanitize_address. - Defined KASAN_SET_TAG and KASAN_RESET_TAG macros for non KASAN builds to avoid duplication of __kimg_to_phys, _virt_addr_is_linear and page_to_virt macros. - Fixed and simplified find_first_bad_addr for generic KASAN. - Use non symbolized example KASAN report in documentation. - Mention clang version requirements for both KASAN modes in the Kconfig options and in the documentation. - Various small fixes. Version v7 got accidentally skipped. Changes in v6: - Rebased onto 050cdc6c (4.19-rc1+). - Added notes regarding patchset testing into the cover letter. Changes in v5: - Rebased onto 1ffaddd029 (4.18-rc8). - Preassign tags for objects from caches with constructors and SLAB_TYPESAFE_BY_RCU caches. - Fix SLAB allocator support by untagging page->s_mem in kasan_poison_slab(). - Performed dynamic testing to find potential places where pointer tagging might result in bugs [1]. - Clarified and fixed memory usage benchmarks in the cover letter. - Added a rationale for having KHWASAN to the cover letter. Changes in v4: - Fixed SPDX comment style in mm/kasan/kasan.h. - Fixed mm/kasan/kasan.h changes being included in a wrong patch. - Swapped "khwasan, arm64: fix up fault handling logic" and "khwasan: add tag related helper functions" patches order. - Rebased onto 6f0d349d (4.18-rc2+). Changes in v3: - Minor documentation fixes. - Fixed CFLAGS variable name in KASAN makefile. - Added a "SPDX-License-Identifier: GPL-2.0" line to all source files under mm/kasan. - Rebased onto 81e97f013 (4.18-rc1+). Changes in v2: - Changed kmalloc_large_node_hook to return tagged pointer instead of using an output argument. - Fix checking whether -fsanitize=hwaddress is supported by the compiler. - Removed duplication of -fno-builtin for KASAN and KHWASAN. - Removed {} block for one line for_each_possible_cpu loop. - Made set_track() static inline as it is used only in common.c. - Moved optimal_redzone() to common.c. - Fixed using tagged pointer for shadow calculation in kasan_unpoison_shadow(). - Restored setting cache->align in kasan_cache_create(), which was accidentally lost. - Simplified __kasan_slab_free(), kasan_alloc_pages() and kasan_kmalloc(). - Removed tagging from kasan_kmalloc_large(). - Added page_kasan_tag_reset() to kasan_poison_slab() and removed !PageSlab() check from page_to_virt. - Reset pointer tag in _virt_addr_is_linear. - Set page tag for each page when multiple pages are allocated or freed. - Added a comment as to why we ignore cma allocated pages. Changes in v1: - Rebased onto 4.17-rc4. - Updated benchmarking stats. - Documented compiler version requirements, memory usage and slowdown. - Dropped kvm patches, as clang + arm64 + kvm is completely broken [1]. Changes in RFC v3: - Renamed CONFIG_KASAN_CLASSIC and CONFIG_KASAN_TAGS to CONFIG_KASAN_GENERIC and CONFIG_KASAN_HW respectively. - Switch to -fsanitize=kernel-hwaddress instead of -fsanitize=hwaddress. - Removed unnecessary excessive shadow initialization. - Removed khwasan_enabled flag (it’s not needed since KHWASAN is initialized before any slab caches are used). - Split out kasan_report.c and khwasan_report.c from report.c. - Moved more common KASAN and KHWASAN functions to common.c. - Added tagging to pagealloc. - Rebased onto 4.17-rc1. - Temporarily dropped patch that adds kvm support (arm64 + kvm + clang combo is broken right now [2]). Changes in RFC v2: - Removed explicit casts to u8 * for kasan_mem_to_shadow() calls. - Introduced KASAN_TCR_FLAGS for setting the TCR_TBI1 flag. - Added a comment regarding the non-atomic RMW sequence in khwasan_random_tag(). - Made all tag related functions accept const void *. - Untagged pointers in __kimg_to_phys, which is used by virt_to_phys. - Untagged pointers in show_ptr in fault handling logic. - Untagged pointers passed to KVM. - Added two reserved tag values: 0xFF and 0xFE. - Used the reserved tag 0xFF to disable validity checking (to resolve the issue with pointer tag being lost after page_address + kmap usage). - Used the reserved tag 0xFE to mark redzones and freed objects. - Added mnemonics for esr manipulation in KHWASAN brk handler. - Added a comment about the -recover flag. - Some minor cleanups and fixes. - Rebased onto 3215b9d5 (4.16-rc6+). - Tested on real hardware (Odroid C2 board). - Added better benchmarks. [1] https://lkml.org/lkml/2018/7/18/765 [2] https://lkml.org/lkml/2018/4/19/775 Andrey Konovalov (20): kasan, mm: change hooks signatures kasan: move common generic and tag-based code to common.c kasan: rename source files to reflect the new naming scheme kasan: add CONFIG_KASAN_GENERIC and CONFIG_KASAN_SW_TAGS kasan, arm64: adjust shadow size for tag-based mode kasan: initialize shadow to 0xff for tag-based mode kasan, arm64: untag address in __kimg_to_phys and _virt_addr_is_linear kasan: add tag related helper functions kasan: preassign tags to objects with ctors or SLAB_TYPESAFE_BY_RCU mm: move obj_to_index to include/linux/slab_def.h kasan, arm64: fix up fault handling logic kasan, arm64: enable top byte ignore for the kernel kasan, mm: perform untagged pointers comparison in krealloc kasan: split out generic_report.c from report.c kasan: add bug reporting routines for tag-based mode kasan: add hooks implementation for tag-based mode kasan, arm64: add brk handler for inline instrumentation kasan, mm, arm64: tag non slab memory allocated via pagealloc kasan: update documentation kasan: add SPDX-License-Identifier mark to source files Documentation/dev-tools/kasan.rst | 232 +++++---- arch/arm64/Kconfig | 1 + arch/arm64/Makefile | 2 +- arch/arm64/include/asm/brk-imm.h | 2 + arch/arm64/include/asm/memory.h | 36 +- arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/kernel/traps.c | 68 ++- arch/arm64/mm/fault.c | 3 + arch/arm64/mm/kasan_init.c | 18 +- arch/arm64/mm/proc.S | 8 +- include/linux/compiler-clang.h | 5 +- include/linux/kasan.h | 83 +++- include/linux/mm.h | 29 ++ include/linux/page-flags-layout.h | 10 + include/linux/slab_def.h | 13 + lib/Kconfig.kasan | 87 +++- mm/cma.c | 11 + mm/kasan/Makefile | 15 +- mm/kasan/{kasan.c => common.c} | 651 +++++++++---------------- mm/kasan/generic.c | 344 +++++++++++++ mm/kasan/generic_report.c | 153 ++++++ mm/kasan/{kasan_init.c => init.c} | 1 + mm/kasan/kasan.h | 83 +++- mm/kasan/quarantine.c | 1 + mm/kasan/report.c | 272 +++-------- mm/kasan/tags.c | 161 ++++++ mm/kasan/tags_report.c | 58 +++ mm/page_alloc.c | 1 + mm/slab.c | 29 +- mm/slab.h | 2 +- mm/slab_common.c | 6 +- mm/slub.c | 41 +- scripts/Makefile.kasan | 27 +- 33 files changed, 1629 insertions(+), 825 deletions(-) rename mm/kasan/{kasan.c => common.c} (59%) create mode 100644 mm/kasan/generic.c create mode 100644 mm/kasan/generic_report.c rename mm/kasan/{kasan_init.c => init.c} (99%) create mode 100644 mm/kasan/tags.c create mode 100644 mm/kasan/tags_report.c