Patchwork [-v2,2/2] make the compaction "skip ahead" logic robust

login
register
mail settings
Submitter Rik van Riel
Date Sept. 13, 2012, 7:54 p.m.
Message ID <20120913155450.7634148f@cuia.bos.redhat.com>
Download mbox | patch
Permalink /patch/183706/
State New
Headers show

Comments

Rik van Riel - Sept. 13, 2012, 7:54 p.m.
Argh. And of course I send out the version from _before_ the compile test,
instead of the one after! I am not used to caffeine any more and have had
way too much tea...

---8<---

Make the "skip ahead" logic in compaction resistant to compaction
wrapping around to the end of the zone.  This can lead to less
efficient compaction when one thread has wrapped around to the
end of the zone, and another simultaneous compactor has not done
so yet. However, it should ensure that we do not suffer quadratic
behaviour any more.

Signed-off-by: Rik van Riel <riel@redhat.com>
Reported-by: Richard Davies <richard@daviesmail.org>
Richard Davies - Sept. 15, 2012, 3:55 p.m.
Hi Rik, Mel and Shaohua,

Thank you for your latest patches. I attach my latest perf report for a slow
boot with all of these applied.

Mel asked for timings of the slow boots. It's very hard to give anything
useful here! A normal boot would be a minute or so, and many are like that,
but the slowest that I have seen (on 3.5.x) was several hours. Basically, I
just test many times until I get one which is noticeably slow than normal
and then run perf record on that one.

The latest perf report for a slow boot is below. For the fast boots, most of
the time is in clean_page_c in do_huge_pmd_anonymous_page, but for this slow
one there is a lot of lock contention above that.

Thanks,

Richard.


# ========
# captured on: Sat Sep 15 15:40:54 2012
# os release : 3.6.0-rc5-elastic+
# perf version : 3.5.2
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : AMD Opteron(tm) Processor 6128
# cpuid : AuthenticAMD,16,9,1
# total memory : 131973280 kB
# cmdline : /home/root/bin/perf record -g -a 
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80 }
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# ========
#
# Samples: 3M of event 'cycles'
# Event count (approx.): 1457256240581
#
# Overhead          Command         Shared Object                                          Symbol
# ........  ...............  ....................  ..............................................
#
    58.49%         qemu-kvm  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave                    
                   |
                   --- _raw_spin_lock_irqsave
                      |          
                      |--95.07%-- compact_checklock_irqsave
                      |          |          
                      |          |--70.03%-- isolate_migratepages_range
                      |          |          compact_zone
                      |          |          compact_zone_order
                      |          |          try_to_compact_pages
                      |          |          __alloc_pages_direct_compact
                      |          |          __alloc_pages_nodemask
                      |          |          alloc_pages_vma
                      |          |          do_huge_pmd_anonymous_page
                      |          |          handle_mm_fault
                      |          |          __get_user_pages
                      |          |          get_user_page_nowait
                      |          |          hva_to_pfn.isra.17
                      |          |          __gfn_to_pfn
                      |          |          gfn_to_pfn_async
                      |          |          try_async_pf
                      |          |          tdp_page_fault
                      |          |          kvm_mmu_page_fault
                      |          |          pf_interception
                      |          |          handle_exit
                      |          |          kvm_arch_vcpu_ioctl_run
                      |          |          kvm_vcpu_ioctl
                      |          |          do_vfs_ioctl
                      |          |          sys_ioctl
                      |          |          system_call_fastpath
                      |          |          ioctl
                      |          |          |          
                      |          |          |--92.76%-- 0x10100000006
                      |          |          |          
                      |          |           --7.24%-- 0x10100000002
                      |          |          
                      |           --29.97%-- compaction_alloc
                      |                     migrate_pages
                      |                     compact_zone
                      |                     compact_zone_order
                      |                     try_to_compact_pages
                      |                     __alloc_pages_direct_compact
                      |                     __alloc_pages_nodemask
                      |                     alloc_pages_vma
                      |                     do_huge_pmd_anonymous_page
                      |                     handle_mm_fault
                      |                     __get_user_pages
                      |                     get_user_page_nowait
                      |                     hva_to_pfn.isra.17
                      |                     __gfn_to_pfn
                      |                     gfn_to_pfn_async
                      |                     try_async_pf
                      |                     tdp_page_fault
                      |                     kvm_mmu_page_fault
                      |                     pf_interception
                      |                     handle_exit
                      |                     kvm_arch_vcpu_ioctl_run
                      |                     kvm_vcpu_ioctl
                      |                     do_vfs_ioctl
                      |                     sys_ioctl
                      |                     system_call_fastpath
                      |                     ioctl
                      |                     |          
                      |                     |--90.69%-- 0x10100000006
                      |                     |          
                      |                      --9.31%-- 0x10100000002
                      |          
                      |--4.53%-- isolate_migratepages_range
                      |          compact_zone
                      |          compact_zone_order
                      |          try_to_compact_pages
                      |          __alloc_pages_direct_compact
                      |          __alloc_pages_nodemask
                      |          alloc_pages_vma
                      |          do_huge_pmd_anonymous_page
                      |          handle_mm_fault
                      |          __get_user_pages
                      |          get_user_page_nowait
                      |          hva_to_pfn.isra.17
                      |          __gfn_to_pfn
                      |          gfn_to_pfn_async
                      |          try_async_pf
                      |          tdp_page_fault
                      |          kvm_mmu_page_fault
                      |          pf_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--92.22%-- 0x10100000006
                      |          |          
                      |           --7.78%-- 0x10100000002
                       --0.40%-- [...]
    13.14%         qemu-kvm  [kernel.kallsyms]     [k] clear_page_c                              
                   |
                   --- clear_page_c
                      |          
                      |--99.38%-- do_huge_pmd_anonymous_page
                      |          handle_mm_fault
                      |          __get_user_pages
                      |          get_user_page_nowait
                      |          hva_to_pfn.isra.17
                      |          __gfn_to_pfn
                      |          gfn_to_pfn_async
                      |          try_async_pf
                      |          tdp_page_fault
                      |          kvm_mmu_page_fault
                      |          pf_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--51.86%-- 0x10100000006
                      |          |          
                      |          |--48.14%-- 0x10100000002
                      |           --0.01%-- [...]
                      |          
                       --0.62%-- __alloc_pages_nodemask
                                 |          
                                 |--76.27%-- alloc_pages_vma
                                 |          handle_pte_fault
                                 |          |          
                                 |          |--99.57%-- handle_mm_fault
                                 |          |          |          
                                 |          |          |--99.65%-- __get_user_pages
                                 |          |          |          get_user_page_nowait
                                 |          |          |          hva_to_pfn.isra.17
                                 |          |          |          __gfn_to_pfn
                                 |          |          |          gfn_to_pfn_async
                                 |          |          |          try_async_pf
                                 |          |          |          tdp_page_fault
                                 |          |          |          kvm_mmu_page_fault
                                 |          |          |          pf_interception
                                 |          |          |          handle_exit
                                 |          |          |          kvm_arch_vcpu_ioctl_run
                                 |          |          |          kvm_vcpu_ioctl
                                 |          |          |          do_vfs_ioctl
                                 |          |          |          sys_ioctl
                                 |          |          |          system_call_fastpath
                                 |          |          |          ioctl
                                 |          |          |          |          
                                 |          |          |          |--91.77%-- 0x10100000006
                                 |          |          |          |          
                                 |          |          |           --8.23%-- 0x10100000002
                                 |          |           --0.35%-- [...]
                                 |           --0.43%-- [...]
                                 |          
                                  --23.73%-- alloc_pages_current
                                            |          
                                            |--99.20%-- pte_alloc_one
                                            |          |          
                                            |          |--98.68%-- do_huge_pmd_anonymous_page
                                            |          |          handle_mm_fault
                                            |          |          __get_user_pages
                                            |          |          get_user_page_nowait
                                            |          |          hva_to_pfn.isra.17
                                            |          |          __gfn_to_pfn
                                            |          |          gfn_to_pfn_async
                                            |          |          try_async_pf
                                            |          |          tdp_page_fault
                                            |          |          kvm_mmu_page_fault
                                            |          |          pf_interception
                                            |          |          handle_exit
                                            |          |          kvm_arch_vcpu_ioctl_run
                                            |          |          kvm_vcpu_ioctl
                                            |          |          do_vfs_ioctl
                                            |          |          sys_ioctl
                                            |          |          system_call_fastpath
                                            |          |          ioctl
                                            |          |          |          
                                            |          |          |--58.61%-- 0x10100000002
                                            |          |          |          
                                            |          |           --41.39%-- 0x10100000006
                                            |          |          
                                            |           --1.32%-- __pte_alloc
                                            |                     do_huge_pmd_anonymous_page
                                            |                     handle_mm_fault
                                            |                     __get_user_pages
                                            |                     get_user_page_nowait
                                            |                     hva_to_pfn.isra.17
                                            |                     __gfn_to_pfn
                                            |                     gfn_to_pfn_async
                                            |                     try_async_pf
                                            |                     tdp_page_fault
                                            |                     kvm_mmu_page_fault
                                            |                     pf_interception
                                            |                     handle_exit
                                            |                     kvm_arch_vcpu_ioctl_run
                                            |                     kvm_vcpu_ioctl
                                            |                     do_vfs_ioctl
                                            |                     sys_ioctl
                                            |                     system_call_fastpath
                                            |                     ioctl
                                            |                     0x10100000006
                                            |          
                                            |--0.69%-- __vmalloc_node_range
                                            |          __vmalloc_node
                                            |          vzalloc
                                            |          __kvm_set_memory_region
                                            |          kvm_set_memory_region
                                            |          kvm_vm_ioctl_set_memory_region
                                            |          kvm_vm_ioctl
                                            |          do_vfs_ioctl
                                            |          sys_ioctl
                                            |          system_call_fastpath
                                            |          ioctl
                                             --0.12%-- [...]
     6.31%         qemu-kvm  [kernel.kallsyms]     [k] isolate_freepages_block                   
                   |
                   --- isolate_freepages_block
                      |          
                      |--99.98%-- compaction_alloc
                      |          migrate_pages
                      |          compact_zone
                      |          compact_zone_order
                      |          try_to_compact_pages
                      |          __alloc_pages_direct_compact
                      |          __alloc_pages_nodemask
                      |          alloc_pages_vma
                      |          do_huge_pmd_anonymous_page
                      |          handle_mm_fault
                      |          __get_user_pages
                      |          get_user_page_nowait
                      |          hva_to_pfn.isra.17
                      |          __gfn_to_pfn
                      |          gfn_to_pfn_async
                      |          try_async_pf
                      |          tdp_page_fault
                      |          kvm_mmu_page_fault
                      |          pf_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--91.13%-- 0x10100000006
                      |          |          
                      |           --8.87%-- 0x10100000002
                       --0.02%-- [...]
     1.68%         qemu-kvm  [kernel.kallsyms]     [k] yield_to                                  
                   |
                   --- yield_to
                      |          
                      |--99.65%-- kvm_vcpu_yield_to
                      |          kvm_vcpu_on_spin
                      |          pause_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--88.78%-- 0x10100000006
                      |          |          
                      |           --11.22%-- 0x10100000002
                       --0.35%-- [...]
     1.24%             ksmd  [kernel.kallsyms]     [k] memcmp                                    
                       |
                       --- memcmp
                          |          
                          |--99.78%-- memcmp_pages
                          |          |          
                          |          |--77.17%-- ksm_scan_thread
                          |          |          kthread
                          |          |          kernel_thread_helper
                          |          |          
                          |           --22.83%-- try_to_merge_with_ksm_page
                          |                     ksm_scan_thread
                          |                     kthread
                          |                     kernel_thread_helper
                           --0.22%-- [...]
     1.09%         qemu-kvm  [kernel.kallsyms]     [k] svm_vcpu_run                              
                   |
                   --- svm_vcpu_run
                      |          
                      |--99.44%-- kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--82.15%-- 0x10100000006
                      |          |          
                      |          |--17.85%-- 0x10100000002
                      |           --0.00%-- [...]
                      |          
                       --0.56%-- kvm_vcpu_ioctl
                                 do_vfs_ioctl
                                 sys_ioctl
                                 system_call_fastpath
                                 ioctl
                                 |          
                                 |--75.21%-- 0x10100000006
                                 |          
                                  --24.79%-- 0x10100000002
     1.09%          swapper  [kernel.kallsyms]     [k] default_idle                              
                    |
                    --- default_idle
                       |          
                       |--99.74%-- cpu_idle
                       |          |          
                       |          |--76.31%-- start_secondary
                       |          |          
                       |           --23.69%-- rest_init
                       |                     start_kernel
                       |                     x86_64_start_reservations
                       |                     x86_64_start_kernel
                        --0.26%-- [...]
     1.08%             ksmd  [kernel.kallsyms]     [k] smp_call_function_many                    
                       |
                       --- smp_call_function_many
                          |          
                          |--99.97%-- native_flush_tlb_others
                          |          |          
                          |          |--99.78%-- flush_tlb_page
                          |          |          ptep_clear_flush
                          |          |          try_to_merge_with_ksm_page
                          |          |          ksm_scan_thread
                          |          |          kthread
                          |          |          kernel_thread_helper
                          |           --0.22%-- [...]
                           --0.03%-- [...]
     0.77%         qemu-kvm  [kernel.kallsyms]     [k] kvm_vcpu_on_spin                          
                   |
                   --- kvm_vcpu_on_spin
                      |          
                      |--99.36%-- pause_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--90.08%-- 0x10100000006
                      |          |          
                      |          |--9.92%-- 0x10100000002
                      |           --0.00%-- [...]
                      |          
                       --0.64%-- handle_exit
                                 kvm_arch_vcpu_ioctl_run
                                 kvm_vcpu_ioctl
                                 do_vfs_ioctl
                                 sys_ioctl
                                 system_call_fastpath
                                 ioctl
                                 |          
                                 |--87.37%-- 0x10100000006
                                 |          
                                  --12.63%-- 0x10100000002
     0.75%         qemu-kvm  [kernel.kallsyms]     [k] compact_zone                              
                   |
                   --- compact_zone
                      |          
                      |--99.98%-- compact_zone_order
                      |          try_to_compact_pages
                      |          __alloc_pages_direct_compact
                      |          __alloc_pages_nodemask
                      |          alloc_pages_vma
                      |          do_huge_pmd_anonymous_page
                      |          handle_mm_fault
                      |          __get_user_pages
                      |          get_user_page_nowait
                      |          hva_to_pfn.isra.17
                      |          __gfn_to_pfn
                      |          gfn_to_pfn_async
                      |          try_async_pf
                      |          tdp_page_fault
                      |          kvm_mmu_page_fault
                      |          pf_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--91.29%-- 0x10100000006
                      |          |          
                      |           --8.71%-- 0x10100000002
                       --0.02%-- [...]
     0.68%         qemu-kvm  [kernel.kallsyms]     [k] _raw_spin_lock                            
                   |
                   --- _raw_spin_lock
                      |          
                      |--39.71%-- yield_to
                      |          kvm_vcpu_yield_to
                      |          kvm_vcpu_on_spin
                      |          pause_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--90.52%-- 0x10100000006
                      |          |          
                      |           --9.48%-- 0x10100000002
                      |          
                      |--15.63%-- kvm_vcpu_yield_to
                      |          kvm_vcpu_on_spin
                      |          pause_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--90.96%-- 0x10100000006
                      |          |          
                      |           --9.04%-- 0x10100000002
                      |          
                      |--6.55%-- tdp_page_fault
                      |          kvm_mmu_page_fault
                      |          pf_interception
                      |          handle_exit
                      |          kvm_arch_vcpu_ioctl_run
                      |          kvm_vcpu_ioctl
                      |          do_vfs_ioctl
                      |          sys_ioctl
                      |          system_call_fastpath
                      |          ioctl
                      |          |          
                      |          |--78.78%-- 0x10100000006
                      |          |          
                      |           --21.22%-- 0x10100000002
                      |          
                      |--4.87%-- free_pcppages_bulk
                      |          |          
                      |          |--51.10%-- free_hot_cold_page
                      |          |          |          
                      |          |          |--83.60%-- free_hot_cold_page_list
                      |          |          |          |          
                      |          |          |          |--62.17%-- release_pages
                      |          |          |          |          pagevec_lru_move_fn
                      |          |          |          |          __pagevec_lru_add
                      |          |          |          |          |          
                      |          |          |          |          |--99.22%-- __lru_cache_add
                      |          |          |          |          |          lru_cache_add_lru
                      |          |          |          |          |          putback_lru_page
                      |          |          |          |          |          |          
                      |          |          |          |          |          |--99.61%-- migrate_pages
                      |          |          |          |          |          |          compact_zone
                      |          |          |          |          |          |          compact_zone_order
                      |          |          |          |          |          |          try_to_compact_pages
                      |          |          |          |          |          |          __alloc_pages_direct_compact
                      |          |          |          |          |          |          __alloc_pages_nodemask
                      |          |          |          |          |          |          alloc_pages_vma
                      |          |          |          |          |          |          do_huge_pmd_anonymous_page
                      |          |          |          |          |          |          handle_mm_fault
                      |          |          |          |          |          |          __get_user_pages
                      |          |          |          |          |          |          get_user_page_nowait
                      |          |          |          |          |          |          hva_to_pfn.isra.17
                      |          |          |          |          |          |          __gfn_to_pfn
                      |          |          |          |          |          |          gfn_to_pfn_async
                      |          |          |          |          |          |          try_async_pf
                      |          |          |          |          |          |          tdp_page_fault
                      |          |          |          |          |          |          kvm_mmu_page_fault
                      |          |          |          |          |          |          pf_interception
                      |          |          |          |          |          |          handle_exit
                      |          |          |          |          |          |          kvm_arch_vcpu_ioctl_run
                      |          |          |          |          |          |          kvm_vcpu_ioctl
                      |          |          |          |          |          |          do_vfs_ioctl
                      |          |          |          |          |          |          sys_ioctl
                      |          |          |          |          |          |          system_call_fastpath
                      |          |          |          |          |          |          ioctl
                      |          |          |          |          |          |          |          
                      |          |          |          |          |          |          |--88.98%-- 0x10100000006
                      |          |          |          |          |          |          |          
                      |          |          |          |          |          |           --11.02%-- 0x10100000002
                      |          |          |          |          |           --0.39%-- [...]
                      |          |          |          |          |          
                      |          |          |          |           --0.78%-- lru_add_drain_cpu
                      |          |          |          |                     lru_add_drain
                      |          |          |          |                     migrate_prep_local
                      |          |          |          |                     compact_zone
                      |          |          |          |                     compact_zone_order
                      |          |          |          |                     try_to_compact_pages
                      |          |          |          |                     __alloc_pages_direct_compact
                      |          |          |          |                     __alloc_pages_nodemask
                      |          |          |          |                     alloc_pages_vma
                      |          |          |          |                     do_huge_pmd_anonymous_page
                      |          |          |          |                     handle_mm_fault
                      |          |          |          |                     __get_user_pages
                      |          |          |          |                     get_user_page_nowait
                      |          |          |          |                     hva_to_pfn.isra.17
                      |          |          |          |                     __gfn_to_pfn
                      |          |          |          |                     gfn_to_pfn_async
                      |          |          |          |                     try_async_pf
                      |          |          |          |                     tdp_page_fault
                      |          |          |          |                     kvm_mmu_page_fault
                      |          |          |          |                     pf_interception
                      |          |          |          |                     handle_exit
                      |          |          |          |                     kvm_arch_vcpu_ioctl_run
                      |          |          |          |                     kvm_vcpu_ioctl
                      |          |          |          |                     do_vfs_ioctl
                      |          |          |          |                     sys_ioctl
                      |          |          |          |                     system_call_fastpath
                      |          |          |          |                     ioctl
                      |          |          |          |                     0x10100000006
                      |          |          |          |          
                      |          |          |           --37.83%-- shrink_page_list
                      |          |          |                     shrink_inactive_list
                      |          |          |                     shrink_lruvec
                      |          |          |                     try_to_free_pages
                      |          |          |                     __alloc_pages_nodemask
                      |          |          |                     alloc_pages_vma
                      |          |          |                     do_huge_pmd_anonymous_page
                      |          |          |                     handle_mm_fault
                      |          |          |                     __get_user_pages
                      |          |          |                     get_user_page_nowait
                      |          |          |                     hva_to_pfn.isra.17
                      |          |          |                     __gfn_to_pfn
                      |          |          |                     gfn_to_pfn_async
                      |          |          |                     try_async_pf
                      |          |          |                     tdp_page_fault
                      |          |          |                     kvm_mmu_page_fault
                      |          |          |                     pf_interception
                      |          |          |                     handle_exit
                      |          |          |                     kvm_arch_vcpu_ioctl_run
                      |          |          |                     kvm_vcpu_ioctl
                      |          |          |                     do_vfs_ioctl
                      |          |          |                     sys_ioctl
                      |          |          |                     system_call_fastpath
                      |          |          |                     ioctl
                      |          |          |                     |          
                      |          |          |                     |--86.38%-- 0x10100000006
                      |          |          |                     |          
                      |          |          |                      --13.62%-- 0x10100000002
                      |          |          |          
                      |          |          |--12.96%-- __free_pages
                      |          |          |          |          
                      |          |          |          |--98.43%-- release_freepages
                      |          |          |          |          compact_zone
                      |          |          |          |          compact_zone_order
                      |          |          |          |          try_to_compact_pages
                      |          |          |          |          __alloc_pages_direct_compact
                      |          |          |          |          __alloc_pages_nodemask
                      |          |          |          |          alloc_pages_vma
                      |          |          |          |          do_huge_pmd_anonymous_page
                      |          |          |          |          handle_mm_fault
                      |          |          |          |          __get_user_pages
                      |          |          |          |          get_user_page_nowait
                      |          |          |          |          hva_to_pfn.isra.17
                      |          |          |          |          __gfn_to_pfn
                      |          |          |          |          gfn_to_pfn_async
                      |          |          |          |          try_async_pf
                      |          |          |          |          tdp_page_fault
                      |          |          |          |          kvm_mmu_page_fault
                      |          |          |          |          pf_interception
                      |          |          |          |          handle_exit
                      |          |          |          |          kvm_arch_vcpu_ioctl_run
                      |          |          |          |          kvm_vcpu_ioctl
                      |          |          |          |          do_vfs_ioctl
                      |          |          |          |          sys_ioctl
                      |          |          |          |          system_call_fastpath
                      |          |          |          |          ioctl
                      |          |          |          |          |          
                      |          |          |          |          |--90.49%-- 0x10100000006
                      |          |          |          |          |          
                      |          |          |          |           --9.51%-- 0x10100000002
                      |          |          |          |          
                      |          |          |           --1.57%-- __free_slab
                      |          |          |                     discard_slab
                      |          |          |                     unfreeze_partials
                      |          |          |                     put_cpu_partial
                      |          |          |                     __slab_free
                      |          |          |                     kmem_cache_free
                      |          |          |                     free_buffer_head
                      |          |          |                     try_to_free_buffers
                      |          |          |                     jbd2_journal_try_to_free_buffers
                      |          |          |                     bdev_try_to_free_page
                      |          |          |                     blkdev_releasepage
                      |          |          |                     try_to_release_page
                      |          |          |                     move_to_new_page
                      |          |          |                     migrate_pages
                      |          |          |                     compact_zone
                      |          |          |                     compact_zone_order
                      |          |          |                     try_to_compact_pages
                      |          |          |                     __alloc_pages_direct_compact
                      |          |          |                     __alloc_pages_nodemask
                      |          |          |                     alloc_pages_vma
                      |          |          |                     do_huge_pmd_anonymous_page
                      |          |          |                     handle_mm_fault
                      |          |          |                     __get_user_pages
                      |          |          |                     get_user_page_nowait
                      |          |          |                     hva_to_pfn.isra.17
                      |          |          |                     __gfn_to_pfn
                      |          |          |                     gfn_to_pfn_async
                      |          |          |                     try_async_pf
                      |          |          |                     tdp_page_fault
                      |          |          |                     kvm_mmu_page_fault
                      |          |          |                     pf_interception
                      |          |          |                     handle_exit
                      |          |          |                     kvm_arch_vcpu_ioctl_run
                      |          |          |                     kvm_vcpu_ioctl
                      |          |          |                     do_vfs_ioctl
                      |          |          |                     sys_ioctl
                      |          |          |                     system_call_fastpath
                      |          |          |                     ioctl
                      |          |          |                     0x10100000006
                      |          |          |          
                      |          |           --3.44%-- __put_single_page
                      |          |                     put_page
                      |          |                     putback_lru_page
                      |          |                     migrate_pages
                      |          |                     compact_zone
                      |          |                     compact_zone_order
                      |          |                     try_to_compact_pages
                      |          |                     __alloc_pages_direct_compact
                      |          |                     __alloc_pages_nodemask
                      |          |                     alloc_pages_vma
                      |          |                     do_huge_pmd_anonymous_page
                      |          |                     handle_mm_fault
                      |          |                     __get_user_pages
                      |          |                     get_user_page_nowait
                      |          |                     hva_to_pfn.isra.17
                      |          |                     __gfn_to_pfn
                      |          |                     gfn_to_pfn_async
                      |          |                     try_async_pf
                      |          |                     tdp_page_fault
                      |          |                     kvm_mmu_page_fault
                      |          |                     pf_interception
                      |          |                     handle_exit
                      |          |                     kvm_arch_vcpu_ioctl_run
                      |          |                     kvm_vcpu_ioctl
                      |          |                     do_vfs_ioctl
                      |          |                     sys_ioctl
                      |          |                     system_call_fastpath
                      |          |                     ioctl
                      |          |                     |          
                      |          |                     |--88.25%-- 0x10100000006
                      |          |                     |          
                      |          |                      --11.75%-- 0x10100000002
                      |          |          
                      |           --48.90%-- drain_pages
                      |                     |          
                      |                     |--88.65%-- drain_local_pages
                      |                     |          |          
                      |                     |          |--96.33%-- generic_smp_call_function_interrupt
                      |                     |          |          smp_call_function_interrupt
                      |                     |          |          call_function_interrupt
                      |                     |          |          |          
                      |                     |          |          |--23.46%-- __remove_mapping
                      |                     |          |          |          shrink_page_list
                      |                     |          |          |          shrink_inactive_list
                      |                     |          |          |          shrink_lruvec
                      |                     |          |          |          try_to_free_pages
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--93.81%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --6.19%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--19.93%-- kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--93.65%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --6.35%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--14.19%-- compaction_alloc
                      |                     |          |          |          migrate_pages
                      |                     |          |          |          compact_zone
                      |                     |          |          |          compact_zone_order
                      |                     |          |          |          try_to_compact_pages
                      |                     |          |          |          __alloc_pages_direct_compact
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--89.88%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --10.12%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--8.57%-- isolate_migratepages_range
                      |                     |          |          |          compact_zone
                      |                     |          |          |          compact_zone_order
                      |                     |          |          |          try_to_compact_pages
                      |                     |          |          |          __alloc_pages_direct_compact
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--92.14%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --7.86%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--5.05%-- do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--92.53%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --7.47%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--4.49%-- shrink_inactive_list
                      |                     |          |          |          shrink_lruvec
                      |                     |          |          |          try_to_free_pages
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--94.61%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --5.39%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--2.80%-- free_hot_cold_page_list
                      |                     |          |          |          shrink_page_list
                      |                     |          |          |          shrink_inactive_list
                      |                     |          |          |          shrink_lruvec
                      |                     |          |          |          try_to_free_pages
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--91.24%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --8.76%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--1.96%-- buffer_migrate_page
                      |                     |          |          |          move_to_new_page
                      |                     |          |          |          migrate_pages
                      |                     |          |          |          compact_zone
                      |                     |          |          |          compact_zone_order
                      |                     |          |          |          try_to_compact_pages
                      |                     |          |          |          __alloc_pages_direct_compact
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          |          
                      |                     |          |          |          |--63.14%-- 0x10100000006
                      |                     |          |          |          |          
                      |                     |          |          |           --36.86%-- 0x10100000002
                      |                     |          |          |          
                      |                     |          |          |--1.62%-- try_to_free_buffers
                      |                     |          |          |          jbd2_journal_try_to_free_buffers
                      |                     |          |          |          ext4_releasepage
                      |                     |          |          |          try_to_release_page
                      |                     |          |          |          shrink_page_list
                      |                     |          |          |          shrink_inactive_list
                      |                     |          |          |          shrink_lruvec
                      |                     |          |          |          try_to_free_pages
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          0x10100000006
                      |                     |          |          |          
                      |                     |          |          |--1.49%-- compact_checklock_irqsave
                      |                     |          |          |          isolate_migratepages_range
                      |                     |          |          |          compact_zone
                      |                     |          |          |          compact_zone_order
                      |                     |          |          |          try_to_compact_pages
                      |                     |          |          |          __alloc_pages_direct_compact
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          0x10100000006
                      |                     |          |          |          
                      |                     |          |          |--1.46%-- __mutex_lock_slowpath
                      |                     |          |          |          mutex_lock
                      |                     |          |          |          page_lock_anon_vma
                      |                     |          |          |          page_referenced
                      |                     |          |          |          shrink_active_list
                      |                     |          |          |          shrink_lruvec
                      |                     |          |          |          try_to_free_pages
                      |                     |          |          |          __alloc_pages_nodemask
                      |                     |          |          |          alloc_pages_vma
                      |                     |          |          |          do_huge_pmd_anonymous_page
                      |                     |          |          |          handle_mm_fault
                      |                     |          |          |          __get_user_pages
                      |                     |          |          |          get_user_page_nowait
                      |                     |          |          |          hva_to_pfn.isra.17
                      |                     |          |          |          __gfn_to_pfn
                      |                     |          |          |          gfn_to_pfn_async
                      |                     |          |          |          try_async_pf
                      |                     |          |          |          tdp_page_fault
                      |                     |          |          |          kvm_mmu_page_fault
                      |                     |          |          |          pf_interception
                      |                     |          |          |          handle_exit
                      |                     |          |          |          kvm_arch_vcpu_ioctl_run
                      |                     |          |          |          kvm_vcpu_ioctl
                      |                     |          |          |          do_vfs_ioctl
                      |                     |          |          |          sys_ioctl
                      |                     |          |          |          system_call_fastpath
                      |                     |          |          |          ioctl
                      |                     |          |          |          0x10100000006
                      |                     |          |          |          
                      |                     |          |          |--1.41%-- native_flush_tlb_others
                      |                     |          |          |          flush_tlb_page
                      |                     |          |          |          |          
                      |                     |          |          |          |--67.10%-- ptep_clear_flush
                      |                     |          |          |          |          try_to_unmap_one
                      |                     |          |          |          |          try_to_unmap_anon
                      |                     |          |          |          |          try_to_unmap
                      |                     |          |          |          |          migrate_pages
                      |                     |          |          |          |          compact_zone
                      |                     |          |          |          |          compact_zone_order
                      |                     |          |          |          |          try_to_compact_pages
                      |                     |          |          |          |          __alloc_pages_direct_compact
                      |                     |          |          |          |          __alloc_pages_nodemask
Rik van Riel - Sept. 17, 2012, 1:50 p.m.
On 09/15/2012 11:55 AM, Richard Davies wrote:
> Hi Rik, Mel and Shaohua,
>
> Thank you for your latest patches. I attach my latest perf report for a slow
> boot with all of these applied.
>
> Mel asked for timings of the slow boots. It's very hard to give anything
> useful here! A normal boot would be a minute or so, and many are like that,
> but the slowest that I have seen (on 3.5.x) was several hours. Basically, I
> just test many times until I get one which is noticeably slow than normal
> and then run perf record on that one.
>
> The latest perf report for a slow boot is below. For the fast boots, most of
> the time is in clean_page_c in do_huge_pmd_anonymous_page, but for this slow
> one there is a lot of lock contention above that.

How often do you run into slow boots, vs. fast ones?

> # Overhead          Command         Shared Object                                          Symbol
> # ........  ...............  ....................  ..............................................
> #
>      58.49%         qemu-kvm  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave
>                     |
>                     --- _raw_spin_lock_irqsave
>                        |
>                        |--95.07%-- compact_checklock_irqsave
>                        |          |
>                        |          |--70.03%-- isolate_migratepages_range
>                        |          |          compact_zone
>                        |          |          compact_zone_order
>                        |          |          try_to_compact_pages
>                        |          |          __alloc_pages_direct_compact
>                        |          |          __alloc_pages_nodemask

Looks like it moved from isolate_freepages_block in your last
trace, to isolate_migratepages_range?

Mel, I wonder if we have any quadratic complexity problems
in this part of the code, too?

The isolate_freepages_block CPU use can be fixed by simply
restarting where the last invocation left off, instead of
always starting at the end of the zone.  Could we need
something similar for isolate_migratepages_range?

After all, Richard has a 128GB system, and runs 108GB worth
of KVM guests on it...
Mel Gorman - Sept. 17, 2012, 2:07 p.m.
On Mon, Sep 17, 2012 at 09:50:08AM -0400, Rik van Riel wrote:
> On 09/15/2012 11:55 AM, Richard Davies wrote:
> >Hi Rik, Mel and Shaohua,
> >
> >Thank you for your latest patches. I attach my latest perf report for a slow
> >boot with all of these applied.
> >
> >Mel asked for timings of the slow boots. It's very hard to give anything
> >useful here! A normal boot would be a minute or so, and many are like that,
> >but the slowest that I have seen (on 3.5.x) was several hours. Basically, I
> >just test many times until I get one which is noticeably slow than normal
> >and then run perf record on that one.
> >
> >The latest perf report for a slow boot is below. For the fast boots, most of
> >the time is in clean_page_c in do_huge_pmd_anonymous_page, but for this slow
> >one there is a lot of lock contention above that.
> 
> How often do you run into slow boots, vs. fast ones?
> 
> ># Overhead          Command         Shared Object                                          Symbol
> ># ........  ...............  ....................  ..............................................
> >#
> >     58.49%         qemu-kvm  [kernel.kallsyms]     [k] _raw_spin_lock_irqsave
> >                    |
> >                    --- _raw_spin_lock_irqsave
> >                       |
> >                       |--95.07%-- compact_checklock_irqsave
> >                       |          |
> >                       |          |--70.03%-- isolate_migratepages_range
> >                       |          |          compact_zone
> >                       |          |          compact_zone_order
> >                       |          |          try_to_compact_pages
> >                       |          |          __alloc_pages_direct_compact
> >                       |          |          __alloc_pages_nodemask
> 
> Looks like it moved from isolate_freepages_block in your last
> trace, to isolate_migratepages_range?
> 
> Mel, I wonder if we have any quadratic complexity problems
> in this part of the code, too?
> 

Possibly but right now I'm focusing on the contention even though I recognise
that reducing the amount of scanning implicitly reduces the amount of
contention. I'm running a test at the moment with an additional patch
to record the pageblock being scanned by either the free or migrate page
scanner. This should be enough to both calculate the scanning efficiency
and how many useless blocks are scanned to determine if your "skip"
patches are behaving as expected and from there decide if the migrate
scanner needs similar logic.

Patch

diff --git a/mm/compaction.c b/mm/compaction.c
index 771775d..0656759 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -431,6 +431,24 @@  static bool suitable_migration_target(struct page *page)
 }
 
 /*
+ * We scan the zone in a circular fashion, starting at
+ * zone->compact_cached_free_pfn. Be careful not to skip if
+ * one compacting thread has just wrapped back to the end of the
+ * zone, but another thread has not.
+ */
+static bool compaction_may_skip(struct zone *zone,
+				struct compact_control *cc)
+{
+	if (!cc->wrapped && zone->compact_cached_free_pfn < cc->start_free_pfn)
+		return true;
+
+	if (cc->wrapped && zone->compact_cached_free_pfn > cc->start_free_pfn)
+		return true;
+
+	return false;
+}
+
+/*
  * Based on information in the current compact_control, find blocks
  * suitable for isolating free pages from and then isolate them.
  */
@@ -471,13 +489,9 @@  static void isolate_freepages(struct zone *zone,
 
 		/*
 		 * Skip ahead if another thread is compacting in the area
-		 * simultaneously. If we wrapped around, we can only skip
-		 * ahead if zone->compact_cached_free_pfn also wrapped to
-		 * above our starting point.
+		 * simultaneously, and has finished with this page block.
 		 */
-		if (cc->order > 0 && (!cc->wrapped ||
-				      zone->compact_cached_free_pfn >
-				      cc->start_free_pfn))
+		if (cc->order > 0 && compaction_may_skip(zone, cc))
 			pfn = min(pfn, zone->compact_cached_free_pfn);
 
 		if (!pfn_valid(pfn))