| Message ID | 20190311193112.25527-1-m@lambda.lt |
|---|---|
| State | Changes Requested |
| Delegated to: | BPF Maintainers |
| Headers | show |
| Series | bpf: Try harder when allocating memory for large maps | expand |
On Mon 11-03-19 20:31:12, Martynas Pumputis wrote: > It has been observed that sometimes a higher order memory allocation > for BPF maps fails when there is no obvious memory pressure in a system. > > E.g. the map (BPF_MAP_TYPE_LRU_HASH, key=38, value=56, max_elems=524288) > could not be created due to vmalloc unable to allocate 75497472B, > when the system's memory consumption (in MB) was the following: > > Total: 3942 Used: 837 (21.24%) Free: 138 Buffers: 239 Cached: 2727 > > Later analysis [1] by Michal Hocko showed that the vmalloc was not trying > to reclaim memory from the page cache and was failing prematurely due to > __GFP_NORETRY. > > Considering dcda9b0471 ("mm, tree wide: replace __GFP_REPEAT by > __GFP_RETRY_MAYFAIL with more useful semantic") and [1], we can replace > __GFP_NORETRY with __GFP_RETRY_MAYFAIL, as it won't invoke OOM killer > and will try harder to fulfil allocation requests. > > The change has been tested with the workloads mentioned above and by > observing oom_kill value from /proc/vmstat. Please document why kvmalloc_node is not used. > [1]: https://lore.kernel.org/bpf/20190310071318.GW5232@dhcp22.suse.cz/ > > Signed-off-by: Martynas Pumputis <m@lambda.lt> > --- > kernel/bpf/syscall.c | 18 ++++++++++++------ > 1 file changed, 12 insertions(+), 6 deletions(-) > > diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c > index 62f6bced3a3c..1b0a057ed6d5 100644 > --- a/kernel/bpf/syscall.c > +++ b/kernel/bpf/syscall.c > @@ -136,20 +136,26 @@ static struct bpf_map *find_and_alloc_map(union bpf_attr *attr) > > void *bpf_map_area_alloc(size_t size, int numa_node) > { > - /* We definitely need __GFP_NORETRY, so OOM killer doesn't > - * trigger under memory pressure as we really just want to > - * fail instead. > + /* We definitely need __GFP_NORETRY or __GFP_RETRY_MAYFAIL, so > + * OOM killer doesn't trigger under memory pressure as we really > + * just want to fail instead. > */ > - const gfp_t flags = __GFP_NOWARN | __GFP_NORETRY | __GFP_ZERO; > + const gfp_t flags = __GFP_NOWARN | __GFP_ZERO; > void *area; > > if (size <= (PAGE_SIZE << PAGE_ALLOC_COSTLY_ORDER)) { > - area = kmalloc_node(size, GFP_USER | flags, numa_node); > + /* To avoid bypassing slab alloc for lower order allocs, > + * __GFP_NORETRY is used instead of __GFP_RETRY_MAYFAIL. > + */ > + area = kmalloc_node(size, GFP_USER | __GFP_NORETRY | flags, > + numa_node); > if (area != NULL) > return area; > } > > - return __vmalloc_node_flags_caller(size, numa_node, GFP_KERNEL | flags, > + return __vmalloc_node_flags_caller(size, numa_node, > + GFP_KERNEL | __GFP_RETRY_MAYFAIL | > + flags, > __builtin_return_address(0)); > } > > -- > 2.21.0 >
On Mon, Mar 11, 2019 at 12:32 PM Martynas Pumputis <m@lambda.lt> wrote: > > It has been observed that sometimes a higher order memory allocation > for BPF maps fails when there is no obvious memory pressure in a system. > > E.g. the map (BPF_MAP_TYPE_LRU_HASH, key=38, value=56, max_elems=524288) > could not be created due to vmalloc unable to allocate 75497472B, > when the system's memory consumption (in MB) was the following: > > Total: 3942 Used: 837 (21.24%) Free: 138 Buffers: 239 Cached: 2727 > > Later analysis [1] by Michal Hocko showed that the vmalloc was not trying > to reclaim memory from the page cache and was failing prematurely due to > __GFP_NORETRY. > > Considering dcda9b0471 ("mm, tree wide: replace __GFP_REPEAT by > __GFP_RETRY_MAYFAIL with more useful semantic") and [1], we can replace > __GFP_NORETRY with __GFP_RETRY_MAYFAIL, as it won't invoke OOM killer > and will try harder to fulfil allocation requests. > > The change has been tested with the workloads mentioned above and by > observing oom_kill value from /proc/vmstat. > > [1]: https://lore.kernel.org/bpf/20190310071318.GW5232@dhcp22.suse.cz/ > > Signed-off-by: Martynas Pumputis <m@lambda.lt> Acked-by: Yonghong Song <yhs@fb.com>
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 62f6bced3a3c..1b0a057ed6d5 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -136,20 +136,26 @@ static struct bpf_map *find_and_alloc_map(union bpf_attr *attr) void *bpf_map_area_alloc(size_t size, int numa_node) { - /* We definitely need __GFP_NORETRY, so OOM killer doesn't - * trigger under memory pressure as we really just want to - * fail instead. + /* We definitely need __GFP_NORETRY or __GFP_RETRY_MAYFAIL, so + * OOM killer doesn't trigger under memory pressure as we really + * just want to fail instead. */ - const gfp_t flags = __GFP_NOWARN | __GFP_NORETRY | __GFP_ZERO; + const gfp_t flags = __GFP_NOWARN | __GFP_ZERO; void *area; if (size <= (PAGE_SIZE << PAGE_ALLOC_COSTLY_ORDER)) { - area = kmalloc_node(size, GFP_USER | flags, numa_node); + /* To avoid bypassing slab alloc for lower order allocs, + * __GFP_NORETRY is used instead of __GFP_RETRY_MAYFAIL. + */ + area = kmalloc_node(size, GFP_USER | __GFP_NORETRY | flags, + numa_node); if (area != NULL) return area; } - return __vmalloc_node_flags_caller(size, numa_node, GFP_KERNEL | flags, + return __vmalloc_node_flags_caller(size, numa_node, + GFP_KERNEL | __GFP_RETRY_MAYFAIL | + flags, __builtin_return_address(0)); }
It has been observed that sometimes a higher order memory allocation for BPF maps fails when there is no obvious memory pressure in a system. E.g. the map (BPF_MAP_TYPE_LRU_HASH, key=38, value=56, max_elems=524288) could not be created due to vmalloc unable to allocate 75497472B, when the system's memory consumption (in MB) was the following: Total: 3942 Used: 837 (21.24%) Free: 138 Buffers: 239 Cached: 2727 Later analysis [1] by Michal Hocko showed that the vmalloc was not trying to reclaim memory from the page cache and was failing prematurely due to __GFP_NORETRY. Considering dcda9b0471 ("mm, tree wide: replace __GFP_REPEAT by __GFP_RETRY_MAYFAIL with more useful semantic") and [1], we can replace __GFP_NORETRY with __GFP_RETRY_MAYFAIL, as it won't invoke OOM killer and will try harder to fulfil allocation requests. The change has been tested with the workloads mentioned above and by observing oom_kill value from /proc/vmstat. [1]: https://lore.kernel.org/bpf/20190310071318.GW5232@dhcp22.suse.cz/ Signed-off-by: Martynas Pumputis <m@lambda.lt> --- kernel/bpf/syscall.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)