[v2] ARC: fix memory nodes topology in case of highmem enabled

Tweak generic node topology in case of CONFIG_HIGHMEM enabled to
prioritize allocations from ZONE_HIGHMEM to avoid ZONE_NORMAL
pressure.

Here is example when we can see problems on ARC with currently
existing topology configuration:

Generic statements:
 - *NOT* every memory allocation which could be done from
   ZONE_NORMAL also could be done from ZONE_HIGHMEM.
 - Every memory allocation which could be done from ZONE_HIGHMEM
   also could be done from ZONE_NORMAL (In other words ZONE_NORMAL
   is more universal than ZONE_HIGHMEM)

ARC statements:
In case of CONFIG_HIGHMEM enabled we have 2 memory nodes:
 - "node 0" has only ZONE_NORMAL memory.
 - "node 1" has only ZONE_HIGHMEM memory.

Steps to reproduce the problem:
1) Let's try to allocate some memory from userspace. It can be
   allocate from anywhere (ZONE_HIGHMEM/ZONE_NORMAL).
2) Kernel tries to allocate memory from the closest memory node
   to this CPU. As we don't have NUMA enabled and don't override
   any define from "include/asm-generic/topology.h" the closest
   memory node to any CPU will be "node 0"
3) OK, we'll allocate memory from "node 0". Let's choose ZONE
   to allocate from. This allocation could be done from both
   ZONE_HIGHMEM / ZONE_NORMAL in this node. The allocation
   priority between zones is ZONE_HIGHMEM > ZONE_NORMAL.
   This is pretty logical - we don't want waste *universal*
   ZONE_NORMAL if we can use ZONE_HIGHMEM. But we don't have
   ZONE_HIGHMEM in "node 0" that's why we rollback to
   ZONE_NORMAL and allocate memory from it.
4) Let's try to allocate a lot of memory [more than we have free
   memory in lowmem] from userspace.
5) Kernel allocates as much memory as it can from the closest
   memory node ("node 0"). But there is no enough memory in
   "node 0". So we'll rollback to another memory node ("node 1")
   and allocate the rest of the amount from it.

   In other words we have following memory lookup path:
      (node 0, ZONE_HIGHMEM) ->
   -> (node 0, ZONE_NORMAL)  ->
   -> (node 1, ZONE_HIGHMEM)

   Now we don't have any free memory in (node 0, ZONE_NORMAL)
   [Actually this is a simplification, but it doesn't matter
   in this example]
6) Oops, some internal kernel memory allocation happen which
   requires ZONE_NORMAL. For example "kmalloc(size, GFP_KERNEL)"
   was called.
   So the we have following memory lookup path:
   (node 0, ZONE_NORMAL) -> ("node 1", ZONE_NORMAL)
   There is no free memory in "node 0". And there is no
   ZONE_NORMAL in "node 1". We only have some free memory in
   (node 1, ZONE_HIGHMEM) but HIGHMEM isn't suitable in this
   case.
7) As we can't allocate memory OOM-Killer is invoked, even if
   we have some free memory in (node 1, ZONE_HIGHMEM).

This patch tweaks generic node topology and mark memory from
"node 1" as the closest to any CPU.

So the we'll have following memory lookup path:
    (node 1, ZONE_HIGHMEM) ->
 -> (node 1, ZONE_NORMAL)  ->
 -> (node 0, ZONE_HIGHMEM) ->
 -> (node 0, ZONE_NORMAL)
In case of node configuration on ARC we obtain the degenerate case
of this path:
(node 1, ZONE_HIGHMEM) -> (node 0, ZONE_NORMAL)

In this case we don't waste *universal* ZONE_NORMAL if we can use
ZONE_HIGHMEM so we don't face with the issue pointed in [5-7]

Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
---
Changes v1->v2:
 * Changes in commit message and comments in a code. No functional
   change intended.

 arch/arc/include/asm/Kbuild     |  1 -
 arch/arc/include/asm/topology.h | 24 ++++++++++++++++++++++++
 2 files changed, 24 insertions(+), 1 deletion(-)
 create mode 100644 arch/arc/include/asm/topology.h

Hi Vineet,

ping.

On Wed, 2019-04-17 at 13:46 +0300, Eugeniy Paltsev wrote:
> Tweak generic node topology in case of CONFIG_HIGHMEM enabled to
> prioritize allocations from ZONE_HIGHMEM to avoid ZONE_NORMAL
> pressure.
> 
> Here is example when we can see problems on ARC with currently
> existing topology configuration:
> 
> Generic statements:
>  - *NOT* every memory allocation which could be done from
>    ZONE_NORMAL also could be done from ZONE_HIGHMEM.
>  - Every memory allocation which could be done from ZONE_HIGHMEM
>    also could be done from ZONE_NORMAL (In other words ZONE_NORMAL
>    is more universal than ZONE_HIGHMEM)
> 
> ARC statements:
> In case of CONFIG_HIGHMEM enabled we have 2 memory nodes:
>  - "node 0" has only ZONE_NORMAL memory.
>  - "node 1" has only ZONE_HIGHMEM memory.
> 
> Steps to reproduce the problem:
> 1) Let's try to allocate some memory from userspace. It can be
>    allocate from anywhere (ZONE_HIGHMEM/ZONE_NORMAL).
> 2) Kernel tries to allocate memory from the closest memory node
>    to this CPU. As we don't have NUMA enabled and don't override
>    any define from "include/asm-generic/topology.h" the closest
>    memory node to any CPU will be "node 0"
> 3) OK, we'll allocate memory from "node 0". Let's choose ZONE
>    to allocate from. This allocation could be done from both
>    ZONE_HIGHMEM / ZONE_NORMAL in this node. The allocation
>    priority between zones is ZONE_HIGHMEM > ZONE_NORMAL.
>    This is pretty logical - we don't want waste *universal*
>    ZONE_NORMAL if we can use ZONE_HIGHMEM. But we don't have
>    ZONE_HIGHMEM in "node 0" that's why we rollback to
>    ZONE_NORMAL and allocate memory from it.
> 4) Let's try to allocate a lot of memory [more than we have free
>    memory in lowmem] from userspace.
> 5) Kernel allocates as much memory as it can from the closest
>    memory node ("node 0"). But there is no enough memory in
>    "node 0". So we'll rollback to another memory node ("node 1")
>    and allocate the rest of the amount from it.
> 
>    In other words we have following memory lookup path:
>       (node 0, ZONE_HIGHMEM) ->
>    -> (node 0, ZONE_NORMAL)  ->
>    -> (node 1, ZONE_HIGHMEM)
> 
>    Now we don't have any free memory in (node 0, ZONE_NORMAL)
>    [Actually this is a simplification, but it doesn't matter
>    in this example]
> 6) Oops, some internal kernel memory allocation happen which
>    requires ZONE_NORMAL. For example "kmalloc(size, GFP_KERNEL)"
>    was called.
>    So the we have following memory lookup path:
>    (node 0, ZONE_NORMAL) -> ("node 1", ZONE_NORMAL)
>    There is no free memory in "node 0". And there is no
>    ZONE_NORMAL in "node 1". We only have some free memory in
>    (node 1, ZONE_HIGHMEM) but HIGHMEM isn't suitable in this
>    case.
> 7) As we can't allocate memory OOM-Killer is invoked, even if
>    we have some free memory in (node 1, ZONE_HIGHMEM).
> 
> This patch tweaks generic node topology and mark memory from
> "node 1" as the closest to any CPU.
> 
> So the we'll have following memory lookup path:
>     (node 1, ZONE_HIGHMEM) ->
>  -> (node 1, ZONE_NORMAL)  ->
>  -> (node 0, ZONE_HIGHMEM) ->
>  -> (node 0, ZONE_NORMAL)
> In case of node configuration on ARC we obtain the degenerate case
> of this path:
> (node 1, ZONE_HIGHMEM) -> (node 0, ZONE_NORMAL)
> 
> In this case we don't waste *universal* ZONE_NORMAL if we can use
> ZONE_HIGHMEM so we don't face with the issue pointed in [5-7]
> 
> Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
> ---
> Changes v1->v2:
>  * Changes in commit message and comments in a code. No functional
>    change intended.
> 
>  arch/arc/include/asm/Kbuild     |  1 -
>  arch/arc/include/asm/topology.h | 24 ++++++++++++++++++++++++
>  2 files changed, 24 insertions(+), 1 deletion(-)
>  create mode 100644 arch/arc/include/asm/topology.h
> 
> diff --git a/arch/arc/include/asm/Kbuild b/arch/arc/include/asm/Kbuild
> index caa270261521..e64e0439baff 100644
> --- a/arch/arc/include/asm/Kbuild
> +++ b/arch/arc/include/asm/Kbuild
> @@ -18,7 +18,6 @@ generic-y += msi.h
>  generic-y += parport.h
>  generic-y += percpu.h
>  generic-y += preempt.h
> -generic-y += topology.h
>  generic-y += trace_clock.h
>  generic-y += user.h
>  generic-y += vga.h
> diff --git a/arch/arc/include/asm/topology.h b/arch/arc/include/asm/topology.h
> new file mode 100644
> index 000000000000..c3b8ab7ed011
> --- /dev/null
> +++ b/arch/arc/include/asm/topology.h
> @@ -0,0 +1,24 @@
> +#ifndef _ASM_ARC_TOPOLOGY_H
> +#define _ASM_ARC_TOPOLOGY_H
> +
> +/*
> + * On ARC (w/o PAE) HIGHMEM addresses are smaller (0x0 based) than addresses in
> + * NORMAL aka low memory (0x8000_0000 based).
> + * Thus HIGHMEM on ARC is implemented with DISCONTIGMEM which requires multiple
> + * nodes. So here is memory node map on ARC:
> + *  - node 0: ZONE_NORMAL  memory (always)
> + *  - node 1: ZONE_HIGHMEM memory (only if CONFIG_HIGHMEM is enabled)
> + *
> + * In case of CONFIG_HIGHMEM enabled we tweak generic node topology and mark
> + * node 1 as the closest to all CPUs to prioritize allocations from ZONE_HIGHMEM
> + * where it is possible to avoid ZONE_NORMAL pressure.
> + */
> +#ifdef CONFIG_HIGHMEM
> +#define cpu_to_node(cpu)	((void)(cpu), 1)
> +#define cpu_to_mem(cpu)		((void)(cpu), 1)
> +#define cpumask_of_node(node)	((node) == 1 ? cpu_online_mask : cpu_none_mask)
> +#endif /* CONFIG_HIGHMEM */
> +
> +#include <asm-generic/topology.h>
> +
> +#endif /* _ASM_ARC_TOPOLOGY_H */