diff mbox series

[kernel] powerpc/pseries/ddw: Extend upper limit for huge DMA window for persistent memory

Message ID 20200331012338.23773-1-aik@ozlabs.ru
State Accepted
Commit 54fc3c681ded9437e4548e2501dc1136b23cfa9a
Headers show
Series [kernel] powerpc/pseries/ddw: Extend upper limit for huge DMA window for persistent memory | expand

Checks

Context Check Description
snowpatch_ozlabs/needsstable success Patch has no Fixes tags
snowpatch_ozlabs/checkpatch warning total: 0 errors, 0 warnings, 1 checks, 15 lines checked
snowpatch_ozlabs/build-pmac32 success Build succeeded
snowpatch_ozlabs/build-ppc64e success Build succeeded
snowpatch_ozlabs/build-ppc64be success Build succeeded
snowpatch_ozlabs/build-ppc64le success Build succeeded
snowpatch_ozlabs/apply_patch success Successfully applied on branch powerpc/merge (c6624071c338732402e8c726df6a4074473eaa0e)

Commit Message

Alexey Kardashevskiy March 31, 2020, 1:23 a.m. UTC
Unlike normal memory ("memory" compatible type in the FDT),
the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
in the guest physical space and it can be used for DMA.

In order to maintain 1:1 mapping via the huge DMA window, we need to
know the maximum physical address at the time of the window setup.
So far we've been looking at "memory" nodes but "ibm,pmemory" does not
have fixed addresses and the persistent memory may be mapped afterwards.

Since the persistent memory is still backed with page structs,
use MAX_PHYSMEM_BITS as the upper limit.

This effectively disables huge DMA window in LPAR under pHyp if
persistent memory is present but this is the best we can do.

Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
---
 arch/powerpc/platforms/pseries/iommu.c | 9 +++++++++
 1 file changed, 9 insertions(+)

Comments

Alexey Kardashevskiy April 3, 2020, 8:06 a.m. UTC | #1
Hi Wen,

Can you please try this? It contains 3 patches from Christoph Hellwig
plus my patch on top, this should improve performance when DMA-ing not
to/from persistent memory:

https://github.com/aik/linux/commits/dma-bypass.3

I am looking for any benchmarks not related to persistent memory. Thanks,



On 03/04/2020 09:09, Wen Xiong wrote:
> I applied the patch on top of the latest upstream kernel. I ran HTX over
> pmem nodes for several hours and it works.
>  
> Tested-by: Wen Xiong<wenxiong@linux.vnet.ibm.com>
>  
> Thanks,
> Wendy
> 
>     ----- Original message -----
>     From: Alexey Kardashevskiy <aik@ozlabs.ru>
>     To: linuxppc-dev@lists.ozlabs.org
>     Cc: Alexey Kardashevskiy <aik@ozlabs.ru>, David Gibson
>     <david@gibson.dropbear.id.au>, Michael Ellerman
>     <mpe@ellerman.id.au>, Oliver O'Halloran <oohall@gmail.com>, "Aneesh
>     Kumar K . V" <aneesh.kumar@linux.ibm.com>, Wen Xiong
>     <wenxiong@us.ibm.com>, Brian J King <bjking1@us.ibm.com>
>     Subject: [EXTERNAL] [PATCH kernel] powerpc/pseries/ddw: Extend upper
>     limit for huge DMA window for persistent memory
>     Date: Mon, Mar 30, 2020 8:23 PM
>      
>     Unlike normal memory ("memory" compatible type in the FDT),
>     the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
>     in the guest physical space and it can be used for DMA.
> 
>     In order to maintain 1:1 mapping via the huge DMA window, we need to
>     know the maximum physical address at the time of the window setup.
>     So far we've been looking at "memory" nodes but "ibm,pmemory" does not
>     have fixed addresses and the persistent memory may be mapped afterwards.
> 
>     Since the persistent memory is still backed with page structs,
>     use MAX_PHYSMEM_BITS as the upper limit.
> 
>     This effectively disables huge DMA window in LPAR under pHyp if
>     persistent memory is present but this is the best we can do.
> 
>     Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
>     ---
>      arch/powerpc/platforms/pseries/iommu.c | 9 +++++++++
>      1 file changed, 9 insertions(+)
> 
>     diff --git a/arch/powerpc/platforms/pseries/iommu.c
>     b/arch/powerpc/platforms/pseries/iommu.c
>     index 2e0a8eab5588..6d47b4a3ce39 100644
>     --- a/arch/powerpc/platforms/pseries/iommu.c
>     +++ b/arch/powerpc/platforms/pseries/iommu.c
>     @@ -945,6 +945,15 @@ static phys_addr_t ddw_memory_hotplug_max(void)
>       phys_addr_t max_addr = memory_hotplug_max();
>       struct device_node *memory;
>      
>     + /*
>     + * The "ibm,pmemory" can appear anywhere in the address space.
>     + * Assuming it is still backed by page structs, set the upper limit
>     + * for the huge DMA window as MAX_PHYSMEM_BITS.
>     + */
>     + if (of_find_node_by_type(NULL, "ibm,pmemory"))
>     + return (sizeof(phys_addr_t) * 8 <= MAX_PHYSMEM_BITS) ?
>     + (phys_addr_t) -1 : (1ULL << MAX_PHYSMEM_BITS);
>     +
>       for_each_node_by_type(memory, "memory") {
>       unsigned long start, size;
>       int n_mem_addr_cells, n_mem_size_cells, len;
>     --
>     2.17.1
>      
> 
>  
>
Vaibhav Jain April 3, 2020, 10:04 a.m. UTC | #2
Alexey Kardashevskiy <aik@ozlabs.ru> writes:

> Unlike normal memory ("memory" compatible type in the FDT),
> the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
> in the guest physical space and it can be used for DMA.
>
> In order to maintain 1:1 mapping via the huge DMA window, we need to
> know the maximum physical address at the time of the window setup.
> So far we've been looking at "memory" nodes but "ibm,pmemory" does not
> have fixed addresses and the persistent memory may be mapped afterwards.
>
> Since the persistent memory is still backed with page structs,
> use MAX_PHYSMEM_BITS as the upper limit.
>
> This effectively disables huge DMA window in LPAR under pHyp if
> persistent memory is present but this is the best we can do.
>
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Thanks for the patch Alexy,
LGTM.

Reviewed-by: Vaibhav Jain <vaibhav@linux.ibm.com>

> ---
>  arch/powerpc/platforms/pseries/iommu.c | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
> index 2e0a8eab5588..6d47b4a3ce39 100644
> --- a/arch/powerpc/platforms/pseries/iommu.c
> +++ b/arch/powerpc/platforms/pseries/iommu.c
> @@ -945,6 +945,15 @@ static phys_addr_t ddw_memory_hotplug_max(void)
>  	phys_addr_t max_addr = memory_hotplug_max();
>  	struct device_node *memory;
>
> +	/*
> +	 * The "ibm,pmemory" can appear anywhere in the address space.
> +	 * Assuming it is still backed by page structs, set the upper limit
> +	 * for the huge DMA window as MAX_PHYSMEM_BITS.
> +	 */
> +	if (of_find_node_by_type(NULL, "ibm,pmemory"))
> +		return (sizeof(phys_addr_t) * 8 <= MAX_PHYSMEM_BITS) ?
> +			(phys_addr_t) -1 : (1ULL << MAX_PHYSMEM_BITS);
> +
>  	for_each_node_by_type(memory, "memory") {
>  		unsigned long start, size;
>  		int n_mem_addr_cells, n_mem_size_cells, len;
> -- 
> 2.17.1
>
Michael Ellerman April 6, 2020, 1:05 p.m. UTC | #3
On Tue, 2020-03-31 at 01:23:38 UTC, Alexey Kardashevskiy wrote:
> Unlike normal memory ("memory" compatible type in the FDT),
> the persistent memory ("ibm,pmemory" in the FDT) can be mapped anywhere
> in the guest physical space and it can be used for DMA.
> 
> In order to maintain 1:1 mapping via the huge DMA window, we need to
> know the maximum physical address at the time of the window setup.
> So far we've been looking at "memory" nodes but "ibm,pmemory" does not
> have fixed addresses and the persistent memory may be mapped afterwards.
> 
> Since the persistent memory is still backed with page structs,
> use MAX_PHYSMEM_BITS as the upper limit.
> 
> This effectively disables huge DMA window in LPAR under pHyp if
> persistent memory is present but this is the best we can do.
> 
> Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>

Applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/54fc3c681ded9437e4548e2501dc1136b23cfa9a

cheers
diff mbox series

Patch

diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index 2e0a8eab5588..6d47b4a3ce39 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -945,6 +945,15 @@  static phys_addr_t ddw_memory_hotplug_max(void)
 	phys_addr_t max_addr = memory_hotplug_max();
 	struct device_node *memory;
 
+	/*
+	 * The "ibm,pmemory" can appear anywhere in the address space.
+	 * Assuming it is still backed by page structs, set the upper limit
+	 * for the huge DMA window as MAX_PHYSMEM_BITS.
+	 */
+	if (of_find_node_by_type(NULL, "ibm,pmemory"))
+		return (sizeof(phys_addr_t) * 8 <= MAX_PHYSMEM_BITS) ?
+			(phys_addr_t) -1 : (1ULL << MAX_PHYSMEM_BITS);
+
 	for_each_node_by_type(memory, "memory") {
 		unsigned long start, size;
 		int n_mem_addr_cells, n_mem_size_cells, len;