[RFC] doc: Document address spaces used by U-Boot

This serves as a reference for developers on how to handle
all those address spaces for U-Boot.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
---
Hi all,

I was trying to clear up long standing myth of virtual
address vs physical adddress mapping on MIPS by having
a clear division betweem virt and phys adddress everywhere.

In this process I would like to confirm my understanding
by writing a document and requesting comments from the
community.

Note that there are some sysmem APIs mentioned in the document
are not in U-Boot tree, namely ``sysmem_addr_t`` and sysmem_to_phys
co. I think they can help with annotating address spaces.

Please kindly comment on the design.

Thanks!
---
 doc/develop/address_spaces.rst | 73 ++++++++++++++++++++++++++++++++++++++++++
 doc/develop/index.rst          |  1 +
 2 files changed, 74 insertions(+)


---
base-commit: ad7dce5abd49ef3b5c93da5303e15449c8c162b4
change-id: 20240517-address-spaces-1c3b9c379b13

Best regards,

On 5/17/24 13:45, Jiaxun Yang wrote:
> This serves as a reference for developers on how to handle
> all those address spaces for U-Boot.
>
> Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
> ---
> Hi all,
>
> I was trying to clear up long standing myth of virtual
> address vs physical adddress mapping on MIPS by having
> a clear division betweem virt and phys adddress everywhere.
>
> In this process I would like to confirm my understanding
> by writing a document and requesting comments from the
> community.
>
> Note that there are some sysmem APIs mentioned in the document
> are not in U-Boot tree, namely ``sysmem_addr_t`` and sysmem_to_phys
> co. I think they can help with annotating address spaces.
>
> Please kindly comment on the design.
>
> Thanks!

Thanks for looking into this.

> ---
>   doc/develop/address_spaces.rst | 73 ++++++++++++++++++++++++++++++++++++++++++
>   doc/develop/index.rst          |  1 +
>   2 files changed, 74 insertions(+)
>
> diff --git a/doc/develop/address_spaces.rst b/doc/develop/address_spaces.rst
> new file mode 100644
> index 000000000000..05c4a1575f27
> --- /dev/null
> +++ b/doc/develop/address_spaces.rst
> @@ -0,0 +1,73 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +.. Copyright (c) 2024 Jiaxun Yang <jiaxun.yang@flygoat.com>
> +
> +Address Spaces
> +==============
> +
> +Introduction
> +------------
> +
> +In U-Boot, we have three address spaces that are used to describe a memory
> +address:
> +
> +    - ``virt``: virtual address space
> +    - ``phys``: physical address space
> +    - ``sysmem``: system memory address space
> +
> +On most architectures, we maintain a 1:1:1 mapping between all three address,
> +with the exception of MIPS and sandbox.
> +
> +There are many API misused in existing code base, this document is to clarify
> +the usage of these address spaces and APIs. Hopefully it will help to reduce
> +the confusion and porting effort.
> +
> +``virt``
> +--------
> +
> +This is the address space that is directly used by U-Boot to access memory.
> +Every pointer in U-Boot must in ``virt`` address space. to get a valid ``virt``

%s/must in/must be in/

%s/to get/To get/

> +address from any other address space, you must use relavant mapping functions

%s/relavant/relevant/

> +to ensure it's being tracked. It is recomanded to use ``ulong`` or pointer types
> +to store ``virt`` address.
> +
> +``phys``
> +--------
> +
> +This is the address space that is used to describe the physical memory address.
> +It's used to describe the physical memory address of a device, or the physical
> +memory address of a memory region. Usual places you would see ``phys`` address
> +are addresses comes from Device Tree and some memory related APIs. It is
> +recommended to use ``fdt_addr_t`` or ``phys_addr_t`` to store ``phys`` address.
> +
> +Following APIs are availble to handle ``phys`` address:

%s/availble/available/

> +
> +    - ``map_physmem()``: Create mapping from physical address to virtual address
> +    - ``unmap_physmem()``: Remove mapping created by ``map_physmem()``
> +    - ``virt_to_phys()``: Find physical address mapping for a virtual address
> +    - ``ioremap()``: Create mapping for I/O memory region
> +    - ``iounmap()``: Remove mapping created by ``ioremap()``


These functions do not reprogram the MMU. In fact these functions are
only used to translate between the sandbox' virtual address space and
the host's virtual address space.

Looking at the sandbox code

static inline void *map_sysmem(phys_addr_t paddr, unsigned long len)
{
         return map_physmem(paddr, len, MAP_WRBACK);
}

there seems to be no difference between sysmem and physmem on the sandbox.

> +
> +``sysmem``
> +----------
> +
> +This is the address space that is used by U-Boot internally to describe RAM
> +address. It helps to abstract some architecture differences and provide a

I would say: It is the address space used in the command line interface
and in environment variables.

It should not be used elsewhere. Specifically these sandbox virtual
addresses must not be used in structures that are passed to external
programs like the kernel or EFI binaries.

> +consistent way to managed RAM. Usual places you would see ``sysmem`` address

The CLI addresses are used for memory mapped devices, too. So 'managed
RAM' is a bit confusing here.

> +are addresses comes from ``gd->bd->bi_dram[]``, addresses comes from ``LMB``
> +addresses being used as memory related config optinos, and addresses being

%s/optinos/options/

Relating to gd->bd->bi_dram[] or LMB just adds a layer of confusion to
the reader. I would not refer to specific internal structures in this
document.

> +used as arguments to load commands. It is recommended to use ``sysmem_addr_t``
> +to store ``sysmem`` address.

In most places long is used.

> +
> +It is further used by sandbox to simulate memory, and by MIPS to handle
> +differences between ``virt`` and ``phys`` address spaces.

You have been submitting patches to implement EFI for MIPS.

The EFI specification explicitly requires a 1:1 mapping in the MMU
between. virtual and physical addresses.

In what respect does MIPS not comply to this?

In how far do MIPS addresses shown in the CLI differ from the physical
addresses (i.e. addresses on the electrical address lines)?

CONFIG_ARCH_MAP_SYSMEM seems only to be set on the sandbox and not on
MIPS. We should try to keep it that way.

> +
> +Following APIs are availble to handle ``sysmem`` address:

%s/availble/available/




To summarize. If I have not mistaken anything about MIPS:

All physical devices are identity mapped.

The sandbox uses a virtual address space incorrectly called 'physical'
which is mapped to the virtual address space of the host. This sandbox
virtual address space is used in

* the command line interface
* in the sandbox device-tree
* in environment variable

This allows using the same address values in sandbox tests irrespective
of what part of the host RAM was mapped via mmap().

Best regards

Heinrich

> +
> +    - ``map_sysmem()``: Create mapping from system memory address to virtual address
> +    - ``unmap_sysmem()``: Remove mapping created by ``map_sysmem()``
> +    - ``map_to_sysmem()``: Find system memory address mapping for a virtual address
> +    - ``sysmem_to_phys()``: Find physical address mapping for a system memory address
> +    - ``phys_to_sysmem()``: Find system memory address mapping for a physical address
> +    - ``nomap_sysmem()``: Pass through an address unchanged (For sandbox tracking)
> +    - ``nomap_to_sysmem()``: Pass through an address unchanged
> +
> diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> index f82e148b101c..1b8e11cd2f24 100644
> --- a/doc/develop/index.rst
> +++ b/doc/develop/index.rst
> @@ -27,6 +27,7 @@ Implementation
>   .. toctree::
>      :maxdepth: 1
>
> +   address_spaces
>      directories
>      bloblist
>      bootstd
>
> ---
> base-commit: ad7dce5abd49ef3b5c93da5303e15449c8c162b4
> change-id: 20240517-address-spaces-1c3b9c379b13
>
> Best regards,

在2024年5月18日五月 上午9:08，Heinrich Schuchardt写道：
> On 5/17/24 13:45, Jiaxun Yang wrote:
[...]

Hi Heinrich,

Ah sorry I should really spell check it before sending it off....
It was copied directly from my personal note.

> %s/optinos/options/
>
> Relating to gd->bd->bi_dram[] or LMB just adds a layer of confusion to
> the reader. I would not refer to specific internal structures in this
> document.

So it's a developer oriented document and I decided to add some examples.

>
>> +used as arguments to load commands. It is recommended to use ``sysmem_addr_t``
>> +to store ``sysmem`` address.
>
> In most places long is used.
>
>> +
>> +It is further used by sandbox to simulate memory, and by MIPS to handle
>> +differences between ``virt`` and ``phys`` address spaces.
>
> You have been submitting patches to implement EFI for MIPS.
>
> The EFI specification explicitly requires a 1:1 mapping in the MMU
> between. virtual and physical addresses.
>
> In what respect does MIPS not comply to this?
>
> In how far do MIPS addresses shown in the CLI differ from the physical
> addresses (i.e. addresses on the electrical address lines)?

So for MIPS we need to apply a fixed offset to physical address to get a
usable virtual address, i.e. 0x1000 physical is 0x80001000 virtual. That
mapping is guaranteed by the hardware, and you can't disable it even after
enabling MMU.

All CLIs are currently using virtual address.

For EFI all table address passed are virtual-ish address.

>
> CONFIG_ARCH_MAP_SYSMEM seems only to be set on the sandbox and not on
> MIPS. We should try to keep it that way.
>

I'm actually trying to introduce ARCH_MAP_SYSMEM abstraction to MIPS
because currently it's totally a mess about virtual vs physical everywhere.

I think eventually we should convert most internal data structure (and cmd
arguments) to use physical address for consistency, but converting all
existing boards to that model cost an arm and a leg. So my plan is to introduce
an option telling if sysmem is virtual or physical and let boards opt-in
that option.

Thanks

>
> To summarize. If I have not mistaken anything about MIPS:
>
> All physical devices are identity mapped.
>
> The sandbox uses a virtual address space incorrectly called 'physical'
> which is mapped to the virtual address space of the host. This sandbox
> virtual address space is used in
>
> * the command line interface
> * in the sandbox device-tree
> * in environment variable
>
> This allows using the same address values in sandbox tests irrespective
> of what part of the host RAM was mapped via mmap().
>
> Best regards
>
> Heinrich
>
[...]