| Message ID | 20221011231832.149839-6-gshan@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | hw/arm/virt: Improve address assignment for high memory regions | expand |
On Wed, Oct 12 2022, Gavin Shan <gshan@redhat.com> wrote: > There are three high memory regions, which are VIRT_HIGH_REDIST2, > VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses > are floating on highest RAM address. However, they can be disabled > in several cases. > > (1) One specific high memory region is disabled by developer by > toggling vms->highmem_{redists, ecam, mmio}. > > (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is > 'virt-2.12' or ealier than it. > > (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded > on 32-bits system. > > (4) One specific high memory region is disabled when it breaks the > PA space limit. > > The current implementation of virt_set_memmap() isn't comprehensive > because the space for one specific high memory region is always > reserved from the PA space for case (1), (2) and (3). In the code, > 'base' and 'vms->highest_gpa' are always increased for those three > cases. It's unnecessary since the assigned space of the disabled > high memory region won't be used afterwards. > > This improves the address assignment for those three high memory > region by skipping the address assignment for one specific high > memory region if it has been disabled in case (1), (2) and (3). > 'vms->high_compact' is false for now, meaning that we don't have > any behavior changes until it becomes configurable through property > 'compact-highmem' in next patch. > > Signed-off-by: Gavin Shan <gshan@redhat.com> > Tested-by: Zhenyu Zhang <zhenyzha@redhat.com> > --- > hw/arm/virt.c | 23 +++++++++++++++-------- > include/hw/arm/virt.h | 1 + > 2 files changed, 16 insertions(+), 8 deletions(-) Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Hi Gavin On 10/12/22 01:18, Gavin Shan wrote: > There are three high memory regions, which are VIRT_HIGH_REDIST2, > VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses > are floating on highest RAM address. However, they can be disabled > in several cases. > > (1) One specific high memory region is disabled by developer by > toggling vms->highmem_{redists, ecam, mmio}. I would replace the above sentence by One specific high memory region is likely to be disabled by the code by toggling vms->highmem_{redists, ecam, mmio}: > > (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is > 'virt-2.12' or ealier than it. > > (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded > on 32-bits system. > > (4) One specific high memory region is disabled when it breaks the > PA space limit. > > The current implementation of virt_set_memmap() isn't comprehensive > because the space for one specific high memory region is always > reserved from the PA space for case (1), (2) and (3). I would suggest: isn't optimized because the high memory region PA range is always reserved whatever the actual state of the corresponding vms->highmem_ * flag. > In the code, > 'base' and 'vms->highest_gpa' are always increased for those three > cases. It's unnecessary since the assigned space of the disabled > high memory region won't be used afterwards. > > This improves the address assignment for those three high memory s/This improves/Improve > region by skipping the address assignment for one specific high > memory region if it has been disabled in case (1), (2) and (3). > 'vms->high_compact' is false for now, meaning that we don't have s/hight_compat/highmem_compact You also may justify the introduction of this new field. > any behavior changes until it becomes configurable through property > 'compact-highmem' in next patch. > > Signed-off-by: Gavin Shan <gshan@redhat.com> > Tested-by: Zhenyu Zhang <zhenyzha@redhat.com> > --- > hw/arm/virt.c | 23 +++++++++++++++-------- > include/hw/arm/virt.h | 1 + > 2 files changed, 16 insertions(+), 8 deletions(-) > > diff --git a/hw/arm/virt.c b/hw/arm/virt.c > index ee98a8a3b6..c05cfb5314 100644 > --- a/hw/arm/virt.c > +++ b/hw/arm/virt.c > @@ -1717,22 +1717,29 @@ static void virt_set_high_memmap(VirtMachineState *vms, > region_base = ROUND_UP(base, extended_memmap[i].size); > region_size = extended_memmap[i].size; > > - vms->memmap[i].base = region_base; > - vms->memmap[i].size = region_size; > - > /* > * Check each device to see if they fit in the PA space, while we are at it, you can change s/they fit/it fits > - * moving highest_gpa as we go. > + * moving highest_gpa as we go. For compatibility, move > + * highest_gpa for disabled fitting devices as well, if > + * the compact layout has been disabled. > * > * For each device that doesn't fit, disable it. > */ > fits = (region_base + region_size) <= BIT_ULL(pa_bits); > - if (fits) { > + if (*region_enabled && fits) { > + vms->memmap[i].base = region_base; > + vms->memmap[i].size = region_size; > vms->highest_gpa = region_base + region_size - 1; > + base = region_base + region_size; > + } else { > + *region_enabled = false; > + if (!vms->highmem_compact) { > + base = region_base + region_size; > + if (fits) { > + vms->highest_gpa = region_base + region_size - 1; > + } > + } > } > - > - *region_enabled &= fits; > - base = region_base + region_size; > } > } This looks quite complicated to me. It is not obvious for instance we have the same code as before when highmem_compact is not set. Typically vms->memmap[i].base/size are not always set as they were to be and impact on the rest of the code must be double checked. Could this be rewritten in that way (pseudocode totally untested). static void fit_highmem_slot(vms, *base, i, pa_bits) { region_enabled = virt_get_high_memmap_enabled(vms, i); region_base = ROUND_UP(*base, extended_memmap[i].size); region_size = extended_memmap[i].size; fits = (region_base + region_size) <= BIT_ULL(pa_bits); *region_enabled &= fits; vms->memmap[i].base = region_base; vms->memmap[i].size = region_size; /* compact layout only allocates space for the region if this latter is enabled & fits*/ if (vms->highmem_compact && !region_enabled) { return; } /* account for the region and update the base address/highest_gpa if needed*/ *base = region_base + region_size; if (fits) { vms->highest_gpa = *base - 1; } } static void virt_set_high_memmap(VirtMachineState *vms, hwaddr base, int pa_bits) { hwaddr region_base, region_size; bool *region_enabled, fits; int i; for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) { /* we do not break in case the region does not fit since fit_highmem_slot also updates the enabled status of the region */ fit_highmem_slot(vms, &base, i, pa_bits); } } > > diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h > index 6ec479ca2b..709f623741 100644 > --- a/include/hw/arm/virt.h > +++ b/include/hw/arm/virt.h > @@ -144,6 +144,7 @@ struct VirtMachineState { > PFlashCFI01 *flash[2]; > bool secure; > bool highmem; > + bool highmem_compact; > bool highmem_ecam; > bool highmem_mmio; > bool highmem_redists; Thanks Eric
Hi Eric, On 10/20/22 4:07 AM, Eric Auger wrote: > On 10/12/22 01:18, Gavin Shan wrote: >> There are three high memory regions, which are VIRT_HIGH_REDIST2, >> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses >> are floating on highest RAM address. However, they can be disabled >> in several cases. >> >> (1) One specific high memory region is disabled by developer by >> toggling vms->highmem_{redists, ecam, mmio}. > I would replace the above sentence by > > One specific high memory region is likely to be disabled by the code by toggling vms->highmem_{redists, ecam, mmio}: > Ok. >> >> (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is >> 'virt-2.12' or ealier than it. >> >> (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded >> on 32-bits system. >> >> (4) One specific high memory region is disabled when it breaks the >> PA space limit. >> >> The current implementation of virt_set_memmap() isn't comprehensive >> because the space for one specific high memory region is always >> reserved from the PA space for case (1), (2) and (3). > I would suggest: > isn't optimized because the high memory region PA range is always > > reserved whatever the actual state of the corresponding vms->highmem_ > * flag. > Ok. I will have something like below in next revision. The current implementation of virt_set_{memmap, high_memmap}() isn't optimized because the high memory region's PA space is always reserved, regardless of whatever the actual state in the corresponding vms->highmem_{redists, ecam, mmio} flag. In the code, .... >> In the code, >> 'base' and 'vms->highest_gpa' are always increased for those three >> cases. It's unnecessary since the assigned space of the disabled >> high memory region won't be used afterwards. >> >> This improves the address assignment for those three high memory > s/This improves/Improve Ok. >> region by skipping the address assignment for one specific high >> memory region if it has been disabled in case (1), (2) and (3). >> 'vms->high_compact' is false for now, meaning that we don't have > s/hight_compat/highmem_compact > > You also may justify the introduction of this new field. Thanks. It should be 'highmem_compact'. Yes, it makes sense to justify the addition of 'vms->highmem_compact'. I will have something like below in next revision. The memory layout may be changed after the improvement is applied, which leads to potential migration breakage. So 'vms->highmem_compact' is added to control if the improvement should be applied. For now, 'vms->highmem_compact' is set to false, meaning that we don't have memory layout change until it becomes configurable through property 'compact-highmem' in next patch. >> any behavior changes until it becomes configurable through property >> 'compact-highmem' in next patch. >>>> Signed-off-by: Gavin Shan <gshan@redhat.com> >> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com> >> --- >> hw/arm/virt.c | 23 +++++++++++++++-------- >> include/hw/arm/virt.h | 1 + >> 2 files changed, 16 insertions(+), 8 deletions(-) >> >> diff --git a/hw/arm/virt.c b/hw/arm/virt.c >> index ee98a8a3b6..c05cfb5314 100644 >> --- a/hw/arm/virt.c >> +++ b/hw/arm/virt.c >> @@ -1717,22 +1717,29 @@ static void virt_set_high_memmap(VirtMachineState *vms, >> region_base = ROUND_UP(base, extended_memmap[i].size); >> region_size = extended_memmap[i].size; >> >> - vms->memmap[i].base = region_base; >> - vms->memmap[i].size = region_size; >> - >> /* >> * Check each device to see if they fit in the PA space, > while we are at it, you can change s/they fit/it fits Agreed. >> - * moving highest_gpa as we go. >> + * moving highest_gpa as we go. For compatibility, move >> + * highest_gpa for disabled fitting devices as well, if >> + * the compact layout has been disabled. >> * >> * For each device that doesn't fit, disable it. >> */ >> fits = (region_base + region_size) <= BIT_ULL(pa_bits); >> - if (fits) { >> + if (*region_enabled && fits) { >> + vms->memmap[i].base = region_base; >> + vms->memmap[i].size = region_size; >> vms->highest_gpa = region_base + region_size - 1; >> + base = region_base + region_size; >> + } else { >> + *region_enabled = false; >> + if (!vms->highmem_compact) { >> + base = region_base + region_size; >> + if (fits) { >> + vms->highest_gpa = region_base + region_size - 1; >> + } >> + } >> } >> - >> - *region_enabled &= fits; >> - base = region_base + region_size; >> } >> } > This looks quite complicated to me. It is not obvious for instance we > have the same code as before when highmem_compact is not set. Typically > > vms->memmap[i].base/size are not always set as they were to be and impact on the rest of the code must be double checked. > > Could this be rewritten in that way (pseudocode totally untested). > > > static void fit_highmem_slot(vms, *base, i, pa_bits) > { > region_enabled = virt_get_high_memmap_enabled(vms, i); > region_base = ROUND_UP(*base, extended_memmap[i].size); > region_size = extended_memmap[i].size; > fits = (region_base + region_size) <= BIT_ULL(pa_bits); > *region_enabled &= fits; > vms->memmap[i].base = region_base; > vms->memmap[i].size = region_size; > > /* compact layout only allocates space for the region if this latter > is enabled & fits*/ > if (vms->highmem_compact && !region_enabled) { > return; > } > > /* account for the region and update the base address/highest_gpa if > needed*/ > *base = region_base + region_size; > if (fits) { > vms->highest_gpa = *base - 1; > } > } > > static void virt_set_high_memmap(VirtMachineState *vms, > hwaddr base, int pa_bits) > { > hwaddr region_base, region_size; > bool *region_enabled, fits; > int i; > > for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) { > /* we do not break in case the region does not fit since > fit_highmem_slot also updates the enabled status of the region */ > fit_highmem_slot(vms, &base, i, pa_bits); > } > } > >> >> diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h >> index 6ec479ca2b..709f623741 100644 >> --- a/include/hw/arm/virt.h >> +++ b/include/hw/arm/virt.h >> @@ -144,6 +144,7 @@ struct VirtMachineState { >> PFlashCFI01 *flash[2]; >> bool secure; >> bool highmem; >> + bool highmem_compact; >> bool highmem_ecam; >> bool highmem_mmio; >> bool highmem_redists; I checked it again. When the corresponding high memory region is disabled, skipping to setting vms->memmap[i].base/size won't affect the left code. I would avoid introducing another helper here, meaning I want to squeeze everything into existing virt_set_high_memmap(). With additional helper, there are too many function calls in the stack to accomplish this sort of simple task (address assignment for high memory regions). Without the additional helper, the changes can be simply as below. diff --git a/hw/arm/virt.c b/hw/arm/virt.c index ee98a8a3b6..da9e23a8ad 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1721,18 +1721,23 @@ static void virt_set_high_memmap(VirtMachineState *vms, vms->memmap[i].size = region_size; /* - * Check each device to see if they fit in the PA space, - * moving highest_gpa as we go. + * Check each device to see if it fits in the PA space, + * moving highest_gpa as we go. For compatibility, move + * highest_gpa for disabled fitting devices as well, if + * the compact layout has been disabled. * * For each device that doesn't fit, disable it. */ fits = (region_base + region_size) <= BIT_ULL(pa_bits); - if (fits) { - vms->highest_gpa = region_base + region_size - 1; + *region_enabled &= fits; + if (vms->highmem_compact && !*region_enabled) { + continue; } - *region_enabled &= fits; base = region_base + region_size; + if (fits) { + vms->highest_gpa = region_base + region_size - 1; + } } } Thanks, Gavin
diff --git a/hw/arm/virt.c b/hw/arm/virt.c index ee98a8a3b6..c05cfb5314 100644 --- a/hw/arm/virt.c +++ b/hw/arm/virt.c @@ -1717,22 +1717,29 @@ static void virt_set_high_memmap(VirtMachineState *vms, region_base = ROUND_UP(base, extended_memmap[i].size); region_size = extended_memmap[i].size; - vms->memmap[i].base = region_base; - vms->memmap[i].size = region_size; - /* * Check each device to see if they fit in the PA space, - * moving highest_gpa as we go. + * moving highest_gpa as we go. For compatibility, move + * highest_gpa for disabled fitting devices as well, if + * the compact layout has been disabled. * * For each device that doesn't fit, disable it. */ fits = (region_base + region_size) <= BIT_ULL(pa_bits); - if (fits) { + if (*region_enabled && fits) { + vms->memmap[i].base = region_base; + vms->memmap[i].size = region_size; vms->highest_gpa = region_base + region_size - 1; + base = region_base + region_size; + } else { + *region_enabled = false; + if (!vms->highmem_compact) { + base = region_base + region_size; + if (fits) { + vms->highest_gpa = region_base + region_size - 1; + } + } } - - *region_enabled &= fits; - base = region_base + region_size; } } diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h index 6ec479ca2b..709f623741 100644 --- a/include/hw/arm/virt.h +++ b/include/hw/arm/virt.h @@ -144,6 +144,7 @@ struct VirtMachineState { PFlashCFI01 *flash[2]; bool secure; bool highmem; + bool highmem_compact; bool highmem_ecam; bool highmem_mmio; bool highmem_redists;
There are three high memory regions, which are VIRT_HIGH_REDIST2, VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses are floating on highest RAM address. However, they can be disabled in several cases. (1) One specific high memory region is disabled by developer by toggling vms->highmem_{redists, ecam, mmio}. (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is 'virt-2.12' or ealier than it. (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded on 32-bits system. (4) One specific high memory region is disabled when it breaks the PA space limit. The current implementation of virt_set_memmap() isn't comprehensive because the space for one specific high memory region is always reserved from the PA space for case (1), (2) and (3). In the code, 'base' and 'vms->highest_gpa' are always increased for those three cases. It's unnecessary since the assigned space of the disabled high memory region won't be used afterwards. This improves the address assignment for those three high memory region by skipping the address assignment for one specific high memory region if it has been disabled in case (1), (2) and (3). 'vms->high_compact' is false for now, meaning that we don't have any behavior changes until it becomes configurable through property 'compact-highmem' in next patch. Signed-off-by: Gavin Shan <gshan@redhat.com> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com> --- hw/arm/virt.c | 23 +++++++++++++++-------- include/hw/arm/virt.h | 1 + 2 files changed, 16 insertions(+), 8 deletions(-)