Patchwork [RFC,v2,05/21,SeaBIOS] pciinit: Fix pcimem_start value

login
register
mail settings
Submitter Vasilis Liaskovitis
Date July 11, 2012, 10:31 a.m.
Message ID <1342002726-18258-6-git-send-email-vasilis.liaskovitis@profitbricks.com>
Download mbox | patch
Permalink /patch/170422/
State New
Headers show

Comments

Vasilis Liaskovitis - July 11, 2012, 10:31 a.m.
In order to hotplug memory between RamSize and BUILD_PCIMEM_START, the pci
window needs to start at BUILD_PCIMEM_START (0xe0000000).
Otherwise, the guest cannot online new dimms at those ranges due to pci_root
window conflicts. (workaround for linux guest is booting with pci=nocrs)

Signed-off-by: Vasilis Liaskovitis <vasilis.liaskovitis@profitbricks.com>
---
 src/pciinit.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
Gerd Hoffmann - July 11, 2012, 11:56 a.m.
On 07/11/12 12:31, Vasilis Liaskovitis wrote:
> In order to hotplug memory between RamSize and BUILD_PCIMEM_START, the pci
> window needs to start at BUILD_PCIMEM_START (0xe0000000).
> Otherwise, the guest cannot online new dimms at those ranges due to pci_root
> window conflicts. (workaround for linux guest is booting with pci=nocrs)

>  static void pci_bios_map_devices(struct pci_bus *busses)
>  {
> -    pcimem_start = RamSize;
> +    pcimem_start = BUILD_PCIMEM_START;

It isn't that simple.  For the 32bit pci window it will work, but will
leaves address space unused instead of assigning it to the 32bit pci
window.  For the 64bit pci window it will not work.

You have to walk the dimms and figure what the highest used address is,
for both below-4g and above-4g.  Then fill two variable with it and make
the pci init code use that instead of RamSize and RamSizeOver4G.

cheers,
  Gerd
Vasilis Liaskovitis - July 11, 2012, 4:45 p.m.
Hi,

On Wed, Jul 11, 2012 at 01:56:19PM +0200, Gerd Hoffmann wrote:
> On 07/11/12 12:31, Vasilis Liaskovitis wrote:
> > In order to hotplug memory between RamSize and BUILD_PCIMEM_START, the pci
> > window needs to start at BUILD_PCIMEM_START (0xe0000000).
> > Otherwise, the guest cannot online new dimms at those ranges due to pci_root
> > window conflicts. (workaround for linux guest is booting with pci=nocrs)
> 
> >  static void pci_bios_map_devices(struct pci_bus *busses)
> >  {
> > -    pcimem_start = RamSize;
> > +    pcimem_start = BUILD_PCIMEM_START;
> 
> It isn't that simple.  For the 32bit pci window it will work, but will
> leaves address space unused instead of assigning it to the 32bit pci
> window.  For the 64bit pci window it will not work.
> 
> You have to walk the dimms and figure what the highest used address is,
> for both below-4g and above-4g.  Then fill two variable with it and make
> the pci init code use that instead of RamSize and RamSizeOver4G.

I see. I already have these values values computed in qemu-kvm, so I can pass
them in a paravirt struct, or infer them from the dimm/srat paravirt info that I
already pass to seabios. 

If i understand correctly, we would like the pcimem windows to use the maximum
possible address space (constrained by the exact dimms/ranges which are defined)
instead of leaving unused space.

thanks,

- Vasilis
Gerd Hoffmann - July 12, 2012, 7:22 a.m.
On 07/11/12 18:45, Vasilis Liaskovitis wrote:
> Hi,
> 
> On Wed, Jul 11, 2012 at 01:56:19PM +0200, Gerd Hoffmann wrote:
>> On 07/11/12 12:31, Vasilis Liaskovitis wrote:
>>> In order to hotplug memory between RamSize and BUILD_PCIMEM_START, the pci
>>> window needs to start at BUILD_PCIMEM_START (0xe0000000).
>>> Otherwise, the guest cannot online new dimms at those ranges due to pci_root
>>> window conflicts. (workaround for linux guest is booting with pci=nocrs)
>>
>>>  static void pci_bios_map_devices(struct pci_bus *busses)
>>>  {
>>> -    pcimem_start = RamSize;
>>> +    pcimem_start = BUILD_PCIMEM_START;
>>
>> It isn't that simple.  For the 32bit pci window it will work, but will
>> leaves address space unused instead of assigning it to the 32bit pci
>> window.  For the 64bit pci window it will not work.
>>
>> You have to walk the dimms and figure what the highest used address is,
>> for both below-4g and above-4g.  Then fill two variable with it and make
>> the pci init code use that instead of RamSize and RamSizeOver4G.
> 
> I see. I already have these values values computed in qemu-kvm, so I can pass
> them in a paravirt struct, or infer them from the dimm/srat paravirt info that I
> already pass to seabios. 

I'd suggest to infer from the dimm info, to limit the amout of
information which needs to be passed from qemu to seabios.

> If i understand correctly, we would like the pcimem windows to use the maximum
> possible address space (constrained by the exact dimms/ranges which are defined)
> instead of leaving unused space.

Yes, for the 32bit pci window.

The 64bit pci window is mapped above all memory, and it must likewise
consider defined+unfilled dimms so the start address doesn't collide
with memory hot-plugged above 4G later on.

cheers,
  Gerd
Vasilis Liaskovitis - July 12, 2012, 9:09 a.m.
On Thu, Jul 12, 2012 at 09:22:14AM +0200, Gerd Hoffmann wrote:
> On 07/11/12 18:45, Vasilis Liaskovitis wrote:
> > Hi,
> > 
> > On Wed, Jul 11, 2012 at 01:56:19PM +0200, Gerd Hoffmann wrote:
> >> On 07/11/12 12:31, Vasilis Liaskovitis wrote:
> >>> In order to hotplug memory between RamSize and BUILD_PCIMEM_START, the pci
> >>> window needs to start at BUILD_PCIMEM_START (0xe0000000).
> >>> Otherwise, the guest cannot online new dimms at those ranges due to pci_root
> >>> window conflicts. (workaround for linux guest is booting with pci=nocrs)
> >>
> >>>  static void pci_bios_map_devices(struct pci_bus *busses)
> >>>  {
> >>> -    pcimem_start = RamSize;
> >>> +    pcimem_start = BUILD_PCIMEM_START;
> >>
> >> It isn't that simple.  For the 32bit pci window it will work, but will
> >> leaves address space unused instead of assigning it to the 32bit pci
> >> window.  For the 64bit pci window it will not work.
> >>
> >> You have to walk the dimms and figure what the highest used address is,
> >> for both below-4g and above-4g.  Then fill two variable with it and make
> >> the pci init code use that instead of RamSize and RamSizeOver4G.
> > 
> > I see. I already have these values values computed in qemu-kvm, so I can pass
> > them in a paravirt struct, or infer them from the dimm/srat paravirt info that I
> > already pass to seabios. 
> 
> I'd suggest to infer from the dimm info, to limit the amout of
> information which needs to be passed from qemu to seabios.

ok.Currently dimm info is processed in bios_init_tables(), which is called after
pci_setup(). I 'll see if i can do the processing earlier.

> 
> > If i understand correctly, we would like the pcimem windows to use the maximum
> > possible address space (constrained by the exact dimms/ranges which are defined)
> > instead of leaving unused space.
> 
> Yes, for the 32bit pci window.
> 
> The 64bit pci window is mapped above all memory, and it must likewise
> consider defined+unfilled dimms so the start address doesn't collide
> with memory hot-plugged above 4G later on.

yes, understood.

thanks,

- Vasilis

Patch

diff --git a/src/pciinit.c b/src/pciinit.c
index 68f302a..0b11cbe 100644
--- a/src/pciinit.c
+++ b/src/pciinit.c
@@ -592,7 +592,7 @@  static void pci_region_map_entries(struct pci_bus *busses, struct pci_region *r)
 
 static void pci_bios_map_devices(struct pci_bus *busses)
 {
-    pcimem_start = RamSize;
+    pcimem_start = BUILD_PCIMEM_START;
 
     if (pci_bios_init_root_regions(busses)) {
         struct pci_region r64_mem, r64_pref;