| Message ID | 1426001534-7151-29-git-send-email-marcel@redhat.com |
|---|---|
| State | New |
| Headers | show |
On 03/10/2015 07:42 PM, Michael S. Tsirkin wrote: > On Tue, Mar 10, 2015 at 06:21:14PM +0200, Marcel Apfelbaum wrote: >> On 03/10/2015 05:47 PM, Michael S. Tsirkin wrote: >>> On Tue, Mar 10, 2015 at 05:32:14PM +0200, Marcel Apfelbaum wrote: >>>> Signed-off-by: Marcel Apfelbaum <marcel@redhat.com> >>>> --- >>>> docs/pci_expander_bridge.txt | 52 ++++++++++++++++++++++++++++++++++++++++++++ >>>> 1 file changed, 52 insertions(+) >>>> create mode 100644 docs/pci_expander_bridge.txt >>>> >>>> diff --git a/docs/pci_expander_bridge.txt b/docs/pci_expander_bridge.txt >>>> new file mode 100644 >>>> index 0000000..58bf7a8 >>>> --- /dev/null >>>> +++ b/docs/pci_expander_bridge.txt >>>> @@ -0,0 +1,52 @@ >>>> +PCI EXPANDER BRIDGE (PXB) >>>> +========================= >>>> + >>>> +Description >>>> +=========== >>>> +PXB is a "light-weight" host bridge in the same PCI domain >>>> +as the main host bridge whose purpose is to enable >>>> +the main host bridge to support multiple PCI root buses. >>>> +It is implemented only for i440fx. >>> >>> BTW what makes it i440fx specific? >>> Also, what happens if you try to use it >>> with a different machine type? >> Is is i440fx specific, please look at patch 22/28. >> Also we have a specific check for i440fx, so CRS >> will not be emitted for other machine types. >> >> Thanks, >> Marcel > > In fact it won't work at all. Need to think about it, > maybe we can make it work more generally. > For CRS, should be possible to emit for q35 too? We can make it work, but not on the scope of this series. However, I'll add a IHostBridgeSnoop interface that will make the device work only with associated bus and this will make it less general. Thanks, Marcel > >>> >>>> + >>>> +As opposed to PCI-2-PCI bridge's secondary bus, PXB's bus >>>> +is a primary bus and can be associated with a NUMA node >>>> +(different from the main host bridge) allowing the guest OS >>>> +to recognize the proximity of a pass-through device to >>>> +other resources as RAM and CPUs. >>>> + >>>> +Usage >>>> +===== >>>> +A detailed command line would be: >>>> + >>>> +[qemu-bin + storage options] >>>> +-bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/ >>>> +-m 2G >>>> +-object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 -numa node,nodeid=0,cpus=0,memdev=ram-node0 >>>> +-object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 -numa node,nodeid=1,cpus=1,memdev=ram-node1 >>>> +-device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd >>>> +-device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device e1000,bus=bridge2,addr=0x3 >>>> +-device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive if=none,id=drive0,file=[img] -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 >>>> + >>>> +Here you have: >>>> + - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node in host, but you can and should put it in different host NUMA nodes) >>>> + - a pxb host bridge attached to NUMA 1 with an e1000 behind it >>>> + - a pxb host bridge attached to NUMA 0 with an e1000 behind it >>>> + - a pxb host bridge not attached to any NUMA with a hard drive behind it. >>>> + >>>> +Implementation >>>> +============== >>>> +The PXB is composed by: >>>> +- HostBridge (TYPE_PXB_HOST) >>>> + The host bridge allows to register and query the PXB's rPCI root bus in QEMU. >>>> +- PXBDev(TYPE_PXB_DEVICE) >>>> + It is a regular PCI Device that resides on the piix host-bridge bus and its bus uses the same PCI domain. >>>> + However, the bus behind is exposed through ACPI as a primary PCI bus and starts a new PCI hierarchy. >>>> + The interrupts from devices behind the PXB are routed through this device the same as if it were a >>>> + PCI-2-PCI bridge. The _PRT follows the i440fx model. >>>> +- PCIBridgeDev(TYPE_PCI_BRIDGE_DEV) >>>> + Created automatically as part of init sequence. >>>> + When adding a device to PXB it is attached to the bridge for two reasons: >>>> + - Using the bridge will enable hotplug support >>>> + - All the devices behind the bridge will use bridge's IO/MEM windows compacting >>>> + the PCI address space. >>>> + >>>> -- >>>> 2.1.0
On Mon, Mar 16, 2015 at 02:16:40PM +0200, Marcel Apfelbaum wrote: > On 03/10/2015 07:42 PM, Michael S. Tsirkin wrote: > >On Tue, Mar 10, 2015 at 06:21:14PM +0200, Marcel Apfelbaum wrote: > >>On 03/10/2015 05:47 PM, Michael S. Tsirkin wrote: > >>>On Tue, Mar 10, 2015 at 05:32:14PM +0200, Marcel Apfelbaum wrote: > >>>>Signed-off-by: Marcel Apfelbaum <marcel@redhat.com> > >>>>--- > >>>> docs/pci_expander_bridge.txt | 52 ++++++++++++++++++++++++++++++++++++++++++++ > >>>> 1 file changed, 52 insertions(+) > >>>> create mode 100644 docs/pci_expander_bridge.txt > >>>> > >>>>diff --git a/docs/pci_expander_bridge.txt b/docs/pci_expander_bridge.txt > >>>>new file mode 100644 > >>>>index 0000000..58bf7a8 > >>>>--- /dev/null > >>>>+++ b/docs/pci_expander_bridge.txt > >>>>@@ -0,0 +1,52 @@ > >>>>+PCI EXPANDER BRIDGE (PXB) > >>>>+========================= > >>>>+ > >>>>+Description > >>>>+=========== > >>>>+PXB is a "light-weight" host bridge in the same PCI domain > >>>>+as the main host bridge whose purpose is to enable > >>>>+the main host bridge to support multiple PCI root buses. > >>>>+It is implemented only for i440fx. > >>> > >>>BTW what makes it i440fx specific? > >>>Also, what happens if you try to use it > >>>with a different machine type? > >>Is is i440fx specific, please look at patch 22/28. > >>Also we have a specific check for i440fx, so CRS > >>will not be emitted for other machine types. > >> > >>Thanks, > >>Marcel > > > >In fact it won't work at all. Need to think about it, > >maybe we can make it work more generally. > >For CRS, should be possible to emit for q35 too? > We can make it work, but not on the scope of this series. > However, I'll add a IHostBridgeSnoop interface that will > make the device work only with associated bus and > this will make it less general. > > Thanks, > Marcel OK, this works too. Do you mean PCIHostBridgeSnoop? > > > >>> > >>>>+ > >>>>+As opposed to PCI-2-PCI bridge's secondary bus, PXB's bus > >>>>+is a primary bus and can be associated with a NUMA node > >>>>+(different from the main host bridge) allowing the guest OS > >>>>+to recognize the proximity of a pass-through device to > >>>>+other resources as RAM and CPUs. > >>>>+ > >>>>+Usage > >>>>+===== > >>>>+A detailed command line would be: > >>>>+ > >>>>+[qemu-bin + storage options] > >>>>+-bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/ > >>>>+-m 2G > >>>>+-object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 -numa node,nodeid=0,cpus=0,memdev=ram-node0 > >>>>+-object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 -numa node,nodeid=1,cpus=1,memdev=ram-node1 > >>>>+-device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd > >>>>+-device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device e1000,bus=bridge2,addr=0x3 > >>>>+-device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive if=none,id=drive0,file=[img] -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 > >>>>+ > >>>>+Here you have: > >>>>+ - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node in host, but you can and should put it in different host NUMA nodes) > >>>>+ - a pxb host bridge attached to NUMA 1 with an e1000 behind it > >>>>+ - a pxb host bridge attached to NUMA 0 with an e1000 behind it > >>>>+ - a pxb host bridge not attached to any NUMA with a hard drive behind it. > >>>>+ > >>>>+Implementation > >>>>+============== > >>>>+The PXB is composed by: > >>>>+- HostBridge (TYPE_PXB_HOST) > >>>>+ The host bridge allows to register and query the PXB's rPCI root bus in QEMU. > >>>>+- PXBDev(TYPE_PXB_DEVICE) > >>>>+ It is a regular PCI Device that resides on the piix host-bridge bus and its bus uses the same PCI domain. > >>>>+ However, the bus behind is exposed through ACPI as a primary PCI bus and starts a new PCI hierarchy. > >>>>+ The interrupts from devices behind the PXB are routed through this device the same as if it were a > >>>>+ PCI-2-PCI bridge. The _PRT follows the i440fx model. > >>>>+- PCIBridgeDev(TYPE_PCI_BRIDGE_DEV) > >>>>+ Created automatically as part of init sequence. > >>>>+ When adding a device to PXB it is attached to the bridge for two reasons: > >>>>+ - Using the bridge will enable hotplug support > >>>>+ - All the devices behind the bridge will use bridge's IO/MEM windows compacting > >>>>+ the PCI address space. > >>>>+ > >>>>-- > >>>>2.1.0
On 03/16/2015 05:28 PM, Michael S. Tsirkin wrote: > On Mon, Mar 16, 2015 at 02:16:40PM +0200, Marcel Apfelbaum wrote: >> On 03/10/2015 07:42 PM, Michael S. Tsirkin wrote: >>> On Tue, Mar 10, 2015 at 06:21:14PM +0200, Marcel Apfelbaum wrote: >>>> On 03/10/2015 05:47 PM, Michael S. Tsirkin wrote: >>>>> On Tue, Mar 10, 2015 at 05:32:14PM +0200, Marcel Apfelbaum wrote: >>>>>> Signed-off-by: Marcel Apfelbaum <marcel@redhat.com> >>>>>> --- >>>>>> docs/pci_expander_bridge.txt | 52 ++++++++++++++++++++++++++++++++++++++++++++ >>>>>> 1 file changed, 52 insertions(+) >>>>>> create mode 100644 docs/pci_expander_bridge.txt >>>>>> >>>>>> diff --git a/docs/pci_expander_bridge.txt b/docs/pci_expander_bridge.txt >>>>>> new file mode 100644 >>>>>> index 0000000..58bf7a8 >>>>>> --- /dev/null >>>>>> +++ b/docs/pci_expander_bridge.txt >>>>>> @@ -0,0 +1,52 @@ >>>>>> +PCI EXPANDER BRIDGE (PXB) >>>>>> +========================= >>>>>> + >>>>>> +Description >>>>>> +=========== >>>>>> +PXB is a "light-weight" host bridge in the same PCI domain >>>>>> +as the main host bridge whose purpose is to enable >>>>>> +the main host bridge to support multiple PCI root buses. >>>>>> +It is implemented only for i440fx. >>>>> >>>>> BTW what makes it i440fx specific? >>>>> Also, what happens if you try to use it >>>>> with a different machine type? >>>> Is is i440fx specific, please look at patch 22/28. >>>> Also we have a specific check for i440fx, so CRS >>>> will not be emitted for other machine types. >>>> >>>> Thanks, >>>> Marcel >>> >>> In fact it won't work at all. Need to think about it, >>> maybe we can make it work more generally. >>> For CRS, should be possible to emit for q35 too? >> We can make it work, but not on the scope of this series. >> However, I'll add a IHostBridgeSnoop interface that will >> make the device work only with associated bus and >> this will make it less general. >> >> Thanks, >> Marcel > > OK, this works too. Do you mean PCIHostBridgeSnoop? Sure. Thanks, Marcel > >>> >>>>> >>>>>> + >>>>>> +As opposed to PCI-2-PCI bridge's secondary bus, PXB's bus >>>>>> +is a primary bus and can be associated with a NUMA node >>>>>> +(different from the main host bridge) allowing the guest OS >>>>>> +to recognize the proximity of a pass-through device to >>>>>> +other resources as RAM and CPUs. >>>>>> + >>>>>> +Usage >>>>>> +===== >>>>>> +A detailed command line would be: >>>>>> + >>>>>> +[qemu-bin + storage options] >>>>>> +-bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/ >>>>>> +-m 2G >>>>>> +-object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 -numa node,nodeid=0,cpus=0,memdev=ram-node0 >>>>>> +-object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 -numa node,nodeid=1,cpus=1,memdev=ram-node1 >>>>>> +-device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd >>>>>> +-device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device e1000,bus=bridge2,addr=0x3 >>>>>> +-device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive if=none,id=drive0,file=[img] -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 >>>>>> + >>>>>> +Here you have: >>>>>> + - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node in host, but you can and should put it in different host NUMA nodes) >>>>>> + - a pxb host bridge attached to NUMA 1 with an e1000 behind it >>>>>> + - a pxb host bridge attached to NUMA 0 with an e1000 behind it >>>>>> + - a pxb host bridge not attached to any NUMA with a hard drive behind it. >>>>>> + >>>>>> +Implementation >>>>>> +============== >>>>>> +The PXB is composed by: >>>>>> +- HostBridge (TYPE_PXB_HOST) >>>>>> + The host bridge allows to register and query the PXB's rPCI root bus in QEMU. >>>>>> +- PXBDev(TYPE_PXB_DEVICE) >>>>>> + It is a regular PCI Device that resides on the piix host-bridge bus and its bus uses the same PCI domain. >>>>>> + However, the bus behind is exposed through ACPI as a primary PCI bus and starts a new PCI hierarchy. >>>>>> + The interrupts from devices behind the PXB are routed through this device the same as if it were a >>>>>> + PCI-2-PCI bridge. The _PRT follows the i440fx model. >>>>>> +- PCIBridgeDev(TYPE_PCI_BRIDGE_DEV) >>>>>> + Created automatically as part of init sequence. >>>>>> + When adding a device to PXB it is attached to the bridge for two reasons: >>>>>> + - Using the bridge will enable hotplug support >>>>>> + - All the devices behind the bridge will use bridge's IO/MEM windows compacting >>>>>> + the PCI address space. >>>>>> + >>>>>> -- >>>>>> 2.1.0
diff --git a/docs/pci_expander_bridge.txt b/docs/pci_expander_bridge.txt new file mode 100644 index 0000000..58bf7a8 --- /dev/null +++ b/docs/pci_expander_bridge.txt @@ -0,0 +1,52 @@ +PCI EXPANDER BRIDGE (PXB) +========================= + +Description +=========== +PXB is a "light-weight" host bridge in the same PCI domain +as the main host bridge whose purpose is to enable +the main host bridge to support multiple PCI root buses. +It is implemented only for i440fx. + +As opposed to PCI-2-PCI bridge's secondary bus, PXB's bus +is a primary bus and can be associated with a NUMA node +(different from the main host bridge) allowing the guest OS +to recognize the proximity of a pass-through device to +other resources as RAM and CPUs. + +Usage +===== +A detailed command line would be: + +[qemu-bin + storage options] +-bios [seabios-dir]/out/bios.bin -L [seabios-dir]/out/ +-m 2G +-object memory-backend-ram,size=1024M,policy=bind,host-nodes=0,id=ram-node0 -numa node,nodeid=0,cpus=0,memdev=ram-node0 +-object memory-backend-ram,size=1024M,policy=interleave,host-nodes=0,id=ram-node1 -numa node,nodeid=1,cpus=1,memdev=ram-node1 +-device pxb-device,id=bridge1,bus=pci.0,numa_node=1,bus_nr=4 -netdev user,id=nd-device e1000,bus=bridge1,addr=0x4,netdev=nd +-device pxb-device,id=bridge2,bus=pci.0,numa_node=0,bus_nr=8 -device e1000,bus=bridge2,addr=0x3 +-device pxb-device,id=bridge3,bus=pci.0,bus_nr=40 -drive if=none,id=drive0,file=[img] -device virtio-blk-pci,drive=drive0,scsi=off,bus=bridge3,addr=1 + +Here you have: + - 2 NUMA nodes for the guest, 0 and 1. (both mapped to the same NUMA node in host, but you can and should put it in different host NUMA nodes) + - a pxb host bridge attached to NUMA 1 with an e1000 behind it + - a pxb host bridge attached to NUMA 0 with an e1000 behind it + - a pxb host bridge not attached to any NUMA with a hard drive behind it. + +Implementation +============== +The PXB is composed by: +- HostBridge (TYPE_PXB_HOST) + The host bridge allows to register and query the PXB's rPCI root bus in QEMU. +- PXBDev(TYPE_PXB_DEVICE) + It is a regular PCI Device that resides on the piix host-bridge bus and its bus uses the same PCI domain. + However, the bus behind is exposed through ACPI as a primary PCI bus and starts a new PCI hierarchy. + The interrupts from devices behind the PXB are routed through this device the same as if it were a + PCI-2-PCI bridge. The _PRT follows the i440fx model. +- PCIBridgeDev(TYPE_PCI_BRIDGE_DEV) + Created automatically as part of init sequence. + When adding a device to PXB it is attached to the bridge for two reasons: + - Using the bridge will enable hotplug support + - All the devices behind the bridge will use bridge's IO/MEM windows compacting + the PCI address space. +
Signed-off-by: Marcel Apfelbaum <marcel@redhat.com> --- docs/pci_expander_bridge.txt | 52 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 docs/pci_expander_bridge.txt