| Message ID | 1317360376-12090-1-git-send-email-david@gibson.dropbear.id.au |
|---|---|
| State | New |
| Headers | show |
On 09/30/2011 12:26 AM, David Gibson wrote: > Currently, virtio devices are usually presented to the guest as an > emulated PCI device, virtio_pci. Although the actual IO operations > are done through system memory, the configuration of the virtio device > is done through the one PCI IO space BAR that virtio_pci presents. > > But PCI IO space (aka PIO) is deprecated for modern PCI devices, and > on some systems with many PCI domains accessing PIO space can be > problematic. For example on the existing PowerVM implementation of > the PAPR spec, PCI PIO access is not supported at all. We're hoping > that our KVM implementation will support PCI PIO (once we support PCI > at all), but it will probably have some irritating limitations. > > This patch, therefore, extends the virtio_pci device to have a PCI > memory space (MMIO) BAR as well as the IO BAR. The MMIO BAR contains > exactly the same registers, in exactly the same layout as the existing > PIO BAR. > > Because the PIO BAR is still present, existing guest drivers should > still work fine. With this change in place, future guest drivers can > check for an MMIO BAR and use that if present (falling back to PIO > when possible to support older qemu versions). > > Signed-off-by: David Gibson<david@gibson.dropbear.id.au> Seems harmless for QEMU, so applied. You should update the virtio-pci spec too. Regards, Anthony Liguori > --- > hw/virtio-pci.c | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > diff --git a/hw/virtio-pci.c b/hw/virtio-pci.c > index df27c19..68cd5bc 100644 > --- a/hw/virtio-pci.c > +++ b/hw/virtio-pci.c > @@ -659,6 +659,8 @@ void virtio_init_pci(VirtIOPCIProxy *proxy, VirtIODevice *vdev) > "virtio-pci", size); > pci_register_bar(&proxy->pci_dev, 0, PCI_BASE_ADDRESS_SPACE_IO, > &proxy->bar); > + pci_register_bar(&proxy->pci_dev, 2, PCI_BASE_ADDRESS_SPACE_MEMORY, > +&proxy->bar); > > if (!kvm_has_many_ioeventfds()) { > proxy->flags&= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
On Tue, Nov 01, 2011 at 05:20:06PM -0500, Anthony Liguori wrote: > On 09/30/2011 12:26 AM, David Gibson wrote: > >Currently, virtio devices are usually presented to the guest as an > >emulated PCI device, virtio_pci. Although the actual IO operations > >are done through system memory, the configuration of the virtio device > >is done through the one PCI IO space BAR that virtio_pci presents. > > > >But PCI IO space (aka PIO) is deprecated for modern PCI devices, and > >on some systems with many PCI domains accessing PIO space can be > >problematic. For example on the existing PowerVM implementation of > >the PAPR spec, PCI PIO access is not supported at all. We're hoping > >that our KVM implementation will support PCI PIO (once we support PCI > >at all), but it will probably have some irritating limitations. > > > >This patch, therefore, extends the virtio_pci device to have a PCI > >memory space (MMIO) BAR as well as the IO BAR. The MMIO BAR contains > >exactly the same registers, in exactly the same layout as the existing > >PIO BAR. > > > >Because the PIO BAR is still present, existing guest drivers should > >still work fine. With this change in place, future guest drivers can > >check for an MMIO BAR and use that if present (falling back to PIO > >when possible to support older qemu versions). > > > >Signed-off-by: David Gibson<david@gibson.dropbear.id.au> > > Seems harmless for QEMU, so applied. You should update the > virtio-pci spec too. Ugh, sorry. This is the old version of the patch which breaks when you actually use it because it attempts to give proxy->bar two different parents. I have a new version which does it correctly. So, do you want to revert and reapply, or should I just make a fix patch?
On 11/01/2011 07:16 PM, David Gibson wrote: > On Tue, Nov 01, 2011 at 05:20:06PM -0500, Anthony Liguori wrote: >> On 09/30/2011 12:26 AM, David Gibson wrote: >>> Currently, virtio devices are usually presented to the guest as an >>> emulated PCI device, virtio_pci. Although the actual IO operations >>> are done through system memory, the configuration of the virtio device >>> is done through the one PCI IO space BAR that virtio_pci presents. >>> >>> But PCI IO space (aka PIO) is deprecated for modern PCI devices, and >>> on some systems with many PCI domains accessing PIO space can be >>> problematic. For example on the existing PowerVM implementation of >>> the PAPR spec, PCI PIO access is not supported at all. We're hoping >>> that our KVM implementation will support PCI PIO (once we support PCI >>> at all), but it will probably have some irritating limitations. >>> >>> This patch, therefore, extends the virtio_pci device to have a PCI >>> memory space (MMIO) BAR as well as the IO BAR. The MMIO BAR contains >>> exactly the same registers, in exactly the same layout as the existing >>> PIO BAR. >>> >>> Because the PIO BAR is still present, existing guest drivers should >>> still work fine. With this change in place, future guest drivers can >>> check for an MMIO BAR and use that if present (falling back to PIO >>> when possible to support older qemu versions). >>> >>> Signed-off-by: David Gibson<david@gibson.dropbear.id.au> >> >> Seems harmless for QEMU, so applied. You should update the >> virtio-pci spec too. > > Ugh, sorry. This is the old version of the patch which breaks when > you actually use it because it attempts to give proxy->bar two > different parents. I have a new version which does it correctly. > > So, do you want to revert and reapply, or should I just make a fix > patch? I'll revert. Thanks. Regards, Anthony Liguori
On Tue, 01 Nov 2011 17:20:06 -0500, Anthony Liguori <aliguori@us.ibm.com> wrote: > On 09/30/2011 12:26 AM, David Gibson wrote: > > Currently, virtio devices are usually presented to the guest as an > > emulated PCI device, virtio_pci. Although the actual IO operations > > are done through system memory, the configuration of the virtio device > > is done through the one PCI IO space BAR that virtio_pci presents. > > > > But PCI IO space (aka PIO) is deprecated for modern PCI devices, and > > on some systems with many PCI domains accessing PIO space can be > > problematic. For example on the existing PowerVM implementation of > > the PAPR spec, PCI PIO access is not supported at all. We're hoping > > that our KVM implementation will support PCI PIO (once we support PCI > > at all), but it will probably have some irritating limitations. > > > > This patch, therefore, extends the virtio_pci device to have a PCI > > memory space (MMIO) BAR as well as the IO BAR. The MMIO BAR contains > > exactly the same registers, in exactly the same layout as the existing > > PIO BAR. > > > > Because the PIO BAR is still present, existing guest drivers should > > still work fine. With this change in place, future guest drivers can > > check for an MMIO BAR and use that if present (falling back to PIO > > when possible to support older qemu versions). > > > > Signed-off-by: David Gibson<david@gibson.dropbear.id.au> > > Seems harmless for QEMU, so applied. You should update the virtio-pci spec too. > > Regards, Can you revert this? We just reverted the kernel part, figuring we need a rethink. Thanks, Rusty.
On 11/01/2011 10:22 PM, Rusty Russell wrote: > On Tue, 01 Nov 2011 17:20:06 -0500, Anthony Liguori<aliguori@us.ibm.com> wrote: >> On 09/30/2011 12:26 AM, David Gibson wrote: >>> Currently, virtio devices are usually presented to the guest as an >>> emulated PCI device, virtio_pci. Although the actual IO operations >>> are done through system memory, the configuration of the virtio device >>> is done through the one PCI IO space BAR that virtio_pci presents. >>> >>> But PCI IO space (aka PIO) is deprecated for modern PCI devices, and >>> on some systems with many PCI domains accessing PIO space can be >>> problematic. For example on the existing PowerVM implementation of >>> the PAPR spec, PCI PIO access is not supported at all. We're hoping >>> that our KVM implementation will support PCI PIO (once we support PCI >>> at all), but it will probably have some irritating limitations. >>> >>> This patch, therefore, extends the virtio_pci device to have a PCI >>> memory space (MMIO) BAR as well as the IO BAR. The MMIO BAR contains >>> exactly the same registers, in exactly the same layout as the existing >>> PIO BAR. >>> >>> Because the PIO BAR is still present, existing guest drivers should >>> still work fine. With this change in place, future guest drivers can >>> check for an MMIO BAR and use that if present (falling back to PIO >>> when possible to support older qemu versions). >>> >>> Signed-off-by: David Gibson<david@gibson.dropbear.id.au> >> >> Seems harmless for QEMU, so applied. You should update the virtio-pci spec too. >> >> Regards, > > Can you revert this? We just reverted the kernel part, figuring we need > a rethink. Already did. Regards, Anthony Liguori > > Thanks, > Rusty. >
On Wed, Nov 02, 2011 at 01:52:55PM +1030, Rusty Russell wrote: > On Tue, 01 Nov 2011 17:20:06 -0500, Anthony Liguori <aliguori@us.ibm.com> wrote: > > On 09/30/2011 12:26 AM, David Gibson wrote: > > > Currently, virtio devices are usually presented to the guest as an > > > emulated PCI device, virtio_pci. Although the actual IO operations > > > are done through system memory, the configuration of the virtio device > > > is done through the one PCI IO space BAR that virtio_pci presents. > > > > > > But PCI IO space (aka PIO) is deprecated for modern PCI devices, and > > > on some systems with many PCI domains accessing PIO space can be > > > problematic. For example on the existing PowerVM implementation of > > > the PAPR spec, PCI PIO access is not supported at all. We're hoping > > > that our KVM implementation will support PCI PIO (once we support PCI > > > at all), but it will probably have some irritating limitations. > > > > > > This patch, therefore, extends the virtio_pci device to have a PCI > > > memory space (MMIO) BAR as well as the IO BAR. The MMIO BAR contains > > > exactly the same registers, in exactly the same layout as the existing > > > PIO BAR. > > > > > > Because the PIO BAR is still present, existing guest drivers should > > > still work fine. With this change in place, future guest drivers can > > > check for an MMIO BAR and use that if present (falling back to PIO > > > when possible to support older qemu versions). > > > > > > Signed-off-by: David Gibson<david@gibson.dropbear.id.au> > > > > Seems harmless for QEMU, so applied. You should update the virtio-pci spec too. > > > > Regards, > > Can you revert this? We just reverted the kernel part, figuring we need > a rethink. > > Thanks, > Rusty. By the way, it looks like linux guests already do pci_iomap to get at the BAR. And that is supposed to work fine with both memory and IO. So a very simple solution, without touching linux guests, would be to allow changing BAR 0 type to memory in QEMU. It'll break non-linux guests unfortunately, but maybe that's not an issue for PowerVM?
On 11/02/2011 12:20 AM, Anthony Liguori wrote: > > Seems harmless for QEMU, so applied. You should update the virtio-pci > spec too. Should be the other way round.
On 11/03/2011 05:36 AM, Avi Kivity wrote: > On 11/02/2011 12:20 AM, Anthony Liguori wrote: >> >> Seems harmless for QEMU, so applied. You should update the virtio-pci >> spec too. > > Should be the other way round. Am not entirely sure. Having worked code that's been reviewed will make for a better spec. Writing the spec and committing to the spec change before getting either side of the implementation merged seems to be asking for trouble to me. We could use a better agreement on the processor for making virtio changes. Should it go (1) virtio spec (2) kernel (3) qemu, or should it go (2), (1), (3)? Regards, Anthony Liguori >
On 11/03/2011 03:07 PM, Anthony Liguori wrote: > On 11/03/2011 05:36 AM, Avi Kivity wrote: >> On 11/02/2011 12:20 AM, Anthony Liguori wrote: >>> >>> Seems harmless for QEMU, so applied. You should update the virtio-pci >>> spec too. >> >> Should be the other way round. > > Am not entirely sure. Having worked code that's been reviewed will > make for a better spec. Writing the spec and committing to the spec > change before getting either side of the implementation merged seems > to be asking for trouble to me. Maybe. But committed code before a spec proposal? I can see the spec maintainer requiring prototype code (for both guest and host). But committing code before a spec change is even seen is not the right way to do this. > We could use a better agreement on the processor for making virtio > changes. Should it go (1) virtio spec (2) kernel (3) qemu, or should > it go (2), (1), (3)? 1. Informal discussion 2. Proposed spec patch, kernel change, qemu change 3. Buy-ins from spec maintainer, kernel driver maintainer, qemu device maintainer (only regarding the ABI, not the code) 4. Spec change committed 5. kernel and qemu code submitted and committed through the normal process Note there are multiple implementations of the device code (qemu virtio, linux vhost) and driver code (linux, windows). The only real synchronization point is the spec.
On 11/03/2011 08:13 AM, Avi Kivity wrote: > On 11/03/2011 03:07 PM, Anthony Liguori wrote: >> On 11/03/2011 05:36 AM, Avi Kivity wrote: >>> On 11/02/2011 12:20 AM, Anthony Liguori wrote: >>>> >>>> Seems harmless for QEMU, so applied. You should update the virtio-pci >>>> spec too. >>> >>> Should be the other way round. >> >> Am not entirely sure. Having worked code that's been reviewed will >> make for a better spec. Writing the spec and committing to the spec >> change before getting either side of the implementation merged seems >> to be asking for trouble to me. > > Maybe. But committed code before a spec proposal? > > I can see the spec maintainer requiring prototype code (for both guest > and host). But committing code before a spec change is even seen is not > the right way to do this. > >> We could use a better agreement on the processor for making virtio >> changes. Should it go (1) virtio spec (2) kernel (3) qemu, or should >> it go (2), (1), (3)? > > 1. Informal discussion Where? Is this lkml? There were a number of virtio changes recently that never involved qemu-devel. > 2. Proposed spec patch, kernel change, qemu change > 3. Buy-ins from spec maintainer, kernel driver maintainer, qemu device > maintainer (only regarding the ABI, not the code) I don't think this is how it's working today. I would be happy with a flow like this. Regards, Anthony Liguori > 4. Spec change committed > 5. kernel and qemu code submitted and committed through the normal process > > Note there are multiple implementations of the device code (qemu virtio, > linux vhost) and driver code (linux, windows). The only real > synchronization point is the spec. >
On 11/03/2011 03:38 PM, Anthony Liguori wrote: >> >>> We could use a better agreement on the processor for making virtio >>> changes. Should it go (1) virtio spec (2) kernel (3) qemu, or should >>> it go (2), (1), (3)? >> >> 1. Informal discussion > > > Where? Is this lkml? There were a number of virtio changes recently > that never involved qemu-devel. Theoretically, virtualization@lists.linux-foundation.org, if it still exists. Maybe we need a virtio list. qemu-devel@, kvm@, lkml could be copied. The point is that we can't drive virtio from either qemu or the kernel any more. The spec represents the "virtual hardware manufacturer", which qemu and linux/vhost (and others) emulate, and which linux (and others) write drivers for. > >> 2. Proposed spec patch, kernel change, qemu change >> 3. Buy-ins from spec maintainer, kernel driver maintainer, qemu device >> maintainer (only regarding the ABI, not the code) > > I don't think this is how it's working today. I would be happy with a > flow like this. If Michael and Rusty agree, we can adopt it immediately.
On 11/03/2011 08:45 AM, Avi Kivity wrote: > On 11/03/2011 03:38 PM, Anthony Liguori wrote: >>> >>>> We could use a better agreement on the processor for making virtio >>>> changes. Should it go (1) virtio spec (2) kernel (3) qemu, or should >>>> it go (2), (1), (3)? >>> >>> 1. Informal discussion >> >> >> Where? Is this lkml? There were a number of virtio changes recently >> that never involved qemu-devel. > > Theoretically, virtualization@lists.linux-foundation.org, if it still > exists. Maybe we need a virtio list. qemu-devel@, kvm@, lkml could be > copied. Perhaps it's time to create a virtio@vger? Just have a simple process that all spec changes to there the appropriate kernel, QEMU, virtio-win, or NKT maintainers can require any virtio change to also have a committed spec change first. > The point is that we can't drive virtio from either qemu or the kernel > any more. The spec represents the "virtual hardware manufacturer", > which qemu and linux/vhost (and others) emulate, and which linux (and > others) write drivers for. Yup. We need to be more rigorous about using the spec for that as we've not done a great job historically here. Regards, Anthony Liguori > >> >>> 2. Proposed spec patch, kernel change, qemu change >>> 3. Buy-ins from spec maintainer, kernel driver maintainer, qemu device >>> maintainer (only regarding the ABI, not the code) >> >> I don't think this is how it's working today. I would be happy with a >> flow like this. > > If Michael and Rusty agree, we can adopt it immediately. >
On Thu, Nov 03, 2011 at 08:49:31AM -0500, Anthony Liguori wrote: > On 11/03/2011 08:45 AM, Avi Kivity wrote: > >On 11/03/2011 03:38 PM, Anthony Liguori wrote: > >>> > >>>>We could use a better agreement on the processor for making virtio > >>>>changes. Should it go (1) virtio spec (2) kernel (3) qemu, or should > >>>>it go (2), (1), (3)? > >>> > >>>1. Informal discussion > >> > >> > >>Where? Is this lkml? There were a number of virtio changes recently > >>that never involved qemu-devel. > > > >Theoretically, virtualization@lists.linux-foundation.org, if it still > >exists. Maybe we need a virtio list. qemu-devel@, kvm@, lkml could be > >copied. > > Perhaps it's time to create a virtio@vger? Just have a simple > process that all spec changes to there the appropriate kernel, QEMU, > virtio-win, or NKT maintainers can require any virtio change to also > have a committed spec change first. Historically virtualization@lists.linux-foundation.org was used, that is still low enough volume. It was down but seems to be up now. It's easy enough for me to subscribe to yet another list but my concern is that if we move we might loose some people that don't notice the change. Anything wrong with just using the old list? It's on the MAINTAINERS file, I'm not sure why it was bypassed in this case. Maybe linux-foundation.org was still down when the patch was posted? > >The point is that we can't drive virtio from either qemu or the kernel > >any more. The spec represents the "virtual hardware manufacturer", > >which qemu and linux/vhost (and others) emulate, and which linux (and > >others) write drivers for. > > Yup. We need to be more rigorous about using the spec for that as > we've not done a great job historically here. > > Regards, > > Anthony Liguori > > > > >> > >>>2. Proposed spec patch, kernel change, qemu change > >>>3. Buy-ins from spec maintainer, kernel driver maintainer, qemu device > >>>maintainer (only regarding the ABI, not the code) > >> > >>I don't think this is how it's working today. I would be happy with a > >>flow like this. > > > >If Michael and Rusty agree, we can adopt it immediately. > > If I understand the proposal, what is suggested is that all of spec, kvm and virtio patches are posted on list and acked before merging any one of them? Sure, this makes sense.
On 11/03/2011 09:31 AM, Michael S. Tsirkin wrote: > On Thu, Nov 03, 2011 at 08:49:31AM -0500, Anthony Liguori wrote: >> On 11/03/2011 08:45 AM, Avi Kivity wrote: >>> On 11/03/2011 03:38 PM, Anthony Liguori wrote: >>>>> >>>>>> We could use a better agreement on the processor for making virtio >>>>>> changes. Should it go (1) virtio spec (2) kernel (3) qemu, or should >>>>>> it go (2), (1), (3)? >>>>> >>>>> 1. Informal discussion >>>> >>>> >>>> Where? Is this lkml? There were a number of virtio changes recently >>>> that never involved qemu-devel. >>> >>> Theoretically, virtualization@lists.linux-foundation.org, if it still >>> exists. Maybe we need a virtio list. qemu-devel@, kvm@, lkml could be >>> copied. >> >> Perhaps it's time to create a virtio@vger? Just have a simple >> process that all spec changes to there the appropriate kernel, QEMU, >> virtio-win, or NKT maintainers can require any virtio change to also >> have a committed spec change first. > > Historically virtualization@lists.linux-foundation.org was used, > that is still low enough volume. It was down but seems to be > up now. It's easy enough for me to subscribe to yet > another list but my concern is that if we move we might loose some > people that don't notice the change. > > Anything wrong with just using the old list? No, but it's so old, I have no idea who admin's it these days. > It's on the MAINTAINERS file, I'm not sure why it was bypassed in this case. > Maybe linux-foundation.org was still down when the patch was posted? The list has been down for a while. >>>>> 2. Proposed spec patch, kernel change, qemu change >>>>> 3. Buy-ins from spec maintainer, kernel driver maintainer, qemu device >>>>> maintainer (only regarding the ABI, not the code) >>>> >>>> I don't think this is how it's working today. I would be happy with a >>>> flow like this. >>> >>> If Michael and Rusty agree, we can adopt it immediately. >>> > > If I understand the proposal, what is suggested is that > all of spec, kvm and virtio patches are posted on list and > acked before merging any one of them? > > Sure, this makes sense. Well, what's needed before the spec is changed is an interesting question, but I think the main thing is, don't commit any virtio ABI changes to vhost, QEMU, NKT, or the kernel until the spec for the change has been committed. It would be nice to have a working implementation before committing a spec change. Even nicer would be to have Acked-by's a maintainer in each area affected. Regards, Anthony Liguori
On 11/03/2011 04:37 PM, Anthony Liguori wrote: > >>>>>> 2. Proposed spec patch, kernel change, qemu change >>>>>> 3. Buy-ins from spec maintainer, kernel driver maintainer, qemu >>>>>> device >>>>>> maintainer (only regarding the ABI, not the code) >>>>> >>>>> I don't think this is how it's working today. I would be happy >>>>> with a >>>>> flow like this. >>>> >>>> If Michael and Rusty agree, we can adopt it immediately. >>>> >> >> If I understand the proposal, what is suggested is that >> all of spec, kvm and virtio patches are posted on list and >> acked before merging any one of them? >> >> Sure, this makes sense. > > Well, what's needed before the spec is changed is an interesting > question, but I think the main thing is, don't commit any virtio ABI > changes to vhost, QEMU, NKT, or the kernel until the spec for the > change has been committed. > > It would be nice to have a working implementation before committing a > spec change. Even nicer would be to have Acked-by's a maintainer in > each area affected. > Those are steps 2 and 3 of the proposed process.
diff --git a/hw/virtio-pci.c b/hw/virtio-pci.c index df27c19..68cd5bc 100644 --- a/hw/virtio-pci.c +++ b/hw/virtio-pci.c @@ -659,6 +659,8 @@ void virtio_init_pci(VirtIOPCIProxy *proxy, VirtIODevice *vdev) "virtio-pci", size); pci_register_bar(&proxy->pci_dev, 0, PCI_BASE_ADDRESS_SPACE_IO, &proxy->bar); + pci_register_bar(&proxy->pci_dev, 2, PCI_BASE_ADDRESS_SPACE_MEMORY, + &proxy->bar); if (!kvm_has_many_ioeventfds()) { proxy->flags &= ~VIRTIO_PCI_FLAG_USE_IOEVENTFD;
Currently, virtio devices are usually presented to the guest as an emulated PCI device, virtio_pci. Although the actual IO operations are done through system memory, the configuration of the virtio device is done through the one PCI IO space BAR that virtio_pci presents. But PCI IO space (aka PIO) is deprecated for modern PCI devices, and on some systems with many PCI domains accessing PIO space can be problematic. For example on the existing PowerVM implementation of the PAPR spec, PCI PIO access is not supported at all. We're hoping that our KVM implementation will support PCI PIO (once we support PCI at all), but it will probably have some irritating limitations. This patch, therefore, extends the virtio_pci device to have a PCI memory space (MMIO) BAR as well as the IO BAR. The MMIO BAR contains exactly the same registers, in exactly the same layout as the existing PIO BAR. Because the PIO BAR is still present, existing guest drivers should still work fine. With this change in place, future guest drivers can check for an MMIO BAR and use that if present (falling back to PIO when possible to support older qemu versions). Signed-off-by: David Gibson <david@gibson.dropbear.id.au> --- hw/virtio-pci.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)