| Message ID | 1360669793-6921-2-git-send-email-sjur.brandeland@stericsson.com |
|---|---|
| State | Not Applicable, archived |
| Delegated to: | David Miller |
| Headers | show |
Hi Sjur, On Tue, Feb 12, 2013 at 1:49 PM, <sjur.brandeland@stericsson.com> wrote: > From: Sjur Brændeland <sjur.brandeland@stericsson.com> > > Add functions for creating, deleting and kicking host-side virtio rings. > > The host ring is not integrated with virtiqueues and cannot be managed > through virtio-config. Is that an inherent design/issue of vringh or just a description of the current vringh code ? > Remoteproc must export functions for handling the host-side virtio rings. Have you considered exporting this via virtio instead ? > The functions rproc_virtio_get_vringh(), rproc_virtio_del_vringh(), > rproc_virtio_kick_vringh() are added to remoteproc_virtio.c. I wonder if this is the way we want things to work. Following this design, virtio drivers that use these rproc_* functions will be coupled with the remoteproc framework. One issue with this is what happens if, e.g., a VIRTIO_ID_CAIF vdev is added by other than remoteproc (e.g. by virtio_pci or virtio_mmio). Not sure how probable this really is, and whether there's anything that prevents this, but things will go awry if this happens. But maybe the important aspect to consider is whether we really want to couple virtio drivers (such as the upcoming caif one) with the remoteproc framework. If you'll take a look at the rpmsg virtio driver, there's nothing there which couples it with remoteproc. It's just a standard virtio driver, that can be easily used with traditional virtio hosts as well. This is possible of course thanks to the abstraction provided by virtio: remoteproc only implements a set of callbacks which virtio invokes when needed. Do we not want to follow a similar design scheme with vringh ? I have some other questions as well but maybe it's better to discuss first the bigger picture. Thanks! Ohad. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Feb 20, 2013 at 5:05 PM, Ohad Ben-Cohen <ohad@wizery.com> wrote: > Hi Sjur, > > On Tue, Feb 12, 2013 at 1:49 PM, <sjur.brandeland@stericsson.com> wrote: >> From: Sjur Brændeland <sjur.brandeland@stericsson.com> >> >> Add functions for creating, deleting and kicking host-side virtio rings. >> >> The host ring is not integrated with virtiqueues and cannot be managed >> through virtio-config. > > Is that an inherent design/issue of vringh or just a description of > the current vringh code ? > >> Remoteproc must export functions for handling the host-side virtio rings. > > Have you considered exporting this via virtio instead ? Rusty should comment on this... I asked Rusty the same question a while a go, see http://lkml.org/lkml/2013/1/11/559 AFAIK, using the vringh API directly is a deliberate design choice. [Sjur:] >> How do you see the in-kernel API for this? I would like to see >> something similar to my previous patches, where we extend >> the virtqueue API. E.g. something like this: >> struct virtqueue *vring_new_virtqueueh(...)... [Rusty:] >I was just going to create _kernel variants of all the _user helpers, >and let you drive it directly like that. > >If we get a second in-kernel user, we create wrappers (I'd prefer not to >overload struct virtqueue though). >> The functions rproc_virtio_get_vringh(), rproc_virtio_del_vringh(), >> rproc_virtio_kick_vringh() are added to remoteproc_virtio.c. > > I wonder if this is the way we want things to work. > > Following this design, virtio drivers that use these rproc_* functions > will be coupled with the remoteproc framework. > > One issue with this is what happens if, e.g., a VIRTIO_ID_CAIF vdev is > added by other than remoteproc (e.g. by virtio_pci or virtio_mmio). > Not sure how probable this really is, and whether there's anything > that prevents this, but things will go awry if this happens. Yes, if you insert a "malicious device" like that you can make it crash, but wouldn't most drivers do if you try to register a malicious device...? If we really want to protect from this, we could perhaps validate the vdev pointer in function rproc_virtio_new_vringh() by looking through the vdevs of the registered rprocs. > But maybe the important aspect to consider is whether we really want > to couple virtio drivers (such as the upcoming caif one) with the > remoteproc framework. I'm not sure this is an issue for the CAIF driver. It would be very nice if someone else could make use of it, but right now cannot see the CAIF driver being used outside the remoteproc framework. This driver is designed specifically to work with the STE-modem using the CAIF protocol over a shared memory interface. > If you'll take a look at the rpmsg virtio driver, there's nothing > there which couples it with remoteproc. It's just a standard virtio > driver, that can be easily used with traditional virtio hosts as well. > > This is possible of course thanks to the abstraction provided by > virtio: remoteproc only implements a set of callbacks which virtio > invokes when needed. Yes, and generalizing the use of virtio devices in remoteproc has been useful. It has enabled me to let remoteproc manage both virtio_serial and virtio_caif devices :-) > > Do we not want to follow a similar design scheme with vringh ? I know some of my colleagues has been working on symmetric vring for rpmsg. Last I heard from them they were going to use the same approach I've done for CAIF, by "reversing" the direction of the rings. AFAIK this means that the current API I have proposed will work for them as well. If some other driver is showing up using the vringh kernel API where the current API don't fit, I guess it's time to create some abstractions and wrappers... But I hope the current approach will do for now? > I have some other questions as well but maybe it's better to discuss > first the bigger picture. OK, but please don't hesitate to address this. I'm still aiming for this to go into 3.9. Regards, Sjur -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Ohad Ben-Cohen <ohad@wizery.com> writes: > Hi Sjur, > > On Tue, Feb 12, 2013 at 1:49 PM, <sjur.brandeland@stericsson.com> wrote: >> From: Sjur Brændeland <sjur.brandeland@stericsson.com> >> >> Add functions for creating, deleting and kicking host-side virtio rings. >> >> The host ring is not integrated with virtiqueues and cannot be managed >> through virtio-config. > > Is that an inherent design/issue of vringh or just a description of > the current vringh code ? It's by design. The producer (virtqueue) and consumer (vringh) are two sides of the same coin, but they do different things. virtqueue is a slightly higher level abstraction which assumes a virtio_device, because every user so far has had one. vringh doesn't, because it's also aimed to underlie vhost.c which doesn't really have one. > This is possible of course thanks to the abstraction provided by > virtio: remoteproc only implements a set of callbacks which virtio > invokes when needed. > > Do we not want to follow a similar design scheme with vringh ? Hmm... I clearly jumped the gun, assuming consensus was already reached. I have put these patches *back* into pending-rebases, and they will not be merged this merge window. Cheers, Rusty. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Feb 21, 2013 at 1:01 AM, Sjur Brændeland <sjur.brandeland@stericsson.com> wrote: > [Sjur:] >>> How do you see the in-kernel API for this? I would like to see >>> something similar to my previous patches, where we extend >>> the virtqueue API. E.g. something like this: >>> struct virtqueue *vring_new_virtqueueh(...)... > > [Rusty:] >>I was just going to create _kernel variants of all the _user helpers, >>and let you drive it directly like that. >> >>If we get a second in-kernel user, we create wrappers (I'd prefer not to >>overload struct virtqueue though). The wrappers suggestion sounds good. One of the things we truly enjoyed about virtio is how easy it was to reuse its drivers for communication with a remote processor. It'd be great if we can stick to this design and keep vringh virtio drivers decoupled from the low level implementation they are developed with (specifically caif and remoteproc in this case). > Yes, if you insert a "malicious device" like that you can make it crash, > but wouldn't most drivers do if you try to register a malicious device...? > > If we really want to protect from this, we could perhaps validate the vdev > pointer in function rproc_virtio_new_vringh() by looking through the vdevs > of the registered rprocs. It sounds like we can instead just avoid this altogether if we follow Rusty's wrappers suggestion. The result would probably look better, and I suspect it might be very minimal code. > If some other driver is showing up using the vringh kernel API where > the current API don't fit, I guess it's time to create some abstractions > and wrappers... But I hope the current approach will do for now? Unless I'm missing something here, adding wrappers should be straight forward and quick. Coupling a virtio driver with remoteproc doesn't look great to me, probably because I appreciate so much the elegance of virtio and how easy we could have reused its vanilla drivers with a use case it wasn't originally designed to support. >> I have some other questions as well but maybe it's better to discuss >> first the bigger picture. > > OK, but please don't hesitate to address this. I'm still aiming for this > to go into 3.9. I was wondering - can you please explain your motivation for using vringh in caif ? We have internally discussed supporting multiple remote processors talking to each other using rpmsg, and in that scenario using vringh can considerably simplifies the solution (no need to decide in advance which side is the "guest" and which is the "host"). Was this the general incentive in using vringh in caif too or something else? Thanks, Ohad. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Feb 21, 2013 at 8:37 AM, Rusty Russell <rusty@rustcorp.com.au> wrote: > Hmm... I clearly jumped the gun, assuming consensus was already reached. > I have put these patches *back* into pending-rebases, and they will not > be merged this merge window. Thanks. What do you think about creating some virtio-level wrappers for the vringh handlers? I don't think we're going to stop with caif as the only vringh user, and it could be nice if we follow the virtio spirit of decoupling the drivers from the low level implementation. It sure did prove itself when the remoteproc use cases started showing up, and it's neat. Thanks, Ohad. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Ohad, > I was wondering - can you please explain your motivation for using > vringh in caif ? > > We have internally discussed supporting multiple remote processors > talking to each other using rpmsg, and in that scenario using vringh > can considerably simplifies the solution (no need to decide in advance > which side is the "guest" and which is the "host"). Was this the > general incentive in using vringh in caif too or something else? The motivation for using vringh was to avoid copying buffers when sending data from the modem to the host. By using vringh the modem can transfer datagrams to host only by transfering buffer pointers via the rings. We use a carveout to declare a memory area that is passed to the modem internal memory allocator. This way we get both modem and host to work on the shared memory region. For TX traffic we use normal virtio queues for the same reason. TX buffers can be handled without copying as well. We are seeing a nice performance gain on the modem side after implementing this. Host size zero-copy is more tricky. It's hard to handle ownership of buffers, if the modem crashes. But we might look into this in the future as well. Regards, Sjur -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Sjur, On Thu, Feb 21, 2013 at 7:28 PM, Sjur Brændeland <sjurbren@gmail.com> wrote: > The motivation for using vringh was to avoid copying buffers > when sending data from the modem to the host. I may be missing something here, but why do you need vringh for that? With rpmsg (which uses two regular vrings) both ends send huge amount of data without copying it - we just send the pointers across. Thanks, Ohad. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Ohad, On Thu, Feb 21, 2013 at 6:55 PM, Ohad Ben-Cohen <ohad@wizery.com> wrote: > On Thu, Feb 21, 2013 at 7:28 PM, Sjur Brændeland <sjurbren@gmail.com> wrote: >> The motivation for using vringh was to avoid copying buffers >> when sending data from the modem to the host. > > I may be missing something here, but why do you need vringh for that? > > With rpmsg (which uses two regular vrings) both ends send huge amount > of data without copying it - we just send the pointers across. OK, We did carefully consider using the normal vrings, but concluded it was not doable. I'll try to give you some of the background that I can recall from top of my head. (I can dig out more if you're still not convinced :) The modem we're integrating with is a complex beast (multi mode modem LTE, HSPA+, etc). When a packet is received deep down in the radio stack, it allocates packet buffers using the internal slab allocator.The packet may contain anything from control, voice, or IP packets. If the packet contains IP-payload it will travel to a different asymmetric CPU that handles the IPC towards the modem. If the packet is not a IP-packet it will be processed internally and eventually freed. What we have done to integrate virtio is to inject the carveout into the the modem internal slab-allocator. Using the reversed ring allows us to introduce zero-copy for virtio with virtually no impact on the radio-stack, only a small change on the modem slab allocator. It supports dynamic size buffer allocation, and modem internal messaging using data allocated from the shared region. It also allows modem to manage it's own memory, without any dependency on host side allocators. Thanks, Sjur -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Ohad Ben-Cohen <ohad@wizery.com> writes: > On Thu, Feb 21, 2013 at 8:37 AM, Rusty Russell <rusty@rustcorp.com.au> wrote: >> Hmm... I clearly jumped the gun, assuming consensus was already reached. >> I have put these patches *back* into pending-rebases, and they will not >> be merged this merge window. > > Thanks. > > What do you think about creating some virtio-level wrappers for the > vringh handlers? > > I don't think we're going to stop with caif as the only vringh user, > and it could be nice if we follow the virtio spirit of decoupling the > drivers from the low level implementation. It sure did prove itself > when the remoteproc use cases started showing up, and it's neat. The problem space is a bit different. My immediate concern is getting vhost (and thus vhost_net/blk) to use vringh: I wanted to unify the in-userspace and in-kernelspace ring implementations. We don't have that issue in virtqueue.c. vhost is (will be) the higher abstraction for in-userspace rings, perhaps we want an equivalent for in-kernelspace rings. I'm happy to look at patches, but I don't immediately see what it would look like... Thanks, Rusty. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Feb 21, 2013 at 9:36 PM, Sjur Brændeland <sjurbren@gmail.com> wrote: > OK, We did carefully consider using the normal vrings, but concluded it was > not doable. I'll try to give you some of the background that I can > recall from top of my head. > (I can dig out more if you're still not convinced :) > > The modem we're integrating with is a complex beast (multi mode modem > LTE, HSPA+, etc). > When a packet is received deep down in the radio stack, it allocates packet > buffers using the internal slab allocator.The packet may contain > anything from control, voice, > or IP packets. If the packet contains IP-payload it will travel to a > different asymmetric CPU that > handles the IPC towards the modem. If the packet is not a IP-packet it will be > processed internally and eventually freed. > > What we have done to integrate virtio is to inject the carveout into the > the modem internal slab-allocator. > > Using the reversed ring allows us to introduce zero-copy for virtio > with virtually no > impact on the radio-stack, only a small change on the modem slab allocator. > It supports dynamic size buffer allocation, and modem internal > messaging using data > allocated from the shared region. It also allows modem to manage it's > own memory, > without any dependency on host side allocators. Thanks for the description, Sjur. It sounds like this mostly simplifies your modem-side code, especially if it talks to other cores as well. My impression is that persistently sticking to guest vrings is also possible, but it makes things cumbersome. We will most probably adopt vringh in rpmsg too when a real multicore use case shows up. Ohad. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/remoteproc/Kconfig b/drivers/remoteproc/Kconfig index 96ce101..c7d1d36 100644 --- a/drivers/remoteproc/Kconfig +++ b/drivers/remoteproc/Kconfig @@ -7,6 +7,9 @@ config REMOTEPROC depends on HAS_DMA select FW_CONFIG select VIRTIO + select VHOST_RING + +source drivers/vhost/Kconfig config OMAP_REMOTEPROC tristate "OMAP remoteproc support" diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c index 9e198e5..fa7bf7b 100644 --- a/drivers/remoteproc/remoteproc_virtio.c +++ b/drivers/remoteproc/remoteproc_virtio.c @@ -23,6 +23,7 @@ #include <linux/virtio_config.h> #include <linux/virtio_ids.h> #include <linux/virtio_ring.h> +#include <linux/vringh.h> #include <linux/err.h> #include <linux/kref.h> #include <linux/slab.h> @@ -60,10 +61,15 @@ irqreturn_t rproc_vq_interrupt(struct rproc *rproc, int notifyid) dev_dbg(&rproc->dev, "vq index %d is interrupted\n", notifyid); rvring = idr_find(&rproc->notifyids, notifyid); - if (!rvring || !rvring->vq) + if (!rvring) return IRQ_NONE; - return vring_interrupt(0, rvring->vq); + if (rvring->vringh && rvring->vringh_cb) + return rvring->vringh_cb(&rvring->rvdev->vdev, rvring->vringh); + else if (rvring->vq) + return vring_interrupt(0, rvring->vq); + else + return IRQ_NONE; } EXPORT_SYMBOL(rproc_vq_interrupt); @@ -149,14 +155,21 @@ static int rproc_virtio_find_vqs(struct virtio_device *vdev, unsigned nvqs, const char *names[]) { struct rproc *rproc = vdev_to_rproc(vdev); - int i, ret; + struct rproc_vdev *rvdev = vdev_to_rvdev(vdev); + int rng, id, ret, nrings = ARRAY_SIZE(rvdev->vring); + + for (id = 0, rng = 0; rng < nrings; ++rng) { + struct rproc_vring *rvring = &rvdev->vring[rng]; + /* Skip the host side rings */ + if (rvring->vringh) + continue; - for (i = 0; i < nvqs; ++i) { - vqs[i] = rp_find_vq(vdev, i, callbacks[i], names[i]); - if (IS_ERR(vqs[i])) { - ret = PTR_ERR(vqs[i]); + vqs[id] = rp_find_vq(vdev, rng, callbacks[id], names[id]); + if (IS_ERR(vqs[id])) { + ret = PTR_ERR(vqs[id]); goto error; } + ++id; } /* now that the vqs are all set, boot the remote processor */ @@ -173,6 +186,106 @@ error: return ret; } +/** + * rproc_virtio_new_vringh() - create a reversed virtio ring. + * @vdev: the virtio device + * @index: the virtio ring index + * @cb: callback for the reversed virtio ring + * + * This function should be called by the virtio-driver + * before calling find_vqs(). It returns a struct vringh for + * accessing the virtio ring. + * + * Return: struct vhost, or NULL upon error. + */ +struct vringh * +rproc_virtio_new_vringh(struct virtio_device *vdev, unsigned index, + irqreturn_t (*cb)(struct virtio_device *vdev, + struct vringh *vring)) +{ + struct rproc_vdev *rvdev = vdev_to_rvdev(vdev); + struct rproc_vring *rvring; + struct vringh *vrh; + int err; + + if (index > ARRAY_SIZE(rvdev->vring)) { + dev_err(&rvdev->vdev.dev, "bad vring index: %d\n", index); + return NULL; + } + + vrh = kzalloc(sizeof(*vrh), GFP_KERNEL); + if (!vrh) + return NULL; + + err = rproc_alloc_vring(rvdev, index); + if (err) + goto free_vring; + + + rvring = &rvdev->vring[index]; + /* zero vring */ + memset(rvring->va, 0, vring_size(rvring->len, rvring->align)); + vring_init(&vrh->vring, rvring->len, rvring->va, rvring->align); + + rvring->vringh_cb = cb; + rvring->vringh = vrh; + + err = vringh_init_kern(vrh, + rvdev->dfeatures, + rvring->len, + false, + vrh->vring.desc, + vrh->vring.avail, + vrh->vring.used); + if (!err) + return vrh; + + dev_err(&vdev->dev, "failed to create vhost: %d\n", err); + rproc_free_vring(rvring); +free_vring: + kfree(vrh); + return NULL; +} +EXPORT_SYMBOL(rproc_virtio_new_vringh); + +/** + * rproc_virtio_del_vringh() - release a reversed virtio ring. + * @vdev: the virtio device + * @index: the virtio ring index + * + * This function release the reversed virtio ring. + */ +void rproc_virtio_del_vringh(struct virtio_device *vdev, unsigned index) +{ + struct rproc_vdev *rvdev = vdev_to_rvdev(vdev); + struct rproc_vring *rvring = &rvdev->vring[index]; + kfree(rvring->vringh); + rproc_free_vring(rvring); + rvring->vringh_cb = NULL; + rvring->vringh = NULL; +} +EXPORT_SYMBOL(rproc_virtio_del_vringh); + +/** + * rproc_virtio_kick_vringh() - kick the remote processor. + * @vdev: the virtio device + * @index: the virtio ring index + * + * kick the remote processor, and let it know which vring to poke at + */ +void rproc_virtio_kick_vringh(struct virtio_device *vdev, unsigned index) +{ + struct rproc_vdev *rvdev = vdev_to_rvdev(vdev); + struct rproc_vring *rvring = &rvdev->vring[index]; + struct rproc *rproc = rvring->rvdev->rproc; + int notifyid = rvring->notifyid; + + dev_dbg(&rproc->dev, "kicking vq index: %d\n", notifyid); + + rproc->ops->kick(rproc, notifyid); +} +EXPORT_SYMBOL(rproc_virtio_kick_vringh); + /* * We don't support yet real virtio status semantics. * diff --git a/include/linux/remoteproc.h b/include/linux/remoteproc.h index faf3332..414a1fd 100644 --- a/include/linux/remoteproc.h +++ b/include/linux/remoteproc.h @@ -39,7 +39,9 @@ #include <linux/klist.h> #include <linux/mutex.h> #include <linux/virtio.h> +#include <linux/vringh.h> #include <linux/completion.h> +#include <linux/interrupt.h> #include <linux/idr.h> /** @@ -444,6 +446,8 @@ struct rproc { * @notifyid: rproc-specific unique vring index * @rvdev: remote vdev * @vq: the virtqueue of this vring + * @vringh_cb: callback used when device has kicked + * @vringh: the reversed host-side vring */ struct rproc_vring { void *va; @@ -454,6 +458,9 @@ struct rproc_vring { int notifyid; struct rproc_vdev *rvdev; struct virtqueue *vq; + irqreturn_t (*vringh_cb)(struct virtio_device *vdev, + struct vringh *vring); + struct vringh *vringh; }; /** @@ -485,6 +492,13 @@ int rproc_boot(struct rproc *rproc); void rproc_shutdown(struct rproc *rproc); void rproc_report_crash(struct rproc *rproc, enum rproc_crash_type type); +struct vringh * +rproc_virtio_new_vringh(struct virtio_device *vdev, unsigned index, + irqreturn_t (*cb)(struct virtio_device *vdev, + struct vringh *vring)); +void rproc_virtio_del_vringh(struct virtio_device *vdev, unsigned index); +void rproc_virtio_kick_vringh(struct virtio_device *vdev, unsigned index); + static inline struct rproc_vdev *vdev_to_rvdev(struct virtio_device *vdev) { return container_of(vdev, struct rproc_vdev, vdev);