| Message ID | 20170402201012.76473-4-willemdebruijn.kernel@gmail.com |
|---|---|
| State | Changes Requested, archived |
| Delegated to: | David Miller |
| Headers | show |
On Sun, Apr 02, 2017 at 04:10:12PM -0400, Willem de Bruijn wrote: > From: Willem de Bruijn <willemb@google.com> > > Amortize the cost of virtual interrupts by doing both rx and tx work > on reception of a receive interrupt if tx napi is enabled. With > VIRTIO_F_EVENT_IDX, this suppresses most explicit tx completion > interrupts for bidirectional workloads. > > Signed-off-by: Willem de Bruijn <willemb@google.com> > --- > drivers/net/virtio_net.c | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > > diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c > index 95d938e82080..af830eb212bf 100644 > --- a/drivers/net/virtio_net.c > +++ b/drivers/net/virtio_net.c > @@ -1030,12 +1030,34 @@ static int virtnet_receive(struct receive_queue *rq, int budget) > return received; > } > > +static void free_old_xmit_skbs(struct send_queue *sq); > + Could you pls re-arrange code to avoid forward declarations? > +static void virtnet_poll_cleantx(struct receive_queue *rq) > +{ > + struct virtnet_info *vi = rq->vq->vdev->priv; > + unsigned int index = vq2rxq(rq->vq); > + struct send_queue *sq = &vi->sq[index]; > + struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index); > + > + if (!sq->napi.weight) > + return; > + > + __netif_tx_lock(txq, smp_processor_id()); > + free_old_xmit_skbs(sq); > + __netif_tx_unlock(txq); > + > + if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) > + netif_wake_subqueue(vi->dev, vq2txq(sq->vq)); > +} > + Looks very similar to virtnet_poll_tx. I think this might be waking the tx queue too early, so it will tend to stay almost full for long periods of time. Why not defer wakeup until queue is at least half empty? I wonder whether it's worth it to handle very short queues correctly - they previously made very slow progress, not they are never woken up. I'm a bit concerned about the cost of these wakeups and locking. I note that this wake is called basically every time queue is not full. Would it make sense to limit the amount of tx polling? Maybe use trylock to reduce the conflict with xmit? > static int virtnet_poll(struct napi_struct *napi, int budget) > { > struct receive_queue *rq = > container_of(napi, struct receive_queue, napi); > unsigned int received; > > + virtnet_poll_cleantx(rq); > + > received = virtnet_receive(rq, budget); > > /* Out of packets? */ > -- > 2.12.2.564.g063fe858b8-goog
On Sun, Apr 2, 2017 at 10:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Sun, Apr 02, 2017 at 04:10:12PM -0400, Willem de Bruijn wrote: >> From: Willem de Bruijn <willemb@google.com> >> >> Amortize the cost of virtual interrupts by doing both rx and tx work >> on reception of a receive interrupt if tx napi is enabled. With >> VIRTIO_F_EVENT_IDX, this suppresses most explicit tx completion >> interrupts for bidirectional workloads. >> >> Signed-off-by: Willem de Bruijn <willemb@google.com> >> --- >> drivers/net/virtio_net.c | 22 ++++++++++++++++++++++ >> 1 file changed, 22 insertions(+) >> >> diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c >> index 95d938e82080..af830eb212bf 100644 >> --- a/drivers/net/virtio_net.c >> +++ b/drivers/net/virtio_net.c >> @@ -1030,12 +1030,34 @@ static int virtnet_receive(struct receive_queue *rq, int budget) >> return received; >> } >> >> +static void free_old_xmit_skbs(struct send_queue *sq); >> + > > Could you pls re-arrange code to avoid forward declarations? Okay. I'll do the move in a separate patch to simplify review. >> +static void virtnet_poll_cleantx(struct receive_queue *rq) >> +{ >> + struct virtnet_info *vi = rq->vq->vdev->priv; >> + unsigned int index = vq2rxq(rq->vq); >> + struct send_queue *sq = &vi->sq[index]; >> + struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index); >> + >> + if (!sq->napi.weight) >> + return; >> + >> + __netif_tx_lock(txq, smp_processor_id()); >> + free_old_xmit_skbs(sq); >> + __netif_tx_unlock(txq); >> + >> + if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) >> + netif_wake_subqueue(vi->dev, vq2txq(sq->vq)); >> +} >> + > > Looks very similar to virtnet_poll_tx. > > I think this might be waking the tx queue too early, so > it will tend to stay almost full for long periods of time. > Why not defer wakeup until queue is at least half empty? I'll test that. Delaying wake-up longer than necessary can cause queue build up at the qdisc and higher tail latency, I imagine. But it may reduce the number of __netif_schedule calls. > I wonder whether it's worth it to handle very short queues > correctly - they previously made very slow progress, > not they are never woken up. > > I'm a bit concerned about the cost of these wakeups > and locking. I note that this wake is called basically > every time queue is not full. > > Would it make sense to limit the amount of tx polling? > Maybe use trylock to reduce the conflict with xmit? Yes, that sounds good. I did test that previously and saw no difference then. But when multiple cpus contend for a single txq it should help.
On Mon, Apr 03, 2017 at 01:02:13AM -0400, Willem de Bruijn wrote: > On Sun, Apr 2, 2017 at 10:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Sun, Apr 02, 2017 at 04:10:12PM -0400, Willem de Bruijn wrote: > >> From: Willem de Bruijn <willemb@google.com> > >> > >> Amortize the cost of virtual interrupts by doing both rx and tx work > >> on reception of a receive interrupt if tx napi is enabled. With > >> VIRTIO_F_EVENT_IDX, this suppresses most explicit tx completion > >> interrupts for bidirectional workloads. > >> > >> Signed-off-by: Willem de Bruijn <willemb@google.com> This is a popular approach, but I think this will only work well if tx and rx interrupts are processed on the same CPU and if tx queue is per cpu. If they target different CPUs or if tx queue is used from multiple CPUs they will conflict on the shared locks. This can even change dynamically as CPUs/queues are reconfigured. How about adding a flag and skipping the tx poll if there's no match?
On Fri, Apr 7, 2017 at 3:28 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Mon, Apr 03, 2017 at 01:02:13AM -0400, Willem de Bruijn wrote: >> On Sun, Apr 2, 2017 at 10:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> > On Sun, Apr 02, 2017 at 04:10:12PM -0400, Willem de Bruijn wrote: >> >> From: Willem de Bruijn <willemb@google.com> >> >> >> >> Amortize the cost of virtual interrupts by doing both rx and tx work >> >> on reception of a receive interrupt if tx napi is enabled. With >> >> VIRTIO_F_EVENT_IDX, this suppresses most explicit tx completion >> >> interrupts for bidirectional workloads. >> >> >> >> Signed-off-by: Willem de Bruijn <willemb@google.com> > > This is a popular approach, but I think this will only work well if tx > and rx interrupts are processed on the same CPU and if tx queue is per > cpu. If they target different CPUs or if tx queue is used from multiple > CPUs they will conflict on the shared locks. Yes. As a result of this discussion I started running a few vcpu affinity tests. The data is not complete. In particular, I don't have the data yet to compare having tx and rx irq on the same cpu (0,0) vs on different (0,2) for this patchset. Which is the relevant data to your point. Initial results for unmodified upstream driver at {1, 10, 100}x TCP_STREAM, for irq cpu affinity (rx,tx). Process is always pinned to cpu 1. This is a 4 vcpu system pinned by the host to 4 cores on the same socket. The previously reported results were obtained with txq, rtx and process on different vcpus (0,2). Running all on the same vcpu lower cycle count considerably: irq 0,0 1 throughput_Mbps=29767.14 391,488,924,526 cycles 10 throughput_Mbps=40808.64 424,530,251,896 cycles 100 throughput_Mbps=33475.13 414,622,071,167 cycles irq 0,2 1 throughput_Mbps=30176.05 395,673,200,747 cycles 10 throughput_Mbps=40729.26 433,948,374,991 cycles 100 throughput_Mbps=33758.68 436,291,949,393 cycles irq 1,1 1 throughput_Mbps=26635.20 269,071,002,844 cycles 10 throughput_Mbps=42385.05 299,945,944,516 cycles 100 throughput_Mbps=33580.98 283,272,895,507 cycles With this patch set applied, cpu (1,1) 1 throughput_Mbps=34980.76 276,504,805,414 cycles 10 throughput_Mbps=42519.92 298,105,889,785 cycles 100 throughput_Mbps=35268.86 296,670,598,712 cycles I will need to get data for (0,2) vs (0,0). > This can even change dynamically as CPUs/queues are reconfigured. > How about adding a flag and skipping the tx poll if there's no match? I suspect that even with the cache invalidations this optimization will be an improvement over handling all tx interrupts in the tx napi handler. I will get the datapoint for that. That said, we can make this conditional. What flag exactly do you propose? Compare raw_smp_processor_id() in the rx softint with one previously stored in the napi tx callback?
On Fri, Apr 07, 2017 at 04:59:58PM -0400, Willem de Bruijn wrote: > On Fri, Apr 7, 2017 at 3:28 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Mon, Apr 03, 2017 at 01:02:13AM -0400, Willem de Bruijn wrote: > >> On Sun, Apr 2, 2017 at 10:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > >> > On Sun, Apr 02, 2017 at 04:10:12PM -0400, Willem de Bruijn wrote: > >> >> From: Willem de Bruijn <willemb@google.com> > >> >> > >> >> Amortize the cost of virtual interrupts by doing both rx and tx work > >> >> on reception of a receive interrupt if tx napi is enabled. With > >> >> VIRTIO_F_EVENT_IDX, this suppresses most explicit tx completion > >> >> interrupts for bidirectional workloads. > >> >> > >> >> Signed-off-by: Willem de Bruijn <willemb@google.com> > > > > This is a popular approach, but I think this will only work well if tx > > and rx interrupts are processed on the same CPU and if tx queue is per > > cpu. If they target different CPUs or if tx queue is used from multiple > > CPUs they will conflict on the shared locks. > > Yes. As a result of this discussion I started running a few vcpu affinity tests. > > The data is not complete. In particular, I don't have the data yet to > compare having tx and rx irq on the same cpu (0,0) vs on different > (0,2) for this patchset. Which is the relevant data to your point. > > Initial results for unmodified upstream driver at {1, 10, 100}x > TCP_STREAM, for irq cpu affinity (rx,tx). Process is always pinned to cpu > 1. This is a 4 vcpu system pinned by the host to 4 cores on the same > socket. The previously reported results were obtained with txq, rtx > and process on different vcpus (0,2). Running all on the same vcpu > lower cycle count considerably: > > irq 0,0 > 1 throughput_Mbps=29767.14 391,488,924,526 cycles > 10 throughput_Mbps=40808.64 424,530,251,896 cycles > 100 throughput_Mbps=33475.13 414,622,071,167 cycles > > irq 0,2 > 1 throughput_Mbps=30176.05 395,673,200,747 cycles > 10 throughput_Mbps=40729.26 433,948,374,991 cycles > 100 throughput_Mbps=33758.68 436,291,949,393 cycles > > irq 1,1 > 1 throughput_Mbps=26635.20 269,071,002,844 cycles > 10 throughput_Mbps=42385.05 299,945,944,516 cycles > 100 throughput_Mbps=33580.98 283,272,895,507 cycles > > With this patch set applied, cpu (1,1) > > 1 throughput_Mbps=34980.76 276,504,805,414 cycles > 10 throughput_Mbps=42519.92 298,105,889,785 cycles > 100 throughput_Mbps=35268.86 296,670,598,712 cycles > > I will need to get data for (0,2) vs (0,0). > > > This can even change dynamically as CPUs/queues are reconfigured. > > How about adding a flag and skipping the tx poll if there's no match? > > I suspect that even with the cache invalidations this optimization > will be an improvement over handling all tx interrupts in the tx napi > handler. I will get the datapoint for that. > > That said, we can make this conditional. What flag exactly do you > propose? Compare raw_smp_processor_id() in the rx softint with one > previously stored in the napi tx callback? I'm not sure. Another idea is to check vi->affinity_hint_set. If set we know rq and sq are on the same CPU.
On Fri, Apr 7, 2017 at 5:10 PM, Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Apr 07, 2017 at 04:59:58PM -0400, Willem de Bruijn wrote: >> On Fri, Apr 7, 2017 at 3:28 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> > On Mon, Apr 03, 2017 at 01:02:13AM -0400, Willem de Bruijn wrote: >> >> On Sun, Apr 2, 2017 at 10:47 PM, Michael S. Tsirkin <mst@redhat.com> wrote: >> >> > On Sun, Apr 02, 2017 at 04:10:12PM -0400, Willem de Bruijn wrote: >> >> >> From: Willem de Bruijn <willemb@google.com> >> >> >> >> >> >> Amortize the cost of virtual interrupts by doing both rx and tx work >> >> >> on reception of a receive interrupt if tx napi is enabled. With >> >> >> VIRTIO_F_EVENT_IDX, this suppresses most explicit tx completion >> >> >> interrupts for bidirectional workloads. >> >> >> >> >> >> Signed-off-by: Willem de Bruijn <willemb@google.com> >> > >> > This is a popular approach, but I think this will only work well if tx >> > and rx interrupts are processed on the same CPU and if tx queue is per >> > cpu. If they target different CPUs or if tx queue is used from multiple >> > CPUs they will conflict on the shared locks. >> >> Yes. As a result of this discussion I started running a few vcpu affinity tests. >> >> > This can even change dynamically as CPUs/queues are reconfigured. >> > How about adding a flag and skipping the tx poll if there's no match? >> >> I suspect that even with the cache invalidations this optimization >> will be an improvement over handling all tx interrupts in the tx napi >> handler. I will get the datapoint for that. >> >> That said, we can make this conditional. What flag exactly do you >> propose? Compare raw_smp_processor_id() in the rx softint with one >> previously stored in the napi tx callback? > > I'm not sure. Another idea is to check vi->affinity_hint_set. > If set we know rq and sq are on the same CPU. I was not aware of that flag, thanks. Yes, that looks like it should work.
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 95d938e82080..af830eb212bf 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -1030,12 +1030,34 @@ static int virtnet_receive(struct receive_queue *rq, int budget) return received; } +static void free_old_xmit_skbs(struct send_queue *sq); + +static void virtnet_poll_cleantx(struct receive_queue *rq) +{ + struct virtnet_info *vi = rq->vq->vdev->priv; + unsigned int index = vq2rxq(rq->vq); + struct send_queue *sq = &vi->sq[index]; + struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index); + + if (!sq->napi.weight) + return; + + __netif_tx_lock(txq, smp_processor_id()); + free_old_xmit_skbs(sq); + __netif_tx_unlock(txq); + + if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) + netif_wake_subqueue(vi->dev, vq2txq(sq->vq)); +} + static int virtnet_poll(struct napi_struct *napi, int budget) { struct receive_queue *rq = container_of(napi, struct receive_queue, napi); unsigned int received; + virtnet_poll_cleantx(rq); + received = virtnet_receive(rq, budget); /* Out of packets? */