[{"id":1775526,"web_url":"http://patchwork.ozlabs.org/comment/1775526/","msgid":"<20170926164055-mutt-send-email-mst@kernel.org>","list_archive_url":null,"date":"2017-09-26T13:45:54","subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","submitter":{"id":2235,"url":"http://patchwork.ozlabs.org/api/people/2235/","name":"Michael S. Tsirkin","email":"mst@redhat.com"},"content":"On Fri, Sep 22, 2017 at 04:02:30PM +0800, Jason Wang wrote:\n> Hi:\n> \n> This series tries to implement basic tx batched processing. This is\n> done by prefetching descriptor indices and update used ring in a\n> batch. This intends to speed up used ring updating and improve the\n> cache utilization.\n\nInteresting, thanks for the patches. So IIUC most of the gain is really\novercoming some of the shortcomings of virtio 1.0 wrt cache utilization?\n\nWhich is fair enough (1.0 is already deployed) but I would like to avoid\nmaking 1.1 support harder, and this patchset does this unfortunately,\nsee comments on individual patches. I'm sure it can be addressed though.\n\n> Test shows about ~22% improvement in tx pss.\n\nIs this with or without tx napi in guest?\n\n> Please review.\n> \n> Jason Wang (5):\n> vhost: split out ring head fetching logic\n> vhost: introduce helper to prefetch desc index\n> vhost: introduce vhost_add_used_idx()\n> vhost_net: rename VHOST_RX_BATCH to VHOST_NET_BATCH\n> vhost_net: basic tx virtqueue batched processing\n> \n> drivers/vhost/net.c | 221 ++++++++++++++++++++++++++++----------------------\n> drivers/vhost/vhost.c | 165 +++++++++++++++++++++++++++++++------\n> drivers/vhost/vhost.h | 9 ++\n> 3 files changed, 270 insertions(+), 125 deletions(-)\n> \n> -- \n> 2.7.4","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=mst@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3y1j0J4MNNz9t3m\n\tfor ;\n\tTue, 26 Sep 2017 23:46:08 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S969214AbdIZNp6 (ORCPT );\n\tTue, 26 Sep 2017 09:45:58 -0400","from mx1.redhat.com ([209.132.183.28]:54612 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S969198AbdIZNp4 (ORCPT );\n\tTue, 26 Sep 2017 09:45:56 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id DE54281233;\n\tTue, 26 Sep 2017 13:45:55 +0000 (UTC)","from redhat.com (ovpn-122-9.rdu2.redhat.com [10.10.122.9])\n\tby smtp.corp.redhat.com (Postfix) with SMTP id 440D466D52;\n\tTue, 26 Sep 2017 13:45:55 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com DE54281233","Date":"Tue, 26 Sep 2017 16:45:54 +0300","From":"\"Michael S. Tsirkin\" ","To":"Jason Wang ","Cc":"virtualization@lists.linux-foundation.org, netdev@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org, kvm@vger.kernel.org","Subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","Message-ID":"<20170926164055-mutt-send-email-mst@kernel.org>","References":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.25]);\n\tTue, 26 Sep 2017 13:45:56 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1775804,"web_url":"http://patchwork.ozlabs.org/comment/1775804/","msgid":"<20170926222528-mutt-send-email-mst@kernel.org>","list_archive_url":null,"date":"2017-09-26T19:26:40","subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","submitter":{"id":2235,"url":"http://patchwork.ozlabs.org/api/people/2235/","name":"Michael S. Tsirkin","email":"mst@redhat.com"},"content":"On Fri, Sep 22, 2017 at 04:02:30PM +0800, Jason Wang wrote:\n> Hi:\n> \n> This series tries to implement basic tx batched processing. This is\n> done by prefetching descriptor indices and update used ring in a\n> batch. This intends to speed up used ring updating and improve the\n> cache utilization. Test shows about ~22% improvement in tx pss.\n\n\n\n\n> Please review.\n> \n> Jason Wang (5):\n> vhost: split out ring head fetching logic\n> vhost: introduce helper to prefetch desc index\n> vhost: introduce vhost_add_used_idx()\n\nPlease squash these new APIs into where they are used.\nThis split-up just makes review harder for me as\nI can't figure out how the new APIs are used.\n\n\n> vhost_net: rename VHOST_RX_BATCH to VHOST_NET_BATCH\n\nThis is ok as a separate patch.\n\n> vhost_net: basic tx virtqueue batched processing\n> \n> drivers/vhost/net.c | 221 ++++++++++++++++++++++++++++----------------------\n> drivers/vhost/vhost.c | 165 +++++++++++++++++++++++++++++++------\n> drivers/vhost/vhost.h | 9 ++\n> 3 files changed, 270 insertions(+), 125 deletions(-)\n> \n> -- \n> 2.7.4","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ext-mx05.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx05.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=mst@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3y1rYT25NZz9t3m\n\tfor ;\n\tWed, 27 Sep 2017 05:26:53 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S970218AbdIZT0n (ORCPT );\n\tTue, 26 Sep 2017 15:26:43 -0400","from mx1.redhat.com ([209.132.183.28]:52182 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S965192AbdIZT0l (ORCPT );\n\tTue, 26 Sep 2017 15:26:41 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 8216215552;\n\tTue, 26 Sep 2017 19:26:41 +0000 (UTC)","from redhat.com (ovpn-122-9.rdu2.redhat.com [10.10.122.9])\n\tby smtp.corp.redhat.com (Postfix) with SMTP id CC61317484;\n\tTue, 26 Sep 2017 19:26:40 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 8216215552","Date":"Tue, 26 Sep 2017 22:26:40 +0300","From":"\"Michael S. Tsirkin\" ","To":"Jason Wang ","Cc":"virtualization@lists.linux-foundation.org, netdev@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org, kvm@vger.kernel.org","Subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","Message-ID":"<20170926222528-mutt-send-email-mst@kernel.org>","References":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.15","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.29]);\n\tTue, 26 Sep 2017 19:26:41 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1775961,"web_url":"http://patchwork.ozlabs.org/comment/1775961/","msgid":"","list_archive_url":null,"date":"2017-09-27T00:27:37","subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","submitter":{"id":5225,"url":"http://patchwork.ozlabs.org/api/people/5225/","name":"Jason Wang","email":"jasowang@redhat.com"},"content":"On 2017年09月26日 21:45, Michael S. Tsirkin wrote:\n> On Fri, Sep 22, 2017 at 04:02:30PM +0800, Jason Wang wrote:\n>> Hi:\n>>\n>> This series tries to implement basic tx batched processing. This is\n>> done by prefetching descriptor indices and update used ring in a\n>> batch. This intends to speed up used ring updating and improve the\n>> cache utilization.\n> Interesting, thanks for the patches. So IIUC most of the gain is really\n> overcoming some of the shortcomings of virtio 1.0 wrt cache utilization?\n\nYes.\n\nActually, looks like batching in 1.1 is not as easy as in 1.0.\n\nIn 1.0, we could do something like:\n\nbatch update used ring by user copy_to_user()\nsmp_wmb()\nupdate used_idx\n\nIn 1.1, we need more memory barriers, can't benefit from fast copy helpers?\n\nfor () {\n     update desc.addr\n     smp_wmb()\n     update desc.flag\n}\n\n>\n> Which is fair enough (1.0 is already deployed) but I would like to avoid\n> making 1.1 support harder, and this patchset does this unfortunately,\n\nI think the new APIs do not expose more internal data structure of \nvirtio than before? (vq->heads has already been used by vhost_net for \nyears). Consider the layout is re-designed completely, I don't see an \neasy method to reuse current 1.0 API for 1.1.\n\n> see comments on individual patches. I'm sure it can be addressed though.\n>\n>> Test shows about ~22% improvement in tx pss.\n> Is this with or without tx napi in guest?\n\nMoonGen is used in guest for better numbers.\n\nThanks\n\n>\n>> Please review.\n>>\n>> Jason Wang (5):\n>> vhost: split out ring head fetching logic\n>> vhost: introduce helper to prefetch desc index\n>> vhost: introduce vhost_add_used_idx()\n>> vhost_net: rename VHOST_RX_BATCH to VHOST_NET_BATCH\n>> vhost_net: basic tx virtqueue batched processing\n>>\n>> drivers/vhost/net.c | 221 ++++++++++++++++++++++++++++----------------------\n>> drivers/vhost/vhost.c | 165 +++++++++++++++++++++++++++++++------\n>> drivers/vhost/vhost.h | 9 ++\n>> 3 files changed, 270 insertions(+), 125 deletions(-)\n>>\n>> -- \n>> 2.7.4","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ext-mx06.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx06.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=jasowang@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3y1zDt60vqz9t4b\n\tfor ;\n\tWed, 27 Sep 2017 10:27:58 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S970735AbdI0A1t (ORCPT );\n\tTue, 26 Sep 2017 20:27:49 -0400","from mx1.redhat.com ([209.132.183.28]:41296 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S964956AbdI0A1q (ORCPT );\n\tTue, 26 Sep 2017 20:27:46 -0400","from smtp.corp.redhat.com\n\t(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 1C1BC356CC;\n\tWed, 27 Sep 2017 00:27:46 +0000 (UTC)","from [10.72.12.40] (ovpn-12-40.pek2.redhat.com [10.72.12.40])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 8C9947D4E0;\n\tWed, 27 Sep 2017 00:27:41 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 1C1BC356CC","Subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","To":"\"Michael S. Tsirkin\" ","Cc":"virtualization@lists.linux-foundation.org, netdev@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org, kvm@vger.kernel.org","References":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>\n\t<20170926164055-mutt-send-email-mst@kernel.org>","From":"Jason Wang ","Message-ID":"","Date":"Wed, 27 Sep 2017 08:27:37 +0800","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20170926164055-mutt-send-email-mst@kernel.org>","Content-Type":"text/plain; charset=utf-8; format=flowed","Content-Transfer-Encoding":"8bit","Content-Language":"en-US","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.11","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.30]);\n\tWed, 27 Sep 2017 00:27:46 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1775990,"web_url":"http://patchwork.ozlabs.org/comment/1775990/","msgid":"","list_archive_url":null,"date":"2017-09-27T02:06:43","subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","submitter":{"id":5225,"url":"http://patchwork.ozlabs.org/api/people/5225/","name":"Jason Wang","email":"jasowang@redhat.com"},"content":"On 2017年09月27日 03:26, Michael S. Tsirkin wrote:\n> On Fri, Sep 22, 2017 at 04:02:30PM +0800, Jason Wang wrote:\n>> Hi:\n>>\n>> This series tries to implement basic tx batched processing. This is\n>> done by prefetching descriptor indices and update used ring in a\n>> batch. This intends to speed up used ring updating and improve the\n>> cache utilization. Test shows about ~22% improvement in tx pss.\n>\n>\n>\n>> Please review.\n>>\n>> Jason Wang (5):\n>> vhost: split out ring head fetching logic\n>> vhost: introduce helper to prefetch desc index\n>> vhost: introduce vhost_add_used_idx()\n> Please squash these new APIs into where they are used.\n> This split-up just makes review harder for me as\n> I can't figure out how the new APIs are used.\n\nOk.\n\nThanks\n\n>\n>\n>> vhost_net: rename VHOST_RX_BATCH to VHOST_NET_BATCH\n> This is ok as a separate patch.\n>\n>> vhost_net: basic tx virtqueue batched processing\n>>\n>> drivers/vhost/net.c | 221 ++++++++++++++++++++++++++++----------------------\n>> drivers/vhost/vhost.c | 165 +++++++++++++++++++++++++++++++------\n>> drivers/vhost/vhost.h | 9 ++\n>> 3 files changed, 270 insertions(+), 125 deletions(-)\n>>\n>> -- \n>> 2.7.4","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=jasowang@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3y21RD61krz9t3m\n\tfor ;\n\tWed, 27 Sep 2017 12:07:04 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S969960AbdI0CGv (ORCPT );\n\tTue, 26 Sep 2017 22:06:51 -0400","from mx1.redhat.com ([209.132.183.28]:33850 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S968086AbdI0CGt (ORCPT );\n\tTue, 26 Sep 2017 22:06:49 -0400","from smtp.corp.redhat.com\n\t(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id B722880F90;\n\tWed, 27 Sep 2017 02:06:49 +0000 (UTC)","from [10.72.12.40] (ovpn-12-40.pek2.redhat.com [10.72.12.40])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id CBD7978DDB;\n\tWed, 27 Sep 2017 02:06:45 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com B722880F90","Subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","To":"\"Michael S. Tsirkin\" ","Cc":"virtualization@lists.linux-foundation.org, netdev@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org, kvm@vger.kernel.org","References":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>\n\t<20170926222528-mutt-send-email-mst@kernel.org>","From":"Jason Wang ","Message-ID":"","Date":"Wed, 27 Sep 2017 10:06:43 +0800","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20170926222528-mutt-send-email-mst@kernel.org>","Content-Type":"text/plain; charset=utf-8; format=flowed","Content-Transfer-Encoding":"8bit","Content-Language":"en-US","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.12","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.27]);\n\tWed, 27 Sep 2017 02:06:49 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1776662,"web_url":"http://patchwork.ozlabs.org/comment/1776662/","msgid":"<20170928012009-mutt-send-email-mst@kernel.org>","list_archive_url":null,"date":"2017-09-27T22:28:54","subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","submitter":{"id":2235,"url":"http://patchwork.ozlabs.org/api/people/2235/","name":"Michael S. Tsirkin","email":"mst@redhat.com"},"content":"On Wed, Sep 27, 2017 at 08:27:37AM +0800, Jason Wang wrote:\n> \n> \n> On 2017年09月26日 21:45, Michael S. Tsirkin wrote:\n> > On Fri, Sep 22, 2017 at 04:02:30PM +0800, Jason Wang wrote:\n> > > Hi:\n> > > \n> > > This series tries to implement basic tx batched processing. This is\n> > > done by prefetching descriptor indices and update used ring in a\n> > > batch. This intends to speed up used ring updating and improve the\n> > > cache utilization.\n> > Interesting, thanks for the patches. So IIUC most of the gain is really\n> > overcoming some of the shortcomings of virtio 1.0 wrt cache utilization?\n> \n> Yes.\n> \n> Actually, looks like batching in 1.1 is not as easy as in 1.0.\n> \n> In 1.0, we could do something like:\n> \n> batch update used ring by user copy_to_user()\n> smp_wmb()\n> update used_idx\n> In 1.1, we need more memory barriers, can't benefit from fast copy helpers?\n> \n> for () {\n>     update desc.addr\n>     smp_wmb()\n>     update desc.flag\n> }\n\nYes but smp_wmb is a NOP on e.g. x86. We can switch to other types of\nbarriers as well. We do need to do the updates in order, so we might\nneed new APIs for that to avoid re-doing the translation all the time.\n\nIn 1.0 the last update is a cache miss always. You need batching to get\nless misses. In 1.1 you don't have it so fundamentally there is less\nneed for batching. But batching does not always work. DPDK guys (which\nbatch things aggressively) already tried 1.1 and saw performance gains\nso we do not need to argue theoretically.\n\n\n\n> > \n> > Which is fair enough (1.0 is already deployed) but I would like to avoid\n> > making 1.1 support harder, and this patchset does this unfortunately,\n> \n> I think the new APIs do not expose more internal data structure of virtio\n> than before? (vq->heads has already been used by vhost_net for years).\n\nFor sure we might need to change vring_used_elem.\n\n> Consider the layout is re-designed completely, I don't see an easy method to\n> reuse current 1.0 API for 1.1.\n\nCurrent API just says you get buffers then you use them. It is not tied\nto actual separate used ring.\n\n\n> > see comments on individual patches. I'm sure it can be addressed though.\n> > \n> > > Test shows about ~22% improvement in tx pss.\n> > Is this with or without tx napi in guest?\n> \n> MoonGen is used in guest for better numbers.\n> \n> Thanks\n\nNot sure I understand. Did you set napi_tx to true or false?\n\n> > \n> > > Please review.\n> > > \n> > > Jason Wang (5):\n> > > vhost: split out ring head fetching logic\n> > > vhost: introduce helper to prefetch desc index\n> > > vhost: introduce vhost_add_used_idx()\n> > > vhost_net: rename VHOST_RX_BATCH to VHOST_NET_BATCH\n> > > vhost_net: basic tx virtqueue batched processing\n> > > \n> > > drivers/vhost/net.c | 221 ++++++++++++++++++++++++++++----------------------\n> > > drivers/vhost/vhost.c | 165 +++++++++++++++++++++++++++++++------\n> > > drivers/vhost/vhost.h | 9 ++\n> > > 3 files changed, 270 insertions(+), 125 deletions(-)\n> > > \n> > > -- \n> > > 2.7.4","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx10.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=mst@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3y2XYL5SRjz9t6W\n\tfor ;\n\tThu, 28 Sep 2017 08:29:10 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752166AbdI0W25 (ORCPT );\n\tWed, 27 Sep 2017 18:28:57 -0400","from mx1.redhat.com ([209.132.183.28]:41096 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1751355AbdI0W24 (ORCPT );\n\tWed, 27 Sep 2017 18:28:56 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id CB099257E95;\n\tWed, 27 Sep 2017 22:28:55 +0000 (UTC)","from redhat.com (ovpn-123-149.rdu2.redhat.com [10.10.123.149])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 3CF5E8BE24;\n\tWed, 27 Sep 2017 22:28:55 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com CB099257E95","Date":"Thu, 28 Sep 2017 01:28:54 +0300","From":"\"Michael S. Tsirkin\" ","To":"Jason Wang ","Cc":"virtualization@lists.linux-foundation.org, netdev@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org, kvm@vger.kernel.org","Subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","Message-ID":"<20170928012009-mutt-send-email-mst@kernel.org>","References":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>\n\t<20170926164055-mutt-send-email-mst@kernel.org>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=utf-8","Content-Disposition":"inline","Content-Transfer-Encoding":"8bit","In-Reply-To":"","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.14","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.39]);\n\tWed, 27 Sep 2017 22:28:55 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1776815,"web_url":"http://patchwork.ozlabs.org/comment/1776815/","msgid":"<030a3f75-a957-2502-a8cb-e7781827a61c@redhat.com>","list_archive_url":null,"date":"2017-09-28T07:16:59","subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","submitter":{"id":5225,"url":"http://patchwork.ozlabs.org/api/people/5225/","name":"Jason Wang","email":"jasowang@redhat.com"},"content":"On 2017年09月28日 06:28, Michael S. Tsirkin wrote:\n> On Wed, Sep 27, 2017 at 08:27:37AM +0800, Jason Wang wrote:\n>>\n>> On 2017年09月26日 21:45, Michael S. Tsirkin wrote:\n>>> On Fri, Sep 22, 2017 at 04:02:30PM +0800, Jason Wang wrote:\n>>>> Hi:\n>>>>\n>>>> This series tries to implement basic tx batched processing. This is\n>>>> done by prefetching descriptor indices and update used ring in a\n>>>> batch. This intends to speed up used ring updating and improve the\n>>>> cache utilization.\n>>> Interesting, thanks for the patches. So IIUC most of the gain is really\n>>> overcoming some of the shortcomings of virtio 1.0 wrt cache utilization?\n>> Yes.\n>>\n>> Actually, looks like batching in 1.1 is not as easy as in 1.0.\n>>\n>> In 1.0, we could do something like:\n>>\n>> batch update used ring by user copy_to_user()\n>> smp_wmb()\n>> update used_idx\n>> In 1.1, we need more memory barriers, can't benefit from fast copy helpers?\n>>\n>> for () {\n>>     update desc.addr\n>>     smp_wmb()\n>>     update desc.flag\n>> }\n> Yes but smp_wmb is a NOP on e.g. x86. We can switch to other types of\n> barriers as well.\n\nWe need consider non x86 platforms as well. And we meet similar things \non batch reading.\n\n> We do need to do the updates in order, so we might\n> need new APIs for that to avoid re-doing the translation all the time.\n>\n> In 1.0 the last update is a cache miss always. You need batching to get\n> less misses. In 1.1 you don't have it so fundamentally there is less\n> need for batching. But batching does not always work. DPDK guys (which\n> batch things aggressively) already tried 1.1 and saw performance gains\n> so we do not need to argue theoretically.\n\nDo they test on non-x86 CPUs? And the prototype should be reviewed \ncarefully since a bug can boost the performance in this case.\n\n>\n>\n>\n>>> Which is fair enough (1.0 is already deployed) but I would like to avoid\n>>> making 1.1 support harder, and this patchset does this unfortunately,\n>> I think the new APIs do not expose more internal data structure of virtio\n>> than before? (vq->heads has already been used by vhost_net for years).\n> For sure we might need to change vring_used_elem.\n>\n>> Consider the layout is re-designed completely, I don't see an easy method to\n>> reuse current 1.0 API for 1.1.\n> Current API just says you get buffers then you use them. It is not tied\n> to actual separate used ring.\n\nSo do this patch I think? It introduces:\n\nint vhost_prefetch_desc_indices(struct vhost_virtqueue *vq,\n                 struct vring_used_elem *heads,\n                 u16 num);\nint vhost_add_used_idx(struct vhost_virtqueue *vq, int n);\n\nThere's nothing new exposed to vhost_net. (vring_used_elem has been used \nby net.c at many places without this patch).\n\n>\n>\n>>> see comments on individual patches. I'm sure it can be addressed though.\n>>>\n>>>> Test shows about ~22% improvement in tx pss.\n>>> Is this with or without tx napi in guest?\n>> MoonGen is used in guest for better numbers.\n>>\n>> Thanks\n> Not sure I understand. Did you set napi_tx to true or false?\n\nMoonGen uses dpdk, so virtio-net pmd is used in guest. (See \nhttp://conferences.sigcomm.org/imc/2015/papers/p275.pdf)\n\nThanks\n\n>\n>>>> Please review.\n>>>>\n>>>> Jason Wang (5):\n>>>> vhost: split out ring head fetching logic\n>>>> vhost: introduce helper to prefetch desc index\n>>>> vhost: introduce vhost_add_used_idx()\n>>>> vhost_net: rename VHOST_RX_BATCH to VHOST_NET_BATCH\n>>>> vhost_net: basic tx virtqueue batched processing\n>>>>\n>>>> drivers/vhost/net.c | 221 ++++++++++++++++++++++++++++----------------------\n>>>> drivers/vhost/vhost.c | 165 +++++++++++++++++++++++++++++++------\n>>>> drivers/vhost/vhost.h | 9 ++\n>>>> 3 files changed, 270 insertions(+), 125 deletions(-)\n>>>>\n>>>> -- \n>>>> 2.7.4","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx07.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=jasowang@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3y2mGt40ygz9t5x\n\tfor ;\n\tThu, 28 Sep 2017 17:17:26 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751692AbdI1HRM (ORCPT );\n\tThu, 28 Sep 2017 03:17:12 -0400","from mx1.redhat.com ([209.132.183.28]:38612 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1750914AbdI1HRL (ORCPT );\n\tThu, 28 Sep 2017 03:17:11 -0400","from smtp.corp.redhat.com\n\t(int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id D9667C1027E8;\n\tThu, 28 Sep 2017 07:17:10 +0000 (UTC)","from [10.72.12.139] (ovpn-12-139.pek2.redhat.com [10.72.12.139])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id D8B758F364;\n\tThu, 28 Sep 2017 07:17:03 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com D9667C1027E8","Subject":"Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net","To":"\"Michael S. Tsirkin\" ","Cc":"virtualization@lists.linux-foundation.org, netdev@vger.kernel.org,\n\tlinux-kernel@vger.kernel.org, kvm@vger.kernel.org","References":"<1506067355-5771-1-git-send-email-jasowang@redhat.com>\n\t<20170926164055-mutt-send-email-mst@kernel.org>\n\t\n\t<20170928012009-mutt-send-email-mst@kernel.org>","From":"Jason Wang ","Message-ID":"<030a3f75-a957-2502-a8cb-e7781827a61c@redhat.com>","Date":"Thu, 28 Sep 2017 15:16:59 +0800","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<20170928012009-mutt-send-email-mst@kernel.org>","Content-Type":"text/plain; charset=utf-8; format=flowed","Content-Transfer-Encoding":"8bit","Content-Language":"en-US","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.11","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.31]);\n\tThu, 28 Sep 2017 07:17:11 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}}]