[{"id":1761436,"web_url":"http://patchwork.ozlabs.org/comment/1761436/","msgid":"<20170901021007.pkbb2gsipbprf4w7@ast-mbp>","list_archive_url":null,"date":"2017-09-01T02:10:09","subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","submitter":{"id":42586,"url":"http://patchwork.ozlabs.org/api/people/42586/","name":"Alexei Starovoitov","email":"alexei.starovoitov@gmail.com"},"content":"On Thu, Aug 31, 2017 at 05:00:13PM -0400, Willem de Bruijn wrote:\n> From: Willem de Bruijn \n> \n> Documentation for this feature was missing from the patchset.\n> Copied a lot from the netdev 2.1 paper, addressing some small\n> interface changes since then.\n> \n> Signed-off-by: Willem de Bruijn \n...\n> +Notification Batching\n> +~~~~~~~~~~~~~~~~~~~~~\n> +\n> +Multiple outstanding packets can be read at once using the recvmmsg\n> +call. This is often not needed. In each message the kernel returns not\n> +a single value, but a range. It coalesces consecutive notifications\n> +while one is outstanding for reception on the error queue.\n> +\n> +When a new notification is about to be queued, it checks whether the\n> +new value extends the range of the notification at the tail of the\n> +queue. If so, it drops the new notification packet and instead increases\n> +the range upper value of the outstanding notification.\n\nWould it make sense to mention that max notification range is 32-bit?\nSo each 4Gbyte of xmit bytes there will be a notification.\nIn modern 40Gbps NICs it's not a lot. Means that there will be\nat least one notification every second.\nOr I misread the code?\n\nThanks for the doc!\n\nAcked-by: Alexei Starovoitov ","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"rkDh2+Z+\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xk2kw0Hdyz9sPm\n\tfor ;\n\tFri, 1 Sep 2017 12:10:16 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751009AbdIACKO (ORCPT );\n\tThu, 31 Aug 2017 22:10:14 -0400","from mail-pg0-f66.google.com ([74.125.83.66]:36329 \"EHLO\n\tmail-pg0-f66.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750762AbdIACKN (ORCPT\n\t); Thu, 31 Aug 2017 22:10:13 -0400","by mail-pg0-f66.google.com with SMTP id 83so839769pgb.3\n\tfor ; Thu, 31 Aug 2017 19:10:12 -0700 (PDT)","from ast-mbp ([2620:10d:c090:180::1:f66])\n\tby smtp.gmail.com with ESMTPSA id\n\tw184sm1100659pgb.29.2017.08.31.19.10.10\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tThu, 31 Aug 2017 19:10:11 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:in-reply-to:user-agent;\n\tbh=d1D16NYS7VKvLE9nwTtf6HsC3YuRrSnyLVXFdx3O/Qg=;\n\tb=rkDh2+Z++VarYo8cI9sAh5PjHDKN9LlIGxXP1yHIJdwBgUp0I87+No8BsiogTxM6kR\n\tQhWr3X8K0ltIDnnV6mDqJhWnC2NcKiwX999c4NvUhQIumEYLKjMAVPFeJYV2xOyppSHm\n\tkIcpdlX2hROmws9txe2Qv9psXZ4xw0mCQkZarZeU3Xqotxml9VxEWdOObYuwKGljRI6V\n\tfnIYUVbvOK2OuHCeMr3FEo5k0FuNiCpuF8JdWXBxBuz27v7NjwEU64YDgYbbgKF9nddG\n\ty4Oso7+L2DbT9Z/tGEjq2HjCwkfACDSsVx0Do3Izykn5lp08ceKTw5EbmjX1ZFPDlamo\n\tu3RQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:in-reply-to:user-agent;\n\tbh=d1D16NYS7VKvLE9nwTtf6HsC3YuRrSnyLVXFdx3O/Qg=;\n\tb=eQZy3ih7/DDgw4rnUsScpH6SZN7aaBW2C3b0T37t+nx4TAFOM2q0ss8zv/UmDzDWKp\n\tvrIAFrP/kmVAxWpFj2whXM59EZR3Mjh3unT3G1pkdsrvIy3CC4x7xC6NM6ZM4l9RQh1g\n\t6kk4Ix/1mI6IgRspYoptbJ5N/NXONx5j+MKK7r1EA6odn1aLxF6/5I4lC8QD9pKwO5KF\n\tJPDMafabcsWLa8GunOJ6zOD7G6/4z/LfI4tDPJT4wpDx2nwB7rIRh5ZESGmr19JChXR6\n\tqHQgLQe8xQu/leM4dpbVWodlSnxIo0qmQfQRBBgPjUfP7dmv0YQDywsvT0l/d3VvpsuD\n\tTDKQ==","X-Gm-Message-State":"AHPjjUgVTP8R07LJHuJhKvNuP7guF10PckBLjvd56igjL/JKLbrAS347\n\tyRXb2FxFEXCEwQmK6Ek=","X-Google-Smtp-Source":"ADKCNb4EBNF+DriiLGr2KXhrDvioQnUpLrifogqnfQ73I1lRb+WoyJkrLlRYRoINd6xPT29QM4BTMg==","X-Received":"by 10.84.214.143 with SMTP id j15mr541280pli.338.1504231812575; \n\tThu, 31 Aug 2017 19:10:12 -0700 (PDT)","Date":"Thu, 31 Aug 2017 19:10:09 -0700","From":"Alexei Starovoitov ","To":"Willem de Bruijn ","Cc":"netdev@vger.kernel.org, davem@davemloft.net,\n\tWillem de Bruijn ","Subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","Message-ID":"<20170901021007.pkbb2gsipbprf4w7@ast-mbp>","References":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>","User-Agent":"NeoMutt/20170421 (1.8.2)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761447,"web_url":"http://patchwork.ozlabs.org/comment/1761447/","msgid":"","list_archive_url":null,"date":"2017-09-01T03:04:41","subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","submitter":{"id":67615,"url":"http://patchwork.ozlabs.org/api/people/67615/","name":"Willem de Bruijn","email":"willemdebruijn.kernel@gmail.com"},"content":"On Thu, Aug 31, 2017 at 10:10 PM, Alexei Starovoitov\n wrote:\n> On Thu, Aug 31, 2017 at 05:00:13PM -0400, Willem de Bruijn wrote:\n>> From: Willem de Bruijn \n>>\n>> Documentation for this feature was missing from the patchset.\n>> Copied a lot from the netdev 2.1 paper, addressing some small\n>> interface changes since then.\n>>\n>> Signed-off-by: Willem de Bruijn \n> ...\n>> +Notification Batching\n>> +~~~~~~~~~~~~~~~~~~~~~\n>> +\n>> +Multiple outstanding packets can be read at once using the recvmmsg\n>> +call. This is often not needed. In each message the kernel returns not\n>> +a single value, but a range. It coalesces consecutive notifications\n>> +while one is outstanding for reception on the error queue.\n>> +\n>> +When a new notification is about to be queued, it checks whether the\n>> +new value extends the range of the notification at the tail of the\n>> +queue. If so, it drops the new notification packet and instead increases\n>> +the range upper value of the outstanding notification.\n>\n> Would it make sense to mention that max notification range is 32-bit?\n> So each 4Gbyte of xmit bytes there will be a notification.\n> In modern 40Gbps NICs it's not a lot. Means that there will be\n> at least one notification every second.\n> Or I misread the code?\n\nYou're right. The doc does mention that the counter and range\nare 32-bit. I can state more explicitly that that bounds the working\nset size to 4GB. Do you expect this to be problematic? Processing\na single notification per 4GB of data should not be a significant\ncost in itself.\n\n> Thanks for the doc!\n\nThanks for reviewing :)\n\n>\n> Acked-by: Alexei Starovoitov \n>","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"fXMvFchW\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xk3yZ4BMMz9s83\n\tfor ;\n\tFri, 1 Sep 2017 13:05:26 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751431AbdIADFY (ORCPT );\n\tThu, 31 Aug 2017 23:05:24 -0400","from mail-oi0-f66.google.com ([209.85.218.66]:35344 \"EHLO\n\tmail-oi0-f66.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751243AbdIADFW (ORCPT\n\t); Thu, 31 Aug 2017 23:05:22 -0400","by mail-oi0-f66.google.com with SMTP id n18so865522oig.2\n\tfor ; Thu, 31 Aug 2017 20:05:22 -0700 (PDT)","by 10.168.52.10 with HTTP; Thu, 31 Aug 2017 20:04:41 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc; bh=N8D0P2Jr5EPxtKRB9ktBPs/4gN4nJeG658lxE2T2Tak=;\n\tb=fXMvFchW6Kz6JB9PROAQZjfeEZGQ7t/Zs+HlqnQdmJnh3Ttky2/dJEfy7f80X3OUOO\n\tHJVmDJ3oTMCVPlpBP1ZgvvmaN2vMxtqoOgVSMxH8/mGxS7FjW1EZagqzkYunxppOVDQN\n\txlBRiX1Mmc6P5sPQrgAkzBq+sv43xJptDAPtBXk5E2C7rLWwuB8TFHeSnqxdGfelfA//\n\tjm0f6rRvw2zlyMV3doC1WP09sYyYHoFufYJNFj5NriGaNTlmsMoJPrxD2oAISNjll8g9\n\t+/8k0fINcHmF+76VyGftrPeWaeDr77brPp5dTA3SuOpKFkYdL/eBI1CkdHW8uNt8TR7I\n\ttKMw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc;\n\tbh=N8D0P2Jr5EPxtKRB9ktBPs/4gN4nJeG658lxE2T2Tak=;\n\tb=rfVaewDud5AvUdWuxczmTVSlGrWOczICktZ8dociTzIfkP6u9NFeLQ/ZvSrALzQckB\n\tPeBXwta4eaQxFGBOdWX5IJ/eYWvbHjdjYoQLR8NvB1MiDQVOLE1NV9agYAAxsccGPur4\n\tjN5nC1M5uQFKyMWRxlODoYr3mMFKF2kl0nwPZlW2e3kOx6ntGCttq78rXpGx3OenD8g5\n\t5dCjS6pWPmJHkbiBnexjD+Nq7MrIG7YBLdMT4+18FXeBjOg/Oa5bw26gR75nLenfUawm\n\t3L68/QQYj9uyu+WC9pnprjwCNEHxEABC5uC4Crc6ctXqTDOuc7tkY08hC+ln1DxyxMEw\n\tt4Hw==","X-Gm-Message-State":"AHPjjUgOhYejQq4qUG3Iv23SC/ltjVOzgbaTSfYi3OXPBxTiqnXGRQq8\n\t170s4g4aCKaRqv2TpbbU9MSwVM46tRgQ8wc=","X-Google-Smtp-Source":"ADKCNb40spZBS13rR7fuEq1RkUzybl3mHxSsPbPq5M0vQryQ3yBa8NXgx8irVsgPb80kYHWJ9QHwPxotRMKGd8tAbqI=","X-Received":"by 10.202.48.194 with SMTP id w185mr629678oiw.196.1504235121882; \n\tThu, 31 Aug 2017 20:05:21 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170901021007.pkbb2gsipbprf4w7@ast-mbp>","References":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>\n\t<20170901021007.pkbb2gsipbprf4w7@ast-mbp>","From":"Willem de Bruijn ","Date":"Thu, 31 Aug 2017 23:04:41 -0400","Message-ID":"","Subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","To":"Alexei Starovoitov ","Cc":"Network Development ,\n\tDavid Miller , Willem de Bruijn ","Content-Type":"text/plain; charset=\"UTF-8\"","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761452,"web_url":"http://patchwork.ozlabs.org/comment/1761452/","msgid":"<20170901031009.2p4vv6fncma2y2l7@ast-mbp>","list_archive_url":null,"date":"2017-09-01T03:10:11","subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","submitter":{"id":42586,"url":"http://patchwork.ozlabs.org/api/people/42586/","name":"Alexei Starovoitov","email":"alexei.starovoitov@gmail.com"},"content":"On Thu, Aug 31, 2017 at 11:04:41PM -0400, Willem de Bruijn wrote:\n> On Thu, Aug 31, 2017 at 10:10 PM, Alexei Starovoitov\n> wrote:\n> > On Thu, Aug 31, 2017 at 05:00:13PM -0400, Willem de Bruijn wrote:\n> >> From: Willem de Bruijn \n> >>\n> >> Documentation for this feature was missing from the patchset.\n> >> Copied a lot from the netdev 2.1 paper, addressing some small\n> >> interface changes since then.\n> >>\n> >> Signed-off-by: Willem de Bruijn \n> > ...\n> >> +Notification Batching\n> >> +~~~~~~~~~~~~~~~~~~~~~\n> >> +\n> >> +Multiple outstanding packets can be read at once using the recvmmsg\n> >> +call. This is often not needed. In each message the kernel returns not\n> >> +a single value, but a range. It coalesces consecutive notifications\n> >> +while one is outstanding for reception on the error queue.\n> >> +\n> >> +When a new notification is about to be queued, it checks whether the\n> >> +new value extends the range of the notification at the tail of the\n> >> +queue. If so, it drops the new notification packet and instead increases\n> >> +the range upper value of the outstanding notification.\n> >\n> > Would it make sense to mention that max notification range is 32-bit?\n> > So each 4Gbyte of xmit bytes there will be a notification.\n> > In modern 40Gbps NICs it's not a lot. Means that there will be\n> > at least one notification every second.\n> > Or I misread the code?\n> \n> You're right. The doc does mention that the counter and range\n> are 32-bit. I can state more explicitly that that bounds the working\n> set size to 4GB. Do you expect this to be problematic? Processing\n> a single notification per 4GB of data should not be a significant\n> cost in itself.\n\nI think 4GB is fine. Just there was an idea that in cases when\nnotification of transmission can be known by other means the user space\ncould have skipped reading errqeuee completely, but looks like it\nstill needs to poll. That's fine.\n\n> > Thanks for the doc!\n> \n> Thanks for reviewing :)\n> \n> >\n> > Acked-by: Alexei Starovoitov \n> >","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"qNk2I6g2\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xk44965jWz9sPt\n\tfor ;\n\tFri, 1 Sep 2017 13:10:17 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751345AbdIADKP (ORCPT );\n\tThu, 31 Aug 2017 23:10:15 -0400","from mail-pg0-f68.google.com ([74.125.83.68]:32939 \"EHLO\n\tmail-pg0-f68.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750925AbdIADKO (ORCPT\n\t); Thu, 31 Aug 2017 23:10:14 -0400","by mail-pg0-f68.google.com with SMTP id m15so924753pgc.0\n\tfor ; Thu, 31 Aug 2017 20:10:14 -0700 (PDT)","from ast-mbp ([2620:10d:c090:180::1:f66])\n\tby smtp.gmail.com with ESMTPSA id\n\tc201sm1193286pfb.149.2017.08.31.20.10.12\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tThu, 31 Aug 2017 20:10:13 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=date:from:to:cc:subject:message-id:references:mime-version\n\t:content-disposition:in-reply-to:user-agent;\n\tbh=o96moG6wU7Kz2EoLSicrV5MgEUWRv7/LJ+E3G18apgU=;\n\tb=qNk2I6g2v6AWIfX+XYH0kVFtC9aKfxuvdXH1a16xfL/QAbiwwB+Ok5w2C2RhZu8bud\n\t21d/UsRuH1lmrHnce0/gHzTAW6+PEkfIk4nLvZ8C2FgXjNQoXw1Q+Sx3GrhUliPG4L6a\n\trHDUFBM1cQXJehQTnaNZcXY9b4CXIsdh+qkLjWBYi0ZIGAbUwS3E/W2LayHPw67lSPN7\n\tGX40/WVZMtlZoHfYgXgyclcOXIERLLeHQCQv+q7VZ+xtcETlHyv7wMO3Lobye7UKMOB9\n\t07CMztFH0egaeypgOTcq/W38K66S6qllsI9koB2kE/wIXV5MzXo3RrOkRibBNAiqUTxg\n\tDhXg==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:references\n\t:mime-version:content-disposition:in-reply-to:user-agent;\n\tbh=o96moG6wU7Kz2EoLSicrV5MgEUWRv7/LJ+E3G18apgU=;\n\tb=gey62eHPvi6UjDLyAI3INBH4culyauA8KDZjJFea3Si8U8fAezNet51rYhzMfghd6l\n\typ+05T29zV86q5OdX+1oS4KQg/MdgUd7DZ905aApE00KdYF89Rzw82Hw4VcYIELR5S+Y\n\tJhRzCNZXCIatHQoZgp48T12c70pOFYKmZYDNk3YSiO8GvKfr+WFQVKYVkHmqtszbhlT3\n\tFw46V/LEdcvec/B5Kv+CVOBT69ReQvjkYqbl8whPPxUfnU1/Z8mX2NCtc01JlVi/30EN\n\ttf3sgnW30AdgeEb4FFzmpjqs4SQnStv/TUGjufJvP/sJN0FrWU8owZcHEfeoWLcYUcYs\n\t/n2A==","X-Gm-Message-State":"AHPjjUjHF7H0fajgfK2h2mf1IxLc4vtBedmygIubYo0C+1VfES2SrPpD\n\tCE09U7wYiKx0/w==","X-Google-Smtp-Source":"ADKCNb42xbkSahs9AHNibTqPKcDc4V5cHidnMMsiwM4g/i+Nb6nbExbAfmiTndE2kx+zd+fpPsG1Og==","X-Received":"by 10.99.0.83 with SMTP id 80mr731638pga.82.1504235414302;\n\tThu, 31 Aug 2017 20:10:14 -0700 (PDT)","Date":"Thu, 31 Aug 2017 20:10:11 -0700","From":"Alexei Starovoitov ","To":"Willem de Bruijn ","Cc":"Network Development ,\n\tDavid Miller , Willem de Bruijn ","Subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","Message-ID":"<20170901031009.2p4vv6fncma2y2l7@ast-mbp>","References":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>\n\t<20170901021007.pkbb2gsipbprf4w7@ast-mbp>\n\t","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"","User-Agent":"NeoMutt/20170421 (1.8.2)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761456,"web_url":"http://patchwork.ozlabs.org/comment/1761456/","msgid":"","list_archive_url":null,"date":"2017-09-01T03:31:54","subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","submitter":{"id":67615,"url":"http://patchwork.ozlabs.org/api/people/67615/","name":"Willem de Bruijn","email":"willemdebruijn.kernel@gmail.com"},"content":"On Thu, Aug 31, 2017 at 11:10 PM, Alexei Starovoitov\n wrote:\n> On Thu, Aug 31, 2017 at 11:04:41PM -0400, Willem de Bruijn wrote:\n>> On Thu, Aug 31, 2017 at 10:10 PM, Alexei Starovoitov\n>> wrote:\n>> > On Thu, Aug 31, 2017 at 05:00:13PM -0400, Willem de Bruijn wrote:\n>> >> From: Willem de Bruijn \n>> >>\n>> >> Documentation for this feature was missing from the patchset.\n>> >> Copied a lot from the netdev 2.1 paper, addressing some small\n>> >> interface changes since then.\n>> >>\n>> >> Signed-off-by: Willem de Bruijn \n>> > ...\n>> >> +Notification Batching\n>> >> +~~~~~~~~~~~~~~~~~~~~~\n>> >> +\n>> >> +Multiple outstanding packets can be read at once using the recvmmsg\n>> >> +call. This is often not needed. In each message the kernel returns not\n>> >> +a single value, but a range. It coalesces consecutive notifications\n>> >> +while one is outstanding for reception on the error queue.\n>> >> +\n>> >> +When a new notification is about to be queued, it checks whether the\n>> >> +new value extends the range of the notification at the tail of the\n>> >> +queue. If so, it drops the new notification packet and instead increases\n>> >> +the range upper value of the outstanding notification.\n>> >\n>> > Would it make sense to mention that max notification range is 32-bit?\n>> > So each 4Gbyte of xmit bytes there will be a notification.\n>> > In modern 40Gbps NICs it's not a lot. Means that there will be\n>> > at least one notification every second.\n>> > Or I misread the code?\n>>\n>> You're right. The doc does mention that the counter and range\n>> are 32-bit. I can state more explicitly that that bounds the working\n>> set size to 4GB. Do you expect this to be problematic? Processing\n>> a single notification per 4GB of data should not be a significant\n>> cost in itself.\n>\n> I think 4GB is fine. Just there was an idea that in cases when\n> notification of transmission can be known by other means\n\nSome kind of unspoofable response from the peer (i.e., not just\na tcp ack), or a kernel mechanism independent from the error\nqueue? The first does not guarantee that a retransmit is\nnot in progress.\n\n> the user space\n> could have skipped reading errqeuee completely, but looks like it\n> still needs to poll.\n\nIf a process has no need to see the notification, say because\nit is sending out a buffer that is constant for the process lifetime,\nthen it could conceivably skip the recv, and poll with it. The code\nas written will not coalesce more than 4GB of data, but that could\nbe revised.","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"kdFZh2k9\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xk4Yy2lqfz9sNr\n\tfor ;\n\tFri, 1 Sep 2017 13:32:38 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751258AbdIADcg (ORCPT );\n\tThu, 31 Aug 2017 23:32:36 -0400","from mail-oi0-f68.google.com ([209.85.218.68]:36448 \"EHLO\n\tmail-oi0-f68.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750925AbdIADcf (ORCPT\n\t); Thu, 31 Aug 2017 23:32:35 -0400","by mail-oi0-f68.google.com with SMTP id x184so1215285oia.3\n\tfor ; Thu, 31 Aug 2017 20:32:34 -0700 (PDT)","by 10.168.52.10 with HTTP; Thu, 31 Aug 2017 20:31:54 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc; bh=5tbTDJ0vd8DMOxKDErgfRPYeyVwh1I8/EHQzyyeZ5fM=;\n\tb=kdFZh2k9I6omxhsBkNS/Tgx2kJskZIRLNHCmBkQR/GcC8dF0lE09ndyEYSMRkXAhzh\n\tyjF/9MNtPPMQtoflc1Y/TUKZ8zq+KE7bGJBoXQwNzQmQo+8+RMxXxd1dN2x8lWMfHVo9\n\tXZmcQtmqJkLQbO5tP1qg7b8lqeMUZ6nrjtJfVT5sjd5Ziz6XaUGNYLcYni6cBxZI5uhn\n\tTCZfSMLazZACTWdQDna6vVRXZAgI4c9UCF4gNwAjtvqm1obZkFfwKHut6Ot3WCAvmn53\n\tm+B91lPd2b/hDhI9Pa5eDZQMA46bmg0gZh21+1gRprqD1N1DOQXsZOeAkIpOTA5AZl9c\n\tXAvw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc;\n\tbh=5tbTDJ0vd8DMOxKDErgfRPYeyVwh1I8/EHQzyyeZ5fM=;\n\tb=YligkEMrCCVDm0bohsijXlQj6JDXqQ4UKJP2KpjWqgXgI00lVfwb1BmkHtjUKXfWEO\n\tbtDrBaf2mjywh2c3nY7ieDKsVTL7fdt1glSKUwQchWu9IexIfSAHhXDgsvoIlV8O+0qN\n\t9o6TCjiNWXgu0OH+LqJbThNMQ6jClauwWyozxKARX2q3VtT6L5lGwhITG8GX4OVClnoL\n\tW1ta0+2P6v/u+coF89EMZEDdSy9Zylhcl6bvIeupDRZtxT/GjUpPvWlGp/Jtb99pIm8V\n\tq/bHKQY+84EcwFGd+gI5MF4kftDZHhPyUGzYoFMg1mRFL6Qr15U+m5NtUHGMtznOTh5L\n\tLaXA==","X-Gm-Message-State":"AHPjjUhmYSk3ilFUWb9k5iC4i68DBlVC+1bXGBgOiBUBGqRjoQro9rlp\n\t7wQ3hhLj6Q7qt5emQSmsi0yI1X8NPw==","X-Google-Smtp-Source":"ADKCNb7n9a8TOth5FPdgJCXYrCQXKKrxo4lKchvoKb9T1IJPWM4CZbIsBKRvrzz2VIY9V28antMcAxmCUqtRdlIP+zE=","X-Received":"by 10.202.169.17 with SMTP id s17mr490736oie.165.1504236754325; \n\tThu, 31 Aug 2017 20:32:34 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170901031009.2p4vv6fncma2y2l7@ast-mbp>","References":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>\n\t<20170901021007.pkbb2gsipbprf4w7@ast-mbp>\n\t\n\t<20170901031009.2p4vv6fncma2y2l7@ast-mbp>","From":"Willem de Bruijn ","Date":"Thu, 31 Aug 2017 23:31:54 -0400","Message-ID":"","Subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","To":"Alexei Starovoitov ","Cc":"Network Development ,\n\tDavid Miller , Willem de Bruijn ","Content-Type":"text/plain; charset=\"UTF-8\"","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761654,"web_url":"http://patchwork.ozlabs.org/comment/1761654/","msgid":"<20170901144424.72f9c9e9@redhat.com>","list_archive_url":null,"date":"2017-09-01T12:44:24","subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","submitter":{"id":13625,"url":"http://patchwork.ozlabs.org/api/people/13625/","name":"Jesper Dangaard Brouer","email":"brouer@redhat.com"},"content":"On Thu, 31 Aug 2017 17:00:13 -0400\nWillem de Bruijn wrote:\n\n> +More Info\n> +---------\n> +\n> +Much of this document was derived from a longer paper presented at\n> +netdev 2.1. For more in-depth information see that paper and talk,\n> +the excellent reporting over at LWN.net or read the original code.\n> +\n> + paper, slides, video\n> + https://netdevconf.org/2.1/session.html?debruijn\n> +\n> + LWN article\n> + https://lwn.net/Articles/726917/\n> +\n> + patchset\n> + [PATCH net-next v4 0/9] socket sendmsg MSG_ZEROCOPY\n> + https://lwn.net/Articles/730010/\n> + https://www.spinics.net/lists/netdev/msg447552.html\n\nIMHO I think it would be better to use the type links also used in the\ngit log. If you look at the kernel git log, then the \"Link:\" tag have\nthe form: http://lkml.kernel.org/r/\nAnd you can simply append the \"Message-Id:\" email header.\n\nIn this case the link would be:\n http://lkml.kernel.org/r/20170803202945.70750-1-willemdebruijn.kernel@gmail.com","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ext-mx04.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx04.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=brouer@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xkJpp2hQ2z9s7c\n\tfor ;\n\tFri, 1 Sep 2017 22:44:34 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751937AbdIAMoc (ORCPT );\n\tFri, 1 Sep 2017 08:44:32 -0400","from mx1.redhat.com ([209.132.183.28]:50608 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1751778AbdIAMob (ORCPT );\n\tFri, 1 Sep 2017 08:44:31 -0400","from smtp.corp.redhat.com\n\t(int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 64EB9821C3;\n\tFri, 1 Sep 2017 12:44:31 +0000 (UTC)","from localhost (ovpn-200-27.brq.redhat.com [10.40.200.27])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id 714647B150;\n\tFri, 1 Sep 2017 12:44:26 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 64EB9821C3","Date":"Fri, 1 Sep 2017 14:44:24 +0200","From":"Jesper Dangaard Brouer ","To":"Willem de Bruijn ","Cc":"brouer@redhat.com, netdev@vger.kernel.org, davem@davemloft.net,\n\tWillem de Bruijn ","Subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","Message-ID":"<20170901144424.72f9c9e9@redhat.com>","In-Reply-To":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>","References":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.16","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.28]);\n\tFri, 01 Sep 2017 12:44:31 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761693,"web_url":"http://patchwork.ozlabs.org/comment/1761693/","msgid":"","list_archive_url":null,"date":"2017-09-01T13:37:29","subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","submitter":{"id":67615,"url":"http://patchwork.ozlabs.org/api/people/67615/","name":"Willem de Bruijn","email":"willemdebruijn.kernel@gmail.com"},"content":"On Fri, Sep 1, 2017 at 8:44 AM, Jesper Dangaard Brouer\n wrote:\n> On Thu, 31 Aug 2017 17:00:13 -0400\n> Willem de Bruijn wrote:\n>\n>> +More Info\n>> +---------\n>> +\n>> +Much of this document was derived from a longer paper presented at\n>> +netdev 2.1. For more in-depth information see that paper and talk,\n>> +the excellent reporting over at LWN.net or read the original code.\n>> +\n>> + paper, slides, video\n>> + https://netdevconf.org/2.1/session.html?debruijn\n>> +\n>> + LWN article\n>> + https://lwn.net/Articles/726917/\n>> +\n>> + patchset\n>> + [PATCH net-next v4 0/9] socket sendmsg MSG_ZEROCOPY\n>> + https://lwn.net/Articles/730010/\n>> + https://www.spinics.net/lists/netdev/msg447552.html\n>\n> IMHO I think it would be better to use the type links also used in the\n> git log. If you look at the kernel git log, then the \"Link:\" tag have\n> the form: http://lkml.kernel.org/r/\n> And you can simply append the \"Message-Id:\" email header.\n>\n> In this case the link would be:\n> http://lkml.kernel.org/r/20170803202945.70750-1-willemdebruijn.kernel@gmail.com\n\nI was not aware of that. Thanks, I'll send a v2.","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"XChL6iTY\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xkL0j2c2Gz9s2G\n\tfor ;\n\tFri, 1 Sep 2017 23:38:13 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752192AbdIANiL (ORCPT );\n\tFri, 1 Sep 2017 09:38:11 -0400","from mail-oi0-f66.google.com ([209.85.218.66]:34021 \"EHLO\n\tmail-oi0-f66.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1752061AbdIANiK (ORCPT\n\t); Fri, 1 Sep 2017 09:38:10 -0400","by mail-oi0-f66.google.com with SMTP id w10so203824oie.1\n\tfor ; Fri, 01 Sep 2017 06:38:10 -0700 (PDT)","by 10.168.52.10 with HTTP; Fri, 1 Sep 2017 06:37:29 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc; bh=6YshzHw05uNaJET5TN+NOs3i5qDlcZBz+NmRU7x0Kbc=;\n\tb=XChL6iTY8IYg9xb3G2UQd0pHdzS1CkcXBdJ/1CpLHbMutAxAUdMbqy6MZ+zNOkg5fm\n\tx4DricQUKSrlCfnngA0xfBAGqYOaJz10Nu16e2iE9pwLhU0H4C+P4eA9ykY9i3nH5Y9a\n\teLURclWSJFsWGRuEju8SPN3ekZ/QeycBOIbSgVt3hf0tYp9k4sRSspiRpwulkfr52/fg\n\tXWaxMz0Ih4RaFOKfhSHIZO3d6ZqHJf4C8XHv3vv6jUhTgUIL25JtglWJmO4JXLyYg30n\n\tw5H7aaXcHwk3JAK51TUpCVVhOBqG9MdUuxZotS0zCHpBml97lms4W66MeXFvB0F86/Td\n\tiQ0w==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc;\n\tbh=6YshzHw05uNaJET5TN+NOs3i5qDlcZBz+NmRU7x0Kbc=;\n\tb=NgdWtzYYEFnVDBuCXlQLH3XK3DF7ASVQkGXI1NU+TXb/fCEfVRxXqFRxK3XRa2GVO5\n\thfZYNurl+rzs+8qumCSWgQgybwCyVubkHLXscygxWkQngtyVY0eM72lVzHXAkF6Gd5Mx\n\t+NX/Y8YdwoAUFtnF6pnoVUtP0IAJYV7t0n2B7thQBbVNwjKzq/EiSflAVxqRJUm+ncOc\n\tcJNR1oQeFk/phMxWF2s3P5eRGz2NrnijhUYM18RE7Ez6MGAQ8QkKPQ5qPH8d22dZT6GG\n\t/9im4jCyj3d3YsQkAKusc3i156EEzB6eiiG+jBGgokQlyxrmzL5ZvpZuF8AvSqN/u582\n\tvq6g==","X-Gm-Message-State":"AHPjjUhmGCypJT5Tq2uIUlNQU1XbW/3NLdFbCFfBBKQheMVWmjRpMSg3\n\togK7SqdZoo3ss4fwfA5DVlZ849JQJwEp","X-Google-Smtp-Source":"ADKCNb6ft9QiDWr0ZFU6+44NXmWN6J/IzxZAQazMlT1AOgEie/IN8ixgacn6VMhGl0AiIYlpn9qfkKEYTEhQiP49yX4=","X-Received":"by 10.202.206.12 with SMTP id e12mr2085258oig.126.1504273089407; \n\tFri, 01 Sep 2017 06:38:09 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170901144424.72f9c9e9@redhat.com>","References":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>\n\t<20170901144424.72f9c9e9@redhat.com>","From":"Willem de Bruijn ","Date":"Fri, 1 Sep 2017 09:37:29 -0400","Message-ID":"","Subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","To":"Jesper Dangaard Brouer ","Cc":"Network Development ,\n\tDavid Miller , Willem de Bruijn ","Content-Type":"text/plain; charset=\"UTF-8\"","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761773,"web_url":"http://patchwork.ozlabs.org/comment/1761773/","msgid":"","list_archive_url":null,"date":"2017-09-01T15:47:06","subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","submitter":{"id":67615,"url":"http://patchwork.ozlabs.org/api/people/67615/","name":"Willem de Bruijn","email":"willemdebruijn.kernel@gmail.com"},"content":"On Thu, Aug 31, 2017 at 11:10 PM, Alexei Starovoitov\n wrote:\n> On Thu, Aug 31, 2017 at 11:04:41PM -0400, Willem de Bruijn wrote:\n>> On Thu, Aug 31, 2017 at 10:10 PM, Alexei Starovoitov\n>> wrote:\n>> > On Thu, Aug 31, 2017 at 05:00:13PM -0400, Willem de Bruijn wrote:\n>> >> From: Willem de Bruijn \n>> >>\n>> >> Documentation for this feature was missing from the patchset.\n>> >> Copied a lot from the netdev 2.1 paper, addressing some small\n>> >> interface changes since then.\n>> >>\n>> >> Signed-off-by: Willem de Bruijn \n>> > ...\n>> >> +Notification Batching\n>> >> +~~~~~~~~~~~~~~~~~~~~~\n>> >> +\n>> >> +Multiple outstanding packets can be read at once using the recvmmsg\n>> >> +call. This is often not needed. In each message the kernel returns not\n>> >> +a single value, but a range. It coalesces consecutive notifications\n>> >> +while one is outstanding for reception on the error queue.\n>> >> +\n>> >> +When a new notification is about to be queued, it checks whether the\n>> >> +new value extends the range of the notification at the tail of the\n>> >> +queue. If so, it drops the new notification packet and instead increases\n>> >> +the range upper value of the outstanding notification.\n>> >\n>> > Would it make sense to mention that max notification range is 32-bit?\n>> > So each 4Gbyte of xmit bytes there will be a notification.\n>> > In modern 40Gbps NICs it's not a lot. Means that there will be\n>> > at least one notification every second.\n>> > Or I misread the code?\n>>\n>> You're right. The doc does mention that the counter and range\n>> are 32-bit. I can state more explicitly that that bounds the working\n>> set size to 4GB. Do you expect this to be problematic? Processing\n>> a single notification per 4GB of data should not be a significant\n>> cost in itself.\n\nActually, the counter is not a byte counter. It is incremented on each\nsystem call that sends data with MSG_ZEROCOPY. So the 4GB limit\nwould only hold if a caller sends single byte requests at a time.\n\nI will make this more clear in v2.\n\n>\n> I think 4GB is fine. Just there was an idea that in cases when\n> notification of transmission can be known by other means the user space\n> could have skipped reading errqeuee completely, but looks like it\n> still needs to poll. That's fine.\n>> > Thanks for the doc!\n>>\n>> Thanks for reviewing :)\n>>\n>> >\n>> > Acked-by: Alexei Starovoitov \n>> >","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"ph/mcuSA\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xkNtJ0jZyz9t2d\n\tfor ;\n\tSat, 2 Sep 2017 01:47:52 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752425AbdIAPrt (ORCPT );\n\tFri, 1 Sep 2017 11:47:49 -0400","from mail-oi0-f66.google.com ([209.85.218.66]:34036 \"EHLO\n\tmail-oi0-f66.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1752324AbdIAPrr (ORCPT\n\t); Fri, 1 Sep 2017 11:47:47 -0400","by mail-oi0-f66.google.com with SMTP id w10so473934oie.1\n\tfor ; Fri, 01 Sep 2017 08:47:47 -0700 (PDT)","by 10.168.52.10 with HTTP; Fri, 1 Sep 2017 08:47:06 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc; bh=Inmz7kHGoPvSS/nsNhHWfpHmHYMs2/dzdXNoGT30H6U=;\n\tb=ph/mcuSA5GgjnZMhQDEajXhc3WqhPDz1Z26QBMmi3l12f1iyIN1fCL38cUP5CNw7pI\n\tkWeMJ9jfyvlO2MnIg5mqMtz84oXPiNIuh1MrNbRHExfGO8dP0ar2tNy/Lp8wncLGu4eQ\n\t9fpHiCVRfcHj5DZEIRAksp0xjBWeRslnvGK4CHWzOH2f1iHoXCgekSMkYbWRMhBEIK5o\n\tgizUfZD+zpEx9H2QAu4JU7BUPG1BfZs6+C0pIU0TuMCW5M84a4CBB3Wz7rpeGiMc6D8b\n\tpdB+v456Yzo9dBTCNho/3VVqTNRwj4LUkbeTBdxrS0V6GPRj5sAq9kqM+8NnPrl/gCs2\n\t2M+g==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc;\n\tbh=Inmz7kHGoPvSS/nsNhHWfpHmHYMs2/dzdXNoGT30H6U=;\n\tb=lkAg5VBcl3r5lXDqj73SfEqsjmqZI2iZG1VVidpcTB14cim81l7nNGBA2aVg3cXouq\n\tiVwqoERnbc9qTSTJ3HUukzYiosz4wxlMCmUsSDXNiwxPtAUDEJgTKhtSiR2Ee8zbQ4B7\n\th5TVnAYOIG0R1ZSGvAErI5iESnJRAQHAQ9aL059Q/XRqRtn0eZfJgHtXJpItsu18x/iC\n\tcyFyYxXK7jQgExL/te8ZFGEGoVobegt8LfuTs64i229L1oRMgmXk++aZ4rntXbkwu/xw\n\t/csxSeqZLAbxBOEDlayp7nnXil457ba3TSYvmeBdr3LKvafZz0SV4Fr/CLc1nxkobBMq\n\tK51Q==","X-Gm-Message-State":"AHPjjUj8TF2RyywLfu1GUA1WcU2BIiTV+XSDIF3I2hHqa2cUWue7EH84\n\tpGf/Ww72axFhVIw7SHCdXHdVtFXrJg==","X-Google-Smtp-Source":"ADKCNb7j48SBtTYBmFYX73lho9ckCxkWkgLy3YSUnynxSThp4XVfVry91ZwVXfRqMF0ruk55lRs/95S3LM0ikOGlS5c=","X-Received":"by 10.202.206.12 with SMTP id e12mr2470027oig.126.1504280866381; \n\tFri, 01 Sep 2017 08:47:46 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170901031009.2p4vv6fncma2y2l7@ast-mbp>","References":"<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>\n\t<20170901021007.pkbb2gsipbprf4w7@ast-mbp>\n\t\n\t<20170901031009.2p4vv6fncma2y2l7@ast-mbp>","From":"Willem de Bruijn ","Date":"Fri, 1 Sep 2017 11:47:06 -0400","Message-ID":"","Subject":"Re: [PATCH net-next] doc: document MSG_ZEROCOPY","To":"Alexei Starovoitov ","Cc":"Network Development ,\n\tDavid Miller , Willem de Bruijn ","Content-Type":"text/plain; charset=\"UTF-8\"","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}}]