Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/808430/?format=api
{ "id": 808430, "url": "http://patchwork.ozlabs.org/api/patches/808430/?format=api", "web_url": "http://patchwork.ozlabs.org/project/netdev/patch/20170831210013.85220-1-willemdebruijn.kernel@gmail.com/", "project": { "id": 7, "url": "http://patchwork.ozlabs.org/api/projects/7/?format=api", "name": "Linux network development", "link_name": "netdev", "list_id": "netdev.vger.kernel.org", "list_email": "netdev@vger.kernel.org", "web_url": null, "scm_url": null, "webscm_url": null, "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>", "list_archive_url": null, "date": "2017-08-31T21:00:13", "name": "[net-next] doc: document MSG_ZEROCOPY", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "64a3428474484ce2e7b394a400946230acd5313f", "submitter": { "id": 67615, "url": "http://patchwork.ozlabs.org/api/people/67615/?format=api", "name": "Willem de Bruijn", "email": "willemdebruijn.kernel@gmail.com" }, "delegate": { "id": 34, "url": "http://patchwork.ozlabs.org/api/users/34/?format=api", "username": "davem", "first_name": "David", "last_name": "Miller", "email": "davem@davemloft.net" }, "mbox": "http://patchwork.ozlabs.org/project/netdev/patch/20170831210013.85220-1-willemdebruijn.kernel@gmail.com/mbox/", "series": [ { "id": 908, "url": "http://patchwork.ozlabs.org/api/series/908/?format=api", "web_url": "http://patchwork.ozlabs.org/project/netdev/list/?series=908", "date": "2017-08-31T21:00:13", "name": "[net-next] doc: document MSG_ZEROCOPY", "version": 1, "mbox": "http://patchwork.ozlabs.org/series/908/mbox/" } ], "comments": "http://patchwork.ozlabs.org/api/patches/808430/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/808430/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<netdev-owner@vger.kernel.org>", "X-Original-To": "patchwork-incoming@ozlabs.org", "Delivered-To": "patchwork-incoming@ozlabs.org", "Authentication-Results": [ "ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)", "ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"F0zkg/pH\"; dkim-atps=neutral" ], "Received": [ "from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjvsV3HZpz9t1t\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri, 1 Sep 2017 07:00:30 +1000 (AEST)", "(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751685AbdHaVA2 (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 17:00:28 -0400", "from mail-qt0-f174.google.com ([209.85.216.174]:35070 \"EHLO\n\tmail-qt0-f174.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751001AbdHaVA0 (ORCPT\n\t<rfc822;netdev@vger.kernel.org>); Thu, 31 Aug 2017 17:00:26 -0400", "by mail-qt0-f174.google.com with SMTP id x36so3776836qtx.2\n\tfor <netdev@vger.kernel.org>; Thu, 31 Aug 2017 14:00:26 -0700 (PDT)", "from willemb1.nyc.corp.google.com ([100.101.212.81])\n\tby smtp.gmail.com with ESMTPSA id\n\tr22sm6177494qkl.12.2017.08.31.14.00.24\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128);\n\tThu, 31 Aug 2017 14:00:24 -0700 (PDT)" ], "DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=from:to:cc:subject:date:message-id;\n\tbh=70/1YKtwVlgC06X104tLGGnROFLG+x8Jv9zfMhSIIM4=;\n\tb=F0zkg/pHMxgUgONxY4dfUCiZyuVNqrpTuYa9Z0SXx/sg7LZNolA48mEp/hoeXqhpfB\n\twJNcgkt/topxT7PHDFR+RR3ru6RoFwxeadSCm/eefU1eMZTlJTiS1QvbFtN+yGODNi0O\n\tVL1wIJBpQeCkNLLaLqCrPC46rERUmTT5zd8b4UYH9NPStk3aN1l/9CztFNBotKUz4l+M\n\tQltRoow3pGDoQE0n3EQo45oU/xpfcR4WOt7XYn+ZmtSdWRqlsBlkfz01jP69HZh3owVF\n\tbzTY0ClWy+TxdPeZw1nRZ1WrKqCOrJFOvuQ58tnxpawdUvvbuYUUcMLwhiJv1asET5vd\n\tbxxA==", "X-Google-DKIM-Signature": "v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:from:to:cc:subject:date:message-id;\n\tbh=70/1YKtwVlgC06X104tLGGnROFLG+x8Jv9zfMhSIIM4=;\n\tb=Lo23+cHYo6L47h06ByOf9CoUKp+yxo/CHU8N5m0FhtsJ+qdz6Ux9wbPySjHcJKpWhU\n\tV+yz2v2klx/HCbQiUGHn8ocf94SXtITO/gnFltxG0HF2sKYD2vVuu7hJMdyuWoe/0qbB\n\t9T//lDltJD6uhUqzSWGTvAcjDQAPchWZUe/+Hc7cNRb8mrYvNfdFTIUuXobcFoljs6ZU\n\tu+Z2EROXsvgNpij2P37Q0mvcuRlOEEnX3SJ/+h1QJz+wrme47rYu7dMe5RsigBKCd62h\n\tJosmzVNG/iXhofRg2h2hjq+5bvA0T/ksvJ5AAOHFLdzvkgcdWd4PUzmQJLvdzIDKTTfd\n\tu0SQ==", "X-Gm-Message-State": "AHYfb5h/NDQrFJnmNw5CJqeqkC2xIKpRhhn/IMREJ6LfIyLhzMc7ngZe\n\tkaGInE9S8mfH5YvD49E=", "X-Google-Smtp-Source": "ADKCNb6bNwOuejUe+vzc912dztFEAHkJ9KnxrD+38op04d7X/BptrwABWQCPNTKlYkrZFOIqkjewPA==", "X-Received": "by 10.200.56.122 with SMTP id r55mr8922441qtb.60.1504213225067; \n\tThu, 31 Aug 2017 14:00:25 -0700 (PDT)", "From": "Willem de Bruijn <willemdebruijn.kernel@gmail.com>", "To": "netdev@vger.kernel.org", "Cc": "davem@davemloft.net, Willem de Bruijn <willemb@google.com>", "Subject": "[PATCH net-next] doc: document MSG_ZEROCOPY", "Date": "Thu, 31 Aug 2017 17:00:13 -0400", "Message-Id": "<20170831210013.85220-1-willemdebruijn.kernel@gmail.com>", "X-Mailer": "git-send-email 2.14.1.581.gf28d330327-goog", "Sender": "netdev-owner@vger.kernel.org", "Precedence": "bulk", "List-ID": "<netdev.vger.kernel.org>", "X-Mailing-List": "netdev@vger.kernel.org" }, "content": "From: Willem de Bruijn <willemb@google.com>\n\nDocumentation for this feature was missing from the patchset.\nCopied a lot from the netdev 2.1 paper, addressing some small\ninterface changes since then.\n\nSigned-off-by: Willem de Bruijn <willemb@google.com>\n---\n Documentation/networking/msg_zerocopy.rst | 254 ++++++++++++++++++++++++++++++\n 1 file changed, 254 insertions(+)\n create mode 100644 Documentation/networking/msg_zerocopy.rst", "diff": "diff --git a/Documentation/networking/msg_zerocopy.rst b/Documentation/networking/msg_zerocopy.rst\nnew file mode 100644\nindex 000000000000..2e2a3410b947\n--- /dev/null\n+++ b/Documentation/networking/msg_zerocopy.rst\n@@ -0,0 +1,254 @@\n+\n+============\n+MSG_ZEROCOPY\n+============\n+\n+Intro\n+=====\n+\n+The MSG_ZEROCOPY flag enables copy avoidance for socket send calls.\n+The feature is currently implemented for TCP sockets.\n+\n+\n+Opportunity and Caveats\n+-----------------------\n+\n+Copying large buffers between user process and kernel can be\n+expensive. Linux supports various interfaces that eschew copying,\n+such as sendpage and splice. The MSG_ZEROCOPY flag extends the\n+underlying copy avoidance mechanism to common socket send calls.\n+\n+Copy avoidance is not a free lunch. As implemented, with page pinning,\n+it replaces per byte copy cost with page accounting and completion\n+notification overhead. As a result, MSG_ZEROCOPY is generally only\n+effective at writes over around 10 KB.\n+\n+Page pinning also changes system call semantics. It temporarily shares\n+the buffer between process and network stack. Unlike with copying, the\n+process cannot immediately overwrite the buffer after system call\n+return without possibly modifying the data in flight. Kernel integrity\n+is not affected, but a buggy program can possibly corrupt its own data\n+stream.\n+\n+The kernel returns a notification when it is safe to modify data.\n+Converting an existing application to MSG_ZEROCOPY is not always as\n+trivial as just passing the flag, then.\n+\n+\n+More Info\n+---------\n+\n+Much of this document was derived from a longer paper presented at\n+netdev 2.1. For more in-depth information see that paper and talk,\n+the excellent reporting over at LWN.net or read the original code.\n+\n+ paper, slides, video\n+ https://netdevconf.org/2.1/session.html?debruijn\n+\n+ LWN article\n+ https://lwn.net/Articles/726917/\n+\n+ patchset\n+ [PATCH net-next v4 0/9] socket sendmsg MSG_ZEROCOPY\n+ https://lwn.net/Articles/730010/\n+ https://www.spinics.net/lists/netdev/msg447552.html\n+\n+\n+Interface\n+=========\n+\n+Passing the MSG_ZEROCOPY flag is the most obvious step to enable copy\n+avoidance, but not the only one.\n+\n+Socket Setup\n+------------\n+\n+The kernel is permissive when applications pass undefined flags to the\n+send system call. By default it simply ignores these. To avoid enabling\n+copy avoidance mode for legacy processes that accidentally pass this\n+flag, a process must first signal intent by setting a socket option:\n+\n+::\n+\n+\tif (setsockopt(fd, SOL_SOCKET, SO_ZEROCOPY, &one, sizeof(one)))\n+\t\terror(1, errno, \"setsockopt zerocopy\");\n+\n+\n+Transmission\n+------------\n+\n+The change to send (or sendto, sendmsg, sendmmsg) itself is trivial.\n+Pass the new flag.\n+\n+::\n+\n+\tret = send(fd, buf, sizeof(buf), MSG_ZEROCOPY);\n+\tif (ret != sizeof(buf))\n+\t\terror(1, errno, \"send\");\n+\n+\n+Mixing copy avoidance and copying\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Many workloads have a mixture of large and small buffers. Because copy\n+avoidance is more expensive than copying for small packets, the\n+feature is implemented as a flag. It is safe to mix calls with the flag\n+with those without.\n+\n+\n+Notifications\n+-------------\n+\n+The kernel has to notify the process when it is safe to reuse a\n+previously passed buffer. It queues completion notifications on the\n+socket error queue, akin to the transmit timestamping interface.\n+\n+The notification itself is a simple scalar value. Each socket\n+maintains an internal 32-bit counter. Each send call with MSG_ZEROCOPY\n+that successfully sends data increments the counter. The counter is\n+not incremented on failure or if called with length zero.\n+\n+\n+Notification Reception\n+~~~~~~~~~~~~~~~~~~~~~~\n+\n+The below snippet demonstrates the API. In the simplest case, each\n+send syscall is followed by a poll and recvmsg on the error queue.\n+\n+Reading from the error queue is always a non-blocking operation. The\n+poll call is there to block until an error is outstanding. It will set\n+POLLERR in its output flags. That flag does not have to be set in the\n+events field: errors are signaled unconditionally.\n+\n+::\n+\n+\tpfd.fd = fd;\n+\tpfd.events = 0;\n+\tif (poll(&pfd, 1, -1) != 1 || pfd.revents & POLLERR == 0)\n+\t\terror(1, errno, \"poll\");\n+\n+\tret = recvmsg(fd, &msg, MSG_ERRQUEUE);\n+\tif (ret == -1)\n+\t\terror(1, errno, \"recvmsg\");\n+\n+\tread_notification(msg);\n+\n+The example is for demonstration purpose only. In practice, it is more\n+efficient to not wait for notifications, but read without blocking\n+every couple of send calls.\n+\n+Notifications can be processed out of order with other operations on\n+the socket. A socket that has an error queued would normally block\n+other operations until the error is read. Zerocopy notifications have\n+a zero error code, however, to not block send and recv calls.\n+\n+\n+Notification Batching\n+~~~~~~~~~~~~~~~~~~~~~\n+\n+Multiple outstanding packets can be read at once using the recvmmsg\n+call. This is often not needed. In each message the kernel returns not\n+a single value, but a range. It coalesces consecutive notifications\n+while one is outstanding for reception on the error queue.\n+\n+When a new notification is about to be queued, it checks whether the\n+new value extends the range of the notification at the tail of the\n+queue. If so, it drops the new notification packet and instead increases\n+the range upper value of the outstanding notification.\n+\n+For protocols that acknowledge data in-order, like TCP, each\n+notification can be squashed into the previous one, so that no more\n+than one notification is outstanding at any one point.\n+\n+Ordered delivery is the common case, but not guaranteed. Notifications\n+may arrive out of order on retransmission and socket teardown.\n+\n+\n+Notification Parsing\n+~~~~~~~~~~~~~~~~~~~~\n+\n+The below snippet demonstrates how to parse the control message: the\n+read_notification() call in the previous snippet. A notification\n+is encoded in the standard error format, sock_extended_err.\n+\n+The level and type fields in the control data are protocol family\n+specific, IP_RECVERR or IPV6_RECVERR.\n+\n+Error origin is the new type SO_EE_ORIGIN_ZEROCOPY. The errno is zero,\n+as explained before, to avoid blocking read and write system calls on\n+the socket.\n+\n+The 32-bit notification range is encoded as [ee_info, ee_data]. This\n+range is inclusive. Other fields in the struct must be treated as\n+undefined, bar for ee_code, as discussed below.\n+\n+::\n+\n+\tstruct sock_extended_err *serr;\n+\tstruct cmsghdr *cm;\n+\n+\tcm = CMSG_FIRSTHDR(msg);\n+\tif (cm->cmsg_level != SOL_IP &&\n+\t cm->cmsg_type != IP_RECVERR)\n+\t\terror(1, 0, \"cmsg\");\n+\n+\tserr = (void *) CMSG_DATA(cm);\n+\tif (serr->ee_errno != 0 ||\n+\t serr->ee_origin != SO_EE_ORIGIN_ZEROCOPY)\n+\t\terror(1, 0, \"serr\");\n+\n+\tprintf(\"completed: %u..%u\\n\", serr->ee_info, serr->ee_data);\n+\n+\n+Deferred copies\n+~~~~~~~~~~~~~~~\n+\n+Passing flag MSG_ZEROCOPY is a hint to the kernel to apply copy\n+avoidance, and a contract that the kernel will queue a completion\n+notification. It is not a guarantee that the copy is elided.\n+\n+Copy avoidance is not always feasible. Devices that do not support\n+scatter-gather I/O cannot send packets made up of kernel generated\n+protocol headers plus zerocopy user data. A packet may need to be\n+converted to having a private copy of data deep in the stack, say to\n+compute a checksum.\n+\n+In all these cases, the kernel returns a completion notification when\n+it releases its hold on the shared pages. The notification may arrive\n+before the (copied) data is fully transmitted. A zerocopy completion\n+notification is not a transmit completion notification, therefore.\n+\n+Deferred copies can be more expensive than a copy immediately in the\n+system call, if the data is no longer warm in the cache. The process\n+also incurs notification processing cost for no benefit. For this\n+reason, the kernel signals if data was completed with a copy, by\n+setting flag SO_EE_CODE_ZEROCOPY_COPIED in field ee_code on return.\n+A process may use this signal to stop passing flag MSG_ZEROCOPY on\n+subsequent requests on the same socket.\n+\n+Implementation\n+==============\n+\n+Loopback\n+--------\n+\n+Data sent to local sockets can be queued indefinitely if the receive\n+process does not read its socket. Unbound notification latency is not\n+acceptable. For this reason all packets generated with MSG_ZEROCOPY\n+that are looped to a local socket will incur a deferred copy. This\n+includes looping onto packet sockets (e.g., tcpdump) and tun devices.\n+\n+\n+Testing\n+=======\n+\n+More realistic example code can be found in the kernel source under\n+tools/testing/selftests/net/msg_zerocopy.c.\n+\n+Be cognizant of the loopback constraint. The test can be run between\n+a pair of hosts. But if run between a local pair of processes, for\n+instance when run with msg_zerocopy.sh between a veth pair across\n+namespaces, the test will not show any improvement. For testing, the\n+loopback restriction can be temporarily avoided by making\n+skb_orphan_frags_rx identical to skb_orphan_frags.\n+\n", "prefixes": [ "net-next" ] }