Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/813779/?format=api
{ "id": 813779, "url": "http://patchwork.ozlabs.org/api/patches/813779/?format=api", "web_url": "http://patchwork.ozlabs.org/project/netdev/patch/1505385988-94522-6-git-send-email-ilyal@mellanox.com/", "project": { "id": 7, "url": "http://patchwork.ozlabs.org/api/projects/7/?format=api", "name": "Linux network development", "link_name": "netdev", "list_id": "netdev.vger.kernel.org", "list_email": "netdev@vger.kernel.org", "web_url": null, "scm_url": null, "webscm_url": null, "list_archive_url": "", "list_archive_url_format": "", "commit_url_format": "" }, "msgid": "<1505385988-94522-6-git-send-email-ilyal@mellanox.com>", "list_archive_url": null, "date": "2017-09-14T10:46:28", "name": "[net-next,5/5] tls: Add generic NIC offload infrastructure.", "commit_ref": null, "pull_url": null, "state": "deferred", "archived": true, "hash": "d09a1e65165f62a8773820d9a25dc516941709ed", "submitter": { "id": 67931, "url": "http://patchwork.ozlabs.org/api/people/67931/?format=api", "name": "Ilya Lesokhin", "email": "ilyal@mellanox.com" }, "delegate": { "id": 34, "url": "http://patchwork.ozlabs.org/api/users/34/?format=api", "username": "davem", "first_name": "David", "last_name": "Miller", "email": "davem@davemloft.net" }, "mbox": "http://patchwork.ozlabs.org/project/netdev/patch/1505385988-94522-6-git-send-email-ilyal@mellanox.com/mbox/", "series": [ { "id": 3066, "url": "http://patchwork.ozlabs.org/api/series/3066/?format=api", "web_url": "http://patchwork.ozlabs.org/project/netdev/list/?series=3066", "date": "2017-09-14T10:46:26", "name": "tls: Add generic NIC offload infrastructure", "version": 1, "mbox": "http://patchwork.ozlabs.org/series/3066/mbox/" } ], "comments": "http://patchwork.ozlabs.org/api/patches/813779/comments/", "check": "pending", "checks": "http://patchwork.ozlabs.org/api/patches/813779/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<netdev-owner@vger.kernel.org>", "X-Original-To": "patchwork-incoming@ozlabs.org", "Delivered-To": "patchwork-incoming@ozlabs.org", "Authentication-Results": "ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)", "Received": [ "from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xtFf66VnXz9sP1\n\tfor <patchwork-incoming@ozlabs.org>;\n\tThu, 14 Sep 2017 20:49:34 +1000 (AEST)", "(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1752067AbdINKtc (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 14 Sep 2017 06:49:32 -0400", "from mail-il-dmz.mellanox.com ([193.47.165.129]:37234 \"EHLO\n\tmellanox.co.il\" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org\n\twith ESMTP id S1751829AbdINKqf (ORCPT\n\t<rfc822;netdev@vger.kernel.org>); Thu, 14 Sep 2017 06:46:35 -0400", "from Internal Mail-Server by MTLPINE1 (envelope-from\n\tilyal@mellanox.com)\n\twith ESMTPS (AES256-SHA encrypted); 14 Sep 2017 13:46:31 +0300", "from gen-l-vrt-094.mtl.labs.mlnx (gen-l-vrt-094.mtl.labs.mlnx\n\t[10.137.9.1])\n\tby labmailer.mlnx (8.13.8/8.13.8) with ESMTP id v8EAkUp2020670;\n\tThu, 14 Sep 2017 13:46:30 +0300" ], "From": "Ilya Lesokhin <ilyal@mellanox.com>", "To": "netdev@vger.kernel.org, davem@davemloft.net", "Cc": "davejwatson@fb.com, tom@herbertland.com,\n\thannes@stressinduktion.org, borisp@mellanox.com,\n\tilyal@mellanox.com, aviadye@mellanox.com, liranl@mellanox.com", "Subject": "[PATCH net-next 5/5] tls: Add generic NIC offload infrastructure.", "Date": "Thu, 14 Sep 2017 13:46:28 +0300", "Message-Id": "<1505385988-94522-6-git-send-email-ilyal@mellanox.com>", "X-Mailer": "git-send-email 1.8.3.1", "In-Reply-To": "<1505385988-94522-1-git-send-email-ilyal@mellanox.com>", "References": "<1505385988-94522-1-git-send-email-ilyal@mellanox.com>", "Sender": "netdev-owner@vger.kernel.org", "Precedence": "bulk", "List-ID": "<netdev.vger.kernel.org>", "X-Mailing-List": "netdev@vger.kernel.org" }, "content": "This patch adds a generic infrastructure to offload TLS crypto to a\nnetwork devices. It enables the kernel TLS socket to skip encryption and\nauthentication operations on the transmit side of the data path. Leaving\nthose computationally expensive operations to the NIC.\n\nThe NIC offload infrastructure builds TLS records and pushes them to the\nTCP layer just like the SW KTLS implementation and using the same API.\nTCP segmentation is mostly unaffected. Currently the only exception is\nthat we prevent mixed SKBs where only part of the payload requires\noffload. In the future we are likely to add a similar restriction\nfollowing a change cipher spec record.\n\nThe notable differences between SW KTLS and NIC offloaded TLS\nimplementations are as follows:\n1. The offloaded implementation builds \"plaintext TLS record\", those\nrecords contain plaintext instead of ciphertext and place holder bytes\ninstead of authentication tags.\n2. The offloaded implementation maintains a mapping from TCP sequence\nnumber to TLS records. Thus given a TCP SKB sent from a NIC offloaded\n TLS socket, we can use the tls NIC offload infrastructure to obtain\nenough context to encrypt the payload of the SKB.\nA TLS record is released when the last byte of the record is ack'ed,\nthis is done through the new icsk_clean_acked callback.\n\nThe infrastructure should be extendable to support various NIC offload\nimplementations. However it is currently written with the\nimplementation below in mind:\nThe NIC assumes that packets from each offloaded stream are sent as\nplaintext and in-order. It keeps track of the TLS records in the TCP\nstream. When a packet marked for offload is transmitted, the NIC\nencrypts the payload in-place and puts authentication tags in the\nrelevant place holders.\n\nThe responsibility for handling out-of-order packets (i.e. TCP\nretransmission, qdisc drops) falls on the netdev driver.\n\nThe netdev driver keeps track of the expected TCP SN from the NIC's\nperspective. If the next packet to transmit matches the expected TCP\nSN, the driver advances the expected TCP SN, and transmits the packet\nwith TLS offload indication.\n\nIf the next packet to transmit does not match the expected TCP SN. The\ndriver calls the TLS layer to obtain the TLS record that includes the\nTCP of the packet for transmission. Using this TLS record, the driver\nposts a work entry on the transmit queue to reconstruct the NIC TLS\nstate required for the offload of the out-of-order packet. It updates\nthe expected TCP SN accordingly and transmit the now in-order packet.\nThe same queue is used for packet transmission and TLS context\nreconstruction to avoid the need for flushing the transmit queue before\nissuing the context reconstruction request.\n\nSigned-off-by: Boris Pismenny <borisp@mellanox.com>\nSigned-off-by: Ilya Lesokhin <ilyal@mellanox.com>\nSigned-off-by: Aviad Yehezkel <aviadye@mellanox.com>\n---\n include/net/tls.h | 41 +++-\n net/tls/Kconfig | 9 +\n net/tls/Makefile | 3 +\n net/tls/tls_device.c | 673 +++++++++++++++++++++++++++++++++++++++++++++++++++\n net/tls/tls_main.c | 63 +++--\n 5 files changed, 771 insertions(+), 18 deletions(-)\n create mode 100644 net/tls/tls_device.c", "diff": "diff --git a/include/net/tls.h b/include/net/tls.h\nindex b89d397..1f83c8e 100644\n--- a/include/net/tls.h\n+++ b/include/net/tls.h\n@@ -71,6 +71,24 @@ struct tls_sw_context {\n \tstruct scatterlist sg_aead_out[2];\n };\n \n+struct tls_record_info {\n+\tstruct list_head list;\n+\tu32 end_seq;\n+\tint len;\n+\tint num_frags;\n+\tskb_frag_t frags[MAX_SKB_FRAGS];\n+};\n+\n+struct tls_offload_context {\n+\tstruct list_head records_list;\n+\tstruct scatterlist sg_tx_data[MAX_SKB_FRAGS];\n+\tvoid (*sk_destruct)(struct sock *sk);\n+\tstruct tls_record_info *open_record;\n+\tstruct tls_record_info *retransmit_hint;\n+\tu32 expected_seq;\n+\tspinlock_t lock;\t/* protects records list */\n+};\n+\n enum {\n \tTLS_PENDING_CLOSED_RECORD\n };\n@@ -81,6 +99,9 @@ struct tls_context {\n \t\tstruct tls12_crypto_info_aes_gcm_128 crypto_send_aes_gcm_128;\n \t};\n \n+\tstruct list_head gclist;\n+\tstruct sock *sk;\n+\tstruct net_device *netdev;\n \tvoid *priv_ctx;\n \n \tu16 prepend_size;\n@@ -123,9 +144,18 @@ int tls_sw_sendpage(struct sock *sk, struct page *page,\n \t\t int offset, size_t size, int flags);\n void tls_sw_close(struct sock *sk, long timeout);\n \n-void tls_sk_destruct(struct sock *sk, struct tls_context *ctx);\n-void tls_icsk_clean_acked(struct sock *sk);\n+void tls_clear_device_offload(struct sock *sk, struct tls_context *ctx);\n+int tls_set_device_offload(struct sock *sk, struct tls_context *ctx);\n+int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size);\n+int tls_device_sendpage(struct sock *sk, struct page *page,\n+\t\t\tint offset, size_t size, int flags);\n+void tls_device_sk_destruct(struct sock *sk);\n+void tls_device_cleanup(void);\n \n+struct tls_record_info *tls_get_record(struct tls_offload_context *context,\n+\t\t\t\t u32 seq);\n+\n+void tls_sk_destruct(struct sock *sk, struct tls_context *ctx);\n int tls_push_sg(struct sock *sk, struct tls_context *ctx,\n \t\tstruct scatterlist *sg, u16 first_offset,\n \t\tint flags);\n@@ -162,6 +192,13 @@ static inline bool tls_is_pending_open_record(struct tls_context *tls_ctx)\n \treturn tls_ctx->pending_open_record_frags;\n }\n \n+static inline bool tls_is_sk_tx_device_offloaded(struct sock *sk)\n+{\n+\t/* matches smp_store_release in tls_set_device_offload */\n+\treturn\tsmp_load_acquire(&sk->sk_destruct) ==\n+\t\t\t&tls_device_sk_destruct;\n+}\n+\n static inline void tls_err_abort(struct sock *sk)\n {\n \tsk->sk_err = -EBADMSG;\ndiff --git a/net/tls/Kconfig b/net/tls/Kconfig\nindex eb58303..1a4ea55c 100644\n--- a/net/tls/Kconfig\n+++ b/net/tls/Kconfig\n@@ -13,3 +13,12 @@ config TLS\n \tencryption handling of the TLS protocol to be done in-kernel.\n \n \tIf unsure, say N.\n+\n+config TLS_DEVICE\n+\tbool \"Transport Layer Security HW offload\"\n+\tdepends on TLS\n+\tdefault n\n+\t---help---\n+\tEnable kernel support for HW offload of the TLS protocol.\n+\n+\tIf unsure, say N.\ndiff --git a/net/tls/Makefile b/net/tls/Makefile\nindex a930fd1..9de5055 100644\n--- a/net/tls/Makefile\n+++ b/net/tls/Makefile\n@@ -5,3 +5,6 @@\n obj-$(CONFIG_TLS) += tls.o\n \n tls-y := tls_main.o tls_sw.o\n+\n+tls-$(CONFIG_TLS_DEVICE) += tls_device.o\n+\ndiff --git a/net/tls/tls_device.c b/net/tls/tls_device.c\nnew file mode 100644\nindex 0000000..94a25c2\n--- /dev/null\n+++ b/net/tls/tls_device.c\n@@ -0,0 +1,673 @@\n+/* Copyright (c) 2016-2017, Mellanox Technologies All rights reserved.\n+ *\n+ * Redistribution and use in source and binary forms, with or\n+ * without modification, are permitted provided that the following\n+ * conditions are met:\n+ *\n+ * - Redistributions of source code must retain the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer.\n+ *\n+ * - Redistributions in binary form must reproduce the above\n+ * copyright notice, this list of conditions and the following\n+ * disclaimer in the documentation and/or other materials\n+ * provided with the distribution.\n+ *\n+ * - Neither the name of the Mellanox Technologies nor the\n+ * names of its contributors may be used to endorse or promote\n+ * products derived from this software without specific prior written\n+ * permission.\n+ *\n+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\n+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,\n+ * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR\n+ * A PARTICULAR PURPOSE ARE DISCLAIMED.\n+ * IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR\n+ * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\n+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\n+ * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)\n+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,\n+ * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING\n+ * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n+ * POSSIBILITY OF SUCH DAMAGE\n+ */\n+\n+#include <linux/module.h>\n+#include <net/tcp.h>\n+#include <net/inet_common.h>\n+#include <linux/highmem.h>\n+#include <linux/netdevice.h>\n+\n+#include <net/tls.h>\n+\n+static void tls_device_gc_task(struct work_struct *work);\n+\n+static DECLARE_WORK(tls_device_gc_work, tls_device_gc_task);\n+static LIST_HEAD(tls_device_gc_list);\n+static DEFINE_SPINLOCK(tls_device_gc_lock);\n+\n+static void tls_device_gc_task(struct work_struct *work)\n+{\n+\tstruct tls_context *ctx, *tmp;\n+\tstruct list_head gc_list;\n+\tunsigned long flags;\n+\n+\tspin_lock_irqsave(&tls_device_gc_lock, flags);\n+\tINIT_LIST_HEAD(&gc_list);\n+\tlist_splice_init(&tls_device_gc_list, &gc_list);\n+\tspin_unlock_irqrestore(&tls_device_gc_lock, flags);\n+\n+\tlist_for_each_entry_safe(ctx, tmp, &gc_list, gclist) {\n+\t\tstruct tls_offload_context *offlad_ctx = tls_offload_ctx(ctx);\n+\t\tvoid (*sk_destruct)(struct sock *sk) = offlad_ctx->sk_destruct;\n+\t\tstruct net_device *netdev = ctx->netdev;\n+\t\tstruct sock *sk = ctx->sk;\n+\n+\t\tnetdev->tlsdev_ops->tls_dev_del(netdev, sk,\n+\t\t\t\t\t\tTLS_OFFLOAD_CTX_DIR_TX);\n+\n+\t\tlist_del(&ctx->gclist);\n+\t\tkfree(offlad_ctx);\n+\t\tkfree(ctx);\n+\t\tsk_destruct(sk);\n+\t}\n+}\n+\n+static void tls_device_queue_ctx_destruction(struct tls_context *ctx)\n+{\n+\tunsigned long flags;\n+\n+\tspin_lock_irqsave(&tls_device_gc_lock, flags);\n+\tlist_add_tail(&ctx->gclist, &tls_device_gc_list);\n+\tspin_unlock_irqrestore(&tls_device_gc_lock, flags);\n+\n+\tschedule_work(&tls_device_gc_work);\n+}\n+\n+/* We assume that the socket is already connected */\n+static struct net_device *get_netdev_for_sock(struct sock *sk)\n+{\n+\tstruct inet_sock *inet = inet_sk(sk);\n+\tstruct net_device *netdev = NULL;\n+\n+\tnetdev = dev_get_by_index(sock_net(sk), inet->cork.fl.flowi_oif);\n+\n+\treturn netdev;\n+}\n+\n+static void detach_sock_from_netdev(struct sock *sk, struct tls_context *ctx)\n+{\n+\tstruct net_device *netdev;\n+\n+\tnetdev = get_netdev_for_sock(sk);\n+\tif (!netdev) {\n+\t\tpr_err(\"got offloaded socket with no netdev\\n\");\n+\t\treturn;\n+\t}\n+\n+\tif (!netdev->tlsdev_ops) {\n+\t\tpr_err(\"attach_sock_to_netdev: netdev %s with no TLS offload\\n\",\n+\t\t netdev->name);\n+\t\treturn;\n+\t}\n+\n+\tnetdev->tlsdev_ops->tls_dev_del(netdev, sk, TLS_OFFLOAD_CTX_DIR_TX);\n+\tdev_put(netdev);\n+}\n+\n+static int attach_sock_to_netdev(struct sock *sk, struct net_device *netdev,\n+\t\t\t\t struct tls_context *ctx)\n+{\n+\tint rc;\n+\n+\trc = netdev->tlsdev_ops->tls_dev_add(\n+\t\t\tnetdev,\n+\t\t\tsk,\n+\t\t\tTLS_OFFLOAD_CTX_DIR_TX,\n+\t\t\t&ctx->crypto_send);\n+\tif (rc) {\n+\t\tpr_err(\"The netdev has refused to offload this socket\\n\");\n+\t\tgoto out;\n+\t}\n+\n+\tsk->sk_bound_dev_if = netdev->ifindex;\n+\tsk_dst_reset(sk);\n+\n+\trc = 0;\n+out:\n+\treturn rc;\n+}\n+\n+static void destroy_record(struct tls_record_info *record)\n+{\n+\tskb_frag_t *frag;\n+\tint nr_frags = record->num_frags;\n+\n+\twhile (nr_frags > 0) {\n+\t\tfrag = &record->frags[nr_frags - 1];\n+\t\t__skb_frag_unref(frag);\n+\t\t--nr_frags;\n+\t}\n+\tkfree(record);\n+}\n+\n+static void delete_all_records(struct tls_offload_context *offload_ctx)\n+{\n+\tstruct tls_record_info *info, *temp;\n+\n+\tlist_for_each_entry_safe(info, temp, &offload_ctx->records_list, list) {\n+\t\tlist_del(&info->list);\n+\t\tdestroy_record(info);\n+\t}\n+\n+\toffload_ctx->retransmit_hint = NULL;\n+}\n+\n+static void tls_icsk_clean_acked(struct sock *sk)\n+{\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\tstruct tls_offload_context *ctx;\n+\tstruct tcp_sock *tp = tcp_sk(sk);\n+\tstruct tls_record_info *info, *temp;\n+\tunsigned long flags;\n+\n+\tif (!tls_ctx)\n+\t\treturn;\n+\n+\tctx = tls_offload_ctx(tls_ctx);\n+\n+\tspin_lock_irqsave(&ctx->lock, flags);\n+\tinfo = ctx->retransmit_hint;\n+\tif (info && !before(tp->snd_una, info->end_seq)) {\n+\t\tctx->retransmit_hint = NULL;\n+\t\tlist_del(&info->list);\n+\t\tdestroy_record(info);\n+\t}\n+\n+\tlist_for_each_entry_safe(info, temp, &ctx->records_list, list) {\n+\t\tif (before(tp->snd_una, info->end_seq))\n+\t\t\tbreak;\n+\t\tlist_del(&info->list);\n+\n+\t\tdestroy_record(info);\n+\t}\n+\n+\tspin_unlock_irqrestore(&ctx->lock, flags);\n+}\n+\n+static void tls_device_free_resources(struct sock *sk)\n+{\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\tstruct tls_offload_context *ctx = tls_offload_ctx(tls_ctx);\n+\n+\tif (ctx->open_record)\n+\t\tdestroy_record(ctx->open_record);\n+}\n+\n+/* At this point, there should be no references on this\n+ * socket and no in-flight SKBs associated with this\n+ * socket, so it is safe to free all the resources.\n+ */\n+void tls_device_sk_destruct(struct sock *sk)\n+{\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\tstruct tls_offload_context *ctx = tls_offload_ctx(tls_ctx);\n+\n+\tdelete_all_records(ctx);\n+\n+\ttls_device_queue_ctx_destruction(tls_ctx);\n+}\n+EXPORT_SYMBOL(tls_device_sk_destruct);\n+\n+static inline void tls_append_frag(struct tls_record_info *record,\n+\t\t\t\t struct page_frag *pfrag,\n+\t\t\t\t int size)\n+{\n+\tskb_frag_t *frag;\n+\n+\tfrag = &record->frags[record->num_frags - 1];\n+\tif (frag->page.p == pfrag->page &&\n+\t frag->page_offset + frag->size == pfrag->offset) {\n+\t\tfrag->size += size;\n+\t} else {\n+\t\t++frag;\n+\t\tfrag->page.p = pfrag->page;\n+\t\tfrag->page_offset = pfrag->offset;\n+\t\tfrag->size = size;\n+\t\t++record->num_frags;\n+\t\tget_page(pfrag->page);\n+\t}\n+\n+\tpfrag->offset += size;\n+\trecord->len += size;\n+}\n+\n+static inline int tls_push_record(struct sock *sk,\n+\t\t\t\t struct tls_context *ctx,\n+\t\t\t\t struct tls_offload_context *offload_ctx,\n+\t\t\t\t struct tls_record_info *record,\n+\t\t\t\t struct page_frag *pfrag,\n+\t\t\t\t int flags,\n+\t\t\t\t unsigned char record_type)\n+{\n+\tskb_frag_t *frag;\n+\tstruct tcp_sock *tp = tcp_sk(sk);\n+\tstruct page_frag fallback_frag;\n+\tstruct page_frag *tag_pfrag = pfrag;\n+\tint i;\n+\n+\t/* fill prepand */\n+\tfrag = &record->frags[0];\n+\ttls_fill_prepend(ctx,\n+\t\t\t skb_frag_address(frag),\n+\t\t\t record->len - ctx->prepend_size,\n+\t\t\t record_type);\n+\n+\tif (unlikely(!skb_page_frag_refill(\n+\t\t\t\tctx->tag_size,\n+\t\t\t\tpfrag, GFP_KERNEL))) {\n+\t\t/* HW doesn't care about the data in the tag\n+\t\t * so in case pfrag has no room\n+\t\t * for a tag and we can't allocate a new pfrag\n+\t\t * just use the page in the first frag\n+\t\t * rather then write a complicated fall back code.\n+\t\t */\n+\t\ttag_pfrag = &fallback_frag;\n+\t\ttag_pfrag->page = skb_frag_page(frag);\n+\t\ttag_pfrag->offset = 0;\n+\t}\n+\n+\ttls_append_frag(record, tag_pfrag, ctx->tag_size);\n+\trecord->end_seq = tp->write_seq + record->len;\n+\tspin_lock_irq(&offload_ctx->lock);\n+\tlist_add_tail(&record->list, &offload_ctx->records_list);\n+\tspin_unlock_irq(&offload_ctx->lock);\n+\toffload_ctx->open_record = NULL;\n+\tset_bit(TLS_PENDING_CLOSED_RECORD, &ctx->flags);\n+\ttls_advance_record_sn(sk, ctx);\n+\n+\tfor (i = 0; i < record->num_frags; i++) {\n+\t\tfrag = &record->frags[i];\n+\t\tsg_unmark_end(&offload_ctx->sg_tx_data[i]);\n+\t\tsg_set_page(&offload_ctx->sg_tx_data[i], skb_frag_page(frag),\n+\t\t\t frag->size, frag->page_offset);\n+\t\tsk_mem_charge(sk, frag->size);\n+\t\tget_page(skb_frag_page(frag));\n+\t}\n+\tsg_mark_end(&offload_ctx->sg_tx_data[record->num_frags - 1]);\n+\n+\t/* all ready, send */\n+\treturn tls_push_sg(sk, ctx, offload_ctx->sg_tx_data, 0, flags);\n+}\n+\n+static inline int tls_create_new_record(\n+\t\tstruct tls_offload_context *offload_ctx,\n+\t\tstruct page_frag *pfrag,\n+\t\tsize_t prepend_size)\n+{\n+\tskb_frag_t *frag;\n+\tstruct tls_record_info *record;\n+\n+\trecord = kmalloc(sizeof(*record), GFP_KERNEL);\n+\tif (!record)\n+\t\treturn -ENOMEM;\n+\n+\tfrag = &record->frags[0];\n+\t__skb_frag_set_page(frag, pfrag->page);\n+\tfrag->page_offset = pfrag->offset;\n+\tskb_frag_size_set(frag, prepend_size);\n+\n+\tget_page(pfrag->page);\n+\tpfrag->offset += prepend_size;\n+\n+\trecord->num_frags = 1;\n+\trecord->len = prepend_size;\n+\toffload_ctx->open_record = record;\n+\treturn 0;\n+}\n+\n+static inline int tls_do_allocation(\n+\t\tstruct sock *sk,\n+\t\tstruct tls_offload_context *offload_ctx,\n+\t\tstruct page_frag *pfrag,\n+\t\tsize_t prepend_size)\n+{\n+\tint ret;\n+\n+\tif (!offload_ctx->open_record) {\n+\t\tif (unlikely(!skb_page_frag_refill(prepend_size, pfrag,\n+\t\t\t\t\t\t sk->sk_allocation))) {\n+\t\t\tsk->sk_prot->enter_memory_pressure(sk);\n+\t\t\tsk_stream_moderate_sndbuf(sk);\n+\t\t\treturn -ENOMEM;\n+\t\t}\n+\n+\t\tret = tls_create_new_record(offload_ctx, pfrag, prepend_size);\n+\t\tif (ret)\n+\t\t\treturn ret;\n+\n+\t\tif (pfrag->size > pfrag->offset)\n+\t\t\treturn 0;\n+\t}\n+\n+\tif (!sk_page_frag_refill(sk, pfrag))\n+\t\treturn -ENOMEM;\n+\n+\treturn 0;\n+}\n+\n+static int tls_push_data(struct sock *sk,\n+\t\t\t struct iov_iter *msg_iter,\n+\t\t\t size_t size, int flags,\n+\t\t\t unsigned char record_type)\n+{\n+\tstruct tls_context *tls_ctx = tls_get_ctx(sk);\n+\tstruct tls_offload_context *ctx = tls_offload_ctx(tls_ctx);\n+\tstruct tls_record_info *record = ctx->open_record;\n+\tstruct page_frag *pfrag;\n+\tint copy, rc = 0;\n+\tsize_t orig_size = size;\n+\tu32 max_open_record_len;\n+\tlong timeo;\n+\tint more = flags & (MSG_SENDPAGE_NOTLAST | MSG_MORE);\n+\tint tls_push_record_flags = flags | MSG_SENDPAGE_NOTLAST;\n+\tbool done = false;\n+\n+\tif (sk->sk_err)\n+\t\treturn -sk->sk_err;\n+\n+\ttimeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT);\n+\trc = tls_complete_pending_work(sk, tls_ctx, flags, &timeo);\n+\tif (rc < 0)\n+\t\treturn rc;\n+\n+\tpfrag = sk_page_frag(sk);\n+\n+\t/* KTLS_TLS_HEADER_SIZE is not counted as part of the TLS record, and\n+\t * we need to leave room for an authentication tag.\n+\t */\n+\tmax_open_record_len = TLS_MAX_PAYLOAD_SIZE +\n+\t\t\t tls_ctx->prepend_size;\n+\tdo {\n+\t\tif (tls_do_allocation(sk, ctx, pfrag,\n+\t\t\t\t tls_ctx->prepend_size)) {\n+\t\t\trc = sk_stream_wait_memory(sk, &timeo);\n+\t\t\tif (!rc)\n+\t\t\t\tcontinue;\n+\n+\t\t\trecord = ctx->open_record;\n+\t\t\tif (!record)\n+\t\t\t\tbreak;\n+handle_error:\n+\t\t\tif (record_type != TLS_RECORD_TYPE_DATA) {\n+\t\t\t\t/* avoid sending partial\n+\t\t\t\t * record with type !=\n+\t\t\t\t * application_data\n+\t\t\t\t */\n+\t\t\t\tsize = orig_size;\n+\t\t\t\tdestroy_record(record);\n+\t\t\t\tctx->open_record = NULL;\n+\t\t\t} else if (record->len > tls_ctx->prepend_size) {\n+\t\t\t\tgoto last_record;\n+\t\t\t}\n+\n+\t\t\tbreak;\n+\t\t}\n+\n+\t\trecord = ctx->open_record;\n+\t\tcopy = min_t(size_t, size, (pfrag->size - pfrag->offset));\n+\t\tcopy = min_t(size_t, copy, (max_open_record_len - record->len));\n+\n+\t\tif (copy_from_iter_nocache(\n+\t\t\t\tpage_address(pfrag->page) + pfrag->offset,\n+\t\t\t\tcopy, msg_iter) != copy) {\n+\t\t\trc = -EFAULT;\n+\t\t\tgoto handle_error;\n+\t\t}\n+\t\ttls_append_frag(record, pfrag, copy);\n+\n+\t\tsize -= copy;\n+\t\tif (!size) {\n+last_record:\n+\t\t\ttls_push_record_flags = flags;\n+\t\t\tif (more) {\n+\t\t\t\ttls_ctx->pending_open_record_frags =\n+\t\t\t\t\t\trecord->num_frags;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\n+\t\t\tdone = true;\n+\t\t}\n+\n+\t\tif ((done) ||\n+\t\t (record->len >= max_open_record_len) ||\n+\t\t (record->num_frags >= MAX_SKB_FRAGS - 1)) {\n+\t\t\trc = tls_push_record(sk,\n+\t\t\t\t\t tls_ctx,\n+\t\t\t\t\t ctx,\n+\t\t\t\t\t record,\n+\t\t\t\t\t pfrag,\n+\t\t\t\t\t tls_push_record_flags,\n+\t\t\t\t\t record_type);\n+\t\t\tif (rc < 0)\n+\t\t\t\tbreak;\n+\t\t}\n+\t} while (!done);\n+\n+\tif (orig_size - size > 0)\n+\t\trc = orig_size - size;\n+\n+\treturn rc;\n+}\n+\n+int tls_device_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)\n+{\n+\tunsigned char record_type = TLS_RECORD_TYPE_DATA;\n+\tint rc = 0;\n+\n+\tlock_sock(sk);\n+\n+\tif (unlikely(msg->msg_controllen)) {\n+\t\trc = tls_proccess_cmsg(sk, msg, &record_type);\n+\t\tif (rc)\n+\t\t\tgoto out;\n+\t}\n+\n+\trc = tls_push_data(sk, &msg->msg_iter, size,\n+\t\t\t msg->msg_flags, record_type);\n+\n+out:\n+\trelease_sock(sk);\n+\treturn rc;\n+}\n+\n+int tls_device_sendpage(struct sock *sk, struct page *page,\n+\t\t\tint offset, size_t size, int flags)\n+{\n+\tstruct iov_iter\tmsg_iter;\n+\tstruct kvec iov;\n+\tchar *kaddr = kmap(page);\n+\tint rc = 0;\n+\n+\tif (flags & MSG_SENDPAGE_NOTLAST)\n+\t\tflags |= MSG_MORE;\n+\n+\tlock_sock(sk);\n+\n+\tif (flags & MSG_OOB) {\n+\t\trc = -ENOTSUPP;\n+\t\tgoto out;\n+\t}\n+\n+\tiov.iov_base = kaddr + offset;\n+\tiov.iov_len = size;\n+\tiov_iter_kvec(&msg_iter, WRITE | ITER_KVEC, &iov, 1, size);\n+\trc = tls_push_data(sk, &msg_iter, size,\n+\t\t\t flags, TLS_RECORD_TYPE_DATA);\n+\tkunmap(page);\n+\n+out:\n+\trelease_sock(sk);\n+\treturn rc;\n+}\n+\n+struct tls_record_info *tls_get_record(struct tls_offload_context *context,\n+\t\t\t\t u32 seq)\n+{\n+\tstruct tls_record_info *info;\n+\n+\tinfo = context->retransmit_hint;\n+\tif (!info ||\n+\t before(seq, info->end_seq - info->len))\n+\t\tinfo = list_first_entry(&context->records_list,\n+\t\t\t\t\tstruct tls_record_info, list);\n+\n+\tlist_for_each_entry_from(info, &context->records_list, list) {\n+\t\tif (before(seq, info->end_seq)) {\n+\t\t\tif (!context->retransmit_hint ||\n+\t\t\t after(info->end_seq,\n+\t\t\t\t context->retransmit_hint->end_seq))\n+\t\t\t\tcontext->retransmit_hint = info;\n+\t\t\treturn info;\n+\t\t}\n+\t}\n+\n+\treturn NULL;\n+}\n+EXPORT_SYMBOL(tls_get_record);\n+\n+static int tls_device_push_pending_record(struct sock *sk, int flags)\n+{\n+\tstruct iov_iter\tmsg_iter;\n+\n+\tiov_iter_kvec(&msg_iter, WRITE | ITER_KVEC, NULL, 0, 0);\n+\treturn tls_push_data(sk, &msg_iter, 0, flags, TLS_RECORD_TYPE_DATA);\n+}\n+\n+int tls_set_device_offload(struct sock *sk, struct tls_context *ctx)\n+{\n+\tstruct tls_crypto_info *crypto_info;\n+\tstruct tls_offload_context *offload_ctx;\n+\tstruct tls_record_info *start_marker_record;\n+\tu16 nonece_size, tag_size, iv_size, rec_seq_size;\n+\tchar *iv, *rec_seq;\n+\tint rc;\n+\tstruct net_device *netdev;\n+\tstruct sk_buff *skb;\n+\n+\tif (!ctx) {\n+\t\trc = -EINVAL;\n+\t\tgoto out;\n+\t}\n+\n+\tif (ctx->priv_ctx) {\n+\t\trc = -EEXIST;\n+\t\tgoto out;\n+\t}\n+\n+\tnetdev = get_netdev_for_sock(sk);\n+\tif (!netdev) {\n+\t\tpr_err(\"%s: netdev not found\\n\", __func__);\n+\t\trc = -EINVAL;\n+\t\tgoto out;\n+\t}\n+\n+\tif (!(netdev->features & NETIF_F_HW_TLS_TX)) {\n+\t\trc = -ENOTSUPP;\n+\t\tgoto release_netdev;\n+\t}\n+\n+\tcrypto_info = &ctx->crypto_send;\n+\tswitch (crypto_info->cipher_type) {\n+\tcase TLS_CIPHER_AES_GCM_128: {\n+\t\tnonece_size = TLS_CIPHER_AES_GCM_128_IV_SIZE;\n+\t\ttag_size = TLS_CIPHER_AES_GCM_128_TAG_SIZE;\n+\t\tiv_size = TLS_CIPHER_AES_GCM_128_IV_SIZE;\n+\t\tiv = ((struct tls12_crypto_info_aes_gcm_128 *)crypto_info)->iv;\n+\t\trec_seq_size = TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE;\n+\t\trec_seq =\n+\t\t ((struct tls12_crypto_info_aes_gcm_128 *)crypto_info)->rec_seq;\n+\t\tbreak;\n+\t}\n+\tdefault:\n+\t\trc = -EINVAL;\n+\t\tgoto release_netdev;\n+\t}\n+\n+\tstart_marker_record = kmalloc(sizeof(*start_marker_record), GFP_KERNEL);\n+\tif (!start_marker_record) {\n+\t\trc = -ENOMEM;\n+\t\tgoto release_netdev;\n+\t}\n+\n+\trc = attach_sock_to_netdev(sk, netdev, ctx);\n+\tif (rc)\n+\t\tgoto free_marker_record;\n+\n+\tctx->netdev = netdev;\n+\tctx->sk = sk;\n+\n+\tctx->prepend_size = TLS_HEADER_SIZE + nonece_size;\n+\tctx->tag_size = tag_size;\n+\tctx->iv_size = iv_size;\n+\tctx->iv = kmalloc(iv_size + TLS_CIPHER_AES_GCM_128_SALT_SIZE,\n+\t\t\t GFP_KERNEL);\n+\tif (!ctx->iv) {\n+\t\trc = -ENOMEM;\n+\t\tgoto detach_sock;\n+\t}\n+\tmemcpy(ctx->iv + TLS_CIPHER_AES_GCM_128_SALT_SIZE, iv, iv_size);\n+\tctx->rec_seq_size = rec_seq_size;\n+\tctx->rec_seq = kmalloc(rec_seq_size, GFP_KERNEL);\n+\tif (!ctx->rec_seq) {\n+\t\trc = -ENOMEM;\n+\t\tgoto err_iv;\n+\t}\n+\tmemcpy(ctx->rec_seq, rec_seq, rec_seq_size);\n+\n+\toffload_ctx = ctx->priv_ctx;\n+\tstart_marker_record->end_seq = tcp_sk(sk)->write_seq;\n+\tstart_marker_record->len = 0;\n+\tstart_marker_record->num_frags = 0;\n+\n+\tINIT_LIST_HEAD(&offload_ctx->records_list);\n+\tlist_add_tail(&start_marker_record->list, &offload_ctx->records_list);\n+\tspin_lock_init(&offload_ctx->lock);\n+\n+\tinet_csk(sk)->icsk_clean_acked = &tls_icsk_clean_acked;\n+\tctx->push_pending_record = tls_device_push_pending_record;\n+\tctx->free_resources = tls_device_free_resources;\n+\toffload_ctx->sk_destruct = sk->sk_destruct;\n+\n+\t/* TLS offload is greatly simplified if we don't send\n+\t * SKBs where only part of the payload needs to be encrypted.\n+\t * So mark the last skb in the write queue as end of record.\n+\t */\n+\tskb = tcp_write_queue_tail(sk);\n+\tif (skb)\n+\t\tTCP_SKB_CB(skb)->eor = 1;\n+\n+\t/* After the next line tls_is_sk_tx_device_offloaded\n+\t * will return true and ndo_start_xmit might access the\n+\t * offload context\n+\t */\n+\tsmp_store_release(&sk->sk_destruct,\n+\t\t\t &tls_device_sk_destruct);\n+\tgoto release_netdev;\n+\n+err_iv:\n+\tkfree(ctx->iv);\n+detach_sock:\n+\tdetach_sock_from_netdev(sk, ctx);\n+free_marker_record:\n+\tkfree(start_marker_record);\n+release_netdev:\n+\tdev_put(netdev);\n+out:\n+\treturn rc;\n+}\n+\n+void __exit tls_device_cleanup(void)\n+{\n+\tflush_work(&tls_device_gc_work);\n+}\ndiff --git a/net/tls/tls_main.c b/net/tls/tls_main.c\nindex ae20ee3..a93a712 100644\n--- a/net/tls/tls_main.c\n+++ b/net/tls/tls_main.c\n@@ -45,8 +45,16 @@\n MODULE_DESCRIPTION(\"Transport Layer Security Support\");\n MODULE_LICENSE(\"Dual BSD/GPL\");\n \n-static struct proto tls_base_prot;\n-static struct proto tls_sw_prot;\n+enum {\n+\tTLS_BASE_TX,\n+\tTLS_SW_TX,\n+#ifdef CONFIG_TLS_DEVICE\n+\tTLS_HW_TX,\n+#endif\n+\tTLS_NUM_CONFIG,\n+};\n+\n+static struct proto tls_prots[TLS_NUM_CONFIG];\n \n int wait_on_pending_writer(struct sock *sk, long *timeo)\n {\n@@ -393,11 +401,19 @@ static int do_tls_setsockopt_tx(struct sock *sk, char __user *optval,\n \n \tctx->sk_proto_close = sk->sk_prot->close;\n \n-\t/* currently SW is default, we will have ethtool in future */\n-\trc = tls_set_sw_offload(sk, ctx);\n-\tprot = &tls_sw_prot;\n-\tif (rc)\n-\t\tgoto err_crypto_info;\n+#ifdef CONFIG_TLS_DEVICE\n+\trc = tls_set_device_offload(sk, ctx);\n+\tprot = &tls_prots[TLS_HW_TX];\n+\tif (rc) {\n+#else\n+\t{\n+#endif\n+\t\t/* if HW offload fails fallback to SW */\n+\t\trc = tls_set_sw_offload(sk, ctx);\n+\t\tprot = &tls_prots[TLS_SW_TX];\n+\t\tif (rc)\n+\t\t\tgoto err_crypto_info;\n+\t}\n \n \tsk->sk_prot = prot;\n \tgoto out;\n@@ -452,7 +468,8 @@ static int tls_init(struct sock *sk)\n \ticsk->icsk_ulp_data = ctx;\n \tctx->setsockopt = sk->sk_prot->setsockopt;\n \tctx->getsockopt = sk->sk_prot->getsockopt;\n-\tsk->sk_prot = &tls_base_prot;\n+\n+\tsk->sk_prot = &tls_prots[TLS_BASE_TX];\n out:\n \treturn rc;\n }\n@@ -463,16 +480,27 @@ static int tls_init(struct sock *sk)\n \t.init\t\t\t= tls_init,\n };\n \n-static int __init tls_register(void)\n+static void build_protos(struct proto *prot, struct proto *base)\n {\n-\ttls_base_prot\t\t\t= tcp_prot;\n-\ttls_base_prot.setsockopt\t= tls_setsockopt;\n-\ttls_base_prot.getsockopt\t= tls_getsockopt;\n+\tprot[TLS_BASE_TX] = *base;\n+\tprot[TLS_BASE_TX].setsockopt = tls_setsockopt;\n+\tprot[TLS_BASE_TX].getsockopt = tls_getsockopt;\n+\n+\tprot[TLS_SW_TX] = prot[TLS_BASE_TX];\n+\tprot[TLS_SW_TX].close\t\t= tls_sk_proto_close;\n+\tprot[TLS_SW_TX].sendmsg\t\t= tls_sw_sendmsg;\n+\tprot[TLS_SW_TX].sendpage\t= tls_sw_sendpage;\n+\n+#ifdef CONFIG_TLS_DEVICE\n+\tprot[TLS_HW_TX] = prot[TLS_SW_TX];\n+\tprot[TLS_HW_TX].sendmsg\t\t= tls_device_sendmsg;\n+\tprot[TLS_HW_TX].sendpage\t= tls_device_sendpage;\n+#endif\n+}\n \n-\ttls_sw_prot\t\t\t= tls_base_prot;\n-\ttls_sw_prot.sendmsg\t\t= tls_sw_sendmsg;\n-\ttls_sw_prot.sendpage = tls_sw_sendpage;\n-\ttls_sw_prot.close = tls_sk_proto_close;\n+static int __init tls_register(void)\n+{\n+\tbuild_protos(tls_prots, &tcp_prot);\n \n \ttcp_register_ulp(&tcp_tls_ulp_ops);\n \n@@ -482,6 +510,9 @@ static int __init tls_register(void)\n static void __exit tls_unregister(void)\n {\n \ttcp_unregister_ulp(&tcp_tls_ulp_ops);\n+#ifdef CONFIG_TLS_DEVICE\n+\ttls_device_cleanup();\n+#endif\n }\n \n module_init(tls_register);\n", "prefixes": [ "net-next", "5/5" ] }