From patchwork Sat Sep 7 05:29:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 1159266 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="nHh7E8NQ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 46QNLL0dLVz9sNf for ; Sat, 7 Sep 2019 15:30:34 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2394402AbfIGFaV (ORCPT ); Sat, 7 Sep 2019 01:30:21 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:36694 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2394256AbfIGFaV (ORCPT ); Sat, 7 Sep 2019 01:30:21 -0400 Received: by mail-wr1-f66.google.com with SMTP id y19so8563764wrd.3 for ; Fri, 06 Sep 2019 22:30:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f2s5rhtwO8YHxyDkRTSrlV6IIFRP4Xm8hU+pa5uONk4=; b=nHh7E8NQtmzN9NMKoDlOHwF3gy/4i6AkJGaHM4tj+0jli7P2POBedUtRfFsWTf2qaa i1v/NMsqwSTDq2axbZPGFB0JzwhPvwyqSkGKc3QJ3dOHwigDzHGAkH9/3gEavwKHUy2S gFslG6w5ytd68XL6HbyzDzUkr5iGY4Dy4/xc7CJVIFLf/Nh5ND42ItjVKFceDidu/ImS 84M54mA9yliVMv0oJYq0+JA2eRf1U9bFBoAePlCTx/dEgGEgM4tbDCTIpKGrCU9fqEqY B6NUKeXojSvp+T415tw3zbMIk8EFMhYLcLtCoB4OCfWc5QHO/rwvzylj2FVjiRYkajyn 0q2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f2s5rhtwO8YHxyDkRTSrlV6IIFRP4Xm8hU+pa5uONk4=; b=IF/2A28kCs1cCaHlP5bDn3AhT9pUgTVy4L0/DZ59ndmDVR53z6ncFXbdDUGgGDf4in nxqIARGUIJ3DtnCKQlJD4oMaTW0lsdRXol58zu+qPmtplpZp6QTt831L9a/nGoAKRMkx sQysnqYCZct9c+L3lb2qBgqh0/cNZ/BiKEfNaPGZ3R5wFobqEJphGGh36eSuFJ5en7ZK o2J+P8rN8o1bGnBpAl+7zlmmi25Nwb1Feb9Rs8VsH08cJY5mR+SH0ALNyKxGFPKfAlj+ wl1qRCvDkZWc8Uwq9fkmE3Y9TLEwaIk5MxBE06tJR/jA1XMsCGPNNRIyAfbXPwnqhg9p Zc3w== X-Gm-Message-State: APjAAAXrCXqan6NmqN2h3fgCqHh1JTqQS0O4n1lL4WGCNyDy80Uihlf1 NBzxFCf/eDfSWP0QDSQYrkTnwQ== X-Google-Smtp-Source: APXvYqzKui96PtaXVeM4Kv/+btM0Ccud81Yoe5Fn0FfURIPAojDiAY8P/iH9hRbt/8OIPyS2rwcDCw== X-Received: by 2002:adf:db01:: with SMTP id s1mr6786333wri.164.1567834218748; Fri, 06 Sep 2019 22:30:18 -0700 (PDT) Received: from jkicinski-Precision-T1700.netronome.com ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id n4sm2446939wmd.45.2019.09.06.22.30.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Sep 2019 22:30:18 -0700 (PDT) From: Jakub Kicinski To: davem@davemloft.net Cc: netdev@vger.kernel.org, oss-drivers@netronome.com, davejwatson@fb.com, borisp@mellanox.com, aviadye@mellanox.com, john.fastabend@gmail.com, daniel@iogearbox.net, Jakub Kicinski , Dirk van der Merwe Subject: [PATCH net-next 1/4] net/tls: unref frags in order Date: Fri, 6 Sep 2019 22:29:57 -0700 Message-Id: <20190907053000.23869-2-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190907053000.23869-1-jakub.kicinski@netronome.com> References: <20190907053000.23869-1-jakub.kicinski@netronome.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org It's generally more cache friendly to walk arrays in order, especially those which are likely not in cache. Signed-off-by: Jakub Kicinski Reviewed-by: Dirk van der Merwe --- net/tls/tls_device.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 41c106e45f01..285c9f9e94e4 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -122,13 +122,10 @@ static struct net_device *get_netdev_for_sock(struct sock *sk) static void destroy_record(struct tls_record_info *record) { - int nr_frags = record->num_frags; - skb_frag_t *frag; + int i; - while (nr_frags-- > 0) { - frag = &record->frags[nr_frags]; - __skb_frag_unref(frag); - } + for (i = 0; i < record->num_frags; i++) + __skb_frag_unref(&record->frags[i]); kfree(record); } From patchwork Sat Sep 7 05:29:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 1159267 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="ppvxpbsX"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 46QNLL46z4z9sNk for ; Sat, 7 Sep 2019 15:30:34 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2394413AbfIGFaY (ORCPT ); Sat, 7 Sep 2019 01:30:24 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:34432 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729239AbfIGFaY (ORCPT ); Sat, 7 Sep 2019 01:30:24 -0400 Received: by mail-wr1-f68.google.com with SMTP id s18so8564546wrn.1 for ; Fri, 06 Sep 2019 22:30:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=58RVaXeoboEIXt5GYSoeQPQsnm9j+sWV+ds+yjKX9b8=; b=ppvxpbsXkrfV8HV867kL2H8pN4Hg+MX5Lb7IAfZDJwvkEpU7AvXZjkRJwg8DlequSk qT3Iw5PZo47EGXA9qdn4xwfjZEnpma0YuiRA5rp43wa+oVYDQRG8yOj/s0GC0ysl8y82 k+R70Uu1w9vsL+TtOu9ryQtoeXEepUj4S+/jPUzOhcpTsC2hF4MFwlzMUqFvfE9KciEA 87lObT93iCFTki2s36xndAOhaC8ZQ/L+RJYcBcN4jlRD3kShfhnCBFUd9yK8tZWZrl1T 2BS0dXxoM+WKQw/XA7yqTU4bIqai3mdyw0IyuqO33N+UlogIBtvzzcTi3LQvLpBjU56S NkOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=58RVaXeoboEIXt5GYSoeQPQsnm9j+sWV+ds+yjKX9b8=; b=VfyAerLvLQAJzVcR0ifT58KV+UsDt0E86rdwG7hFvXNs5SczQYaAbDx8GFO5JTpLX9 1B3bA0Exha/zNM+uwLQAPX1lxJqGFTOwC8x9ZNo6z43iBvow9YPBCn3TyCrurHG2Nn80 H51A9D522nd7rG1xvCgidi7Kra/qH8TJSQBWkf4yiiyqTJD/diwW4QCdYO7t05l/Hq3O 2i3/t9dEfTzFEaFa/+NHj8Ym+L+t2GNRdl2n0qPbk+i/7+36Y11zuVLXzMmMZr3OlvAE lI5Ha9rb5KUJ5Ten6G09a3tew+1KgXcFNoM0rYzmybWeh+39SANajJ1quKw3dqgcJR1C hH/g== X-Gm-Message-State: APjAAAXKYoMjYAEQ/XMa5KBX+q6ZwfZmuux4pvjQTsPf8TYzakHVAepF 9OClbPNp8imR/3iwbr7POJ/RHw== X-Google-Smtp-Source: APXvYqxvM0hknMOzboe+SYCIo9NTMons0mgF3/QPuy2XxZskjFmLI+CwMUL6dNrJfXrhO9apxNFniA== X-Received: by 2002:adf:9043:: with SMTP id h61mr9732119wrh.247.1567834221788; Fri, 06 Sep 2019 22:30:21 -0700 (PDT) Received: from jkicinski-Precision-T1700.netronome.com ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id n4sm2446939wmd.45.2019.09.06.22.30.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Sep 2019 22:30:21 -0700 (PDT) From: Jakub Kicinski To: davem@davemloft.net Cc: netdev@vger.kernel.org, oss-drivers@netronome.com, davejwatson@fb.com, borisp@mellanox.com, aviadye@mellanox.com, john.fastabend@gmail.com, daniel@iogearbox.net, Jakub Kicinski , Dirk van der Merwe Subject: [PATCH net-next 2/4] net/tls: use RCU for the adder to the offload record list Date: Fri, 6 Sep 2019 22:29:58 -0700 Message-Id: <20190907053000.23869-3-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190907053000.23869-1-jakub.kicinski@netronome.com> References: <20190907053000.23869-1-jakub.kicinski@netronome.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org All modifications to TLS record list happen under the socket lock. Since records form an ordered queue readers are only concerned about elements being removed, additions can happen concurrently. Use RCU primitives to ensure the correct access types (READ_ONCE/WRITE_ONCE). Signed-off-by: Jakub Kicinski Reviewed-by: Dirk van der Merwe --- net/tls/tls_device.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 285c9f9e94e4..b11355e00514 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -280,9 +280,7 @@ static int tls_push_record(struct sock *sk, tls_append_frag(record, &dummy_tag_frag, prot->tag_size); record->end_seq = tp->write_seq + record->len; - spin_lock_irq(&offload_ctx->lock); - list_add_tail(&record->list, &offload_ctx->records_list); - spin_unlock_irq(&offload_ctx->lock); + list_add_tail_rcu(&record->list, &offload_ctx->records_list); offload_ctx->open_record = NULL; if (test_bit(TLS_TX_SYNC_SCHED, &ctx->flags)) @@ -535,12 +533,16 @@ struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context, /* if retransmit_hint is irrelevant start * from the beggining of the list */ - info = list_first_entry(&context->records_list, - struct tls_record_info, list); + info = list_first_entry_or_null(&context->records_list, + struct tls_record_info, list); + if (!info) + return NULL; record_sn = context->unacked_record_sn; } - list_for_each_entry_from(info, &context->records_list, list) { + /* We just need the _rcu for the READ_ONCE() */ + rcu_read_lock(); + list_for_each_entry_from_rcu(info, &context->records_list, list) { if (before(seq, info->end_seq)) { if (!context->retransmit_hint || after(info->end_seq, @@ -549,12 +551,15 @@ struct tls_record_info *tls_get_record(struct tls_offload_context_tx *context, context->retransmit_hint = info; } *p_record_sn = record_sn; - return info; + goto exit_rcu_unlock; } record_sn++; } + info = NULL; - return NULL; +exit_rcu_unlock: + rcu_read_unlock(); + return info; } EXPORT_SYMBOL(tls_get_record); From patchwork Sat Sep 7 05:29:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 1159268 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="s9bRZvm3"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 46QNLM0rptz9sP3 for ; Sat, 7 Sep 2019 15:30:35 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2394433AbfIGFa1 (ORCPT ); Sat, 7 Sep 2019 01:30:27 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:44501 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2394420AbfIGFa0 (ORCPT ); Sat, 7 Sep 2019 01:30:26 -0400 Received: by mail-wr1-f66.google.com with SMTP id 30so8515484wrk.11 for ; Fri, 06 Sep 2019 22:30:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SFYVCFDmUtuKidhMHe9RczcjQTKIIorRjG9j3ugVPdw=; b=s9bRZvm3HQhxMVncU//uvmFID8o8X1N1cKK2RH6Qzr4XUrBe0OKfqK6U4Io9qOQzjY t4KitTVnRP79JQKYOc+XKBbucLERi1yYfI+sE/FrOrKHzZS6iI5LBLfYiJXKQrBc30x5 2oxUKBjvJh5xvCCsESDpvzPwKtk2V52UayGALrJRKCVV3OcErZ5shRZXjGpWtUFG6tit cDLCSiJTbQcLl55N9Sl9O2aofEHTZ+NG/ptjq922TiSxEjUkBfzaPycR0zGUm+iwmVPu ZRhucdnNCSYCkakx/nZEFnpCWAil1aXp+q60i0WFj09HqIwf4Zjge/M6vNNwRoWwoXwb hjzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SFYVCFDmUtuKidhMHe9RczcjQTKIIorRjG9j3ugVPdw=; b=UK50r5qWP3gZYzrbCOjvq3npe3D6CcoB+6xLp5b56k8BpsySyw/8tT9XYtjLHJxPsq EV0NB5impiP71gNkDk9bgxV8vHi6e74clWAUW0qGLlS1fuyZOwBxHhFvBGi0M2xx4qzK xpeVHETQHHWezQSPSb/YgqDirTrGSbhDNm6KMgzgPnT06EkChdoBjCa5tLlaDoV5dfLH zyNEpMtnWti4ZLUamDbGoALQZVHvU1p5L05FJD+pHbvqrs6Z9FpF9XQslq/mNLt7bbB1 jpDTUbw5Yh3+zCzKM6vwUbMvq29EpQEyJjHAFuDAjx2veAUHFgUl7mc+MYhbELzwooQi OpAg== X-Gm-Message-State: APjAAAWXisBZyGtGw8pows3eqRPhlCp1bSjwHQrQF4dARTJESY7qqS3x 9pX4V/dSyXb3mzk05j2M3MfqoQ== X-Google-Smtp-Source: APXvYqzH1ZcCtyiHq0xxlUSZwVoVvH6gwGRRH6HP7c1rz3BR3ifCJmqlfu4ufsGFQ01sfpDWnDw4Ww== X-Received: by 2002:adf:fd41:: with SMTP id h1mr9492644wrs.315.1567834224856; Fri, 06 Sep 2019 22:30:24 -0700 (PDT) Received: from jkicinski-Precision-T1700.netronome.com ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id n4sm2446939wmd.45.2019.09.06.22.30.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Sep 2019 22:30:24 -0700 (PDT) From: Jakub Kicinski To: davem@davemloft.net Cc: netdev@vger.kernel.org, oss-drivers@netronome.com, davejwatson@fb.com, borisp@mellanox.com, aviadye@mellanox.com, john.fastabend@gmail.com, daniel@iogearbox.net, Jakub Kicinski , Dirk van der Merwe Subject: [PATCH net-next 3/4] net/tls: remove the record tail optimization Date: Fri, 6 Sep 2019 22:29:59 -0700 Message-Id: <20190907053000.23869-4-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190907053000.23869-1-jakub.kicinski@netronome.com> References: <20190907053000.23869-1-jakub.kicinski@netronome.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org For TLS device offload the tag/message authentication code are filled in by the device. The kernel merely reserves space for them. Because device overwrites it, the contents of the tag make do no matter. Current code tries to save space by reusing the header as the tag. This, however, leads to an additional frag being created and defeats buffer coalescing (which trickles all the way down to the drivers). Remove this optimization, and try to allocate the space for the tag in the usual way, leave the memory uninitialized. If memory allocation fails rewind the record pointer so that we use the already copied user data as tag. Note that the optimization was actually buggy, as the tag for TLS 1.2 is 16 bytes, but header is just 13, so the reuse may had looked past the end of the page.. Signed-off-by: Jakub Kicinski Reviewed-by: Dirk van der Merwe --- net/tls/tls_device.c | 67 +++++++++++++++++++++++++++++++------------- 1 file changed, 47 insertions(+), 20 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index b11355e00514..916c3c0a99f0 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -256,29 +256,13 @@ static int tls_push_record(struct sock *sk, struct tls_context *ctx, struct tls_offload_context_tx *offload_ctx, struct tls_record_info *record, - struct page_frag *pfrag, - int flags, - unsigned char record_type) + int flags) { struct tls_prot_info *prot = &ctx->prot_info; struct tcp_sock *tp = tcp_sk(sk); - struct page_frag dummy_tag_frag; skb_frag_t *frag; int i; - /* fill prepend */ - frag = &record->frags[0]; - tls_fill_prepend(ctx, - skb_frag_address(frag), - record->len - prot->prepend_size, - record_type, - prot->version); - - /* HW doesn't care about the data in the tag, because it fills it. */ - dummy_tag_frag.page = skb_frag_page(frag); - dummy_tag_frag.offset = 0; - - tls_append_frag(record, &dummy_tag_frag, prot->tag_size); record->end_seq = tp->write_seq + record->len; list_add_tail_rcu(&record->list, &offload_ctx->records_list); offload_ctx->open_record = NULL; @@ -302,6 +286,38 @@ static int tls_push_record(struct sock *sk, return tls_push_sg(sk, ctx, offload_ctx->sg_tx_data, 0, flags); } +static int tls_device_record_close(struct sock *sk, + struct tls_context *ctx, + struct tls_record_info *record, + struct page_frag *pfrag, + unsigned char record_type) +{ + struct tls_prot_info *prot = &ctx->prot_info; + int ret; + + /* append tag + * device will fill in the tag, we just need to append a placeholder + * use socket memory to improve coalescing (re-using a single buffer + * increases frag count) + * if we can't allocate memory now, steal some back from data + */ + if (likely(skb_page_frag_refill(prot->tag_size, pfrag, + sk->sk_allocation))) { + ret = 0; + tls_append_frag(record, pfrag, prot->tag_size); + } else { + ret = prot->tag_size; + if (record->len <= prot->overhead_size) + return -ENOMEM; + } + + /* fill prepend */ + tls_fill_prepend(ctx, skb_frag_address(&record->frags[0]), + record->len - prot->overhead_size, + record_type, prot->version); + return ret; +} + static int tls_create_new_record(struct tls_offload_context_tx *offload_ctx, struct page_frag *pfrag, size_t prepend_size) @@ -452,13 +468,24 @@ static int tls_push_data(struct sock *sk, if (done || record->len >= max_open_record_len || (record->num_frags >= MAX_SKB_FRAGS - 1)) { + rc = tls_device_record_close(sk, tls_ctx, record, + pfrag, record_type); + if (rc) { + if (rc > 0) { + size += rc; + } else { + size = orig_size; + destroy_record(record); + ctx->open_record = NULL; + break; + } + } + rc = tls_push_record(sk, tls_ctx, ctx, record, - pfrag, - tls_push_record_flags, - record_type); + tls_push_record_flags); if (rc < 0) break; } From patchwork Sat Sep 7 05:30:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 1159269 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="seRsbNMQ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 46QNLM5JvDz9s4Y for ; Sat, 7 Sep 2019 15:30:35 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2394449AbfIGFaa (ORCPT ); Sat, 7 Sep 2019 01:30:30 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:52487 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2394420AbfIGFaa (ORCPT ); Sat, 7 Sep 2019 01:30:30 -0400 Received: by mail-wm1-f67.google.com with SMTP id t17so8446867wmi.2 for ; Fri, 06 Sep 2019 22:30:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eGHarff9FRFVkSuJ2XYS/AC6bZO7MZEONTwO+G1Ua7A=; b=seRsbNMQqAN+Bz7PLLOYm7tTC18/gfYzeGKHznsfj9sncbl9AtlwkZGuIVwEj+PaM0 Im5X3nWEJvwz+DQ6ZG85dqV0iPv+iA/s3W3GdY4SzdJFJQfY1O3Wy2SBInsaLP8+UvEc bNZFhonNv2jz/o5IKa4m6LYXYki8D+L68cekzaOBJrcTvZWXnmTlGH1srPPdGSOSN92I DkhBIc59Z4v9EGCxI2qmsbJouyVH5Kw23Za/nTNDoPoTedRkjcl9nTmioor/HNbiaTm7 wPVKeQ25ZNpaU+mfygIc5Bghy0yuwLG4E6wRiYYhBhfiTstR0kIk/fau33cAPYNEL3im KmCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=eGHarff9FRFVkSuJ2XYS/AC6bZO7MZEONTwO+G1Ua7A=; b=hOek5NYaM9w+LadXrePXBp3eyfDWF86Op0TOQwyujmKd4h+J//O8GwiLDf+/FgLjSC YxPsaAzPFw3O43AtP6BRLq9TZ62NbCZ2mkztFrbeXQuo1gu7A9cUk9a4/YHNKdCSmWGI 9tYiRFvUfCI6SSLOILe4YnsLhDCXGaVwbiIddgA5NCilT+XTvuCJTGHE5KewZ3XnrFT1 Zci2eSM+oKNVkQrBivmlP9SUXDEjLkIBdxitKQ21yZyYMNelCIa7i/Xs2CaeyZFbemxi ZIao6GTTSi2I12CR9b7Oq7nezc/xjzZM0arCrs/gt4BM0Ouc+y8PvxkCrgI6lpKCI6cP qEYg== X-Gm-Message-State: APjAAAWdk9ZTBFuYU0gx6/dGT+Dsv/Nq60wtyQbyYwu2WY02bDDkwyfT Z0v5B25pCf5ek6fYyTQPn8UKLg== X-Google-Smtp-Source: APXvYqxFCC5+LiVjd9f7WE08SVNUxSpDx1WqnLUJifSC/L+y5jOEB/rVjj6nyQNi2H7k0uktltCCpA== X-Received: by 2002:a1c:1981:: with SMTP id 123mr9791837wmz.88.1567834227955; Fri, 06 Sep 2019 22:30:27 -0700 (PDT) Received: from jkicinski-Precision-T1700.netronome.com ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id n4sm2446939wmd.45.2019.09.06.22.30.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 06 Sep 2019 22:30:27 -0700 (PDT) From: Jakub Kicinski To: davem@davemloft.net Cc: netdev@vger.kernel.org, oss-drivers@netronome.com, davejwatson@fb.com, borisp@mellanox.com, aviadye@mellanox.com, john.fastabend@gmail.com, daniel@iogearbox.net, Jakub Kicinski , Dirk van der Merwe Subject: [PATCH net-next 4/4] net/tls: align non temporal copy to cache lines Date: Fri, 6 Sep 2019 22:30:00 -0700 Message-Id: <20190907053000.23869-5-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190907053000.23869-1-jakub.kicinski@netronome.com> References: <20190907053000.23869-1-jakub.kicinski@netronome.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Unlike normal TCP code TLS has to touch the cache lines it copies into to fill header info. On memory-heavy workloads having non temporal stores and normal accesses targeting the same cache line leads to significant overhead. Measured 3% overhead running 3600 round robin connections with additional memory heavy workload. Signed-off-by: Jakub Kicinski Reviewed-by: Dirk van der Merwe --- net/tls/tls_device.c | 33 ++++++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 5 deletions(-) diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 916c3c0a99f0..f959487c5cd1 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -372,6 +372,31 @@ static int tls_do_allocation(struct sock *sk, return 0; } +static int tls_device_copy_data(void *addr, size_t bytes, struct iov_iter *i) +{ + size_t pre_copy, nocache; + + pre_copy = ~((unsigned long)addr - 1) & (SMP_CACHE_BYTES - 1); + if (pre_copy) { + pre_copy = min(pre_copy, bytes); + if (copy_from_iter(addr, pre_copy, i) != pre_copy) + return -EFAULT; + bytes -= pre_copy; + addr += pre_copy; + } + + nocache = round_down(bytes, SMP_CACHE_BYTES); + if (copy_from_iter_nocache(addr, nocache, i) != nocache) + return -EFAULT; + bytes -= nocache; + addr += nocache; + + if (bytes && copy_from_iter(addr, bytes, i) != bytes) + return -EFAULT; + + return 0; +} + static int tls_push_data(struct sock *sk, struct iov_iter *msg_iter, size_t size, int flags, @@ -445,12 +470,10 @@ static int tls_push_data(struct sock *sk, copy = min_t(size_t, size, (pfrag->size - pfrag->offset)); copy = min_t(size_t, copy, (max_open_record_len - record->len)); - if (copy_from_iter_nocache(page_address(pfrag->page) + - pfrag->offset, - copy, msg_iter) != copy) { - rc = -EFAULT; + rc = tls_device_copy_data(page_address(pfrag->page) + + pfrag->offset, copy, msg_iter); + if (rc) goto handle_error; - } tls_append_frag(record, pfrag, copy); size -= copy;