From patchwork Wed Jan 29 23:15:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Flavio Leitner X-Patchwork-Id: 1231157 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=sysclose.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=sysclose.org header.i=@sysclose.org header.a=rsa-sha256 header.s=201903 header.b=eoRgiPbV; dkim=fail reason="signature verification failed" (2048-bit key) header.d=sysclose.org header.i=@sysclose.org header.a=rsa-sha256 header.s=201903 header.b=Kox/kLRh; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 487K8d1bxjz9sP6 for ; Thu, 30 Jan 2020 10:15:28 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1EA42883C6; Wed, 29 Jan 2020 23:15:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id C6sHlVhW5cNN; Wed, 29 Jan 2020 23:15:25 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id 1F390882F0; Wed, 29 Jan 2020 23:15:25 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 10F2AC0881; Wed, 29 Jan 2020 23:15:25 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5C7C5C0171 for ; Wed, 29 Jan 2020 23:15:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 4A5D122234 for ; Wed, 29 Jan 2020 23:15:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uqZwCsCkaqJZ for ; Wed, 29 Jan 2020 23:15:21 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sysclose.org (smtp.sysclose.org [69.164.214.230]) by silver.osuosl.org (Postfix) with ESMTPS id 90558220D0 for ; Wed, 29 Jan 2020 23:15:21 +0000 (UTC) Received: by sysclose.org (Postfix, from userid 5001) id 435EC4129; Wed, 29 Jan 2020 23:16:02 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 sysclose.org 435EC4129 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sysclose.org; s=201903; t=1580339762; bh=y4bF6gHMy0tv7kwNN7qojz8zeljz4aWh8c0Zbn9NxFM=; h=From:To:Cc:Subject:Date:From; b=eoRgiPbV/BvzKqXBAELK2TwWysH2VZEBRPV0nnDrALiRzs1ijBNpWRH+lLcSiMall tMe+LyAKqaAgLJ5HRtQSWvoDMvfKy1f+wnVCfAzK4sIYORe6ZhmwKVQ1qI/2pH+a/9 cg6QCvn0nH7qmYFpmhxZVx5VasCU9xCb+oZnVTTpn+4P2Rmhg0u+0uZCFPfxcZNTxp vmiLCFO3J/eg2+RrqtFloYEEIGxCKMQnYhWUpH4Nm5+ZJRPF+0heI1XIUxWOV3zxtI FAbF7RAkk/mxtwlGs9rn6206yzpYjHLQPkvZOgWEOW6gOg4fEbCNmBkkPlPKzbg4Sg O1jelCRph9VIQ== Received: from localhost (unknown [45.71.104.84]) by sysclose.org (Postfix) with ESMTPSA id 9034735BD; Wed, 29 Jan 2020 23:15:57 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 sysclose.org 9034735BD DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sysclose.org; s=201903; t=1580339757; bh=y4bF6gHMy0tv7kwNN7qojz8zeljz4aWh8c0Zbn9NxFM=; h=From:To:Cc:Subject:Date:From; b=Kox/kLRh8PgBN73ZTX0xoLyydFxEWsWeBB8QXbXkAv4sH9Wp10y+gSI8PQ7B86PXg +YsYiQOYgLy8/ygo4o65KU2H7zeu9hvoIjlIhQ6jRoygUCcKy2WGeBkixcu+hrPBjT WwfVbbz86WjwQuNRlMaCyuJ2Q57lA5+7L1d2r9H46WvnMz1yWNKOHyGfh4gxUk7SvF o2e5+j1jQTQJSFzbuXUyQf3CjUOInGJcaNegVrJHebjtqLgPcvoaSIeAAcq3GvQA+w yUND//ItQzPturLUQWNn/v7qZTl7g+Fhh0OYP0NdUuAZLyrYXOMwSaG7eqLWRib4bK R9ddRuzPRai4g== From: Flavio Leitner To: dev@openvswitch.org Date: Wed, 29 Jan 2020 20:15:11 -0300 Message-Id: <20200129231511.94186-1-fbl@sysclose.org> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Cc: Ilya Maximets , Flavio Leitner , txfh2007 Subject: [ovs-dev] [PATCH] netdev-linux: Prepend the std packet in the TSO packet X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Usually TSO packets are close to 50k, 60k bytes long, so to to copy less bytes when receiving a packet from the kernel change the approach. Instead of extending the MTU sized packet received and append with remaining TSO data from the TSO buffer, allocate a TSO packet with enough headroom to prepend the std packet data. Suggested-by: Ben Pfaff Signed-off-by: Flavio Leitner --- lib/dp-packet.c | 8 +-- lib/dp-packet.h | 2 + lib/netdev-linux-private.h | 3 +- lib/netdev-linux.c | 117 ++++++++++++++++++++++--------------- 4 files changed, 78 insertions(+), 52 deletions(-) diff --git a/lib/dp-packet.c b/lib/dp-packet.c index 8dfedcb7c..cd2623500 100644 --- a/lib/dp-packet.c +++ b/lib/dp-packet.c @@ -243,8 +243,8 @@ dp_packet_copy__(struct dp_packet *b, uint8_t *new_base, /* Reallocates 'b' so that it has exactly 'new_headroom' and 'new_tailroom' * bytes of headroom and tailroom, respectively. */ -static void -dp_packet_resize__(struct dp_packet *b, size_t new_headroom, size_t new_tailroom) +void +dp_packet_resize(struct dp_packet *b, size_t new_headroom, size_t new_tailroom) { void *new_base, *new_data; size_t new_allocated; @@ -297,7 +297,7 @@ void dp_packet_prealloc_tailroom(struct dp_packet *b, size_t size) { if (size > dp_packet_tailroom(b)) { - dp_packet_resize__(b, dp_packet_headroom(b), MAX(size, 64)); + dp_packet_resize(b, dp_packet_headroom(b), MAX(size, 64)); } } @@ -308,7 +308,7 @@ void dp_packet_prealloc_headroom(struct dp_packet *b, size_t size) { if (size > dp_packet_headroom(b)) { - dp_packet_resize__(b, MAX(size, 64), dp_packet_tailroom(b)); + dp_packet_resize(b, MAX(size, 64), dp_packet_tailroom(b)); } } diff --git a/lib/dp-packet.h b/lib/dp-packet.h index 69ae5dfac..9a9d35183 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -152,6 +152,8 @@ struct dp_packet *dp_packet_clone_with_headroom(const struct dp_packet *, struct dp_packet *dp_packet_clone_data(const void *, size_t); struct dp_packet *dp_packet_clone_data_with_headroom(const void *, size_t, size_t headroom); +void dp_packet_resize(struct dp_packet *b, size_t new_headroom, + size_t new_tailroom); static inline void dp_packet_delete(struct dp_packet *); static inline void *dp_packet_at(const struct dp_packet *, size_t offset, diff --git a/lib/netdev-linux-private.h b/lib/netdev-linux-private.h index 143616ca8..e5c13fc37 100644 --- a/lib/netdev-linux-private.h +++ b/lib/netdev-linux-private.h @@ -44,7 +44,8 @@ struct netdev_rxq_linux { struct netdev_rxq up; bool is_tap; int fd; - char *aux_bufs[NETDEV_MAX_BURST]; /* Batch of preallocated TSO buffers. */ + struct dp_packet *aux_bufs[NETDEV_MAX_BURST]; /* Preallocated TSO + packets. */ }; int netdev_linux_construct(struct netdev *); diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c index 6add3e2fc..c30a22287 100644 --- a/lib/netdev-linux.c +++ b/lib/netdev-linux.c @@ -1052,15 +1052,6 @@ static struct netdev_rxq * netdev_linux_rxq_alloc(void) { struct netdev_rxq_linux *rx = xzalloc(sizeof *rx); - if (userspace_tso_enabled()) { - int i; - - /* Allocate auxiliay buffers to receive TSO packets. */ - for (i = 0; i < NETDEV_MAX_BURST; i++) { - rx->aux_bufs[i] = xmalloc(LINUX_RXQ_TSO_MAX_LEN); - } - } - return &rx->up; } @@ -1172,7 +1163,7 @@ netdev_linux_rxq_destruct(struct netdev_rxq *rxq_) } for (i = 0; i < NETDEV_MAX_BURST; i++) { - free(rx->aux_bufs[i]); + dp_packet_delete(rx->aux_bufs[i]); } } @@ -1238,13 +1229,18 @@ netdev_linux_batch_rxq_recv_sock(struct netdev_rxq_linux *rx, int mtu, virtio_net_hdr_size = 0; } - std_len = VLAN_ETH_HEADER_LEN + mtu + virtio_net_hdr_size; + /* The length here needs to be accounted in the same way when the + * aux_buf is allocated so that it can be prepended to TSO buffer. */ + std_len = virtio_net_hdr_size + VLAN_ETH_HEADER_LEN + mtu; for (i = 0; i < NETDEV_MAX_BURST; i++) { buffers[i] = dp_packet_new_with_headroom(std_len, DP_NETDEV_HEADROOM); iovs[i][IOV_PACKET].iov_base = dp_packet_data(buffers[i]); iovs[i][IOV_PACKET].iov_len = std_len; - iovs[i][IOV_AUXBUF].iov_base = rx->aux_bufs[i]; - iovs[i][IOV_AUXBUF].iov_len = LINUX_RXQ_TSO_MAX_LEN; + if (iovlen == IOV_TSO_SIZE) { + iovs[i][IOV_AUXBUF].iov_base = dp_packet_data(rx->aux_bufs[i]); + iovs[i][IOV_AUXBUF].iov_len = dp_packet_size(rx->aux_bufs[i]); + } + mmsgs[i].msg_hdr.msg_name = NULL; mmsgs[i].msg_hdr.msg_namelen = 0; mmsgs[i].msg_hdr.msg_iov = iovs[i]; @@ -1268,6 +1264,8 @@ netdev_linux_batch_rxq_recv_sock(struct netdev_rxq_linux *rx, int mtu, } for (i = 0; i < retval; i++) { + struct dp_packet *pkt; + if (mmsgs[i].msg_len < ETH_HEADER_LEN) { struct netdev *netdev_ = netdev_rxq_get_netdev(&rx->up); struct netdev_linux *netdev = netdev_linux_cast(netdev_); @@ -1280,29 +1278,29 @@ netdev_linux_batch_rxq_recv_sock(struct netdev_rxq_linux *rx, int mtu, } if (mmsgs[i].msg_len > std_len) { - /* Build a single linear TSO packet by expanding the current packet - * to append the data received in the aux_buf. */ - size_t extra_len = mmsgs[i].msg_len - std_len; - - dp_packet_set_size(buffers[i], dp_packet_size(buffers[i]) - + std_len); - dp_packet_prealloc_tailroom(buffers[i], extra_len); - memcpy(dp_packet_tail(buffers[i]), rx->aux_bufs[i], extra_len); - dp_packet_set_size(buffers[i], dp_packet_size(buffers[i]) - + extra_len); - } else { - dp_packet_set_size(buffers[i], dp_packet_size(buffers[i]) - + mmsgs[i].msg_len); - } + /* Build a single linear TSO packet by prepending the data from + * std_len buffer to the aux_buf. */ + pkt = rx->aux_bufs[i]; + dp_packet_set_size(pkt, mmsgs[i].msg_len - std_len); + dp_packet_push(pkt, dp_packet_data(buffers[i]), std_len); + /* The headroom should be the same in buffers[i], pkt and + * DP_NETDEV_HEADROOM. */ + dp_packet_resize(pkt, DP_NETDEV_HEADROOM, 0); + dp_packet_delete(buffers[i]); + rx->aux_bufs[i] = NULL; + } else { + dp_packet_set_size(buffers[i], mmsgs[i].msg_len); + pkt = buffers[i]; + } - if (virtio_net_hdr_size && netdev_linux_parse_vnet_hdr(buffers[i])) { + if (virtio_net_hdr_size && netdev_linux_parse_vnet_hdr(pkt)) { struct netdev *netdev_ = netdev_rxq_get_netdev(&rx->up); struct netdev_linux *netdev = netdev_linux_cast(netdev_); /* Unexpected error situation: the virtio header is not present * or corrupted. Drop the packet but continue in case next ones * are correct. */ - dp_packet_delete(buffers[i]); + dp_packet_delete(pkt); netdev->rx_dropped += 1; VLOG_WARN_RL(&rl, "%s: Dropped packet: Invalid virtio net header", netdev_get_name(netdev_)); @@ -1325,16 +1323,16 @@ netdev_linux_batch_rxq_recv_sock(struct netdev_rxq_linux *rx, int mtu, struct eth_header *eth; bool double_tagged; - eth = dp_packet_data(buffers[i]); + eth = dp_packet_data(pkt); double_tagged = eth->eth_type == htons(ETH_TYPE_VLAN_8021Q); - eth_push_vlan(buffers[i], + eth_push_vlan(pkt, auxdata_to_vlan_tpid(aux, double_tagged), htons(aux->tp_vlan_tci)); break; } } - dp_packet_batch_add(batch, buffers[i]); + dp_packet_batch_add(batch, pkt); } /* Delete unused buffers. */ @@ -1354,7 +1352,6 @@ static int netdev_linux_batch_rxq_recv_tap(struct netdev_rxq_linux *rx, int mtu, struct dp_packet_batch *batch) { - struct dp_packet *buffer; int virtio_net_hdr_size; ssize_t retval; size_t std_len; @@ -1372,16 +1369,22 @@ netdev_linux_batch_rxq_recv_tap(struct netdev_rxq_linux *rx, int mtu, virtio_net_hdr_size = 0; } - std_len = VLAN_ETH_HEADER_LEN + mtu + virtio_net_hdr_size; + /* The length here needs to be accounted in the same way when the + * aux_buf is allocated so that it can be prepended to TSO buffer. */ + std_len = virtio_net_hdr_size + VLAN_ETH_HEADER_LEN + mtu; for (i = 0; i < NETDEV_MAX_BURST; i++) { + struct dp_packet *buffer; + struct dp_packet *pkt; struct iovec iov[IOV_TSO_SIZE]; /* Assume Ethernet port. No need to set packet_type. */ buffer = dp_packet_new_with_headroom(std_len, DP_NETDEV_HEADROOM); iov[IOV_PACKET].iov_base = dp_packet_data(buffer); iov[IOV_PACKET].iov_len = std_len; - iov[IOV_AUXBUF].iov_base = rx->aux_bufs[i]; - iov[IOV_AUXBUF].iov_len = LINUX_RXQ_TSO_MAX_LEN; + if (iovlen == IOV_TSO_SIZE) { + iov[IOV_AUXBUF].iov_base = dp_packet_data(rx->aux_bufs[i]); + iov[IOV_AUXBUF].iov_len = dp_packet_size(rx->aux_bufs[i]); + } do { retval = readv(rx->fd, iov, iovlen); @@ -1393,33 +1396,36 @@ netdev_linux_batch_rxq_recv_tap(struct netdev_rxq_linux *rx, int mtu, } if (retval > std_len) { - /* Build a single linear TSO packet by expanding the current packet - * to append the data received in the aux_buf. */ - size_t extra_len = retval - std_len; - - dp_packet_set_size(buffer, dp_packet_size(buffer) + std_len); - dp_packet_prealloc_tailroom(buffer, extra_len); - memcpy(dp_packet_tail(buffer), rx->aux_bufs[i], extra_len); - dp_packet_set_size(buffer, dp_packet_size(buffer) + extra_len); + /* Build a single linear TSO packet by prepending the data from + * std_len buffer to the aux_buf. */ + pkt = rx->aux_bufs[i]; + dp_packet_set_size(pkt, retval - std_len); + dp_packet_push(pkt, dp_packet_data(buffer), std_len); + /* The headroom should be the same in buffers[i], pkt and + * DP_NETDEV_HEADROOM. */ + dp_packet_resize(pkt, DP_NETDEV_HEADROOM, 0); + dp_packet_delete(buffer); + rx->aux_bufs[i] = NULL; } else { dp_packet_set_size(buffer, dp_packet_size(buffer) + retval); + pkt = buffer; } - if (virtio_net_hdr_size && netdev_linux_parse_vnet_hdr(buffer)) { + if (virtio_net_hdr_size && netdev_linux_parse_vnet_hdr(pkt)) { struct netdev *netdev_ = netdev_rxq_get_netdev(&rx->up); struct netdev_linux *netdev = netdev_linux_cast(netdev_); /* Unexpected error situation: the virtio header is not present * or corrupted. Drop the packet but continue in case next ones * are correct. */ - dp_packet_delete(buffer); + dp_packet_delete(pkt); netdev->rx_dropped += 1; VLOG_WARN_RL(&rl, "%s: Dropped packet: Invalid virtio net header", netdev_get_name(netdev_)); continue; } - dp_packet_batch_add(batch, buffer); + dp_packet_batch_add(batch, pkt); } if ((i == 0) && (retval < 0)) { @@ -1442,6 +1448,23 @@ netdev_linux_rxq_recv(struct netdev_rxq *rxq_, struct dp_packet_batch *batch, mtu = ETH_PAYLOAD_MAX; } + if (userspace_tso_enabled()) { + /* Allocate TSO packets. The packet has enough headroom to store + * a full non-TSO packet. When a TSO packet is received, the data + * from non-TSO buffer (std_len) is prepended to the TSO packet + * (aux_buf). */ + size_t data_len = LINUX_RXQ_TSO_MAX_LEN - mtu; + size_t std_len = sizeof(struct virtio_net_hdr) + VLAN_ETH_HEADER_LEN + + DP_NETDEV_HEADROOM + mtu; + for (int i = 0; i < NETDEV_MAX_BURST; i++) { + if (rx->aux_bufs[i]) { + continue; + } + + rx->aux_bufs[i] = dp_packet_new_with_headroom(data_len, std_len); + } + } + dp_packet_batch_init(batch); retval = (rx->is_tap ? netdev_linux_batch_rxq_recv_tap(rx, mtu, batch)