From patchwork Tue Sep 26 02:01:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Darrell Ball X-Patchwork-Id: 818416 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VEi4yztK"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y1PMh5zHMz9s7c for ; Tue, 26 Sep 2017 12:01:52 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id CD372ABA; Tue, 26 Sep 2017 02:01:47 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 95D8DA59 for ; Tue, 26 Sep 2017 02:01:46 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pg0-f68.google.com (mail-pg0-f68.google.com [74.125.83.68]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 1765B3FD for ; Tue, 26 Sep 2017 02:01:46 +0000 (UTC) Received: by mail-pg0-f68.google.com with SMTP id d8so5869722pgt.3 for ; Mon, 25 Sep 2017 19:01:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=zA2dh8eefkgNdlkxpcldiBG0zyJfce8yazF61wuMjXI=; b=VEi4yztKYB/NokNKyhe42BxgTgPiOB6EeWX6Bk9Wv/kOLUYgU3ijNS9Lzc4+8J3Gks q+YgNW/lA/nDvnLGntgtLjzeZrR6ts3neltqAm/uZs+DRFCh1iiedac5DHWxWqPnXDZp 1A7Ssilldfbbi7sUAc5NXFl9Sd6PoiHUyxNL2TPIS1q8nBerhgyht5wH4fWIxMqh1CaR 2xrYG68wOXS2P6qsMzD6SGR1bSjKB/lNAu+DdvmFR3LoPfdZOD8b2Wm+Fu7cc4DsPCce DPhQbCqWrCF4EISN5bBE3/OWgiL8YtMPUR8wczHmNSVw/Suu5/tiVzVRVlrfboI47DA2 Qk0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=zA2dh8eefkgNdlkxpcldiBG0zyJfce8yazF61wuMjXI=; b=nGuG+dKY5xcDeLma9hZlS2nB2aQ9WN4V3XIHdKQHouzCgnjdyOK0JXxR4tTbYnYZzj 00/e3TQ3SPdLF8was9miVZWccSN0SHfTqiwZENQ+a9QrTvqfcWy4BJYmOZlh12jAVXBU vRZVIz6tGT74Uc3qDsON74nzZzpmy8Zs4kjG+qMGP93glMaH/BlZmejFm8TeMuzkAEiH STtzcB8X4DDYtVSTlWSBgehUbYp4BGB+pLDHvf1477LibvI/SI75Vws/Yl48xk/au7Lt OCcZ17feZ6geP8YDcTz4K008qHcSqXPdqtRBJSf2p8r6cYkfMD68Bgx3GxvItFa0H7yG y//g== X-Gm-Message-State: AHPjjUhGF6KLytsIG9t11c1KGMRMAGntPL55+EtNJ/yQDiq+14Fow4O9 bLTQLTF3EP4EGVnyWYHPB4Y= X-Google-Smtp-Source: AOwi7QB+l3wzxoP8nAiXnn7ZtyGE+j+qw80FSbmMuQw6ev697uexgB43daW2rtQYc0K0HgCnJC3N3Q== X-Received: by 10.98.198.78 with SMTP id m75mr9398191pfg.76.1506391305546; Mon, 25 Sep 2017 19:01:45 -0700 (PDT) Received: from ubuntu.localdomain ([208.91.2.2]) by smtp.gmail.com with ESMTPSA id k25sm11914920pgf.13.2017.09.25.19.01.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 25 Sep 2017 19:01:44 -0700 (PDT) From: Darrell Ball To: dlu998@gmail.com, dev@openvswitch.org Date: Mon, 25 Sep 2017 19:01:35 -0700 Message-Id: <1506391296-38587-1-git-send-email-dlu998@gmail.com> X-Mailer: git-send-email 1.9.1 X-Spam-Status: No, score=0.7 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, RCVD_IN_SORBS_SPAM autolearn=disabled version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [branch 2.8 1/2] dp-packet: Refactor DPDK packet initialization. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org DPDK uses dp-packet pools and manages the mbuf portion of each packet. When a pool is created, partial initialization is also done on the OVS portion (i.e. non-mbuf). Since packet memory is reused, this is not very useful for transient fields and is also misleading. Furthermore, some of these transient fields are properly initialized for DPDK packets entering OVS anyways, which is the only reasonable way to do this. Another field, cutlen, is initialized in this manner in the pool and intended to be reset when cutlen is applied on sending the packet out. However, if cutlen context is set but the packet is not sent out for some reason, then the packet header would be corrupted in the memory pool. It is better to just reset the cutlen in the packets when received. I did not detect a degradation in performance, however, I would be willing to have some degradation, since this is a proper way to handle this. In addition to initializing cutlen in received packets, the other OVS transient fields are removed from the DPDK pool initialization. Acked-by: Sugesh Chandran Signed-off-by: Darrell Ball --- lib/dp-packet.c | 16 +++++++++------- lib/dp-packet.h | 10 ++++++++++ lib/netdev-dpdk.c | 2 ++ 3 files changed, 21 insertions(+), 7 deletions(-) diff --git a/lib/dp-packet.c b/lib/dp-packet.c index c1f43f3..312f63a 100644 --- a/lib/dp-packet.c +++ b/lib/dp-packet.c @@ -92,16 +92,18 @@ dp_packet_use_const(struct dp_packet *b, const void *data, size_t size) dp_packet_set_size(b, size); } -/* Initializes 'b' as an empty dp_packet that contains the 'allocated' bytes of - * memory starting at 'base'. DPDK allocated dp_packet and *data is allocated - * from one continous memory region, so in memory data start right after - * dp_packet. Therefore there is special method to free this type of - * buffer. dp_packet base, data and size are initialized by dpdk rcv() so no - * need to initialize those fields. */ +/* Initializes 'b' as an empty dp_packet that contains the 'allocated' bytes. + * DPDK allocated dp_packet and *data is allocated from one continous memory + * region as part of memory pool, so in memory data start right after + * dp_packet. Therefore, there is a special method to free this type of + * buffer. Here, non-transient ovs dp-packet fields are initialized for + * packets that are part of a DPDK memory pool. */ void dp_packet_init_dpdk(struct dp_packet *b, size_t allocated) { - dp_packet_init__(b, allocated, DPBUF_DPDK); + dp_packet_set_allocated(b, allocated); + b->source = DPBUF_DPDK; + b->packet_type = htonl(PT_ETH); } /* Initializes 'b' as an empty dp_packet with an initial capacity of 'size' diff --git a/lib/dp-packet.h b/lib/dp-packet.h index 8b3f0ff..75ec305 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -801,6 +801,16 @@ dp_packet_delete_batch(struct dp_packet_batch *batch, bool may_steal) } static inline void +dp_packet_batch_init_cutlen(struct dp_packet_batch *batch) +{ + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (packet, batch) { + dp_packet_reset_cutlen(packet); + } +} + +static inline void dp_packet_batch_apply_cutlen(struct dp_packet_batch *batch) { if (batch->trunc) { diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index 62dbb7e..f58e9be 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -1644,6 +1644,7 @@ netdev_dpdk_vhost_rxq_recv(struct netdev_rxq *rxq, nb_rx, dropped); rte_spinlock_unlock(&dev->stats_lock); + dp_packet_batch_init_cutlen(batch); batch->count = (int) nb_rx; return 0; } @@ -1683,6 +1684,7 @@ netdev_dpdk_rxq_recv(struct netdev_rxq *rxq, struct dp_packet_batch *batch) rte_spinlock_unlock(&dev->stats_lock); } + dp_packet_batch_init_cutlen(batch); batch->count = nb_rx; return 0; From patchwork Tue Sep 26 02:01:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Darrell Ball X-Patchwork-Id: 818417 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="KxY8Il4v"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y1PNM2P4hz9t3t for ; Tue, 26 Sep 2017 12:02:27 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id BEACCB08; Tue, 26 Sep 2017 02:01:48 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 61BF8A59 for ; Tue, 26 Sep 2017 02:01:47 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pg0-f65.google.com (mail-pg0-f65.google.com [74.125.83.65]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id DE98A3FD for ; Tue, 26 Sep 2017 02:01:46 +0000 (UTC) Received: by mail-pg0-f65.google.com with SMTP id j16so5875738pga.2 for ; Mon, 25 Sep 2017 19:01:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=V9RPtqXZaUlqCA95WnWL8Q0Rkw2DlrQ58CPP6tfAx3E=; b=KxY8Il4vxyO9XzRJo9OWlMH02s+TKBnJHdNxYnvzC7zKIYYOEw5wQW6BW9MFH4nV6R HApu2dkTXQZQVn6KusBptdoBTNQeHIkv758tTbolJE+z2k/+mwzA/dhpnMNGvbj6ax9b zI+AIcM+iKQ0QSdbkqa0F+NaxBQ9gbBkn/v2n2SncBez63ih/5mceJhj7rrNSPShMKKx Y1hbuqeJyqpNKEXOWIuKqgIjIrKlvtnxKKGsRnTuLaprFXEjAhsyg6JOndSLteewfNkC oQfMj2ozZ+O7bIW2vWCbaVg5OVVfsNsln2C6ztWGKwChhtczwshGGGRJBu5LF0m2uzRi luOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=V9RPtqXZaUlqCA95WnWL8Q0Rkw2DlrQ58CPP6tfAx3E=; b=S6EgRr1APOscD/KcBJs5Z31DPrSZDWGzlgnWZ175wxHeuzmhZh5yL1OLy6cIlcU8z0 0GHD/8PZuCPQmM1je/Lm+2qXARHXT3Sv8zF96+zsX8+RwnbDjrtqUbA71nA2XI1s3BQJ D/Tl8LB8g4UNr62zMO623s9+l1lZuoiCGfG79bsKOrnPbUQIDzsQfs/Pr4fp/c9nB/Nz oaGeJ9X6CjzYizEmNu9U+trvGxjNtZmQv6UlJZahHFIiaaUMYY4WVbaWoPna8CVXTKfU RPyJIO8M5d0aFi1ESvXl0u0BhMrXSHVGU+PbrP8/c9TRnXmBJqTe2LXih1cVBaqo+Emn L+Lg== X-Gm-Message-State: AHPjjUhUcrZu3ZMNkJEPwmPK1iVhNotq2ApZnvWjiOlukhDgIGibz2Gj rULviZbDmUKyl1HJ5Bjt+sQ= X-Google-Smtp-Source: AOwi7QAjzaJSjgYQpQmJGiGKex0URdP6PiaxFZ+6oaDJb+uswtfov9Ek8tCM2hOYCT/8YbbFgCYJQw== X-Received: by 10.98.166.137 with SMTP id r9mr9567627pfl.86.1506391306513; Mon, 25 Sep 2017 19:01:46 -0700 (PDT) Received: from ubuntu.localdomain ([208.91.2.2]) by smtp.gmail.com with ESMTPSA id k25sm11914920pgf.13.2017.09.25.19.01.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 25 Sep 2017 19:01:46 -0700 (PDT) From: Darrell Ball To: dlu998@gmail.com, dev@openvswitch.org Date: Mon, 25 Sep 2017 19:01:36 -0700 Message-Id: <1506391296-38587-2-git-send-email-dlu998@gmail.com> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1506391296-38587-1-git-send-email-dlu998@gmail.com> References: <1506391296-38587-1-git-send-email-dlu998@gmail.com> X-Spam-Status: No, score=0.7 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU, FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, RCVD_IN_SORBS_SPAM autolearn=disabled version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [branch 2.8 2/2] netdev-dpdk: reset packet_type for reused dp_packets. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org From: Zoltan Balogh DPDK uses dp-packet pool for storing received packets. The pool is reused by rxq_recv funcions of the DPDK netdevs. The datapath is capable to modify the packet_type property of packets. For instance when encapsulated L3 packets are received on a ptap gre port. In this case the packet_type property of struct dp_packet can be modified and later the same dp_packet with the modified packet_type can be reused in the rxq_rec function, so it can contain corrupted data. The dp_packet_batch_init_cutlen() in the rxq_recv functions iterates over dp_packets and sets their cutlen. So I modified this function to set packet_type to Ethernet for the dp_packets as well. I also renamed this function because of the added functionality. The dp_packet_batch_init_cutlen() iterates over batch->count dp_packet. Therefore setting of batch->count = nb_rx needs to be done before the former function is invoked. This is an additional fix. Signed-off-by: Zoltan Balogh Signed-off-by: Laszlo Suru Co-authored-by: Laszlo Suru CC: Jan Scheurich CC: Sugesh Chandran CC: Darrell Ball Signed-off-by: Darrell Ball --- lib/dp-packet.c | 1 - lib/dp-packet.h | 3 ++- lib/netdev-dpdk.c | 7 ++++--- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/lib/dp-packet.c b/lib/dp-packet.c index 312f63a..945b81a 100644 --- a/lib/dp-packet.c +++ b/lib/dp-packet.c @@ -103,7 +103,6 @@ dp_packet_init_dpdk(struct dp_packet *b, size_t allocated) { dp_packet_set_allocated(b, allocated); b->source = DPBUF_DPDK; - b->packet_type = htonl(PT_ETH); } /* Initializes 'b' as an empty dp_packet with an initial capacity of 'size' diff --git a/lib/dp-packet.h b/lib/dp-packet.h index 75ec305..980ef78 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -801,12 +801,13 @@ dp_packet_delete_batch(struct dp_packet_batch *batch, bool may_steal) } static inline void -dp_packet_batch_init_cutlen(struct dp_packet_batch *batch) +dp_packet_batch_init_packet_fields(struct dp_packet_batch *batch) { struct dp_packet *packet; DP_PACKET_BATCH_FOR_EACH (packet, batch) { dp_packet_reset_cutlen(packet); + packet->packet_type = htonl(PT_ETH); } } diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index f58e9be..ccccb9a 100644 --- a/lib/netdev-dpdk.c +++ b/lib/netdev-dpdk.c @@ -1644,8 +1644,9 @@ netdev_dpdk_vhost_rxq_recv(struct netdev_rxq *rxq, nb_rx, dropped); rte_spinlock_unlock(&dev->stats_lock); - dp_packet_batch_init_cutlen(batch); - batch->count = (int) nb_rx; + batch->count = nb_rx; + dp_packet_batch_init_packet_fields(batch); + return 0; } @@ -1684,8 +1685,8 @@ netdev_dpdk_rxq_recv(struct netdev_rxq *rxq, struct dp_packet_batch *batch) rte_spinlock_unlock(&dev->stats_lock); } - dp_packet_batch_init_cutlen(batch); batch->count = nb_rx; + dp_packet_batch_init_packet_fields(batch); return 0; }