From patchwork Thu Jan 17 11:20:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 1026545 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=lab.ntt.co.jp Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gMB80z3pz9sCX for ; Thu, 17 Jan 2019 22:22:40 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726824AbfAQLWi (ORCPT ); Thu, 17 Jan 2019 06:22:38 -0500 Received: from tama500.ecl.ntt.co.jp ([129.60.39.148]:37802 "EHLO tama500.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725889AbfAQLWi (ORCPT ); Thu, 17 Jan 2019 06:22:38 -0500 Received: from vc2.ecl.ntt.co.jp (vc2.ecl.ntt.co.jp [129.60.86.154]) by tama500.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x0HBMYBK015961; Thu, 17 Jan 2019 20:22:34 +0900 Received: from vc2.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id E538F639311; Thu, 17 Jan 2019 20:22:33 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id D9B366392A4; Thu, 17 Jan 2019 20:22:33 +0900 (JST) Received: from makita-ubuntu.m.ecl.ntt.co.jp (unknown [129.60.241.182]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id D0FFB400B79; Thu, 17 Jan 2019 20:22:33 +0900 (JST) From: Toshiaki Makita Subject: [PATCH net 1/7] virtio_net: Don't enable NAPI when interface is down Date: Thu, 17 Jan 2019 20:20:39 +0900 Message-Id: <1547724045-2726-2-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> References: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-CC-Mail-RelayStamp: 1 To: "David S. Miller" , "Michael S. Tsirkin" , Jason Wang Cc: Toshiaki Makita , netdev@vger.kernel.org, virtualization@lists.linux-foundation.org X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Commit 4e09ff536284 ("virtio-net: disable NAPI only when enabled during XDP set") tried to fix inappropriate NAPI enabling/disabling when !netif_running(), but was not complete. On error path virtio_net could enable NAPI even when !netif_running(). This can cause enabling NAPI twice on virtnet_open(), which would trigger BUG_ON() in napi_enable(). Fixes: 4941d472bf95b ("virtio-net: do not reset during XDP set") Signed-off-by: Toshiaki Makita Acked-by: Jason Wang --- drivers/net/virtio_net.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 0237250..a08da9e 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -2430,8 +2430,10 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, return 0; err: - for (i = 0; i < vi->max_queue_pairs; i++) - virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi); + if (netif_running(dev)) { + for (i = 0; i < vi->max_queue_pairs; i++) + virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi); + } if (prog) bpf_prog_sub(prog, vi->max_queue_pairs - 1); return err; From patchwork Thu Jan 17 11:20:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 1026546 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=lab.ntt.co.jp Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gMBl4SXlz9sCX for ; Thu, 17 Jan 2019 22:23:11 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727393AbfAQLXJ (ORCPT ); Thu, 17 Jan 2019 06:23:09 -0500 Received: from tama500.ecl.ntt.co.jp ([129.60.39.148]:37808 "EHLO tama500.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725889AbfAQLXI (ORCPT ); Thu, 17 Jan 2019 06:23:08 -0500 Received: from vc1.ecl.ntt.co.jp (vc1.ecl.ntt.co.jp [129.60.86.153]) by tama500.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x0HBMxrT015964; Thu, 17 Jan 2019 20:22:59 +0900 Received: from vc1.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id 8E6A8EA816E; Thu, 17 Jan 2019 20:22:59 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id 7E90EEA8154; Thu, 17 Jan 2019 20:22:59 +0900 (JST) Received: from makita-ubuntu.m.ecl.ntt.co.jp (unknown [129.60.241.182]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id 7199D400B79; Thu, 17 Jan 2019 20:22:59 +0900 (JST) From: Toshiaki Makita Subject: [PATCH net 2/7] virtio_net: Don't call free_old_xmit_skbs for xdp_frames Date: Thu, 17 Jan 2019 20:20:40 +0900 Message-Id: <1547724045-2726-3-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> References: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-CC-Mail-RelayStamp: 1 To: "David S. Miller" , "Michael S. Tsirkin" , Jason Wang Cc: Toshiaki Makita , netdev@vger.kernel.org, virtualization@lists.linux-foundation.org, Willem de Bruijn X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When napi_tx is enabled, virtnet_poll_cleantx() called free_old_xmit_skbs() even for xdp send queue. This is bogus since the queue has xdp_frames, not sk_buffs, thus mangled device tx bytes counters because skb->len is meaningless value, and even triggered oops due to general protection fault on freeing them. Since xdp send queues do not aquire locks, old xdp_frames should be freed only in virtnet_xdp_xmit(), so just skip free_old_xmit_skbs() for xdp send queues. Similarly virtnet_poll_tx() called free_old_xmit_skbs(). This NAPI handler is called even without calling start_xmit() because cb for tx is by default enabled. Once the handler is called, it enabled the cb again, and then the handler would be called again. We don't need this handler for XDP, so don't enable cb as well as not calling free_old_xmit_skbs(). Also, we need to disable tx NAPI when disabling XDP, so virtnet_poll_tx() can safely access curr_queue_pairs and xdp_queue_pairs, which are not atomically updated while disabling XDP. Fixes: b92f1e6751a6 ("virtio-net: transmit napi") Fixes: 7b0411ef4aa6 ("virtio-net: clean tx descriptors from rx napi") Signed-off-by: Toshiaki Makita Acked-by: Jason Wang --- drivers/net/virtio_net.c | 49 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 33 insertions(+), 16 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index a08da9e..7d35e6d 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -1358,6 +1358,16 @@ static void free_old_xmit_skbs(struct send_queue *sq) u64_stats_update_end(&sq->stats.syncp); } +static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q) +{ + if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs)) + return false; + else if (q < vi->curr_queue_pairs) + return true; + else + return false; +} + static void virtnet_poll_cleantx(struct receive_queue *rq) { struct virtnet_info *vi = rq->vq->vdev->priv; @@ -1365,7 +1375,7 @@ static void virtnet_poll_cleantx(struct receive_queue *rq) struct send_queue *sq = &vi->sq[index]; struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index); - if (!sq->napi.weight) + if (!sq->napi.weight || is_xdp_raw_buffer_queue(vi, index)) return; if (__netif_tx_trylock(txq)) { @@ -1442,8 +1452,16 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) { struct send_queue *sq = container_of(napi, struct send_queue, napi); struct virtnet_info *vi = sq->vq->vdev->priv; - struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, vq2txq(sq->vq)); + unsigned int index = vq2txq(sq->vq); + struct netdev_queue *txq; + if (unlikely(is_xdp_raw_buffer_queue(vi, index))) { + /* We don't need to enable cb for XDP */ + napi_complete_done(napi, 0); + return 0; + } + + txq = netdev_get_tx_queue(vi->dev, index); __netif_tx_lock(txq, raw_smp_processor_id()); free_old_xmit_skbs(sq); __netif_tx_unlock(txq); @@ -2402,9 +2420,12 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, } /* Make sure NAPI is not using any XDP TX queues for RX. */ - if (netif_running(dev)) - for (i = 0; i < vi->max_queue_pairs; i++) + if (netif_running(dev)) { + for (i = 0; i < vi->max_queue_pairs; i++) { napi_disable(&vi->rq[i].napi); + virtnet_napi_tx_disable(&vi->sq[i].napi); + } + } netif_set_real_num_rx_queues(dev, curr_qp + xdp_qp); err = _virtnet_set_queues(vi, curr_qp + xdp_qp); @@ -2423,16 +2444,22 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, } if (old_prog) bpf_prog_put(old_prog); - if (netif_running(dev)) + if (netif_running(dev)) { virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi); + virtnet_napi_tx_enable(vi, vi->sq[i].vq, + &vi->sq[i].napi); + } } return 0; err: if (netif_running(dev)) { - for (i = 0; i < vi->max_queue_pairs; i++) + for (i = 0; i < vi->max_queue_pairs; i++) { virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi); + virtnet_napi_tx_enable(vi, vi->sq[i].vq, + &vi->sq[i].napi); + } } if (prog) bpf_prog_sub(prog, vi->max_queue_pairs - 1); @@ -2615,16 +2642,6 @@ static void free_receive_page_frags(struct virtnet_info *vi) put_page(vi->rq[i].alloc_frag.page); } -static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q) -{ - if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs)) - return false; - else if (q < vi->curr_queue_pairs) - return true; - else - return false; -} - static void free_unused_bufs(struct virtnet_info *vi) { void *buf; From patchwork Thu Jan 17 11:20:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 1026547 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=lab.ntt.co.jp Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gMC9014jz9sCX for ; Thu, 17 Jan 2019 22:23:32 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727775AbfAQLXb (ORCPT ); Thu, 17 Jan 2019 06:23:31 -0500 Received: from tama50.ecl.ntt.co.jp ([129.60.39.147]:47278 "EHLO tama50.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725889AbfAQLXa (ORCPT ); Thu, 17 Jan 2019 06:23:30 -0500 Received: from vc2.ecl.ntt.co.jp (vc2.ecl.ntt.co.jp [129.60.86.154]) by tama50.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x0HBNOOs021754; Thu, 17 Jan 2019 20:23:24 +0900 Received: from vc2.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id AC0A9639311; Thu, 17 Jan 2019 20:23:24 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id A0F3D6392A4; Thu, 17 Jan 2019 20:23:24 +0900 (JST) Received: from makita-ubuntu.m.ecl.ntt.co.jp (unknown [129.60.241.182]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id 99E0B400B79; Thu, 17 Jan 2019 20:23:24 +0900 (JST) From: Toshiaki Makita Subject: [PATCH net 3/7] virtio_net: Fix not restoring real_num_rx_queues Date: Thu, 17 Jan 2019 20:20:41 +0900 Message-Id: <1547724045-2726-4-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> References: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-CC-Mail-RelayStamp: 1 To: "David S. Miller" , "Michael S. Tsirkin" , Jason Wang Cc: Toshiaki Makita , netdev@vger.kernel.org, virtualization@lists.linux-foundation.org X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When _virtnet_set_queues() failed we did not restore real_num_rx_queues. Fix this by placing the change of real_num_rx_queues after _virtnet_set_queues(). This order is also in line with virtnet_set_channels(). Fixes: 4941d472bf95 ("virtio-net: do not reset during XDP set") Signed-off-by: Toshiaki Makita Acked-by: Jason Wang --- drivers/net/virtio_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 7d35e6d..670cc15 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -2427,10 +2427,10 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, } } - netif_set_real_num_rx_queues(dev, curr_qp + xdp_qp); err = _virtnet_set_queues(vi, curr_qp + xdp_qp); if (err) goto err; + netif_set_real_num_rx_queues(dev, curr_qp + xdp_qp); vi->xdp_queue_pairs = xdp_qp; for (i = 0; i < vi->max_queue_pairs; i++) { From patchwork Thu Jan 17 11:20:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 1026548 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=lab.ntt.co.jp Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gMCX74L6z9sCX for ; Thu, 17 Jan 2019 22:23:52 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727520AbfAQLXv (ORCPT ); Thu, 17 Jan 2019 06:23:51 -0500 Received: from tama500.ecl.ntt.co.jp ([129.60.39.148]:37819 "EHLO tama500.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726841AbfAQLXv (ORCPT ); Thu, 17 Jan 2019 06:23:51 -0500 Received: from vc1.ecl.ntt.co.jp (vc1.ecl.ntt.co.jp [129.60.86.153]) by tama500.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x0HBNkh5015982; Thu, 17 Jan 2019 20:23:46 +0900 Received: from vc1.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id C59F2EA816E; Thu, 17 Jan 2019 20:23:46 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id B71F1EA8154; Thu, 17 Jan 2019 20:23:46 +0900 (JST) Received: from makita-ubuntu.m.ecl.ntt.co.jp (unknown [129.60.241.182]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id A7E48400B79; Thu, 17 Jan 2019 20:23:46 +0900 (JST) From: Toshiaki Makita Subject: [PATCH net 4/7] virtio_net: Fix out of bounds access of sq Date: Thu, 17 Jan 2019 20:20:42 +0900 Message-Id: <1547724045-2726-5-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> References: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-CC-Mail-RelayStamp: 1 To: "David S. Miller" , "Michael S. Tsirkin" , Jason Wang Cc: Toshiaki Makita , netdev@vger.kernel.org, virtualization@lists.linux-foundation.org X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When XDP is disabled, curr_queue_pairs + smp_processor_id() can be larger than max_queue_pairs. There is no guarantee that we have enough XDP send queues dedicated for each cpu when XDP is disabled, so do not count drops on sq in that case. Fixes: 5b8f3c8d30a6 ("virtio_net: Add XDP related stats") Signed-off-by: Toshiaki Makita Acked-by: Jason Wang --- drivers/net/virtio_net.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 670cc15..204eedf 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -491,20 +491,17 @@ static int virtnet_xdp_xmit(struct net_device *dev, int ret, err; int i; - sq = virtnet_xdp_sq(vi); - - if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK)) { - ret = -EINVAL; - drops = n; - goto out; - } - /* Only allow ndo_xdp_xmit if XDP is loaded on dev, as this * indicate XDP resources have been successfully allocated. */ xdp_prog = rcu_dereference(rq->xdp_prog); - if (!xdp_prog) { - ret = -ENXIO; + if (!xdp_prog) + return -ENXIO; + + sq = virtnet_xdp_sq(vi); + + if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK)) { + ret = -EINVAL; drops = n; goto out; } From patchwork Thu Jan 17 11:20:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 1026549 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=lab.ntt.co.jp Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gMD70L9Bz9sCX for ; Thu, 17 Jan 2019 22:24:23 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728291AbfAQLYV (ORCPT ); Thu, 17 Jan 2019 06:24:21 -0500 Received: from tama50.ecl.ntt.co.jp ([129.60.39.147]:47286 "EHLO tama50.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725889AbfAQLYU (ORCPT ); Thu, 17 Jan 2019 06:24:20 -0500 Received: from vc1.ecl.ntt.co.jp (vc1.ecl.ntt.co.jp [129.60.86.153]) by tama50.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x0HBOFlT021763; Thu, 17 Jan 2019 20:24:15 +0900 Received: from vc1.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id 62B70EA816E; Thu, 17 Jan 2019 20:24:15 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id 4E510EA816C; Thu, 17 Jan 2019 20:24:15 +0900 (JST) Received: from makita-ubuntu.m.ecl.ntt.co.jp (unknown [129.60.241.182]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id 45F4C400B79; Thu, 17 Jan 2019 20:24:15 +0900 (JST) From: Toshiaki Makita Subject: [PATCH net 5/7] virtio_net: Don't process redirected XDP frames when XDP is disabled Date: Thu, 17 Jan 2019 20:20:43 +0900 Message-Id: <1547724045-2726-6-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> References: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-CC-Mail-RelayStamp: 1 To: "David S. Miller" , "Michael S. Tsirkin" , Jason Wang Cc: Toshiaki Makita , netdev@vger.kernel.org, virtualization@lists.linux-foundation.org, Jesper Dangaard Brouer X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Commit 8dcc5b0ab0ec ("virtio_net: fix ndo_xdp_xmit crash towards dev not ready for XDP") tried to avoid access to unexpected sq while XDP is disabled, but was not complete. There was a small window which causes out of bounds sq access in virtnet_xdp_xmit() while disabling XDP. An example case of - curr_queue_pairs = 6 (2 for SKB and 4 for XDP) - online_cpu_num = xdp_queue_paris = 4 when XDP is enabled: CPU 0 CPU 1 (Disabling XDP) (Processing redirected XDP frames) virtnet_xdp_xmit() virtnet_xdp_set() _virtnet_set_queues() set curr_queue_pairs (2) check if rq->xdp_prog is not NULL virtnet_xdp_sq(vi) qp = curr_queue_pairs - xdp_queue_pairs + smp_processor_id() = 2 - 4 + 1 = -1 sq = &vi->sq[qp] // out of bounds access set xdp_queue_pairs (0) rq->xdp_prog = NULL Basically we should not change curr_queue_pairs and xdp_queue_pairs while someone can read the values. Thus, when disabling XDP, assign NULL to rq->xdp_prog first, and wait for RCU grace period, then change xxx_queue_pairs. Note that we need to keep the current order when enabling XDP though. Fixes: 186b3c998c50 ("virtio-net: support XDP_REDIRECT") Signed-off-by: Toshiaki Makita --- drivers/net/virtio_net.c | 32 +++++++++++++++++++++++++------- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 204eedf..ae93f0e 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -2424,14 +2424,16 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, } } - err = _virtnet_set_queues(vi, curr_qp + xdp_qp); - if (err) - goto err; - netif_set_real_num_rx_queues(dev, curr_qp + xdp_qp); - vi->xdp_queue_pairs = xdp_qp; + old_prog = rtnl_dereference(vi->rq[0].xdp_prog); + if (!old_prog && prog) { + err = _virtnet_set_queues(vi, curr_qp + xdp_qp); + if (err) + goto err_new_prog; + netif_set_real_num_rx_queues(dev, curr_qp + xdp_qp); + vi->xdp_queue_pairs = xdp_qp; + } for (i = 0; i < vi->max_queue_pairs; i++) { - old_prog = rtnl_dereference(vi->rq[i].xdp_prog); rcu_assign_pointer(vi->rq[i].xdp_prog, prog); if (i == 0) { if (!old_prog) @@ -2439,6 +2441,18 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, if (!prog) virtnet_restore_guest_offloads(vi); } + } + + if (old_prog && !prog) { + synchronize_net(); + err = _virtnet_set_queues(vi, curr_qp + xdp_qp); + if (err) + goto err_old_prog; + netif_set_real_num_rx_queues(dev, curr_qp + xdp_qp); + vi->xdp_queue_pairs = xdp_qp; + } + + for (i = 0; i < vi->max_queue_pairs; i++) { if (old_prog) bpf_prog_put(old_prog); if (netif_running(dev)) { @@ -2450,7 +2464,11 @@ static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog, return 0; -err: +err_old_prog: + virtnet_clear_guest_offloads(vi); + for (i = 0; i < vi->max_queue_pairs; i++) + rcu_assign_pointer(vi->rq[i].xdp_prog, old_prog); +err_new_prog: if (netif_running(dev)) { for (i = 0; i < vi->max_queue_pairs; i++) { virtnet_napi_enable(vi->rq[i].vq, &vi->rq[i].napi); From patchwork Thu Jan 17 11:20:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 1026550 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=lab.ntt.co.jp Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gMDd1qyRz9sCX for ; Thu, 17 Jan 2019 22:24:49 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728351AbfAQLYr (ORCPT ); Thu, 17 Jan 2019 06:24:47 -0500 Received: from tama500.ecl.ntt.co.jp ([129.60.39.148]:37829 "EHLO tama500.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725889AbfAQLYr (ORCPT ); Thu, 17 Jan 2019 06:24:47 -0500 Received: from vc1.ecl.ntt.co.jp (vc1.ecl.ntt.co.jp [129.60.86.153]) by tama500.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x0HBOgde016000; Thu, 17 Jan 2019 20:24:42 +0900 Received: from vc1.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id E2D79EA816E; Thu, 17 Jan 2019 20:24:42 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc1.ecl.ntt.co.jp (Postfix) with ESMTP id D86CEEA816C; Thu, 17 Jan 2019 20:24:42 +0900 (JST) Received: from makita-ubuntu.m.ecl.ntt.co.jp (unknown [129.60.241.182]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id D0907400B79; Thu, 17 Jan 2019 20:24:42 +0900 (JST) From: Toshiaki Makita Subject: [PATCH net 6/7] virtio_net: Use xdp_return_frame to free xdp_frames on destroying vqs Date: Thu, 17 Jan 2019 20:20:44 +0900 Message-Id: <1547724045-2726-7-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> References: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-CC-Mail-RelayStamp: 1 To: "David S. Miller" , "Michael S. Tsirkin" , Jason Wang Cc: Toshiaki Makita , netdev@vger.kernel.org, virtualization@lists.linux-foundation.org, Jesper Dangaard Brouer X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org put_page() can work as a fallback for freeing xdp_frames, but the appropriate way is to use xdp_return_frame(). Fixes: cac320c850ef ("virtio_net: convert to use generic xdp_frame and xdp_return_frame API") Signed-off-by: Toshiaki Makita Acked-by: Jason Wang Acked-by: Jesper Dangaard Brouer --- drivers/net/virtio_net.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index ae93f0e..996de69 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -2668,7 +2668,7 @@ static void free_unused_bufs(struct virtnet_info *vi) if (!is_xdp_raw_buffer_queue(vi, i)) dev_kfree_skb(buf); else - put_page(virt_to_head_page(buf)); + xdp_return_frame(buf); } } From patchwork Thu Jan 17 11:20:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Toshiaki Makita X-Patchwork-Id: 1026551 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=lab.ntt.co.jp Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gMFG3Yd7z9sCX for ; Thu, 17 Jan 2019 22:25:22 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728451AbfAQLZV (ORCPT ); Thu, 17 Jan 2019 06:25:21 -0500 Received: from tama50.ecl.ntt.co.jp ([129.60.39.147]:47291 "EHLO tama50.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725889AbfAQLZU (ORCPT ); Thu, 17 Jan 2019 06:25:20 -0500 Received: from vc2.ecl.ntt.co.jp (vc2.ecl.ntt.co.jp [129.60.86.154]) by tama50.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x0HBPEOi021775; Thu, 17 Jan 2019 20:25:14 +0900 Received: from vc2.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id 9B15D639311; Thu, 17 Jan 2019 20:25:14 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id 8FB8B639309; Thu, 17 Jan 2019 20:25:14 +0900 (JST) Received: from makita-ubuntu.m.ecl.ntt.co.jp (unknown [129.60.241.182]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id 88985400B79; Thu, 17 Jan 2019 20:25:14 +0900 (JST) From: Toshiaki Makita Subject: [PATCH net 7/7] virtio_net: Differentiate sk_buff and xdp_frame on freeing Date: Thu, 17 Jan 2019 20:20:45 +0900 Message-Id: <1547724045-2726-8-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> References: <1547724045-2726-1-git-send-email-makita.toshiaki@lab.ntt.co.jp> X-CC-Mail-RelayStamp: 1 To: "David S. Miller" , "Michael S. Tsirkin" , Jason Wang Cc: Toshiaki Makita , netdev@vger.kernel.org, virtualization@lists.linux-foundation.org X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org We do not reset or free up unused buffers when enabling/disabling XDP, so it can happen that xdp_frames are freed after disabling XDP or sk_buffs are freed after enabling XDP on xdp tx queues. Thus we need to handle both forms (xdp_frames and sk_buffs) regardless of XDP setting. One way to trigger this problem is to disable XDP when napi_tx is enabled. In that case, virtnet_xdp_set() calls virtnet_napi_enable() which kicks NAPI. The NAPI handler will call virtnet_poll_cleantx() which invokes free_old_xmit_skbs() for queues which have been used by XDP. Note that even with this change we need to keep skipping free_old_xmit_skbs() from NAPI handlers when XDP is enabled, because XDP tx queues do not aquire queue locks. Fixes: 4941d472bf95 ("virtio-net: do not reset during XDP set") Signed-off-by: Toshiaki Makita Acked-by: Jason Wang --- drivers/net/virtio_net.c | 54 +++++++++++++++++++++++++++++++++++++----------- 1 file changed, 42 insertions(+), 12 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 996de69..6598c25 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -57,6 +57,8 @@ #define VIRTIO_XDP_TX BIT(0) #define VIRTIO_XDP_REDIR BIT(1) +#define VIRTIO_XDP_FLAG BIT(0) + /* RX packet size EWMA. The average packet size is used to determine the packet * buffer size when refilling RX rings. As the entire RX ring may be refilled * at once, the weight is chosen so that the EWMA will be insensitive to short- @@ -252,6 +254,21 @@ struct padded_vnet_hdr { char padding[4]; }; +static bool is_xdp_frame(void *ptr) +{ + return (unsigned long)ptr & VIRTIO_XDP_FLAG; +} + +static void *xdp_to_ptr(struct xdp_frame *ptr) +{ + return (void *)((unsigned long)ptr | VIRTIO_XDP_FLAG); +} + +static struct xdp_frame *ptr_to_xdp(void *ptr) +{ + return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); +} + /* Converting between virtqueue no. and kernel tx/rx queue no. * 0:rx0 1:tx0 2:rx1 3:tx1 ... 2N:rxN 2N+1:txN 2N+2:cvq */ @@ -462,7 +479,8 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, sg_init_one(sq->sg, xdpf->data, xdpf->len); - err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdpf, GFP_ATOMIC); + err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdp_to_ptr(xdpf), + GFP_ATOMIC); if (unlikely(err)) return -ENOSPC; /* Caller handle free/refcnt */ @@ -482,13 +500,13 @@ static int virtnet_xdp_xmit(struct net_device *dev, { struct virtnet_info *vi = netdev_priv(dev); struct receive_queue *rq = vi->rq; - struct xdp_frame *xdpf_sent; struct bpf_prog *xdp_prog; struct send_queue *sq; unsigned int len; int drops = 0; int kicks = 0; int ret, err; + void *ptr; int i; /* Only allow ndo_xdp_xmit if XDP is loaded on dev, as this @@ -507,8 +525,12 @@ static int virtnet_xdp_xmit(struct net_device *dev, } /* Free up any pending old buffers before queueing new ones. */ - while ((xdpf_sent = virtqueue_get_buf(sq->vq, &len)) != NULL) - xdp_return_frame(xdpf_sent); + while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { + if (likely(is_xdp_frame(ptr))) + xdp_return_frame(ptr_to_xdp(ptr)); + else + dev_consume_skb_any(ptr); + } for (i = 0; i < n; i++) { struct xdp_frame *xdpf = frames[i]; @@ -1329,18 +1351,26 @@ static int virtnet_receive(struct receive_queue *rq, int budget, static void free_old_xmit_skbs(struct send_queue *sq) { - struct sk_buff *skb; unsigned int len; unsigned int packets = 0; unsigned int bytes = 0; + void *ptr; - while ((skb = virtqueue_get_buf(sq->vq, &len)) != NULL) { - pr_debug("Sent skb %p\n", skb); + while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { + if (likely(!is_xdp_frame(ptr))) { + struct sk_buff *skb = ptr; - bytes += skb->len; - packets++; + pr_debug("Sent skb %p\n", skb); - dev_consume_skb_any(skb); + bytes += skb->len; + dev_consume_skb_any(skb); + } else { + struct xdp_frame *frame = ptr_to_xdp(ptr); + + bytes += frame->len; + xdp_return_frame(frame); + } + packets++; } /* Avoid overhead when no packets have been processed @@ -2665,10 +2695,10 @@ static void free_unused_bufs(struct virtnet_info *vi) for (i = 0; i < vi->max_queue_pairs; i++) { struct virtqueue *vq = vi->sq[i].vq; while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) { - if (!is_xdp_raw_buffer_queue(vi, i)) + if (!is_xdp_frame(buf)) dev_kfree_skb(buf); else - xdp_return_frame(buf); + xdp_return_frame(ptr_to_xdp(buf)); } }