From patchwork Mon Jul 17 16:26:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Fastabend X-Patchwork-Id: 789598 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3xB7wm6HKLz9s65 for ; Tue, 18 Jul 2017 02:27:04 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="KTZGKEMY"; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751374AbdGQQ1C (ORCPT ); Mon, 17 Jul 2017 12:27:02 -0400 Received: from mail-pg0-f68.google.com ([74.125.83.68]:36737 "EHLO mail-pg0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751284AbdGQQ1B (ORCPT ); Mon, 17 Jul 2017 12:27:01 -0400 Received: by mail-pg0-f68.google.com with SMTP id y129so20044441pgy.3 for ; Mon, 17 Jul 2017 09:27:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=jJfinsLiaWDDPXfWwiz4RmYvRj0m9e0/+LCTUz3IczI=; b=KTZGKEMYlIdY3Iva3kAWcOB9FF0hoq3NBMpqt3XNnTMT1uDXuf3pMYUBw+LG2rWfW1 KNN+a9EEbR77NlvnI+8pGFR90Kcu2D4ZbRavmWz50A2Q7AOlL0GXOGt86pYOqgJWKOZV UYeRE/hFBzMjggGU1BEQvpAl+izqSzFEoUwgXcWWfOy24Sp2a2aADWcrjxys2Z30JPAf T9T/DrjSEehCRRRDVo34Qq4VAxaEDprXJTbKcEvhKsi8eOcPU65KxzDmIv97XMZeRE+a WhPhV4QEri+cwUd1MgtytIZq1eLw2FIIHdKK7sH4ESplSL5otA1SupwhZkvMrdD75oe0 Czqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=jJfinsLiaWDDPXfWwiz4RmYvRj0m9e0/+LCTUz3IczI=; b=kKjK3+6PvRKxEZ+zmE5Ltte79/Gwlte7HNoW7SbxsQw/YxPjw/XI0QAZ4z23H6iOFK 1R4DdkF/xbFW+wNHmC/9hGzJo9gPpQKaRuCmwiJ+Gwpjhs39DbQEoPLBx6uXEF9/Daem WZEZPH9g8mkIjxdXWrk1HSj5eKIDxvPmg3nzF52q2dAPGpRQ13ZLy0UGUuNxyYjP2U0U o7RTnij6IDldaHVIbssn7DwK/z767//GKNDM09jHZmfFBtgqw0tTWWNcfFvz3RMo1JGJ 9+IzoafeFdUgVyQ1KXqARmi8G/cF2Q+Ej349wJ/lWTGWMI76TLrzU7gwY6+1JNZ/d7nH dXSA== X-Gm-Message-State: AIVw111DBvevW7vbYSfcGDaewQuXWXFAjNPvasrprCQkVefJVCVUiWyf QSz2/55ifg1P+g== X-Received: by 10.98.73.65 with SMTP id w62mr19933989pfa.61.1500308820555; Mon, 17 Jul 2017 09:27:00 -0700 (PDT) Received: from [127.0.1.1] ([72.168.145.49]) by smtp.gmail.com with ESMTPSA id a2sm34054383pfe.101.2017.07.17.09.26.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Jul 2017 09:27:00 -0700 (PDT) Subject: [net-next PATCH 02/12] net: xdp: support xdp generic on virtual devices From: John Fastabend To: davem@davemloft.net Cc: daniel@iogearbox.net, ast@fb.com, netdev@vger.kernel.org, john.fastabend@gmail.com, brouer@redhat.com, andy@greyhouse.net Date: Mon, 17 Jul 2017 09:26:45 -0700 Message-ID: <20170717162645.24315.62704.stgit@john-Precision-Tower-5810> In-Reply-To: <20170717160759.24315.7464.stgit@john-Precision-Tower-5810> References: <20170717160759.24315.7464.stgit@john-Precision-Tower-5810> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org XDP generic allows users to test XDP programs and/or run them with degraded performance on devices that do not yet support XDP. For testing I typically test eBPF programs using a set of veth devices. This allows testing topologies that would otherwise be difficult to setup especially in the early stages of development. This patch adds a xdp generic hook to the netif_rx_internal() function which is called from dev_forward_skb(). With this addition attaching XDP programs to veth devices works as expected! Also I noticed multiple drivers using netif_rx(). These devices will also benefit and generic XDP will work for them as well. Signed-off-by: John Fastabend Tested-by: Andy Gospodarek Acked-by: Daniel Borkmann Acked-by: Jesper Dangaard Brouer --- net/core/dev.c | 208 ++++++++++++++++++++++++++++++-------------------------- 1 file changed, 113 insertions(+), 95 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index 0244051..a1ed7b4 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3865,6 +3865,107 @@ static int enqueue_to_backlog(struct sk_buff *skb, int cpu, return NET_RX_DROP; } +static u32 netif_receive_generic_xdp(struct sk_buff *skb, + struct bpf_prog *xdp_prog) +{ + struct xdp_buff xdp; + u32 act = XDP_DROP; + void *orig_data; + int hlen, off; + u32 mac_len; + + /* Reinjected packets coming from act_mirred or similar should + * not get XDP generic processing. + */ + if (skb_cloned(skb)) + return XDP_PASS; + + if (skb_linearize(skb)) + goto do_drop; + + /* The XDP program wants to see the packet starting at the MAC + * header. + */ + mac_len = skb->data - skb_mac_header(skb); + hlen = skb_headlen(skb) + mac_len; + xdp.data = skb->data - mac_len; + xdp.data_end = xdp.data + hlen; + xdp.data_hard_start = skb->data - skb_headroom(skb); + orig_data = xdp.data; + + act = bpf_prog_run_xdp(xdp_prog, &xdp); + + off = xdp.data - orig_data; + if (off > 0) + __skb_pull(skb, off); + else if (off < 0) + __skb_push(skb, -off); + + switch (act) { + case XDP_TX: + __skb_push(skb, mac_len); + /* fall through */ + case XDP_PASS: + break; + + default: + bpf_warn_invalid_xdp_action(act); + /* fall through */ + case XDP_ABORTED: + trace_xdp_exception(skb->dev, xdp_prog, act); + /* fall through */ + case XDP_DROP: + do_drop: + kfree_skb(skb); + break; + } + + return act; +} + +/* When doing generic XDP we have to bypass the qdisc layer and the + * network taps in order to match in-driver-XDP behavior. + */ +static void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog) +{ + struct net_device *dev = skb->dev; + struct netdev_queue *txq; + bool free_skb = true; + int cpu, rc; + + txq = netdev_pick_tx(dev, skb, NULL); + cpu = smp_processor_id(); + HARD_TX_LOCK(dev, txq, cpu); + if (!netif_xmit_stopped(txq)) { + rc = netdev_start_xmit(skb, dev, txq, 0); + if (dev_xmit_complete(rc)) + free_skb = false; + } + HARD_TX_UNLOCK(dev, txq); + if (free_skb) { + trace_xdp_exception(dev, xdp_prog, XDP_TX); + kfree_skb(skb); + } +} + +static struct static_key generic_xdp_needed __read_mostly; + +static int do_xdp_generic(struct sk_buff *skb) +{ + struct bpf_prog *xdp_prog = rcu_dereference(skb->dev->xdp_prog); + + if (xdp_prog) { + u32 act = netif_receive_generic_xdp(skb, xdp_prog); + + if (act != XDP_PASS) { + if (act == XDP_TX) + generic_xdp_tx(skb, xdp_prog); + return XDP_DROP; + } + } + return XDP_PASS; +} + static int netif_rx_internal(struct sk_buff *skb) { int ret; @@ -3872,6 +3973,14 @@ static int netif_rx_internal(struct sk_buff *skb) net_timestamp_check(netdev_tstamp_prequeue, skb); trace_netif_rx(skb); + + if (static_key_false(&generic_xdp_needed)) { + int ret = do_xdp_generic(skb); + + if (ret != XDP_PASS) + return NET_RX_DROP; + } + #ifdef CONFIG_RPS if (static_key_false(&rps_needed)) { struct rps_dev_flow voidflow, *rflow = &voidflow; @@ -4338,8 +4447,6 @@ static int __netif_receive_skb(struct sk_buff *skb) return ret; } -static struct static_key generic_xdp_needed __read_mostly; - static int generic_xdp_install(struct net_device *dev, struct netdev_xdp *xdp) { struct bpf_prog *old = rtnl_dereference(dev->xdp_prog); @@ -4373,89 +4480,6 @@ static int generic_xdp_install(struct net_device *dev, struct netdev_xdp *xdp) return ret; } -static u32 netif_receive_generic_xdp(struct sk_buff *skb, - struct bpf_prog *xdp_prog) -{ - struct xdp_buff xdp; - u32 act = XDP_DROP; - void *orig_data; - int hlen, off; - u32 mac_len; - - /* Reinjected packets coming from act_mirred or similar should - * not get XDP generic processing. - */ - if (skb_cloned(skb)) - return XDP_PASS; - - if (skb_linearize(skb)) - goto do_drop; - - /* The XDP program wants to see the packet starting at the MAC - * header. - */ - mac_len = skb->data - skb_mac_header(skb); - hlen = skb_headlen(skb) + mac_len; - xdp.data = skb->data - mac_len; - xdp.data_end = xdp.data + hlen; - xdp.data_hard_start = skb->data - skb_headroom(skb); - orig_data = xdp.data; - - act = bpf_prog_run_xdp(xdp_prog, &xdp); - - off = xdp.data - orig_data; - if (off > 0) - __skb_pull(skb, off); - else if (off < 0) - __skb_push(skb, -off); - - switch (act) { - case XDP_TX: - __skb_push(skb, mac_len); - /* fall through */ - case XDP_PASS: - break; - - default: - bpf_warn_invalid_xdp_action(act); - /* fall through */ - case XDP_ABORTED: - trace_xdp_exception(skb->dev, xdp_prog, act); - /* fall through */ - case XDP_DROP: - do_drop: - kfree_skb(skb); - break; - } - - return act; -} - -/* When doing generic XDP we have to bypass the qdisc layer and the - * network taps in order to match in-driver-XDP behavior. - */ -static void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog) -{ - struct net_device *dev = skb->dev; - struct netdev_queue *txq; - bool free_skb = true; - int cpu, rc; - - txq = netdev_pick_tx(dev, skb, NULL); - cpu = smp_processor_id(); - HARD_TX_LOCK(dev, txq, cpu); - if (!netif_xmit_stopped(txq)) { - rc = netdev_start_xmit(skb, dev, txq, 0); - if (dev_xmit_complete(rc)) - free_skb = false; - } - HARD_TX_UNLOCK(dev, txq); - if (free_skb) { - trace_xdp_exception(dev, xdp_prog, XDP_TX); - kfree_skb(skb); - } -} - static int netif_receive_skb_internal(struct sk_buff *skb) { int ret; @@ -4468,17 +4492,11 @@ static int netif_receive_skb_internal(struct sk_buff *skb) rcu_read_lock(); if (static_key_false(&generic_xdp_needed)) { - struct bpf_prog *xdp_prog = rcu_dereference(skb->dev->xdp_prog); - - if (xdp_prog) { - u32 act = netif_receive_generic_xdp(skb, xdp_prog); + int ret = do_xdp_generic(skb); - if (act != XDP_PASS) { - rcu_read_unlock(); - if (act == XDP_TX) - generic_xdp_tx(skb, xdp_prog); - return NET_RX_DROP; - } + if (ret != XDP_PASS) { + rcu_read_unlock(); + return NET_RX_DROP; } }