[{"id":1760923,"web_url":"http://patchwork.ozlabs.org/comment/1760923/","msgid":"<20170831143814.68709334@redhat.com>","list_archive_url":null,"date":"2017-08-31T12:38:14","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":13625,"url":"http://patchwork.ozlabs.org/api/people/13625/","name":"Jesper Dangaard Brouer","email":"brouer@redhat.com"},"content":"On Wed, 30 Aug 2017 22:18:13 -0700\nRoopa Prabhu <roopa@cumulusnetworks.com> wrote:\n\n> From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> \n> This extends bridge fdb table tracepoints to also cover\n> learned fdb entries in the br_fdb_update path. Note that\n> unlike other tracepoints I have moved this to when the fdb\n> is modified because this is in the datapath and can generate\n> a lot of noise in the trace output. br_fdb_update is also called\n> from added_by_user context in the NTF_USE case which is already\n> traced ..hence the !added_by_user check.\n> \n> Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>\n> ---\n>  include/trace/events/bridge.h | 31 +++++++++++++++++++++++++++++++\n>  net/bridge/br_fdb.c           |  5 ++++-\n>  net/core/net-traces.c         |  1 +\n>  3 files changed, 36 insertions(+), 1 deletion(-)\n> \n> diff --git a/include/trace/events/bridge.h b/include/trace/events/bridge.h\n> index 0f1cde0..1bee3e7 100644\n> --- a/include/trace/events/bridge.h\n> +++ b/include/trace/events/bridge.h\n> @@ -92,6 +92,37 @@ TRACE_EVENT(fdb_delete,\n>  \t\t  __entry->addr[4], __entry->addr[5], __entry->vid)\n>  );\n>  \n> +TRACE_EVENT(br_fdb_update,\n> +\n> +\tTP_PROTO(struct net_bridge *br, struct net_bridge_port *source,\n> +\t\t const unsigned char *addr, u16 vid, bool added_by_user),\n> +\n> +\tTP_ARGS(br, source, addr, vid, added_by_user),\n> +\n> +\tTP_STRUCT__entry(\n> +\t\t__string(br_dev, br->dev->name)\n> +\t\t__string(dev, source->dev->name)\n\nI have found that using the device string name is \n\n(1) slow as it involves strcpy+strlen\n\n See [1]+[2] where a single dev-name costed me 16 ns, and the base\n overhead of a bpf attached tracepoint is 25 ns (see [3]).\n\n [1] https://git.kernel.org/davem/net-next/c/e7d12ce121a\n [2] https://git.kernel.org/davem/net-next/c/315ec3990ef\n [3] https://git.kernel.org/davem/net-next/c/25d4dae1a64\n\n(2) strings are also harder to work-with/extract when attaching a bpf_prog\n\nSee the trouble I'm in accessing a dev string here napi:napi_poll here:\n https://github.com/netoptimizer/prototype-kernel/blob/103b955a080/kernel/samples/bpf/napi_monitor_kern.c#L52-L58\n\nUsing ifindex'es in userspace is fairly easy see man if_indextoname(3).\n\n> +\t\t__array(unsigned char, addr, ETH_ALEN)\n> +\t\t__field(u16, vid)\n> +\t\t__field(bool, added_by_user)\n> +\t),\n> +\n> +\tTP_fast_assign(\n> +\t\t__assign_str(br_dev, br->dev->name);\n> +\t\t__assign_str(dev, source->dev->name);\n> +\t\tmemcpy(__entry->addr, addr, ETH_ALEN);\n> +\t\t__entry->vid = vid;\n> +\t\t__entry->added_by_user = added_by_user;\n> +\t),\n> +\n> +\tTP_printk(\"br_dev %s source %s addr %02x:%02x:%02x:%02x:%02x:%02x vid %u added_by_user %d\",\n> +\t\t  __get_str(br_dev), __get_str(dev), __entry->addr[0],\n> +\t\t  __entry->addr[1], __entry->addr[2], __entry->addr[3],\n> +\t\t  __entry->addr[4], __entry->addr[5], __entry->vid,\n> +\t\t  __entry->added_by_user)\n> +);","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx03.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=brouer@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjhk909wgz9s7g\n\tfor <patchwork-incoming@ozlabs.org>;\n\tThu, 31 Aug 2017 22:38:25 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751382AbdHaMiW (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 08:38:22 -0400","from mx1.redhat.com ([209.132.183.28]:41766 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1751009AbdHaMiV (ORCPT <rfc822;netdev@vger.kernel.org>);\n\tThu, 31 Aug 2017 08:38:21 -0400","from smtp.corp.redhat.com\n\t(int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id A4F797E43D;\n\tThu, 31 Aug 2017 12:38:21 +0000 (UTC)","from localhost (ovpn-200-27.brq.redhat.com [10.40.200.27])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id DAB3E841C7;\n\tThu, 31 Aug 2017 12:38:15 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com A4F797E43D","Date":"Thu, 31 Aug 2017 14:38:14 +0200","From":"Jesper Dangaard Brouer <brouer@redhat.com>","To":"Roopa Prabhu <roopa@cumulusnetworks.com>","Cc":"brouer@redhat.com, davem@davemloft.net, netdev@vger.kernel.org,\n\tnikolay@cumulusnetworks.com, f.fainelli@gmail.com, andrew@lunn.ch,\n\tbridge@lists.linux-foundation.org","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","Message-ID":"<20170831143814.68709334@redhat.com>","In-Reply-To":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.12","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.27]);\n\tThu, 31 Aug 2017 12:38:21 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761045,"web_url":"http://patchwork.ozlabs.org/comment/1761045/","msgid":"<9a4af592-87ce-20cc-d617-b00cb4f72998@cumulusnetworks.com>","list_archive_url":null,"date":"2017-08-31T14:19:33","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":66448,"url":"http://patchwork.ozlabs.org/api/people/66448/","name":"Nikolay Aleksandrov","email":"nikolay@cumulusnetworks.com"},"content":"On 31/08/17 08:18, Roopa Prabhu wrote:\n> From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> \n> This extends bridge fdb table tracepoints to also cover\n> learned fdb entries in the br_fdb_update path. Note that\n> unlike other tracepoints I have moved this to when the fdb\n> is modified because this is in the datapath and can generate\n> a lot of noise in the trace output. br_fdb_update is also called\n> from added_by_user context in the NTF_USE case which is already\n> traced ..hence the !added_by_user check.\n> \n> Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>\n> ---\n>  include/trace/events/bridge.h | 31 +++++++++++++++++++++++++++++++\n>  net/bridge/br_fdb.c           |  5 ++++-\n>  net/core/net-traces.c         |  1 +\n>  3 files changed, 36 insertions(+), 1 deletion(-)\n> \n\nAcked-by: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ozlabs.org; dkim=pass (1024-bit key;\n\tunprotected) header.d=cumulusnetworks.com\n\theader.i=@cumulusnetworks.com header.b=\"XR1S6kD7\"; \n\tdkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjkz00lg5z9ryk\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 00:19:40 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751684AbdHaOTh (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 10:19:37 -0400","from mail-wm0-f46.google.com ([74.125.82.46]:36058 \"EHLO\n\tmail-wm0-f46.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751392AbdHaOTg (ORCPT\n\t<rfc822;netdev@vger.kernel.org>); Thu, 31 Aug 2017 10:19:36 -0400","by mail-wm0-f46.google.com with SMTP id f127so5868135wmf.1\n\tfor <netdev@vger.kernel.org>; Thu, 31 Aug 2017 07:19:36 -0700 (PDT)","from [192.168.0.103] (46-10-142-144.ip.btc-net.bg. [46.10.142.144])\n\tby smtp.googlemail.com with ESMTPSA id\n\to22sm9068729wrc.51.2017.08.31.07.19.33\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tThu, 31 Aug 2017 07:19:34 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=cumulusnetworks.com; s=google;\n\th=subject:to:references:cc:from:message-id:date:user-agent\n\t:mime-version:in-reply-to:content-transfer-encoding;\n\tbh=IgeZ+Th+H6Ct7euiFvlHT0hKrcBKEOYwd7qNXTd+21s=;\n\tb=XR1S6kD7hnai8M1qcx0GFrr8K0y9LO9zsYfr7U/4M+VO1JJ5nf9YQWbgKiJkg4a5vF\n\tOYtkVlDn8lnTHVg2bNfwE6ZqTeDCM3byygrgvkQTtu7MEz81PNHoi6chCRpnNgeVYBwI\n\t/SIOFMESWFaCM74PM/X/gQudoVhaT+8TR5Ofg=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:subject:to:references:cc:from:message-id:date\n\t:user-agent:mime-version:in-reply-to:content-transfer-encoding;\n\tbh=IgeZ+Th+H6Ct7euiFvlHT0hKrcBKEOYwd7qNXTd+21s=;\n\tb=QigzK6oopWYJNEm2/unrvbwmDwM9Txl1XB9pCNNyb7TFwSpPS+e1nDheGmwRScuF9q\n\ttIT0X2kX6h0ktrHuwdf7lz8/qp5Lvry3lCW/LfdDgw1Qi2L/Z7KLjMUHHgZFnx42kYMs\n\tHv0Jz9LijSuF7hc4/hkRGuEVWW+O8CN15ERKYAS+4DVUz3MTs1hTcaS/Kmink+9qAJ9s\n\tU+gskV+e6D7yd0Yk+Xj26ZLY7kqncELC+5WgyL4hZlkUjO+j44OPfLDQBlO4rieF8Ifk\n\t/yW1SpiV442hT6G14tgn7XUvdCWf9SxPrDRYx15sZ08cLu0S19LC03JgBIiz7NA4k3Et\n\tUQrA==","X-Gm-Message-State":"AHPjjUiW/4LOSXBp5zozvlU8LUxIB9438OzuObRsZk1WPAs2J/JUHqZZ\n\td+5zpv/R8A7bPF7D","X-Google-Smtp-Source":"ADKCNb4yGlqOPLmhIqWvC8VqnC1D+JSDtC8gH9bVB+YS+zdI6fUIiHQrPRrruzP9kXWlMgQVpChDHA==","X-Received":"by 10.28.140.196 with SMTP id o187mr798274wmd.160.1504189175297; \n\tThu, 31 Aug 2017 07:19:35 -0700 (PDT)","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","To":"Roopa Prabhu <roopa@cumulusnetworks.com>, davem@davemloft.net","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>","Cc":"netdev@vger.kernel.org, f.fainelli@gmail.com, andrew@lunn.ch,\n\tbridge@lists.linux-foundation.org","From":"Nikolay Aleksandrov <nikolay@cumulusnetworks.com>","Message-ID":"<9a4af592-87ce-20cc-d617-b00cb4f72998@cumulusnetworks.com>","Date":"Thu, 31 Aug 2017 17:19:33 +0300","User-Agent":"Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101\n\tIcedove/45.6.0","MIME-Version":"1.0","In-Reply-To":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>","Content-Type":"text/plain; charset=windows-1252","Content-Transfer-Encoding":"7bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761106,"web_url":"http://patchwork.ozlabs.org/comment/1761106/","msgid":"<CAJieiUjZ5dTNpBFb8c2KMRgjXRCKVGhJsTfQcpiw+p6fyNVwDw@mail.gmail.com>","list_archive_url":null,"date":"2017-08-31T15:21:30","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":23500,"url":"http://patchwork.ozlabs.org/api/people/23500/","name":"Roopa Prabhu","email":"roopa@cumulusnetworks.com"},"content":"On Thu, Aug 31, 2017 at 5:38 AM, Jesper Dangaard Brouer\n<brouer@redhat.com> wrote:\n> On Wed, 30 Aug 2017 22:18:13 -0700\n> Roopa Prabhu <roopa@cumulusnetworks.com> wrote:\n>\n>> From: Roopa Prabhu <roopa@cumulusnetworks.com>\n>>\n>> This extends bridge fdb table tracepoints to also cover\n>> learned fdb entries in the br_fdb_update path. Note that\n>> unlike other tracepoints I have moved this to when the fdb\n>> is modified because this is in the datapath and can generate\n>> a lot of noise in the trace output. br_fdb_update is also called\n>> from added_by_user context in the NTF_USE case which is already\n>> traced ..hence the !added_by_user check.\n>>\n>> Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>\n>> ---\n>>  include/trace/events/bridge.h | 31 +++++++++++++++++++++++++++++++\n>>  net/bridge/br_fdb.c           |  5 ++++-\n>>  net/core/net-traces.c         |  1 +\n>>  3 files changed, 36 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/include/trace/events/bridge.h b/include/trace/events/bridge.h\n>> index 0f1cde0..1bee3e7 100644\n>> --- a/include/trace/events/bridge.h\n>> +++ b/include/trace/events/bridge.h\n>> @@ -92,6 +92,37 @@ TRACE_EVENT(fdb_delete,\n>>                 __entry->addr[4], __entry->addr[5], __entry->vid)\n>>  );\n>>\n>> +TRACE_EVENT(br_fdb_update,\n>> +\n>> +     TP_PROTO(struct net_bridge *br, struct net_bridge_port *source,\n>> +              const unsigned char *addr, u16 vid, bool added_by_user),\n>> +\n>> +     TP_ARGS(br, source, addr, vid, added_by_user),\n>> +\n>> +     TP_STRUCT__entry(\n>> +             __string(br_dev, br->dev->name)\n>> +             __string(dev, source->dev->name)\n>\n> I have found that using the device string name is\n>\n> (1) slow as it involves strcpy+strlen\n>\n>  See [1]+[2] where a single dev-name costed me 16 ns, and the base\n>  overhead of a bpf attached tracepoint is 25 ns (see [3]).\n>\n>  [1] https://git.kernel.org/davem/net-next/c/e7d12ce121a\n>  [2] https://git.kernel.org/davem/net-next/c/315ec3990ef\n>  [3] https://git.kernel.org/davem/net-next/c/25d4dae1a64\n>\n> (2) strings are also harder to work-with/extract when attaching a bpf_prog\n>\n> See the trouble I'm in accessing a dev string here napi:napi_poll here:\n>  https://github.com/netoptimizer/prototype-kernel/blob/103b955a080/kernel/samples/bpf/napi_monitor_kern.c#L52-L58\n>\n> Using ifindex'es in userspace is fairly easy see man if_indextoname(3).\n>\n\nJesper thanks for the data!. GTK. Looking at include/trace/events,\ncurrently almost all tracepoints use dev->name.\nThese bridge tracepoints in context are primarily for debugging fdb\nupdates only, not for every packet and hence not in the performance\npath.\nIn large scale deployments with thousands of bridge ports and fdb\nentries, dev->name will definately make it easier to trouble-shoot.\nSo, I did like to leave these with dev->name unless there are strong objections.\n\nthanks.","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ozlabs.org; dkim=pass (1024-bit key;\n\tunprotected) header.d=cumulusnetworks.com\n\theader.i=@cumulusnetworks.com header.b=\"Ei6rHxsX\"; \n\tdkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjmLV0WZCz9s83\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 01:21:38 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751412AbdHaPVd (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 11:21:33 -0400","from mail-ua0-f180.google.com ([209.85.217.180]:33787 \"EHLO\n\tmail-ua0-f180.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751011AbdHaPVc (ORCPT\n\t<rfc822;netdev@vger.kernel.org>); Thu, 31 Aug 2017 11:21:32 -0400","by mail-ua0-f180.google.com with SMTP id 37so3033301ual.0\n\tfor <netdev@vger.kernel.org>; Thu, 31 Aug 2017 08:21:32 -0700 (PDT)","by 10.176.73.66 with HTTP; Thu, 31 Aug 2017 08:21:30 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=cumulusnetworks.com; s=google;\n\th=mime-version:in-reply-to:references:from:date:message-id:subject:to\n\t:cc; bh=edeIDO9eSsTavhlB07CtN2J4gph931gQ0FlgZoiHcv0=;\n\tb=Ei6rHxsXpecbMijB2YuSZGbywh53g4Yup8rqOQa7V6zAae60BUIblyEnruytAXtGWm\n\tBwDHFwQYgftBIMfJXFO5NKV+UZt3mN58BHswuKsk2glAEg9JqRzd5fy14eNVwixJUr4T\n\tYpMn3pJmSAUZ2476WBQodfsr0lWKSMnx7uqT8=","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:mime-version:in-reply-to:references:from:date\n\t:message-id:subject:to:cc;\n\tbh=edeIDO9eSsTavhlB07CtN2J4gph931gQ0FlgZoiHcv0=;\n\tb=B9W7Vd6gllPX+XJw0n6Ev+YLcW7oGjLReIq+12mMic6ezKItadrYZ7UYOwG008R6gO\n\txypEflFbfCIp0lWgdaM4FGBru8vsYZdWBKRs/B/vwWASqz8w3Ytm/UGRMj3Xt6QtLYAg\n\tyRoy1bryvMmBaMeJw1X/bD1iVlGBQlzXpVKqUoYSfEnwhywl451fTadLC1AcGWQGiG4o\n\tYtay8z9sOnh1qdgX/SiR7HPL9l+cUxt8v/woFSgC7FcbKREaeff5a2WNZUZ8V1IoWEME\n\tn6Q+5TWa6bgsbtZxAOEFIEhQ1jVsPlm2mcrq8V7Du+Cdp1KnpTfs7aym6RyJ+IT5iDtD\n\tBRcg==","X-Gm-Message-State":"AHPjjUi3+b/vBTrplv3Vtie0eRlvdUEcV7WcbX8LFS3DIBxmA+jWqCw6\n\twyNTJ6KCgf0OjR617TxvNlx4+cZsSVZy","X-Google-Smtp-Source":"ADKCNb4qAtp8Rmc/GYpZOp5oHwSAiZd97RxB/AKv6coB1ZyKifGy4RSHDJBVeFZl3dkCD+6QGVL2cj82YM0vz/izcSg=","X-Received":"by 10.159.59.68 with SMTP id j4mr3635722uah.60.1504192891416;\n\tThu, 31 Aug 2017 08:21:31 -0700 (PDT)","MIME-Version":"1.0","In-Reply-To":"<20170831143814.68709334@redhat.com>","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>\n\t<20170831143814.68709334@redhat.com>","From":"Roopa Prabhu <roopa@cumulusnetworks.com>","Date":"Thu, 31 Aug 2017 08:21:30 -0700","Message-ID":"<CAJieiUjZ5dTNpBFb8c2KMRgjXRCKVGhJsTfQcpiw+p6fyNVwDw@mail.gmail.com>","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","To":"Jesper Dangaard Brouer <brouer@redhat.com>","Cc":"\"davem@davemloft.net\" <davem@davemloft.net>,\n\t\"netdev@vger.kernel.org\" <netdev@vger.kernel.org>,\n\tNikolay Aleksandrov <nikolay@cumulusnetworks.com>,\n\tFlorian Fainelli <f.fainelli@gmail.com>,\n\tAndrew Lunn <andrew@lunn.ch>, bridge@lists.linux-foundation.org","Content-Type":"text/plain; charset=\"UTF-8\"","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761121,"web_url":"http://patchwork.ozlabs.org/comment/1761121/","msgid":"<a9349049-bfd7-b6c0-d1c7-2f70b0b0ab11@gmail.com>","list_archive_url":null,"date":"2017-08-31T15:30:05","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":6918,"url":"http://patchwork.ozlabs.org/api/people/6918/","name":"David Ahern","email":"dsahern@gmail.com"},"content":"On 8/31/17 9:21 AM, Roopa Prabhu wrote:\n> On Thu, Aug 31, 2017 at 5:38 AM, Jesper Dangaard Brouer\n> <brouer@redhat.com> wrote:\n>> On Wed, 30 Aug 2017 22:18:13 -0700\n>> Roopa Prabhu <roopa@cumulusnetworks.com> wrote:\n>>\n>>> From: Roopa Prabhu <roopa@cumulusnetworks.com>\n>>>\n>>> This extends bridge fdb table tracepoints to also cover\n>>> learned fdb entries in the br_fdb_update path. Note that\n>>> unlike other tracepoints I have moved this to when the fdb\n>>> is modified because this is in the datapath and can generate\n>>> a lot of noise in the trace output. br_fdb_update is also called\n>>> from added_by_user context in the NTF_USE case which is already\n>>> traced ..hence the !added_by_user check.\n>>>\n>>> Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>\n>>> ---\n>>>  include/trace/events/bridge.h | 31 +++++++++++++++++++++++++++++++\n>>>  net/bridge/br_fdb.c           |  5 ++++-\n>>>  net/core/net-traces.c         |  1 +\n>>>  3 files changed, 36 insertions(+), 1 deletion(-)\n>>>\n>>> diff --git a/include/trace/events/bridge.h b/include/trace/events/bridge.h\n>>> index 0f1cde0..1bee3e7 100644\n>>> --- a/include/trace/events/bridge.h\n>>> +++ b/include/trace/events/bridge.h\n>>> @@ -92,6 +92,37 @@ TRACE_EVENT(fdb_delete,\n>>>                 __entry->addr[4], __entry->addr[5], __entry->vid)\n>>>  );\n>>>\n>>> +TRACE_EVENT(br_fdb_update,\n>>> +\n>>> +     TP_PROTO(struct net_bridge *br, struct net_bridge_port *source,\n>>> +              const unsigned char *addr, u16 vid, bool added_by_user),\n>>> +\n>>> +     TP_ARGS(br, source, addr, vid, added_by_user),\n>>> +\n>>> +     TP_STRUCT__entry(\n>>> +             __string(br_dev, br->dev->name)\n>>> +             __string(dev, source->dev->name)\n>>\n>> I have found that using the device string name is\n>>\n>> (1) slow as it involves strcpy+strlen\n>>\n>>  See [1]+[2] where a single dev-name costed me 16 ns, and the base\n>>  overhead of a bpf attached tracepoint is 25 ns (see [3]).\n>>\n>>  [1] https://git.kernel.org/davem/net-next/c/e7d12ce121a\n>>  [2] https://git.kernel.org/davem/net-next/c/315ec3990ef\n>>  [3] https://git.kernel.org/davem/net-next/c/25d4dae1a64\n>>\n>> (2) strings are also harder to work-with/extract when attaching a bpf_prog\n>>\n>> See the trouble I'm in accessing a dev string here napi:napi_poll here:\n>>  https://github.com/netoptimizer/prototype-kernel/blob/103b955a080/kernel/samples/bpf/napi_monitor_kern.c#L52-L58\n>>\n>> Using ifindex'es in userspace is fairly easy see man if_indextoname(3).\n>>\n> \n> Jesper thanks for the data!. GTK. Looking at include/trace/events,\n> currently almost all tracepoints use dev->name.\n> These bridge tracepoints in context are primarily for debugging fdb\n> updates only, not for every packet and hence not in the performance\n> path.\n> In large scale deployments with thousands of bridge ports and fdb\n> entries, dev->name will definately make it easier to trouble-shoot.\n> So, I did like to leave these with dev->name unless there are strong objections.\n\n+1 for user friendliness for debugging tracepoints. The device name is\nalso more user friendly when adding filters to the data collection.\n\nBeing able to add bpf everywhere certainly changes the game a bit, but\nwe should not relinquish ease of use and understanding for the potential\nthat someone might want to put a bpf program on the tracepoint and want\nto maintain high performance.","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"QtNWkbJz\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjmXM72Vmz9sD5\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 01:30:11 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751549AbdHaPaJ (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 11:30:09 -0400","from mail-pg0-f68.google.com ([74.125.83.68]:33894 \"EHLO\n\tmail-pg0-f68.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751040AbdHaPaI (ORCPT\n\t<rfc822;netdev@vger.kernel.org>); Thu, 31 Aug 2017 11:30:08 -0400","by mail-pg0-f68.google.com with SMTP id 63so694554pgc.1\n\tfor <netdev@vger.kernel.org>; Thu, 31 Aug 2017 08:30:08 -0700 (PDT)","from dsa-mb.local ([2601:282:800:7292:a063:7aab:a8f8:ebdc])\n\tby smtp.googlemail.com with ESMTPSA id\n\t187sm32379pgj.22.2017.08.31.08.30.05\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tThu, 31 Aug 2017 08:30:06 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=subject:to:cc:references:from:message-id:date:user-agent\n\t:mime-version:in-reply-to:content-language:content-transfer-encoding; \n\tbh=IVxthM3/Mg7avUA7TKUjASKA5OeXPWzv/CBwwYf0YOA=;\n\tb=QtNWkbJzIe/N+pmFChFazTDX45lCW7g/tfP6OW0rn8Wtso0SjE1yZ+Zvr3GvuVFVR1\n\tIScNG+V5pY9zlX7roZHtMlDxpEUSz6wyrQTCUN4qh2Q86LSJVn3zE29Z0ZqHhWJdqlFy\n\teGIsjskPfCSDtIFtIiDGXQsuAFOkgOvssXQiqLYFGoP68QYgFaIfagEXrjQkxxTTSkEt\n\tODoFy38fXde5HI8UkmOWAc1eMViXt+gA6A8YG0RgVY04IDp8GWGo9usjwVuaMuqv+0Sw\n\tfOHo4lcCrJwiukNKV4X3alC8FIDkMxry7/SdvzYGRssDzaK3HkYx3dRFt9GySlyDUGAD\n\tZoxA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:subject:to:cc:references:from:message-id:date\n\t:user-agent:mime-version:in-reply-to:content-language\n\t:content-transfer-encoding;\n\tbh=IVxthM3/Mg7avUA7TKUjASKA5OeXPWzv/CBwwYf0YOA=;\n\tb=L+wNdl4WrrZXIiQLCEYnIh2GRyvQ2TW1z+9F8niF+0t/ixRXsmewZ2Bb5aXtWhS4M9\n\tDrHtHJuezyjbiFBkJTqBMk027awbtPyZEX0Eb7y3nCrx0z0nfNTsZyhNUUXVJWK6YW/x\n\thehr1hu4nCEzkNz9v4afBBQzji2Z/Tx7USugVEjsPt8yUupVV8gTH8igzl+kr6raeJwr\n\tmgCy69qmlqqMewgfXZSlnFzJd98qQISJCjX2iK7T9NA+hkvMGxAayh6pVx6gb4svlenT\n\tFmtsw0DdT1065yGiiHwVRaDrPM1p2NV8QQ6akkBxr3uUKObZnf86lEBWOCKI+lZ/eKzy\n\tOEHQ==","X-Gm-Message-State":"AHYfb5iKhoLJ9n9+c8aDnv7lwdYD67oXLkwZn8D+6++q1j69eUmwjpIe\n\tIahoDrEvOz32AQ==","X-Google-Smtp-Source":"ADKCNb6n6tnwpTFpjip3MbI7zUWmqTkYy6AMzTSTksIDMcWq6dlm6UfTZQwZOC/ks6/9vyNeA8Uwyw==","X-Received":"by 10.99.119.70 with SMTP id s67mr2804704pgc.181.1504193407750; \n\tThu, 31 Aug 2017 08:30:07 -0700 (PDT)","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","To":"Roopa Prabhu <roopa@cumulusnetworks.com>,\n\tJesper Dangaard Brouer <brouer@redhat.com>","Cc":"\"davem@davemloft.net\" <davem@davemloft.net>,\n\t\"netdev@vger.kernel.org\" <netdev@vger.kernel.org>,\n\tNikolay Aleksandrov <nikolay@cumulusnetworks.com>,\n\tFlorian Fainelli <f.fainelli@gmail.com>,\n\tAndrew Lunn <andrew@lunn.ch>, bridge@lists.linux-foundation.org","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>\n\t<20170831143814.68709334@redhat.com>\n\t<CAJieiUjZ5dTNpBFb8c2KMRgjXRCKVGhJsTfQcpiw+p6fyNVwDw@mail.gmail.com>","From":"David Ahern <dsahern@gmail.com>","Message-ID":"<a9349049-bfd7-b6c0-d1c7-2f70b0b0ab11@gmail.com>","Date":"Thu, 31 Aug 2017 09:30:05 -0600","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0)\n\tGecko/20100101 Thunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<CAJieiUjZ5dTNpBFb8c2KMRgjXRCKVGhJsTfQcpiw+p6fyNVwDw@mail.gmail.com>","Content-Type":"text/plain; charset=utf-8","Content-Language":"en-US","Content-Transfer-Encoding":"7bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761158,"web_url":"http://patchwork.ozlabs.org/comment/1761158/","msgid":"<20170831182012.5d321c6a@redhat.com>","list_archive_url":null,"date":"2017-08-31T16:20:12","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":13625,"url":"http://patchwork.ozlabs.org/api/people/13625/","name":"Jesper Dangaard Brouer","email":"brouer@redhat.com"},"content":"On Thu, 31 Aug 2017 09:30:05 -0600\nDavid Ahern <dsahern@gmail.com> wrote:\n\n> On 8/31/17 9:21 AM, Roopa Prabhu wrote:\n> > On Thu, Aug 31, 2017 at 5:38 AM, Jesper Dangaard Brouer\n> > <brouer@redhat.com> wrote:  \n> >> On Wed, 30 Aug 2017 22:18:13 -0700\n> >> Roopa Prabhu <roopa@cumulusnetworks.com> wrote:\n> >>  \n> >>> From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> >>>\n> >>> This extends bridge fdb table tracepoints to also cover\n> >>> learned fdb entries in the br_fdb_update path. Note that\n> >>> unlike other tracepoints I have moved this to when the fdb\n> >>> is modified because this is in the datapath and can generate\n> >>> a lot of noise in the trace output. br_fdb_update is also called\n> >>> from added_by_user context in the NTF_USE case which is already\n> >>> traced ..hence the !added_by_user check.\n> >>>\n> >>> Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>\n> >>> ---\n> >>>  include/trace/events/bridge.h | 31 +++++++++++++++++++++++++++++++\n> >>>  net/bridge/br_fdb.c           |  5 ++++-\n> >>>  net/core/net-traces.c         |  1 +\n> >>>  3 files changed, 36 insertions(+), 1 deletion(-)\n> >>>\n> >>> diff --git a/include/trace/events/bridge.h b/include/trace/events/bridge.h\n> >>> index 0f1cde0..1bee3e7 100644\n> >>> --- a/include/trace/events/bridge.h\n> >>> +++ b/include/trace/events/bridge.h\n> >>> @@ -92,6 +92,37 @@ TRACE_EVENT(fdb_delete,\n> >>>                 __entry->addr[4], __entry->addr[5], __entry->vid)\n> >>>  );\n> >>>\n> >>> +TRACE_EVENT(br_fdb_update,\n> >>> +\n> >>> +     TP_PROTO(struct net_bridge *br, struct net_bridge_port *source,\n> >>> +              const unsigned char *addr, u16 vid, bool added_by_user),\n> >>> +\n> >>> +     TP_ARGS(br, source, addr, vid, added_by_user),\n> >>> +\n> >>> +     TP_STRUCT__entry(\n> >>> +             __string(br_dev, br->dev->name)\n> >>> +             __string(dev, source->dev->name)  \n> >>\n> >> I have found that using the device string name is\n> >>\n> >> (1) slow as it involves strcpy+strlen\n> >>\n> >>  See [1]+[2] where a single dev-name costed me 16 ns, and the base\n> >>  overhead of a bpf attached tracepoint is 25 ns (see [3]).\n> >>\n> >>  [1] https://git.kernel.org/davem/net-next/c/e7d12ce121a\n> >>  [2] https://git.kernel.org/davem/net-next/c/315ec3990ef\n> >>  [3] https://git.kernel.org/davem/net-next/c/25d4dae1a64\n> >>\n> >> (2) strings are also harder to work-with/extract when attaching a bpf_prog\n> >>\n> >> See the trouble I'm in accessing a dev string here napi:napi_poll here:\n> >>  https://github.com/netoptimizer/prototype-kernel/blob/103b955a080/kernel/samples/bpf/napi_monitor_kern.c#L52-L58\n> >>\n> >> Using ifindex'es in userspace is fairly easy see man if_indextoname(3).\n> >>  \n> > \n> > Jesper thanks for the data!. GTK. Looking at include/trace/events,\n> > currently almost all tracepoints use dev->name.\n\nTrue, but with my recent experience and benchmarking, I consider this\ngenerally a bad choice we have made for all these tracepoints.  In your\ncase with 2 strings, 2x16=32ns, you basically introduced a overhead\nthat is larger that to invocation cost.\n\n> > These bridge tracepoints in context are primarily for debugging fdb\n> > updates only, not for every packet and hence not in the performance\n> > path.\n> > In large scale deployments with thousands of bridge ports and fdb\n> > entries, dev->name will definately make it easier to trouble-shoot.\n> > So, I did like to leave these with dev->name unless there are strong objections.  \n> \n> +1 for user friendliness for debugging tracepoints. The device name is\n> also more user friendly when adding filters to the data collection.\n>\n> Being able to add bpf everywhere certainly changes the game a bit, but\n> we should not relinquish ease of use and understanding for the potential\n> that someone might want to put a bpf program on the tracepoint and want\n> to maintain high performance.\n\n(Cc. Acme and Peterz)\nI wonder if we can create a special perf-tracepoint type for ifindex'es\nand the tool reading (e.g. perf-script) can perform the name lookup in\nuserspace (calling if_indextoname(3)) ?\n\nI don't know the perf tools well enough to know if this is possible?","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ext-mx06.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx06.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=brouer@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjnfG3bTWz9sMN\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 02:20:22 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751824AbdHaQUU (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 12:20:20 -0400","from mx1.redhat.com ([209.132.183.28]:58892 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1751717AbdHaQUT (ORCPT <rfc822;netdev@vger.kernel.org>);\n\tThu, 31 Aug 2017 12:20:19 -0400","from smtp.corp.redhat.com\n\t(int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 30158356CE;\n\tThu, 31 Aug 2017 16:20:19 +0000 (UTC)","from localhost (ovpn-200-27.brq.redhat.com [10.40.200.27])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id A95D494B27;\n\tThu, 31 Aug 2017 16:20:13 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 30158356CE","Date":"Thu, 31 Aug 2017 18:20:12 +0200","From":"Jesper Dangaard Brouer <brouer@redhat.com>","To":"David Ahern <dsahern@gmail.com>","Cc":"Roopa Prabhu <roopa@cumulusnetworks.com>,\n\t\"davem@davemloft.net\" <davem@davemloft.net>,\n\t\"netdev@vger.kernel.org\" <netdev@vger.kernel.org>,\n\tNikolay Aleksandrov <nikolay@cumulusnetworks.com>,\n\tFlorian Fainelli <f.fainelli@gmail.com>, Andrew Lunn <andrew@lunn.ch>,\n\tbridge@lists.linux-foundation.org, brouer@redhat.com,\n\tArnaldo Carvalho de Melo <acme@redhat.com>,\n\tPeter Zijlstra <peterz@infradead.org>","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","Message-ID":"<20170831182012.5d321c6a@redhat.com>","In-Reply-To":"<a9349049-bfd7-b6c0-d1c7-2f70b0b0ab11@gmail.com>","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>\n\t<20170831143814.68709334@redhat.com>\n\t<CAJieiUjZ5dTNpBFb8c2KMRgjXRCKVGhJsTfQcpiw+p6fyNVwDw@mail.gmail.com>\n\t<a9349049-bfd7-b6c0-d1c7-2f70b0b0ab11@gmail.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.15","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.30]);\n\tThu, 31 Aug 2017 16:20:19 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761255,"web_url":"http://patchwork.ozlabs.org/comment/1761255/","msgid":"<20170831.114325.2027598728601121750.davem@davemloft.net>","list_archive_url":null,"date":"2017-08-31T18:43:25","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":15,"url":"http://patchwork.ozlabs.org/api/people/15/","name":"David Miller","email":"davem@davemloft.net"},"content":"From: Roopa Prabhu <roopa@cumulusnetworks.com>\nDate: Wed, 30 Aug 2017 22:18:13 -0700\n\n> From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> \n> This extends bridge fdb table tracepoints to also cover\n> learned fdb entries in the br_fdb_update path. Note that\n> unlike other tracepoints I have moved this to when the fdb\n> is modified because this is in the datapath and can generate\n> a lot of noise in the trace output. br_fdb_update is also called\n> from added_by_user context in the NTF_USE case which is already\n> traced ..hence the !added_by_user check.\n> \n> Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>\n\nApplied.\n\nLet's use dev->name for now and if the tooling can eventually\ndo transparent ifindex->name then we can consider redoing\na bunch of networking tracepoints.\n\nThanks.","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjrqP5171z9s7g\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 04:43:29 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751867AbdHaSn1 (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 14:43:27 -0400","from shards.monkeyblade.net ([184.105.139.130]:55690 \"EHLO\n\tshards.monkeyblade.net\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751607AbdHaSn0 (ORCPT\n\t<rfc822;netdev@vger.kernel.org>); Thu, 31 Aug 2017 14:43:26 -0400","from localhost (74-93-104-98-Washington.hfc.comcastbusiness.net\n\t[74.93.104.98]) (using TLSv1 with cipher AES256-SHA (256/256 bits))\n\t(Client did not present a certificate)\n\t(Authenticated sender: davem-davemloft)\n\tby shards.monkeyblade.net (Postfix) with ESMTPSA id E227B1251BC4D;\n\tThu, 31 Aug 2017 11:43:25 -0700 (PDT)"],"Date":"Thu, 31 Aug 2017 11:43:25 -0700 (PDT)","Message-Id":"<20170831.114325.2027598728601121750.davem@davemloft.net>","To":"roopa@cumulusnetworks.com","Cc":"netdev@vger.kernel.org, nikolay@cumulusnetworks.com,\n\tf.fainelli@gmail.com, andrew@lunn.ch, bridge@lists.linux-foundation.org","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","From":"David Miller <davem@davemloft.net>","In-Reply-To":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>","X-Mailer":"Mew version 6.7 on Emacs 25.2 / Mule 6.0 (HANACHIRUSATO)","Mime-Version":"1.0","Content-Type":"Text/Plain; charset=us-ascii","Content-Transfer-Encoding":"7bit","X-Greylist":"Sender succeeded SMTP AUTH, not delayed by\n\tmilter-greylist-4.5.12 (shards.monkeyblade.net\n\t[149.20.54.216]); Thu, 31 Aug 2017 11:43:26 -0700 (PDT)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761262,"web_url":"http://patchwork.ozlabs.org/comment/1761262/","msgid":"<20170831185657.GC2252@redhat.com>","list_archive_url":null,"date":"2017-08-31T18:56:57","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":313,"url":"http://patchwork.ozlabs.org/api/people/313/","name":"Arnaldo Carvalho de Melo","email":"acme@redhat.com"},"content":"Em Thu, Aug 31, 2017 at 06:20:12PM +0200, Jesper Dangaard Brouer escreveu:\n> On Thu, 31 Aug 2017 09:30:05 -0600 David Ahern <dsahern@gmail.com> wrote:\n> > > On Thu, Aug 31, 2017 at 5:38 AM, Jesper Dangaard Brouer <brouer@redhat.com> wrote:  \n> > > These bridge tracepoints in context are primarily for debugging fdb\n> > > updates only, not for every packet and hence not in the performance\n> > > path.\n> > > In large scale deployments with thousands of bridge ports and fdb\n> > > entries, dev->name will definately make it easier to trouble-shoot.\n> > > So, I did like to leave these with dev->name unless there are strong objections.  \n\n> > +1 for user friendliness for debugging tracepoints. The device name is\n> > also more user friendly when adding filters to the data collection.\n\n> > Being able to add bpf everywhere certainly changes the game a bit, but\n> > we should not relinquish ease of use and understanding for the potential\n> > that someone might want to put a bpf program on the tracepoint and want\n> > to maintain high performance.\n \n> (Cc. Acme and Peterz)\n> I wonder if we can create a special perf-tracepoint type for ifindex'es\n> and the tool reading (e.g. perf-script) can perform the name lookup in\n> userspace (calling if_indextoname(3)) ?\n \n> I don't know the perf tools well enough to know if this is possible?\n\nYeah, there are libtraceevent plugins, and that gets used by trace-cmd\nand perf script, perf trace.\n\n[root@jouet ~]# ls -la ~/.traceevent/plugins/\ntotal 192\ndrwxr-xr-x. 2 acme acme  4096 Aug 31 15:29 .\ndrwxr-xr-x. 3 acme acme  4096 Jan 27  2017 ..\n-rwxr-xr-x. 1 acme acme 13744 Aug 31 15:29 plugin_cfg80211.so\n-rwxr-xr-x. 1 acme acme 20192 Aug 31 15:29 plugin_function.so\n-rwxr-xr-x. 1 acme acme 13680 Aug 31 15:29 plugin_hrtimer.so\n-rwxr-xr-x. 1 acme acme 13760 Aug 31 15:29 plugin_jbd2.so\n-rwxr-xr-x. 1 acme acme 13704 Aug 31 15:29 plugin_kmem.so\n-rwxr-xr-x. 1 acme acme 28568 Aug 31 15:29 plugin_kvm.so\n-rwxr-xr-x. 1 acme acme 14184 Aug 31 15:29 plugin_mac80211.so\n-rwxr-xr-x. 1 acme acme 14424 Aug 31 15:29 plugin_sched_switch.so\n-rwxr-xr-x. 1 acme acme 20136 Aug 31 15:29 plugin_scsi.so\n-rwxr-xr-x. 1 acme acme 14504 Aug 31 15:29 plugin_xen.so\n[root@jouet ~]#\n\nBut... that index is something that is mutable, i.e. you'd have to\nsomehow record all the assignments of an index to an interface and then,\nwhen processing the events, get from that state the mapping you want.\n\nSo you don't store the device name by doing lookups at each of those\nhigh volume tracepoints, store just the index, but then, when\nestablishing the mapping, collect that as well and we come up with some\ninfrastructure to get that mapping in a place where a plugin can do the\nlookups at post processing time.\n\nFor instance, the hrtimer plugin will get an address from the kernel,\nand, from a state recorded at the same time as the trace file, will\nlookup elf symbol tables for the kernel or modules and resolve that\nsymbol, etc.\n\n- Arnaldo","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx01.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=acme@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjs782byFz9s7f\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 04:57:08 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751506AbdHaS5F (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 14:57:05 -0400","from mx1.redhat.com ([209.132.183.28]:53122 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1750996AbdHaS5E (ORCPT <rfc822;netdev@vger.kernel.org>);\n\tThu, 31 Aug 2017 14:57:04 -0400","from smtp.corp.redhat.com\n\t(int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id 6A07781DFF;\n\tThu, 31 Aug 2017 18:57:04 +0000 (UTC)","from sandy.ghostprotocols.net (ovpn-112-4.gru2.redhat.com\n\t[10.97.112.4])\n\tby smtp.corp.redhat.com (Postfix) with ESMTPS id 35922A397A;\n\tThu, 31 Aug 2017 18:57:00 +0000 (UTC)","by sandy.ghostprotocols.net (Postfix, from userid 1000)\n\tid 74B0C18FF; Thu, 31 Aug 2017 15:56:57 -0300 (BRT)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com 6A07781DFF","Date":"Thu, 31 Aug 2017 15:56:57 -0300","From":"Arnaldo Carvalho de Melo <acme@redhat.com>","To":"Jesper Dangaard Brouer <brouer@redhat.com>","Cc":"David Ahern <dsahern@gmail.com>,\n\tRoopa Prabhu <roopa@cumulusnetworks.com>,\n\t\"davem@davemloft.net\" <davem@davemloft.net>,\n\t\"netdev@vger.kernel.org\" <netdev@vger.kernel.org>,\n\tNikolay Aleksandrov <nikolay@cumulusnetworks.com>,\n\tFlorian Fainelli <f.fainelli@gmail.com>, Andrew Lunn <andrew@lunn.ch>,\n\tbridge@lists.linux-foundation.org, Peter Zijlstra <peterz@infradead.org>","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","Message-ID":"<20170831185657.GC2252@redhat.com>","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>\n\t<20170831143814.68709334@redhat.com>\n\t<CAJieiUjZ5dTNpBFb8c2KMRgjXRCKVGhJsTfQcpiw+p6fyNVwDw@mail.gmail.com>\n\t<a9349049-bfd7-b6c0-d1c7-2f70b0b0ab11@gmail.com>\n\t<20170831182012.5d321c6a@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=us-ascii","Content-Disposition":"inline","In-Reply-To":"<20170831182012.5d321c6a@redhat.com>","X-Url":"http://acmel.wordpress.com","User-Agent":"Mutt/1.5.20 (2009-12-10)","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.13","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.25]);\n\tThu, 31 Aug 2017 18:57:04 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761342,"web_url":"http://patchwork.ozlabs.org/comment/1761342/","msgid":"<20170831235026.6377bf86@redhat.com>","list_archive_url":null,"date":"2017-08-31T21:50:26","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":13625,"url":"http://patchwork.ozlabs.org/api/people/13625/","name":"Jesper Dangaard Brouer","email":"brouer@redhat.com"},"content":"On Thu, 31 Aug 2017 11:43:25 -0700 (PDT)\nDavid Miller <davem@davemloft.net> wrote:\n\n> From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> Date: Wed, 30 Aug 2017 22:18:13 -0700\n> \n> > From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> > \n> > This extends bridge fdb table tracepoints to also cover\n> > learned fdb entries in the br_fdb_update path. Note that\n> > unlike other tracepoints I have moved this to when the fdb\n> > is modified because this is in the datapath and can generate\n> > a lot of noise in the trace output. br_fdb_update is also called\n> > from added_by_user context in the NTF_USE case which is already\n> > traced ..hence the !added_by_user check.\n> > \n> > Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>  \n> \n> Applied.\n> \n> Let's use dev->name for now and if the tooling can eventually\n> do transparent ifindex->name then we can consider redoing\n> a bunch of networking tracepoints.\n\nI agree! :-)","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tdmarc=none (p=none dis=none) header.from=redhat.com","ext-mx09.extmail.prod.ext.phx2.redhat.com;\n\tspf=fail smtp.mailfrom=brouer@redhat.com"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjwzH5Z75z9s81\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 07:50:35 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751367AbdHaVud (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 17:50:33 -0400","from mx1.redhat.com ([209.132.183.28]:46248 \"EHLO mx1.redhat.com\"\n\trhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP\n\tid S1750908AbdHaVuc (ORCPT <rfc822;netdev@vger.kernel.org>);\n\tThu, 31 Aug 2017 17:50:32 -0400","from smtp.corp.redhat.com\n\t(int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby mx1.redhat.com (Postfix) with ESMTPS id A4F424DD49;\n\tThu, 31 Aug 2017 21:50:32 +0000 (UTC)","from localhost (ovpn-200-27.brq.redhat.com [10.40.200.27])\n\tby smtp.corp.redhat.com (Postfix) with ESMTP id D73515F92D;\n\tThu, 31 Aug 2017 21:50:27 +0000 (UTC)"],"DMARC-Filter":"OpenDMARC Filter v1.3.2 mx1.redhat.com A4F424DD49","Date":"Thu, 31 Aug 2017 23:50:26 +0200","From":"Jesper Dangaard Brouer <brouer@redhat.com>","To":"David Miller <davem@davemloft.net>","Cc":"brouer@redhat.com, roopa@cumulusnetworks.com,\n\tnetdev@vger.kernel.org, nikolay@cumulusnetworks.com,\n\tf.fainelli@gmail.com, andrew@lunn.ch, bridge@lists.linux-foundation.org","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","Message-ID":"<20170831235026.6377bf86@redhat.com>","In-Reply-To":"<20170831.114325.2027598728601121750.davem@davemloft.net>","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>\n\t<20170831.114325.2027598728601121750.davem@davemloft.net>","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","X-Scanned-By":"MIMEDefang 2.79 on 10.5.11.14","X-Greylist":"Sender IP whitelisted, not delayed by milter-greylist-4.5.16\n\t(mx1.redhat.com [10.5.110.38]);\n\tThu, 31 Aug 2017 21:50:32 +0000 (UTC)","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1761357,"web_url":"http://patchwork.ozlabs.org/comment/1761357/","msgid":"<20170831154344.0a1f714e@xeon-e3>","list_archive_url":null,"date":"2017-08-31T22:43:44","subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","submitter":{"id":21389,"url":"http://patchwork.ozlabs.org/api/people/21389/","name":"Stephen Hemminger","email":"stephen@networkplumber.org"},"content":"On Thu, 31 Aug 2017 23:50:26 +0200\nJesper Dangaard Brouer <brouer@redhat.com> wrote:\n\n> On Thu, 31 Aug 2017 11:43:25 -0700 (PDT)\n> David Miller <davem@davemloft.net> wrote:\n> \n> > From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> > Date: Wed, 30 Aug 2017 22:18:13 -0700\n> >   \n> > > From: Roopa Prabhu <roopa@cumulusnetworks.com>\n> > > \n> > > This extends bridge fdb table tracepoints to also cover\n> > > learned fdb entries in the br_fdb_update path. Note that\n> > > unlike other tracepoints I have moved this to when the fdb\n> > > is modified because this is in the datapath and can generate\n> > > a lot of noise in the trace output. br_fdb_update is also called\n> > > from added_by_user context in the NTF_USE case which is already\n> > > traced ..hence the !added_by_user check.\n> > > \n> > > Signed-off-by: Roopa Prabhu <roopa@cumulusnetworks.com>    \n> > \n> > Applied.\n> > \n> > Let's use dev->name for now and if the tooling can eventually\n> > do transparent ifindex->name then we can consider redoing\n> > a bunch of networking tracepoints.  \n> \n> I agree! :-)\n> \n\nAgreed, but it is yet another case of tracepoints not having\na stable ABI. There is no ABI guarantee on tracing, but it still\nmakes for user complaints.","headers":{"Return-Path":"<netdev-owner@vger.kernel.org>","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=<UNKNOWN>)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=networkplumber-org.20150623.gappssmtp.com\n\theader.i=@networkplumber-org.20150623.gappssmtp.com\n\theader.b=\"rlbSKBA8\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xjy8m3xNwz9s4s\n\tfor <patchwork-incoming@ozlabs.org>;\n\tFri,  1 Sep 2017 08:43:52 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751805AbdHaWnt (ORCPT <rfc822;patchwork-incoming@ozlabs.org>);\n\tThu, 31 Aug 2017 18:43:49 -0400","from mail-pf0-f177.google.com ([209.85.192.177]:36480 \"EHLO\n\tmail-pf0-f177.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751205AbdHaWns (ORCPT\n\t<rfc822;netdev@vger.kernel.org>); Thu, 31 Aug 2017 18:43:48 -0400","by mail-pf0-f177.google.com with SMTP id r187so2801887pfr.3\n\tfor <netdev@vger.kernel.org>; Thu, 31 Aug 2017 15:43:47 -0700 (PDT)","from xeon-e3 (76-14-207-240.or.wavecable.com. [76.14.207.240])\n\tby smtp.gmail.com with ESMTPSA id\n\to63sm820606pga.12.2017.08.31.15.43.47\n\t(version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);\n\tThu, 31 Aug 2017 15:43:47 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=networkplumber-org.20150623.gappssmtp.com; s=20150623;\n\th=date:from:to:cc:subject:message-id:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=RMrOS/ZO0N0OoKR8Qz17GYotLnBBVP6YbfyxDJOltrs=;\n\tb=rlbSKBA8uR0sMG1N/K9yr+++uuGR/bzk3pq7Gk9RFJ/fktttSKdj6qBMyyhb/K4riA\n\tnhHpa+Zdt2qW3hQPJTrcM7Qkf5zuVYKcDrQUw8lIr1haqNHIC0TkoHIt8r6Q1L3Cr6yq\n\tvGrnrgnfU3xnm5q7KcsCklCCNVzQPkPNEqJJRVdcRfFFLOgk4zmcyqf9KIIzkorkoq1j\n\tkialA+uMqmUYXLKBBKzDohsXtLVNW1tN4azSF55uFKtlvdIMbNz/EtYG9DdyRQBYHIn0\n\t+hb0EpGiOtkxFisklm1wFigqiREamd6wVs2V2rD3pwXYv4YxZAmQq/kPc0Qf6zyMDSbv\n\t8qQA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=RMrOS/ZO0N0OoKR8Qz17GYotLnBBVP6YbfyxDJOltrs=;\n\tb=Y8HZJHpbpjwwVTrOIjlfc0gs+YmhTG8g0n8d0actMHifCE1jljxGCmY6Gc75MjGIcD\n\tBlMXTGH2grtuShawnUyMK66g9NLjo7+GPr+YkJ5122YZW7Sf+r1rXOjs2WUTeMKdmQ+C\n\tkyuz/NRnn4ayDbD/XcDuRg8fNhUuq5udkpw1L12LZU9ymrfMlPH77IdcRgyZ2ZAwoETY\n\tTPfXEVb3CMkTTyAk5NMaLoXkw81r73nxhekEOzZuhWHCHxXxrvD8sgQMhbgR7TCo7Lm8\n\t+PMUbK6GGpnDMSQ4vi6Cu0JiBTybvBdvguL0Eb5/MvPGJI+kODj9IOgI5V4LcuGE5ekP\n\tb+6A==","X-Gm-Message-State":"AHPjjUhYBBy+wg5EwQEeFGyVWXKJWws65A67NdM3b/tyR2RKulsU7lbr\n\tgyg1Xjaqb+04aDUD","X-Google-Smtp-Source":"ADKCNb5G8XL2vdXoG1oLPRCXM41/7aUcTJbfAclo5LJHquck5xirPgK3mB+jWKdf/W0HGTrN9Qoo3Q==","X-Received":"by 10.84.229.72 with SMTP id d8mr33472pln.2.1504219427564;\n\tThu, 31 Aug 2017 15:43:47 -0700 (PDT)","Date":"Thu, 31 Aug 2017 15:43:44 -0700","From":"Stephen Hemminger <stephen@networkplumber.org>","To":"Jesper Dangaard Brouer <brouer@redhat.com>","Cc":"David Miller <davem@davemloft.net>, roopa@cumulusnetworks.com,\n\tnetdev@vger.kernel.org, nikolay@cumulusnetworks.com,\n\tf.fainelli@gmail.com, andrew@lunn.ch, bridge@lists.linux-foundation.org","Subject":"Re: [PATCH net-next] bridge: add tracepoint in br_fdb_update","Message-ID":"<20170831154344.0a1f714e@xeon-e3>","In-Reply-To":"<20170831235026.6377bf86@redhat.com>","References":"<1504156693-24692-1-git-send-email-roopa@cumulusnetworks.com>\n\t<20170831.114325.2027598728601121750.davem@davemloft.net>\n\t<20170831235026.6377bf86@redhat.com>","MIME-Version":"1.0","Content-Type":"text/plain; charset=US-ASCII","Content-Transfer-Encoding":"7bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"<netdev.vger.kernel.org>","X-Mailing-List":"netdev@vger.kernel.org"}}]