From patchwork Thu Sep 2 10:18:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Roi Dayan X-Patchwork-Id: 1523594 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=Nvidia.com header.i=@Nvidia.com header.a=rsa-sha256 header.s=selector2 header.b=AyknS5D3; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4H0cN45hC4z9sPf for ; Thu, 2 Sep 2021 20:19:00 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 76541614B2; Thu, 2 Sep 2021 10:18:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z3oC7Ri4VeqN; Thu, 2 Sep 2021 10:18:54 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 806B1614BA; Thu, 2 Sep 2021 10:18:50 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 75B67C002F; Thu, 2 Sep 2021 10:18:48 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id EDC3EC0029 for ; Thu, 2 Sep 2021 10:18:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id DD0E542548 for ; Thu, 2 Sep 2021 10:18:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=nvidia.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JtOt20REgWTu for ; Thu, 2 Sep 2021 10:18:44 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2062d.outbound.protection.outlook.com [IPv6:2a01:111:f400:7e8a::62d]) by smtp4.osuosl.org (Postfix) with ESMTPS id 4B19A402A1 for ; Thu, 2 Sep 2021 10:18:42 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C6ucwzNZdN2MOXRS4aovCdWYW++pvfmyC3tfaKk6KQ5tX4bK2MZF/fzUrYlbsHXoqdbsvg5RimWYvMPxAUeWjFsX66NgA0jvH/rKgDjiJjiGlyImIZ2UHtNBwHtM35QZn7uZ8WSA0E4qqSeE6d4X1LPI9l1oE982zKNPp2pux0ONm3Mcu8juzEstF1f3nRey0pfYxAxJur4YWK2m0TqUcjxr0Iu4xqZy4mbLxMkbEOAsLRL9QTMFhO5DAXBPd+oV5Ozbf3DAmep4ezNTn3wa0eyCi9MmI54WvKCyp/Eeozz/NqfALsXfmB35i5IGpRm4X7il7uDv80/A4fnmTP8yMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6IR+J5VUaRdxan15FFCkSTPyS2KzXaSJoA8VJbKXbS8=; b=LWQTN7RO5Oc/v4fbL9N92/Xw2saBQiGUqJm/+gQSAmErSDnGS+nSgRx6DRL4Af8uhbXXujtO5PnxjON/RAF4fQfVe/MBNjOyB69nfGqOkoFGPBXXW+2SNM3PWp/JjS1x1et1cw95GpTk1oamSS5AX9U0OcLgvbgxa/1S0xsKxM3/66bbLzwQ/tX/Cb7JtpapaSET1Zlp2eu78i3+xrH49dGW5JG/Fs7+7vRfRcsuHXuUp8iS/d1b/7a4MFhPxb2VMM9c12C9PzuHA3D94cm3PO4sRlRbKe45V9pODYWLFWCV9/BApp2ORR5hwrBKVoVtJC1PfsAFFwv+WhMNrfG2ug== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=ovn.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6IR+J5VUaRdxan15FFCkSTPyS2KzXaSJoA8VJbKXbS8=; b=AyknS5D3h48g4LawLnnGAMHLJOOY/Ftyz9z3YikiJYiCq62lMFBaN++cA6qKOmXQVrWYntsw8L2FfTi9tHTYT7yJ39saZh+DCO1qdLi7iJlaAWYIXLEQn2NvuSSESVxEou7RfHrtFq7GTVrj2ErIPWWA/cZQ7h7AG8QT2S0wT2g/iO/pwljATDkc77aPEXQaarSTYZHkxFi4nKI3D3plRuKgMPs74Gffmu7mTMSysNbgCLGR+I2H8ByUInEv3yg02W62+905eCmqnfueljJ7y3s6acBfR4dGDlf5Zk3mKCEwqwPNyiCAlWrHXMVN/JLs2Gyf5rA+dVBCkXqtw9osxQ== Received: from MW4PR03CA0268.namprd03.prod.outlook.com (2603:10b6:303:b4::33) by BN8PR12MB3076.namprd12.prod.outlook.com (2603:10b6:408:61::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4457.20; Thu, 2 Sep 2021 10:18:38 +0000 Received: from CO1NAM11FT013.eop-nam11.prod.protection.outlook.com (2603:10b6:303:b4:cafe::d) by MW4PR03CA0268.outlook.office365.com (2603:10b6:303:b4::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4478.20 via Frontend Transport; Thu, 2 Sep 2021 10:18:38 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; ovn.org; dkim=none (message not signed) header.d=none;ovn.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by CO1NAM11FT013.mail.protection.outlook.com (10.13.174.227) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4478.19 via Frontend Transport; Thu, 2 Sep 2021 10:18:37 +0000 Received: from HQMAIL109.nvidia.com (172.20.187.15) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Thu, 2 Sep 2021 10:18:36 +0000 Received: from dev-r-vrt-138.mtr.labs.mlnx (172.20.187.5) by mail.nvidia.com (172.20.187.15) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Thu, 2 Sep 2021 03:18:34 -0700 To: , Ilya Maximets Date: Thu, 2 Sep 2021 13:18:21 +0300 Message-ID: <20210902101823.3403153-5-roid@nvidia.com> X-Mailer: git-send-email 2.26.3 In-Reply-To: <20210902101823.3403153-1-roid@nvidia.com> References: <20210902101823.3403153-1-roid@nvidia.com> MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a43ca6f9-0600-47ec-f1c8-08d96dfb077b X-MS-TrafficTypeDiagnostic: BN8PR12MB3076: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:109; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yXyr00xN2LCErNoriO7SN44nbjUVtLraqHx86XhUmQQv6TwQLfaDO6JqwGuLpPpyjsNPrTvlc3Cq0VYGYKD37P5FFbLmBDXwUNdkgwn/SUk8IwLW/nLFxPN0gjJ82rLZft1JrMU9yxFXI7L3CrOfvj650f3IXqqUzfVbktYLKKIYDGNXmem6zEN2mcXTyUzBXkAkKdN0DMnTSFFfeIgjnKSncePMmhkM7ghEHWK4KVz3BpXf3mQdnMP4hPbsDK7d42oTYAS2tCsnwkPYcynnK+6s8wtOBqLYGa9uvABLPniF8F/6ZJGoKBqsJqhIqa7u9YvpX13uG84REtmns37Qd/zxKc1emxSwzubcqzJFIUP/k8JXKbSfJIf442JxIP9mZ1vx9jLAObRgw0z8FfFgr1P9OWACbZvpQDPzPZlbJN6WNODssDDyJIwI9saVEfBw1tnhgkiVvrDomKlIPKxMpLh7qAqJwRkyJZxSU0Hd9kwBKj1aTR1Jhr0DlQd3QpKjf1Cmd/C5hDwZD4WpsL8/+UkIOrKBvKz2yiraabuLXY9632FyY8f7BtxkdOrCpt6h/2Ey0AeCGrhOvH91mQ8b/R3a7qiKr1oh98mqhOpU3aU2RoCe930pgDdQ9ME0Oh7oPCXcXQcYNa1IxLmgRs9i6t4MEJZw3b4J/AQ9T4I2zT4mNDT2YLN0yGwkd017ddMTD0e2PBTvaiomX+zAO0vPIg== X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid04.nvidia.com; CAT:NONE; SFS:(4636009)(136003)(39860400002)(346002)(396003)(376002)(36840700001)(46966006)(1076003)(47076005)(8676002)(7636003)(83380400001)(6666004)(36860700001)(8936002)(82740400003)(356005)(82310400003)(426003)(336012)(186003)(26005)(70586007)(70206006)(5660300002)(478600001)(316002)(36906005)(4326008)(2906002)(2616005)(54906003)(110136005)(107886003)(36756003)(86362001); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Sep 2021 10:18:37.8378 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a43ca6f9-0600-47ec-f1c8-08d96dfb077b X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT013.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR12MB3076 Cc: Jianbo Liu Subject: [ovs-dev] [PATCH 4/6] dpif-netlink: Offloading meter to tc police action X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Roi Dayan via dev From: Roi Dayan Reply-To: Roi Dayan Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Jianbo Liu OVS meters are created in advance and openflow rules refer to them by their unique ID. In order to offload meters, they are mapped to tc police actions with one-to-one relationship, then these actions can be used in tc filter rules by the index. So a police action is created when meter is created, and deleted after meter is deleted. An id-pool is used to manage all the available police indexes, which are 50000-59999, reserved only for OVS. Signed-off-by: Jianbo Liu Reviewed-by: Roi Dayan --- lib/dpif-netlink.c | 240 +++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 235 insertions(+), 5 deletions(-) diff --git a/lib/dpif-netlink.c b/lib/dpif-netlink.c index 34fc04237333..fa68e69efbbe 100644 --- a/lib/dpif-netlink.c +++ b/lib/dpif-netlink.c @@ -37,6 +37,7 @@ #include "dpif-provider.h" #include "fat-rwlock.h" #include "flow.h" +#include "id-pool.h" #include "netdev-linux.h" #include "netdev-offload.h" #include "netdev-provider.h" @@ -122,6 +123,23 @@ static void dpif_netlink_unixctl_dispatch_mode(struct unixctl_conn *conn, int argc, const char *argv[], void *aux); +#define METER_POLICE_IDS_BASE 50000 +#define METER_POLICE_IDS_MAX 59999 +/* Protects below meter ids pool and hashmaps. */ +static struct ovs_mutex meter_mutex = OVS_MUTEX_INITIALIZER; +static struct id_pool *meter_police_ids; +static struct hmap meter_id_to_police_idx OVS_GUARDED_BY(meter_mutex) + = HMAP_INITIALIZER(&meter_id_to_police_idx); + +struct meter_id_to_police_idx_data { + struct hmap_node meter_id_node; + uint32_t meter_id; + uint32_t police_idx; +}; + +static int +meter_id_lookup(uint32_t meter_id, uint32_t *police_idx); + struct dpif_netlink_flow { /* Generic Netlink header. */ uint8_t cmd; @@ -704,6 +722,7 @@ dpif_netlink_destroy(struct dpif *dpif_) struct dpif_netlink *dpif = dpif_netlink_cast(dpif_); struct dpif_netlink_dp dp; + id_pool_destroy(meter_police_ids); dpif_netlink_dp_init(&dp); dp.cmd = OVS_DP_CMD_DEL; dp.dp_ifindex = dpif->dp_ifindex; @@ -3921,6 +3940,96 @@ dpif_netlink_ct_timeout_policy_dump_done(struct dpif *dpif OVS_UNUSED, * zero. Check for that condition and disable meters on those kernels. */ static bool probe_broken_meters(struct dpif *); +static struct meter_id_to_police_idx_data * +meter_id_find_locked(uint32_t meter_id) + OVS_REQUIRES(meter_mutex) +{ + struct meter_id_to_police_idx_data *data; + size_t hash = hash_int(meter_id, 0); + + HMAP_FOR_EACH_WITH_HASH (data, meter_id_node, hash, + &meter_id_to_police_idx) { + if (data->meter_id == meter_id) { + return data; + } + } + + return NULL; +} + +static int +meter_id_lookup(uint32_t meter_id, uint32_t *police_idx) +{ + struct meter_id_to_police_idx_data *data; + int ret = 0; + + ovs_mutex_lock(&meter_mutex); + data = meter_id_find_locked(meter_id); + if (data) { + *police_idx = data->police_idx; + } else { + ret = ENOENT; + } + ovs_mutex_unlock(&meter_mutex); + + return ret; +} + +static void +meter_id_insert(uint32_t meter_id, uint32_t police_idx) +{ + struct meter_id_to_police_idx_data *data; + + ovs_mutex_lock(&meter_mutex); + data = meter_id_find_locked(meter_id); + if (!data) { + data = xzalloc(sizeof *data); + data->meter_id = meter_id; + data->police_idx = police_idx; + hmap_insert(&meter_id_to_police_idx, &data->meter_id_node, + hash_int(meter_id, 0)); + } else { + VLOG_WARN_RL(&error_rl, + "try to insert meter %d (%d) with different police (%d)", + meter_id, data->police_idx, police_idx); + } + ovs_mutex_unlock(&meter_mutex); +} + +static void +meter_id_remove(uint32_t meter_id) +{ + struct meter_id_to_police_idx_data *data; + + ovs_mutex_lock(&meter_mutex); + data = meter_id_find_locked(meter_id); + if (data) { + hmap_remove(&meter_id_to_police_idx, &data->meter_id_node); + free(data); + } + ovs_mutex_unlock(&meter_mutex); +} + +static bool +meter_alloc_police_index(uint32_t *police_index) +{ + bool ret; + + ovs_mutex_lock(&meter_mutex); + ret = id_pool_alloc_id(meter_police_ids, police_index); + ovs_mutex_unlock(&meter_mutex); + + return ret; +} + +static void +meter_free_police_index(uint32_t police_index) +{ + ovs_mutex_lock(&meter_mutex); + id_pool_free_id(meter_police_ids, police_index); + ovs_mutex_unlock(&meter_mutex); +} + static void dpif_netlink_meter_init(struct dpif_netlink *dpif, struct ofpbuf *buf, void *stub, size_t size, uint32_t command) @@ -4107,14 +4216,75 @@ dpif_netlink_meter_set__(struct dpif *dpif_, ofproto_meter_id meter_id, } static int +dpif_netlink_meter_set_policer(struct dpif *dpif_, ofproto_meter_id meter_id, + struct ofputil_meter_config *config) +{ + uint32_t police_index; + uint32_t rate, burst; + bool add_policer; + int err; + + ovs_assert(config->bands != NULL); + + rate = config->bands[0].rate; + if (config->flags & OFPMF13_BURST) { + burst = config->bands[0].burst_size; + } else { + burst = config->bands[0].rate; + } + + add_policer = (meter_id_lookup(meter_id.uint32, &police_index) == ENOENT); + if (add_policer) { + if (!meter_alloc_police_index(&police_index)) { + VLOG_WARN_RL(&error_rl, "%s: no free police index for meter id %d", + dpif_name(dpif_), meter_id.uint32); + return ENOENT; + } + } + + err = tc_add_policer_action(police_index, + (config->flags & OFPMF13_KBPS) ? rate : 0, + (config->flags & OFPMF13_KBPS) ? burst : 0, + (config->flags & OFPMF13_PKTPS) ? rate : 0, + (config->flags & OFPMF13_PKTPS) ? burst : 0, + !add_policer); + if (err) { + VLOG_WARN_RL(&error_rl, + "%s: failed to %s police %d for meter id %d: %s", + dpif_name(dpif_), add_policer ? "add" : "modify", + police_index, meter_id.uint32, ovs_strerror(err)); + goto err_add_policer; + } + + if (add_policer) { + meter_id_insert(meter_id.uint32, police_index); + } + + return 0; + +err_add_policer: + if (add_policer) { + meter_free_police_index(police_index); + } + return err; +} + +static int dpif_netlink_meter_set(struct dpif *dpif_, ofproto_meter_id meter_id, struct ofputil_meter_config *config) { + int err; + if (probe_broken_meters(dpif_)) { return ENOMEM; } - return dpif_netlink_meter_set__(dpif_, meter_id, config); + err = dpif_netlink_meter_set__(dpif_, meter_id, config); + if (!err && netdev_is_flow_api_enabled()) { + dpif_netlink_meter_set_policer(dpif_, meter_id, config); + } + + return err; } /* Retrieve statistics and/or delete meter 'meter_id'. Statistics are @@ -4202,19 +4372,77 @@ dpif_netlink_meter_get_stats(const struct dpif *dpif_, } static int +dpif_netlink_meter_get_policer(const struct dpif *dpif, + ofproto_meter_id meter_id, + struct ofputil_meter_stats *stats) +{ + uint32_t police_index; + int err = 0; + + if (!meter_id_lookup(meter_id.uint32, &police_index)) { + err = tc_get_policer_action(police_index, stats); + if (err) { + VLOG_WARN_RL(&error_rl, + "%s: failed to get police %d stats for meter %d: %s", + dpif_name(dpif), police_index, meter_id.uint32, + ovs_strerror(err)); + } + } + + return err; +} + +static int dpif_netlink_meter_get(const struct dpif *dpif, ofproto_meter_id meter_id, struct ofputil_meter_stats *stats, uint16_t max_bands) { - return dpif_netlink_meter_get_stats(dpif, meter_id, stats, max_bands, - OVS_METER_CMD_GET); + int err; + + err = dpif_netlink_meter_get_stats(dpif, meter_id, stats, max_bands, + OVS_METER_CMD_GET); + if (!err && netdev_is_flow_api_enabled()) { + dpif_netlink_meter_get_policer(dpif, meter_id, stats); + } + + return err; +} + +static int +dpif_netlink_meter_del_policer(struct dpif *dpif, ofproto_meter_id meter_id, + struct ofputil_meter_stats *stats) +{ + uint32_t police_index; + int err = 0; + + if (!meter_id_lookup(meter_id.uint32, &police_index)) { + err = tc_del_policer_action(police_index, stats); + if (err) { + VLOG_WARN_RL(&error_rl, + "%s: failed to del police %d for meter %d: %s", + dpif_name(dpif), police_index, + meter_id.uint32, ovs_strerror(err)); + } else { + meter_free_police_index(police_index); + } + meter_id_remove(meter_id.uint32); + } + + return err; } static int dpif_netlink_meter_del(struct dpif *dpif, ofproto_meter_id meter_id, struct ofputil_meter_stats *stats, uint16_t max_bands) { - return dpif_netlink_meter_get_stats(dpif, meter_id, stats, max_bands, - OVS_METER_CMD_DEL); + int err; + + err = dpif_netlink_meter_get_stats(dpif, meter_id, stats, + max_bands, OVS_METER_CMD_DEL); + if (!err && netdev_is_flow_api_enabled()) { + dpif_netlink_meter_del_policer(dpif, meter_id, stats); + } + + return err; } static bool @@ -4256,6 +4484,8 @@ probe_broken_meters(struct dpif *dpif) static bool broken_meters = false; if (ovsthread_once_start(&once)) { + meter_police_ids = id_pool_create(METER_POLICE_IDS_BASE, + METER_POLICE_IDS_MAX - METER_POLICE_IDS_BASE + 1); broken_meters = probe_broken_meters__(dpif); ovsthread_once_done(&once); }