From patchwork Wed Nov 20 00:15:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1197700 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=mellanox.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=Mellanox.com header.i=@Mellanox.com header.b="sqS7PZW2"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47Hjrm4Pl2z9sPV for ; Wed, 20 Nov 2019 11:15:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727506AbfKTAPf (ORCPT ); Tue, 19 Nov 2019 19:15:35 -0500 Received: from mail-eopbgr50065.outbound.protection.outlook.com ([40.107.5.65]:50450 "EHLO EUR03-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727324AbfKTAPf (ORCPT ); Tue, 19 Nov 2019 19:15:35 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WNbC3iD5b6R+IoZEcx1qKUoUVD+xxYIH3wyGy8+E84jJtza6Xi/gw8aYhYgC1EgDvXEG879wP8GEQ8ZLNT8yypXsyrK3WfiqT1aZiIPLGXMhKLa1ak3hNbfX28ROkb/6KdEiwdsNuRA1lt6zR9TX1EhZxYsBks+9IBk5+gSMteOrBIp5vC89amlDUldo1hbKZ/PQsMM8un+GxSMMwZEyllmzWT06MmKtPX3tBTFxRqPDbs93vwKQKDpP0cCf3V/qNKgalBt3KqiF9G9cuhsL85aqi7aKrMALIV/rVRnNq9uX2TcKsq1Ex9KZ9BEG2+AdJSrgUxJhFfLhIzcSraH/YQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X4F+VAqlmwgRU/4t8bBDw5GWw5F9LgxyM9JNQedna7U=; b=WDbG2OaDDAhGmZxzbok6aHQrLPezLnIJD91IELsAu3teWuy6Z//V6V8rYsaoR9QQxALwEeMeto7GcZUNnz1Xyq/GMRMm+23RTUGBnBU924MzOEiqr4Fq/MuAzbOjXLmdI+qfLmWQxikwuit1+MJPLf/FXWGirAITu8UNdebVLfAihvQZc306sThWo03uokKfU9NZq7ysiys7s9Abx6WYVYUhVj/SnTcKZq4JJQ85tbkEV5EwefoLfLDCrWel4psZFY/ZxEM6q0FhTZN9PM8GjrXrH58t0bBlD8elKUwBk1U/+LuenBtoTpKmMUZAgJP/5FNUivSOHSAF19xRijPa8Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X4F+VAqlmwgRU/4t8bBDw5GWw5F9LgxyM9JNQedna7U=; b=sqS7PZW2yjYKdpIJMRY1Iqo3gAc4Rg1QUqEULWQO2kT+fHs+cptCCetnIibFGVoHl3ek/KRLfvqX+bdQFRdrRdNwJnarzwGMnsZZ/vL+X/mFW6eHM71nizqfONKaPK2j83tAKVqGP/hlRr7/VR7gS6Uz2wJG2TLUxOuMW7NQcHk= Received: from AM6PR05MB5094.eurprd05.prod.outlook.com (20.177.34.93) by AM6PR05MB5378.eurprd05.prod.outlook.com (20.177.188.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2474.17; Wed, 20 Nov 2019 00:15:17 +0000 Received: from AM6PR05MB5094.eurprd05.prod.outlook.com ([fe80::5cde:d406:1656:17b4]) by AM6PR05MB5094.eurprd05.prod.outlook.com ([fe80::5cde:d406:1656:17b4%6]) with mapi id 15.20.2451.029; Wed, 20 Nov 2019 00:15:17 +0000 From: Saeed Mahameed To: Jesper Dangaard Brouer , "David S. Miller" CC: "netdev@vger.kernel.org" , Saeed Mahameed , Jonathan Lemon , Ilias Apalodimas Subject: [PATCH net-next V3 1/3] page_pool: Add API to update numa node Thread-Topic: [PATCH net-next V3 1/3] page_pool: Add API to update numa node Thread-Index: AQHVnzeWi5SO0+3v3Ei79aDVcE1cmw== Date: Wed, 20 Nov 2019 00:15:17 +0000 Message-ID: <20191120001456.11170-2-saeedm@mellanox.com> References: <20191120001456.11170-1-saeedm@mellanox.com> In-Reply-To: <20191120001456.11170-1-saeedm@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-originating-ip: [209.116.155.178] x-clientproxiedby: BY5PR13CA0008.namprd13.prod.outlook.com (2603:10b6:a03:180::21) To AM6PR05MB5094.eurprd05.prod.outlook.com (2603:10a6:20b:9::29) authentication-results: spf=none (sender IP is ) smtp.mailfrom=saeedm@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 22cc4bf1-8739-4eeb-9f57-08d76d4eb8a7 x-ms-traffictypediagnostic: AM6PR05MB5378:|AM6PR05MB5378: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:5516; x-forefront-prvs: 02272225C5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(366004)(376002)(39860400002)(346002)(396003)(189003)(199004)(15650500001)(6116002)(3846002)(256004)(14444005)(66946007)(110136005)(446003)(305945005)(81156014)(6506007)(386003)(81166006)(36756003)(6512007)(2906002)(54906003)(14454004)(71190400001)(71200400001)(11346002)(316002)(7736002)(64756008)(66556008)(66476007)(66446008)(486006)(476003)(2616005)(102836004)(50226002)(99286004)(52116002)(186003)(5660300002)(6486002)(86362001)(478600001)(8936002)(76176011)(25786009)(6436002)(26005)(4326008)(1076003)(66066001)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR05MB5378; H:AM6PR05MB5094.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: lrEMmBZQwyOnM4CQtzrCgDN6HBPx/uzCpx2TR1ObpiSQjJ9V/UkdU+pMVq5xyHbEkrYf7Zljxcu2ZPdZGKWjhveNYhQgOBa8YlyvuViWfbSu7xbvFmmnbr5n31SjqqJzBGDDA+UPXkigExFNbcU9Z2NT04TXU92DjEB142iLdu8mcyGxeuo70I/oAlht0+OFkPu1/IKf1dBluFjrE9wHwzOyt9oTk4Gk68O2HmNhpo3lHUABaojqD5GQrDDGjC5+BnMoah2/QRve/2JzmY1ci33Fpp3VurIS6CWAGWqf0IIOsSGlHgccpjGAe+M5NSOmVtX2PWhfaUtFviCQ9AH5eVaV1DcjqYPHKqmrjYMG9x+fhOtjrk9h4dlHxLfbYzmygyb4bZv7bhJXx7+RFiSj36msg0iRfrOsMy3W6yH1szMJChfq8inLRHSvUWRnKpp/ MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 22cc4bf1-8739-4eeb-9f57-08d76d4eb8a7 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Nov 2019 00:15:17.3351 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: c9IXo8vxtKFbrBbSyyvOf5XtoH9lHzax5zEGTmN4AvDQTXtKnDJG1XFR4d+rZSFNNfTC9ovc6Nm/Dht6jsBE8g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5378 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add page_pool_update_nid() to be called by page pool consumers when they detect numa node changes. It will update the page pool nid value to start allocating from the new effective numa node. This is to mitigate page pool allocating pages from a wrong numa node, where the pool was originally allocated, and holding on to pages that belong to a different numa node, which causes performance degradation. For pages that are already being consumed and could be returned to the pool by the consumer, in next patch we will add a check per page to avoid recycling them back to the pool and return them to the page allocator. Signed-off-by: Saeed Mahameed Acked-by: Jonathan Lemon Reviewed-by: Ilias Apalodimas Acked-by: Jesper Dangaard Brouer --- include/net/page_pool.h | 7 +++++++ include/trace/events/page_pool.h | 22 ++++++++++++++++++++++ net/core/page_pool.c | 8 ++++++++ 3 files changed, 37 insertions(+) diff --git a/include/net/page_pool.h b/include/net/page_pool.h index ace881c15dcb..e2e1b7b1e8ba 100644 --- a/include/net/page_pool.h +++ b/include/net/page_pool.h @@ -204,4 +204,11 @@ static inline bool page_pool_put(struct page_pool *pool) return refcount_dec_and_test(&pool->user_cnt); } +/* Caller must provide appropriate safe context, e.g. NAPI. */ +void page_pool_update_nid(struct page_pool *pool, int new_nid); +static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) +{ + if (unlikely(pool->p.nid != new_nid)) + page_pool_update_nid(pool, new_nid); +} #endif /* _NET_PAGE_POOL_H */ diff --git a/include/trace/events/page_pool.h b/include/trace/events/page_pool.h index 2f2a10e8eb56..ad0aa7f31675 100644 --- a/include/trace/events/page_pool.h +++ b/include/trace/events/page_pool.h @@ -89,6 +89,28 @@ TRACE_EVENT(page_pool_state_hold, __entry->pool, __entry->page, __entry->pfn, __entry->hold) ); +TRACE_EVENT(page_pool_update_nid, + + TP_PROTO(const struct page_pool *pool, int new_nid), + + TP_ARGS(pool, new_nid), + + TP_STRUCT__entry( + __field(const struct page_pool *, pool) + __field(int, pool_nid) + __field(int, new_nid) + ), + + TP_fast_assign( + __entry->pool = pool; + __entry->pool_nid = pool->p.nid; + __entry->new_nid = new_nid; + ), + + TP_printk("page_pool=%p pool_nid=%d new_nid=%d", + __entry->pool, __entry->pool_nid, __entry->new_nid) +); + #endif /* _TRACE_PAGE_POOL_H */ /* This part must be outside protection */ diff --git a/net/core/page_pool.c b/net/core/page_pool.c index e28db2ef8e12..9b704ea3f4b2 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -436,3 +436,11 @@ void page_pool_destroy(struct page_pool *pool) schedule_delayed_work(&pool->release_dw, DEFER_TIME); } EXPORT_SYMBOL(page_pool_destroy); + +/* Caller must provide appropriate safe context, e.g. NAPI. */ +void page_pool_update_nid(struct page_pool *pool, int new_nid) +{ + trace_page_pool_update_nid(pool, new_nid); + pool->p.nid = new_nid; +} +EXPORT_SYMBOL(page_pool_update_nid); From patchwork Wed Nov 20 00:15:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1197699 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=mellanox.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=Mellanox.com header.i=@Mellanox.com header.b="H9r8e9fq"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47Hjrj36h5z9sPW for ; Wed, 20 Nov 2019 11:15:33 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727471AbfKTAPc (ORCPT ); Tue, 19 Nov 2019 19:15:32 -0500 Received: from mail-eopbgr50065.outbound.protection.outlook.com ([40.107.5.65]:50450 "EHLO EUR03-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727428AbfKTAPb (ORCPT ); Tue, 19 Nov 2019 19:15:31 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QEmZodCKI9MHqgmCoEtnPCUtEAjrUO9fI2dQEVBqz61La/FAwMARg5sB5bveSJJU4e4SZtaGADEURtIc1LkZb/2Ec0mgghj0OW2A9G6Wxi/7sTZch0RL9iHvFIXdebneJd610U2p7tLYTgC/Lt8qGxVgHpFMBDvTxjLw3nq2lHZHVhkWzSaQFbk+GlVGcPqhCAqorZq7YTFrIVI1te8L3yqJ+StNllSF9FaMvXF8O2yCBzUgEbVEyKvLfXek0FXnxRwbB90K9V3XnCSuWK++LFCzuNsNsCwCiPT9ZDALp0zmywvpIydW2ZYN0RSisXhBG3HsZx6gxl0jhmP7YX/tjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EA2eqqC7dIlXyRPyEfmiKTf9OA1E2vpeJgzKJFyfFi4=; b=kepLaTKx5oz7rRyU567FHTcR3x2xMrw4WB5THqlldI2uWay77Mxak/8xAtxzzDzRoaQQOG1/6Qeyp+rqy3FgwaCl/ML2t1gv5RmB64pGop9wlvlH9tJ6a5tRiWlG/qr3XWrBJdQF1h/V1W25icVbY2Fa6/AuP6RnDkSmZHyAiJ56SQAjk5V5IlsH6L5Jii5X3+X2e53I7L9PIycFUlARp7X0bZA0jR4fhYlhPXFx7uwF/aVHlbGKEyWCpF2ahm8f8VEkWoIFLr8YYj+2II4k3tV4bhCraCqAW2xS2WVuNmdeZ3kpzf+yjcNlIZcRTw3Ix83IQuE4FSCUpch/qRb8Ug== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=EA2eqqC7dIlXyRPyEfmiKTf9OA1E2vpeJgzKJFyfFi4=; b=H9r8e9fqTG05NfJVOAWaBOLE3jYjXod/8eonbrT/rfBOgTilScOR70dOgd6sKs5nDPwbYzKIfNirDWoYqSlRN3z+QWnHxzpHXgjsSSPvawzul2aPAgaJqPLhbjCqrBzYhpnG6vBn1lgw7dUd1DXJp8xZpDuYT5IUYEOQehC6waw= Received: from AM6PR05MB5094.eurprd05.prod.outlook.com (20.177.34.93) by AM6PR05MB5378.eurprd05.prod.outlook.com (20.177.188.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2474.17; Wed, 20 Nov 2019 00:15:19 +0000 Received: from AM6PR05MB5094.eurprd05.prod.outlook.com ([fe80::5cde:d406:1656:17b4]) by AM6PR05MB5094.eurprd05.prod.outlook.com ([fe80::5cde:d406:1656:17b4%6]) with mapi id 15.20.2451.029; Wed, 20 Nov 2019 00:15:19 +0000 From: Saeed Mahameed To: Jesper Dangaard Brouer , "David S. Miller" CC: "netdev@vger.kernel.org" , Saeed Mahameed , Jonathan Lemon , Ilias Apalodimas Subject: [PATCH net-next V3 2/3] page_pool: Don't recycle non-reusable pages Thread-Topic: [PATCH net-next V3 2/3] page_pool: Don't recycle non-reusable pages Thread-Index: AQHVnzeXudYWmdhKVEqshEe1j4Fwzw== Date: Wed, 20 Nov 2019 00:15:19 +0000 Message-ID: <20191120001456.11170-3-saeedm@mellanox.com> References: <20191120001456.11170-1-saeedm@mellanox.com> In-Reply-To: <20191120001456.11170-1-saeedm@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-originating-ip: [209.116.155.178] x-clientproxiedby: BY5PR13CA0008.namprd13.prod.outlook.com (2603:10b6:a03:180::21) To AM6PR05MB5094.eurprd05.prod.outlook.com (2603:10a6:20b:9::29) authentication-results: spf=none (sender IP is ) smtp.mailfrom=saeedm@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 0ae0e867-b306-47cf-c75c-08d76d4eba0c x-ms-traffictypediagnostic: AM6PR05MB5378:|AM6PR05MB5378: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 02272225C5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(366004)(376002)(39860400002)(346002)(396003)(189003)(199004)(6116002)(3846002)(256004)(14444005)(66946007)(110136005)(446003)(305945005)(81156014)(6506007)(386003)(81166006)(36756003)(6512007)(2906002)(54906003)(14454004)(71190400001)(71200400001)(11346002)(316002)(7736002)(64756008)(66556008)(66476007)(66446008)(486006)(476003)(2616005)(102836004)(50226002)(99286004)(52116002)(186003)(5660300002)(6486002)(86362001)(478600001)(8936002)(76176011)(25786009)(6436002)(26005)(4326008)(1076003)(66066001)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR05MB5378; H:AM6PR05MB5094.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: pkbnSFvBAb48zH+4eCnb/tkwpgNEC0G45jDbKFvFpshBi/s1wj01akWqAB7XaqcSA55G7a8WikWLvvcq1XQwwT8UFFOmChIB1mnFa+t2DJI60ZYHsyub2HFslym0NZVXU+wDludv3IjI00pp/NShNLIzbVxIyaaOm8kN2WKojEPTRFzTGuXRKlzFAAIzlcOi5CQveX6c5fX7QhVh0MayTSA4jFuUyXB0s5Xp71Aj8cN1f/atwz32mIc5neSKnGeBiUZW93HE6DeCFU7GvPNPwQLHRgEEqPbcR3wqxi4yQcdXIp+EJ0AKtd8qEnI49ESTh6GYg7mGyq4CbF7rfETWEutJxiBS/xf7bFfdpWvT/W2bE/XIBPzjXHMKkGmYUmtt7/XoaT0tMBWrVviHmRgw7RpBWfqnF3RfibEwsGtBeiwHMcY8U88c9VjjfJIKN9w3 MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ae0e867-b306-47cf-c75c-08d76d4eba0c X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Nov 2019 00:15:19.4689 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: goeVihM29lAPwwWA9pl9MeECVBjOMQ/nDUuiyIff3WR2o9jdVac6mnBZ1KYVi1kc9jWAWFX7SFSE34xEkvo/EA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5378 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org A page is NOT reusable when at least one of the following is true: 1) allocated when system was under some pressure. (page_is_pfmemalloc) 2) belongs to a different NUMA node than pool->p.nid. To update pool->p.nid users should call page_pool_update_nid(). Holding on to such pages in the pool will hurt the consumer performance when the pool migrates to a different numa node. Performance testing: XDP drop/tx rate and TCP single/multi stream, on mlx5 driver while migrating rx ring irq from close to far numa: mlx5 internal page cache was locally disabled to get pure page pool results. CPU: Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz NIC: Mellanox Technologies MT27700 Family [ConnectX-4] (100G) XDP Drop/TX single core: NUMA | XDP | Before | After --------------------------------------- Close | Drop | 11 Mpps | 10.9 Mpps Far | Drop | 4.4 Mpps | 5.8 Mpps Close | TX | 6.5 Mpps | 6.5 Mpps Far | TX | 3.5 Mpps | 4 Mpps Improvement is about 30% drop packet rate, 15% tx packet rate for numa far test. No degradation for numa close tests. TCP single/multi cpu/stream: NUMA | #cpu | Before | After -------------------------------------- Close | 1 | 18 Gbps | 18 Gbps Far | 1 | 15 Gbps | 18 Gbps Close | 12 | 80 Gbps | 80 Gbps Far | 12 | 68 Gbps | 80 Gbps In all test cases we see improvement for the far numa case, and no impact on the close numa case. The impact of adding a check per page is very negligible, and shows no performance degradation whatsoever, also functionality wise it seems more correct and more robust for page pool to verify when pages should be recycled, since page pool can't guarantee where pages are coming from. Signed-off-by: Saeed Mahameed Acked-by: Jonathan Lemon Reviewed-by: Ilias Apalodimas Acked-by: Jesper Dangaard Brouer --- net/core/page_pool.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/net/core/page_pool.c b/net/core/page_pool.c index 9b704ea3f4b2..6c7f78bd6421 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -281,6 +281,17 @@ static bool __page_pool_recycle_direct(struct page *page, return true; } +/* page is NOT reusable when: + * 1) allocated when system is under some pressure. (page_is_pfmemalloc) + * 2) belongs to a different NUMA node than pool->p.nid. + * + * To update pool->p.nid users must call page_pool_update_nid. + */ +static bool pool_page_reusable(struct page_pool *pool, struct page *page) +{ + return !page_is_pfmemalloc(page) && page_to_nid(page) == pool->p.nid; +} + void __page_pool_put_page(struct page_pool *pool, struct page *page, bool allow_direct) { @@ -290,7 +301,8 @@ void __page_pool_put_page(struct page_pool *pool, * * refcnt == 1 means page_pool owns page, and can recycle it. */ - if (likely(page_ref_count(page) == 1)) { + if (likely(page_ref_count(page) == 1 && + pool_page_reusable(pool, page))) { /* Read barrier done in page_ref_count / READ_ONCE */ if (allow_direct && in_serving_softirq()) From patchwork Wed Nov 20 00:15:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1197701 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=mellanox.com Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=Mellanox.com header.i=@Mellanox.com header.b="SQ0CaLpJ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47Hjs51VgSz9sPV for ; Wed, 20 Nov 2019 11:15:53 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727534AbfKTAPw (ORCPT ); Tue, 19 Nov 2019 19:15:52 -0500 Received: from mail-eopbgr40042.outbound.protection.outlook.com ([40.107.4.42]:57648 "EHLO EUR03-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727518AbfKTAPw (ORCPT ); Tue, 19 Nov 2019 19:15:52 -0500 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VxJR7blKX135epMuXlr21VsyfjsPBgoZK7PB6OS8yPAPrEYsjy8SFcpL4QdU1Cm8Zenr8VcC3P4a8ZqnIw+qRHZadx5W66TbUQelLa9wVm41GbDZNbVbX7z54vBBjRASKN8+izOnKwN2dWZBRnI0B8jKZTRGJ6VzpeBOReWXPq3z7pG9a8vnG8Ci5yAtNf8Ji3ketvM7jbQzjIKhnoc0VwBerl/ZGaljb11xAxU4CBUHt9VAUZoyunZCtz2IWB4RyjxvoMDGtk9op1rOZiS3dD2kia/KF1dZdmRSPANFxq/ifrglrU4FM74+D768BmqpktgsTTC92cHZCZQlAyVO9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mvqBIzdT/rwsHbZntj7gxefWY8bGVZeJZkyMRoKVyZU=; b=jYN7Lz1qRWnWfc/2ePPSOsqKpjCeemzyQgfltVwf8AWDSj2FC8Dg1x9RXQFBbdC7xIxGhczRC3yXj014Xh+B/EsoisNSBSF43MSJptAGL5B74DjLojISmGzEhMRUhRmLn+9rwKEKBepWguf3NTveDI/v4e0JiXYtpJos6K9QV7jJ31vFQaKhpPW+Bmsv2+LU+cfrqK8ynyhgKB3to1sQeclsqHXSRZeXNFuRu6eE9UEUGhvG8Rw2kusKIrg894Da2Wh6PJ9zhpIoob+k7VbMpKHbRnDeTxMwnSBic71iPeDdn1iTwThzVAzBPsFWcjiabdKpg1C1czwcBMcGPzNTkg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mvqBIzdT/rwsHbZntj7gxefWY8bGVZeJZkyMRoKVyZU=; b=SQ0CaLpJhHqf6Lgm0CiGrm2RPB7BPkFjgO6Ri6Pgx3eYcSWvAk5OmET99R7G1m6nXVEptMAqGI5iBe1QEbeKKm4woQ4N8k3JAc6isZKc5BrQtipNNJzjVXUocRsUGQx5cjaVfSwZ0ab9MrLtXeQvmrPHc5PcGg2yAYRvKx8sIyk= Received: from AM6PR05MB5094.eurprd05.prod.outlook.com (20.177.34.93) by AM6PR05MB5378.eurprd05.prod.outlook.com (20.177.188.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2474.17; Wed, 20 Nov 2019 00:15:21 +0000 Received: from AM6PR05MB5094.eurprd05.prod.outlook.com ([fe80::5cde:d406:1656:17b4]) by AM6PR05MB5094.eurprd05.prod.outlook.com ([fe80::5cde:d406:1656:17b4%6]) with mapi id 15.20.2451.029; Wed, 20 Nov 2019 00:15:21 +0000 From: Saeed Mahameed To: Jesper Dangaard Brouer , "David S. Miller" CC: "netdev@vger.kernel.org" , Saeed Mahameed , Jonathan Lemon Subject: [PATCH net-next V3 3/3] net/mlx5e: Rx, Update page pool numa node when changed Thread-Topic: [PATCH net-next V3 3/3] net/mlx5e: Rx, Update page pool numa node when changed Thread-Index: AQHVnzeYdwBKSlQ+aUKSpbDfXMiUjQ== Date: Wed, 20 Nov 2019 00:15:21 +0000 Message-ID: <20191120001456.11170-4-saeedm@mellanox.com> References: <20191120001456.11170-1-saeedm@mellanox.com> In-Reply-To: <20191120001456.11170-1-saeedm@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-mailer: git-send-email 2.21.0 x-originating-ip: [209.116.155.178] x-clientproxiedby: BY5PR13CA0008.namprd13.prod.outlook.com (2603:10b6:a03:180::21) To AM6PR05MB5094.eurprd05.prod.outlook.com (2603:10a6:20b:9::29) authentication-results: spf=none (sender IP is ) smtp.mailfrom=saeedm@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: d1e044e9-10f2-48ad-a524-08d76d4ebb3b x-ms-traffictypediagnostic: AM6PR05MB5378:|AM6PR05MB5378: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 02272225C5 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(366004)(376002)(39860400002)(346002)(396003)(189003)(199004)(15650500001)(6116002)(3846002)(256004)(14444005)(66946007)(110136005)(446003)(305945005)(81156014)(6506007)(386003)(81166006)(36756003)(6512007)(2906002)(54906003)(14454004)(71190400001)(71200400001)(11346002)(316002)(7736002)(64756008)(66556008)(66476007)(66446008)(486006)(476003)(2616005)(102836004)(50226002)(99286004)(52116002)(186003)(5660300002)(6486002)(86362001)(478600001)(8936002)(76176011)(25786009)(6436002)(26005)(4326008)(1076003)(66066001)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR05MB5378; H:AM6PR05MB5094.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: zrY5CL9W5PaEYfWCAO6bQiLnYby5/DQI9gC5dmoBeZWy5dXM7TKlozDoelTKZd+G3D+GIqfoNvGLREJvRXxMAg8XHq/KHJmLq92sVMrRxmaJEk2VdZNml8WLi+5ml+ZIVEik35SEPXRdPkAAWeCfaaJIPH91JDyZbtee0ctnoEJ9NeotncgEiMwf+S5SUQpEuxfNvk41sOk0EqexFqcCaePhpBqnf1FGLo4LWO3BritBOmlemfY4+RVwcAxQG1oNrtNcd5SUXa0ifXYf+2tnqSbP1hymZ4f/A8nlbUNwlh4DYWlyODi8FpwO2wOMqi/i9AL+J8K1SU9zWqluaOk3leXbEuGVmcDR2IEGiRho9CZV9ytX/Kt18AQ+FKJ/51Gr3cBzUEXoEnBVFV3UC3xIE9rdb+H2E0zit2B5m1ug4oLh1JxVd85TGobK+XQn7jr/ MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: d1e044e9-10f2-48ad-a524-08d76d4ebb3b X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Nov 2019 00:15:21.5357 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: Q28rxqSnHgYuELVg4Nu19fNDb4z+A61BOqlNko0slHzGrTdRm06l2XhH4vH+IEd/zeIU9bN9S1TSXjWevGEVYA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5378 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Once every napi poll cycle, check if numa node is different than the page pool's numa id, and update it using page_pool_update_nid(). Alternatively, we could have registered an irq affinity change handler, but page_pool_update_nid() must be called from napi context anyways, so the handler won't actually help. Performance testing: XDP drop/tx rate and TCP single/multi stream, on mlx5 driver while migrating rx ring irq from close to far numa: mlx5 internal page cache was locally disabled to get pure page pool results. CPU: Intel(R) Xeon(R) CPU E5-2603 v4 @ 1.70GHz NIC: Mellanox Technologies MT27700 Family [ConnectX-4] (100G) XDP Drop/TX single core: NUMA | XDP | Before | After --------------------------------------- Close | Drop | 11 Mpps | 10.9 Mpps Far | Drop | 4.4 Mpps | 5.8 Mpps Close | TX | 6.5 Mpps | 6.5 Mpps Far | TX | 3.5 Mpps | 4 Mpps Improvement is about 30% drop packet rate, 15% tx packet rate for numa far test. No degradation for numa close tests. TCP single/multi cpu/stream: NUMA | #cpu | Before | After -------------------------------------- Close | 1 | 18 Gbps | 18 Gbps Far | 1 | 15 Gbps | 18 Gbps Close | 12 | 80 Gbps | 80 Gbps Far | 12 | 68 Gbps | 80 Gbps In all test cases we see improvement for the far numa case, and no impact on the close numa case. Signed-off-by: Saeed Mahameed Acked-by: Jonathan Lemon Acked-by: Jesper Dangaard Brouer --- drivers/net/ethernet/mellanox/mlx5/core/en_rx.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 82cffb3a9964..9e9960146e5b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -1386,6 +1386,9 @@ int mlx5e_poll_rx_cq(struct mlx5e_cq *cq, int budget) if (unlikely(!test_bit(MLX5E_RQ_STATE_ENABLED, &rq->state))) return 0; + if (rq->page_pool) + page_pool_nid_changed(rq->page_pool, numa_mem_id()); + if (rq->cqd.left) { work_done += mlx5e_decompress_cqes_cont(rq, cqwq, 0, budget); if (rq->cqd.left || work_done >= budget)