[{"id":1761307,"web_url":"http://patchwork.ozlabs.org/comment/1761307/","msgid":"<26D9FDECA4FBDD4AADA65D8E2FC68A4A10EA9525@ORSMSX104.amr.corp.intel.com>","list_archive_url":null,"date":"2017-08-31T20:47:10","subject":"Re: [Intel-wired-lan] [next PATCH S79-V2 02/13] i40e/i40evf: spread\n\tCPU\taffinity hints across online CPUs only","submitter":{"id":66945,"url":"http://patchwork.ozlabs.org/api/people/66945/","name":"Bowers, AndrewX","email":"andrewx.bowers@intel.com"},"content":"> -----Original Message-----\n> From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On\n> Behalf Of Alice Michael\n> Sent: Tuesday, August 29, 2017 2:33 AM\n> To: Michael, Alice ; intel-wired-\n> lan@lists.osuosl.org\n> Subject: [Intel-wired-lan] [next PATCH S79-V2 02/13] i40e/i40evf: spread CPU\n> affinity hints across online CPUs only\n> \n> From: Jacob Keller \n> \n> Currently, when setting up the IRQ for a q_vector, we set an affinity hint\n> based on the v_idx of that q_vector. Meaning a loop iterates on v_idx, which\n> is an incremental value, and the cpumask is created based on this value.\n> \n> This is a problem in systems with multiple logical CPUs per core (like in SMT\n> scenarios). If we disable some logical CPUs, by turning SMT off for example,\n> we will end up with a sparse cpu_online_mask, i.e., only the first CPU in a\n> core is online, and incremental filling in q_vector cpumask might lead to\n> multiple offline CPUs being assigned to q_vectors.\n> \n> Example: if we have a system with 8 cores each one containing 8 logical CPUs\n> (SMT == 8 in this case), we have 64 CPUs in total. But if SMT is disabled, only\n> the 1st CPU in each core remains online, so the cpu_online_mask in this case\n> would have only 8 bits set, in a sparse way.\n> \n> In general case, when SMT is off the cpu_online_mask has only C bits set:\n> 0, 1*N, 2*N, ..., C*(N-1) where\n> C == # of cores;\n> N == # of logical CPUs per core.\n> In our example, only bits 0, 8, 16, 24, 32, 40, 48, 56 would be set.\n> \n> Instead, we should only assign hints for CPUs which are online. Even better,\n> the kernel already provides a function, cpumask_local_spread() which takes\n> an index and returns a CPU, spreading the interrupts across local NUMA\n> nodes first, and then remote ones if necessary.\n> \n> Since we generally have a 1:1 mapping between vectors and CPUs, there is\n> no real advantage to spreading vectors to local CPUs first. In order to avoid\n> mismatch of the default XPS hints, we'll pass -1 so that it spreads across all\n> CPUs without regard to the node locality.\n> \n> Note that we don't need to change the q_vector->affinity_mask as this is\n> initialized to cpu_possible_mask, until an actual affinity is set and then\n> notified back to us.\n> \n> Signed-off-by: Jacob Keller \n> ---\n> drivers/net/ethernet/intel/i40e/i40e_main.c | 16 +++++++++++-----\n> drivers/net/ethernet/intel/i40evf/i40evf_main.c | 9 ++++++---\n> 2 files changed, 17 insertions(+), 8 deletions(-)\n\nTested-by: Andrew Bowers ","headers":{"Return-Path":"","X-Original-To":["incoming@patchwork.ozlabs.org","intel-wired-lan@lists.osuosl.org"],"Delivered-To":["patchwork-incoming@bilbo.ozlabs.org","intel-wired-lan@lists.osuosl.org"],"Authentication-Results":"ozlabs.org;\n\tspf=pass (mailfrom) smtp.mailfrom=osuosl.org\n\t(client-ip=140.211.166.136; helo=silver.osuosl.org;\n\tenvelope-from=intel-wired-lan-bounces@osuosl.org;\n\treceiver=)","Received":["from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136])\n\t(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))\n\t(No client certificate requested)\n\tby ozlabs.org (Postfix) with ESMTPS id 3xjvZH5pqsz9t1t\n\tfor ;\n\tFri, 1 Sep 2017 06:47:19 +1000 (AEST)","from localhost (localhost [127.0.0.1])\n\tby silver.osuosl.org (Postfix) with ESMTP id 8241330A7A;\n\tThu, 31 Aug 2017 20:47:17 +0000 (UTC)","from silver.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id PBFSgRDoUs6B; Thu, 31 Aug 2017 20:47:16 +0000 (UTC)","from ash.osuosl.org (ash.osuosl.org [140.211.166.34])\n\tby silver.osuosl.org (Postfix) with ESMTP id 2209930A71;\n\tThu, 31 Aug 2017 20:47:16 +0000 (UTC)","from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137])\n\tby ash.osuosl.org (Postfix) with ESMTP id 18AF41BFF7A\n\tfor ;\n\tThu, 31 Aug 2017 20:47:15 +0000 (UTC)","from localhost (localhost [127.0.0.1])\n\tby fraxinus.osuosl.org (Postfix) with ESMTP id 08BA687E2C\n\tfor ;\n\tThu, 31 Aug 2017 20:47:15 +0000 (UTC)","from fraxinus.osuosl.org ([127.0.0.1])\n\tby localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024)\n\twith ESMTP id pWFoQ-yWfPNE for ;\n\tThu, 31 Aug 2017 20:47:12 +0000 (UTC)","from mga07.intel.com (mga07.intel.com [134.134.136.100])\n\tby fraxinus.osuosl.org (Postfix) with ESMTPS id 28A81864FD\n\tfor ;\n\tThu, 31 Aug 2017 20:47:12 +0000 (UTC)","from fmsmga001.fm.intel.com ([10.253.24.23])\n\tby orsmga105.jf.intel.com with ESMTP; 31 Aug 2017 13:47:11 -0700","from orsmsx102.amr.corp.intel.com ([10.22.225.129])\n\tby fmsmga001.fm.intel.com with ESMTP; 31 Aug 2017 13:47:11 -0700","from orsmsx104.amr.corp.intel.com ([169.254.4.142]) by\n\tORSMSX102.amr.corp.intel.com ([169.254.3.17]) with mapi id\n\t14.03.0319.002; Thu, 31 Aug 2017 13:47:10 -0700"],"X-Virus-Scanned":["amavisd-new at osuosl.org","amavisd-new at osuosl.org"],"X-Greylist":"domain auto-whitelisted by SQLgrey-1.7.6","X-ExtLoop1":"1","X-IronPort-AV":"E=Sophos; i=\"5.41,455,1498546800\"; d=\"scan'208\";\n\ta=\"1190342502\"","From":"\"Bowers, AndrewX\" ","To":"\"intel-wired-lan@lists.osuosl.org\" ","Thread-Topic":"[Intel-wired-lan] [next PATCH S79-V2 02/13] i40e/i40evf:\n\tspread CPU\taffinity hints across online CPUs only","Thread-Index":"AQHTIO2DI2bseGH3iEWQyHjrwKycRaKe8rEA","Date":"Thu, 31 Aug 2017 20:47:10 +0000","Message-ID":"<26D9FDECA4FBDD4AADA65D8E2FC68A4A10EA9525@ORSMSX104.amr.corp.intel.com>","References":"<20170829093242.41026-1-alice.michael@intel.com>\n\t<20170829093242.41026-2-alice.michael@intel.com>","In-Reply-To":"<20170829093242.41026-2-alice.michael@intel.com>","Accept-Language":"en-US","Content-Language":"en-US","X-MS-Has-Attach":"","X-MS-TNEF-Correlator":"","x-titus-metadata-40":"eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNTk0MTdmNmMtN2JmYS00ZWNjLWIyYjQtMDU1ZDFhMjI5YzIyIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6Ik9nYlpQbkRjWWFlMGFrd1hkRW56M1wvemhMUEJpc3o1YkxqOVBNRm1WM2hrPSJ9","x-ctpclassification":"CTP_IC","dlp-product":"dlpe-windows","dlp-version":"11.0.0.116","dlp-reaction":"no-action","x-originating-ip":"[10.22.254.140]","MIME-Version":"1.0","Subject":"Re: [Intel-wired-lan] [next PATCH S79-V2 02/13] i40e/i40evf: spread\n\tCPU\taffinity hints across online CPUs only","X-BeenThere":"intel-wired-lan@osuosl.org","X-Mailman-Version":"2.1.18-1","Precedence":"list","List-Id":"Intel Wired Ethernet Linux Kernel Driver Development\n\t","List-Unsubscribe":", \n\t","List-Archive":"","List-Post":"","List-Help":"","List-Subscribe":", \n\t","Content-Type":"text/plain; charset=\"us-ascii\"","Content-Transfer-Encoding":"7bit","Errors-To":"intel-wired-lan-bounces@osuosl.org","Sender":"\"Intel-wired-lan\" "}}]