From patchwork Mon May 9 15:50:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Gardner X-Patchwork-Id: 1628712 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=wSDUO8tr; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Kxlxc4QhRz9sGF for ; Tue, 10 May 2022 01:50:28 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1no5eQ-0005wg-RT; Mon, 09 May 2022 15:50:22 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1no5eP-0005vy-Ob for kernel-team@lists.ubuntu.com; Mon, 09 May 2022 15:50:21 +0000 Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 5D0783F6F1 for ; Mon, 9 May 2022 15:50:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1652111421; bh=J4GRtOV+xriPLBx6039HNP6kVHIO0NpyP0enqxYRWaE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=wSDUO8tr1kBrpqiKUfDlpWV9or9bBRuyfh3o2Gx6Xk/sG7o8fTSylTFBtVRJJSBBq QuHUWrW6I1ZGbF2su1ymC8QoEkwCAW5RSsdyynCanKSMeuDYraYu5fBHjA8rJB6uEt 0i1t6BtYSjwqj6T/useUhA6tK2wjf9oC6qD4lNFBcUmncp6ehDNRZ8uXlM4dGZU+/S LaTFe6Gvb/apgdkQZG8LHBF4g7K3zdPsk88rdIgk2zMvucOQYnlcZXeuZCUgF91Sa4 evBdITJfjiThHohr01JWLsDdm3tlvS+zWugC5sNo94CGU7x8C4JAjFMdIvD+8DRO+y OeelICF1uxwFw== Received: by mail-pf1-f200.google.com with SMTP id b184-20020a62cfc1000000b0050d209cb8dcso4919988pfg.3 for ; Mon, 09 May 2022 08:50:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J4GRtOV+xriPLBx6039HNP6kVHIO0NpyP0enqxYRWaE=; b=5IoJMIOWRwOXoBkbGWpXCCQ68v/l9S7vdRJO3J3D4lgFrSQymRQ7L0wqX6fcHnGihc UpeewiOUR8cmjNAkGL5/xQWGi2P/MTH3riToYyrUOPA6z6nEB8d46bt9b7YGrlx5FqxF LG4/TIsKP9OzX/Cd5nQfkF2npBNAo1HQ6xz6N/aI+UK9t5omljEsypQ6OhA8Afc0uMzx JitK5Hchd0KzrPeKNQDs46uq6CKDB7upt1F0o5j1EZ9XgybaZeMcj4laJXOrndZTnPEH yPo4uG4556JSmD00cv5qkYwJ6yql8OwRAQClX1DEqeC4HV2+YfNlX2fpxR7VmhMUgd1d 8/cA== X-Gm-Message-State: AOAM533A7WhHqhFsQROtF81m5ZBkkxIu29q7Jr+9fWRheAefTfQjA/Yq 4Ozx9OwZVSFsRIB96A/mic4tC/ilWWCd4qNGkk9Vp/lgszZIw1a/K+s6LAH4OJIEFP2mIQYxT5/ ILOopykUtx969Odb03VihAlnXWgsKeSzqH5ULeDFdAw== X-Received: by 2002:a17:90a:dd46:b0:1b8:8:7303 with SMTP id u6-20020a17090add4600b001b800087303mr26870516pjv.197.1652111419442; Mon, 09 May 2022 08:50:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw+hsBw8JmwJm3ySJDTUoGzHc1FSBsCQhoybKwZ6S9VVLj9Jcr32Uma5bDtDhOHA3AGMHCSdQ== X-Received: by 2002:a17:90a:dd46:b0:1b8:8:7303 with SMTP id u6-20020a17090add4600b001b800087303mr26870498pjv.197.1652111419206; Mon, 09 May 2022 08:50:19 -0700 (PDT) Received: from localhost.localdomain ([69.163.84.166]) by smtp.gmail.com with ESMTPSA id cd14-20020a056a00420e00b0050dc7628179sm8717462pfb.83.2022.05.09.08.50.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 09 May 2022 08:50:18 -0700 (PDT) From: Tim Gardner To: kernel-team@lists.ubuntu.com Subject: [PATCH] PCI: hv: Do not set PCI_COMMAND_MEMORY to reduce VM boot time Date: Mon, 9 May 2022 09:50:08 -0600 Message-Id: <20220509155008.10017-2-tim.gardner@canonical.com> X-Mailer: git-send-email 2.36.0 In-Reply-To: <20220509155008.10017-1-tim.gardner@canonical.com> References: <20220509155008.10017-1-tim.gardner@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Dexuan Cui BugLink: https://bugs.launchpad.net/bugs/1972662 Currently when the pci-hyperv driver finishes probing and initializing the PCI device, it sets the PCI_COMMAND_MEMORY bit; later when the PCI device is registered to the core PCI subsystem, the core PCI driver's BAR detection and initialization code toggles the bit multiple times, and each toggling of the bit causes the hypervisor to unmap/map the virtual BARs from/to the physical BARs, which can be slow if the BAR sizes are huge, e.g., a Linux VM with 14 GPU devices has to spend more than 3 minutes on BAR detection and initialization, causing a long boot time. Reduce the boot time by not setting the PCI_COMMAND_MEMORY bit when we register the PCI device (there is no need to have it set in the first place). The bit stays off till the PCI device driver calls pci_enable_device(). With this change, the boot time of such a 14-GPU VM is reduced by almost 3 minutes. Link: https://lore.kernel.org/lkml/20220419220007.26550-1-decui@microsoft.com/ Tested-by: Boqun Feng (Microsoft) Signed-off-by: Dexuan Cui Reviewed-by: Michael Kelley Acked-by: Lorenzo Pieralisi Cc: Jake Oshins Link: https://lore.kernel.org/r/20220502074255.16901-1-decui@microsoft.com Signed-off-by: Wei Liu (cherry picked from commit 23e118a48acf7be223e57d98e98da8ac5a4071ac linux-next) Signed-off-by: Tim Gardner --- drivers/pci/controller/pci-hyperv.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c index 47fa343b1fd6..bd6182d9b970 100644 --- a/drivers/pci/controller/pci-hyperv.c +++ b/drivers/pci/controller/pci-hyperv.c @@ -2080,12 +2080,17 @@ static void prepopulate_bars(struct hv_pcibus_device *hbus) } } if (high_size <= 1 && low_size <= 1) { - /* Set the memory enable bit. */ - _hv_pcifront_read_config(hpdev, PCI_COMMAND, 2, - &command); - command |= PCI_COMMAND_MEMORY; - _hv_pcifront_write_config(hpdev, PCI_COMMAND, 2, - command); + /* + * No need to set the PCI_COMMAND_MEMORY bit as + * the core PCI driver doesn't require the bit + * to be pre-set. Actually here we intentionally + * keep the bit off so that the PCI BAR probing + * in the core PCI driver doesn't cause Hyper-V + * to unnecessarily unmap/map the virtual BARs + * from/to the physical BARs multiple times. + * This reduces the VM boot time significantly + * if the BAR sizes are huge. + */ break; } }