From patchwork Mon Sep 19 18:07:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tim Gardner X-Patchwork-Id: 1679622 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=M0TUVQYz; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MWXjn392Lz1ypH for ; Tue, 20 Sep 2022 04:08:45 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1oaLC9-0001wt-Pk; Mon, 19 Sep 2022 18:08:37 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1oaLBv-0001WY-Rv for kernel-team@lists.ubuntu.com; Mon, 19 Sep 2022 18:08:24 +0000 Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id D8DED3F1BD for ; Mon, 19 Sep 2022 18:08:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1663610901; bh=J8K0dP5+BBKiHQwkCuYMbOnQOOPt95ziYmIVcXLXlW8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=M0TUVQYzSBk6NioJiRWIQiqiwL6YiW9VTniqThRG/gI8eUmHKm+dhOIGLKVVOOuJB 6kmkim7V2Vtsrd6DCyDbZXg+KRDHpgqQF8m3GfBXPE1iaFYbT8TvhpkwERrdWn6Jxw YxYW5gN7uxNylmAYiM/b/sk1Uaz49GzwQgDHUVFTOtWHcjTBvfIb0zPqQXUYwDVwjj MBld/ybbUNhq6isBYkC3amS7VP2o+hkafoxR/3X393CPmUPtvnxJ7erDvOi4xzu9qX hnysbR/88jagl30hOUVy2/2DJxa9WuWqt/L3bWZh1pZk+EzZ9uxRXBh1nM85YCSTBe Zo3V6r+Msz3ng== Received: by mail-pl1-f200.google.com with SMTP id d7-20020a170903230700b00177f6dd8472so20027043plh.6 for ; Mon, 19 Sep 2022 11:08:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=J8K0dP5+BBKiHQwkCuYMbOnQOOPt95ziYmIVcXLXlW8=; b=CIm3rAnwU7gXchgqeLbqGNKnlW/E20KWqUMEJha+FkL6KeS/1A9DBMf8zgAAY2pfvl 0yaqIupFkSWOEJnnyL1ZHEy9l2nGxw8kAyKUPWFZOKgsALG4ynjOXdZghDNLR3dDH/wo rXlKkGSIUVhdUO/AAzjqVIlZEgwpCn+ZR1LIIN9sz9RdEIKnGucQZRc2Tq8E7smKdwlm KoXFX/UCXIlPE+PtwuA0UqbVlnc1CKIVN7D/BDSyZ+abCu1h6aPDV1ZfqQEzfCfhexRu 8igXvClXw5dnhyK3wuB9lq+Hj95qXwgujKmFIYVTAoYEgOfcVQQVGFcXM70kiB05aWRL 7ECg== X-Gm-Message-State: ACrzQf0/IX4bFCJVJ+meCwWA0b5omWRwmgZvrSioULIoFlf6Q2236s5Y yGiUZx5W5eBUnYpvhxqxjjY2oJ6z9r/g3qHUQNSJ9qCAbXxGtlGUCe/M4YVNLnq0gBWB9tAJdS0 FOExH1N1IMRO7my0YylaAJvTaU+fx2AewftZ1f/Hf5Q== X-Received: by 2002:a17:90b:4c46:b0:203:7c2a:defe with SMTP id np6-20020a17090b4c4600b002037c2adefemr11271525pjb.40.1663610900135; Mon, 19 Sep 2022 11:08:20 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6e0a1I3c2LumQjyKSdFYFgM8nw0XRLC518J2PRgjDFkP86qBiggMWsYn+tst2qgpjYOua03Q== X-Received: by 2002:a17:90b:4c46:b0:203:7c2a:defe with SMTP id np6-20020a17090b4c4600b002037c2adefemr11271482pjb.40.1663610899702; Mon, 19 Sep 2022 11:08:19 -0700 (PDT) Received: from localhost.localdomain ([69.163.84.166]) by smtp.gmail.com with ESMTPSA id b3-20020a655783000000b0042bd73400b6sm1498390pgr.87.2022.09.19.11.08.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Sep 2022 11:08:18 -0700 (PDT) From: Tim Gardner To: kernel-team@lists.ubuntu.com Subject: [PATCH 12/13] PCI: hv: Fix interrupt mapping for multi-MSI Date: Mon, 19 Sep 2022 12:07:49 -0600 Message-Id: <20220919180750.5432-13-tim.gardner@canonical.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220919180750.5432-1-tim.gardner@canonical.com> References: <20220919180750.5432-1-tim.gardner@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Jeffrey Hugo BugLink: https://bugs.launchpad.net/bugs/1990184 According to Dexuan, the hypervisor folks beleive that multi-msi allocations are not correct. compose_msi_msg() will allocate multi-msi one by one. However, multi-msi is a block of related MSIs, with alignment requirements. In order for the hypervisor to allocate properly aligned and consecutive entries in the IOMMU Interrupt Remapping Table, there should be a single mapping request that requests all of the multi-msi vectors in one shot. Dexuan suggests detecting the multi-msi case and composing a single request related to the first MSI. Then for the other MSIs in the same block, use the cached information. This appears to be viable, so do it. Suggested-by: Dexuan Cui Signed-off-by: Jeffrey Hugo Reviewed-by: Dexuan Cui Tested-by: Michael Kelley Link: https://lore.kernel.org/r/1652282599-21643-1-git-send-email-quic_jhugo@quicinc.com Signed-off-by: Wei Liu (cherry picked from commit a2bad844a67b1c7740bda63e87453baf63c3a7f7) Signed-off-by: Tim Gardner --- drivers/pci/controller/pci-hyperv.c | 60 ++++++++++++++++++++++++----- 1 file changed, 50 insertions(+), 10 deletions(-) diff --git a/drivers/pci/controller/pci-hyperv.c b/drivers/pci/controller/pci-hyperv.c index 70a09287e129..80efdb4f8800 100644 --- a/drivers/pci/controller/pci-hyperv.c +++ b/drivers/pci/controller/pci-hyperv.c @@ -1519,6 +1519,10 @@ static void hv_int_desc_free(struct hv_pci_dev *hpdev, u8 buffer[sizeof(struct pci_delete_interrupt)]; } ctxt; + if (!int_desc->vector_count) { + kfree(int_desc); + return; + } memset(&ctxt, 0, sizeof(ctxt)); int_pkt = (struct pci_delete_interrupt *)&ctxt.pkt.message; int_pkt->message_type.type = @@ -1603,12 +1607,12 @@ static void hv_pci_compose_compl(void *context, struct pci_response *resp, static u32 hv_compose_msi_req_v1( struct pci_create_interrupt *int_pkt, struct cpumask *affinity, - u32 slot, u8 vector) + u32 slot, u8 vector, u8 vector_count) { int_pkt->message_type.type = PCI_CREATE_INTERRUPT_MESSAGE; int_pkt->wslot.slot = slot; int_pkt->int_desc.vector = vector; - int_pkt->int_desc.vector_count = 1; + int_pkt->int_desc.vector_count = vector_count; int_pkt->int_desc.delivery_mode = DELIVERY_MODE; /* @@ -1631,14 +1635,14 @@ static int hv_compose_msi_req_get_cpu(struct cpumask *affinity) static u32 hv_compose_msi_req_v2( struct pci_create_interrupt2 *int_pkt, struct cpumask *affinity, - u32 slot, u8 vector) + u32 slot, u8 vector, u8 vector_count) { int cpu; int_pkt->message_type.type = PCI_CREATE_INTERRUPT_MESSAGE2; int_pkt->wslot.slot = slot; int_pkt->int_desc.vector = vector; - int_pkt->int_desc.vector_count = 1; + int_pkt->int_desc.vector_count = vector_count; int_pkt->int_desc.delivery_mode = DELIVERY_MODE; cpu = hv_compose_msi_req_get_cpu(affinity); int_pkt->int_desc.processor_array[0] = @@ -1650,7 +1654,7 @@ static u32 hv_compose_msi_req_v2( static u32 hv_compose_msi_req_v3( struct pci_create_interrupt3 *int_pkt, struct cpumask *affinity, - u32 slot, u32 vector) + u32 slot, u32 vector, u8 vector_count) { int cpu; @@ -1658,7 +1662,7 @@ static u32 hv_compose_msi_req_v3( int_pkt->wslot.slot = slot; int_pkt->int_desc.vector = vector; int_pkt->int_desc.reserved = 0; - int_pkt->int_desc.vector_count = 1; + int_pkt->int_desc.vector_count = vector_count; int_pkt->int_desc.delivery_mode = DELIVERY_MODE; cpu = hv_compose_msi_req_get_cpu(affinity); int_pkt->int_desc.processor_array[0] = @@ -1689,6 +1693,8 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) struct cpumask *dest; struct compose_comp_ctxt comp; struct tran_int_desc *int_desc; + struct msi_desc *msi_desc; + u8 vector, vector_count; struct { struct pci_packet pci_pkt; union { @@ -1710,7 +1716,8 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) return; } - pdev = msi_desc_to_pci_dev(irq_data_get_msi_desc(data)); + msi_desc = irq_data_get_msi_desc(data); + pdev = msi_desc_to_pci_dev(msi_desc); dest = irq_data_get_effective_affinity_mask(data); pbus = pdev->bus; hbus = container_of(pbus->sysdata, struct hv_pcibus_device, sysdata); @@ -1723,6 +1730,36 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) if (!int_desc) goto drop_reference; + if (!msi_desc->pci.msi_attrib.is_msix && msi_desc->nvec_used > 1) { + /* + * If this is not the first MSI of Multi MSI, we already have + * a mapping. Can exit early. + */ + if (msi_desc->irq != data->irq) { + data->chip_data = int_desc; + int_desc->address = msi_desc->msg.address_lo | + (u64)msi_desc->msg.address_hi << 32; + int_desc->data = msi_desc->msg.data + + (data->irq - msi_desc->irq); + msg->address_hi = msi_desc->msg.address_hi; + msg->address_lo = msi_desc->msg.address_lo; + msg->data = int_desc->data; + put_pcichild(hpdev); + return; + } + /* + * The vector we select here is a dummy value. The correct + * value gets sent to the hypervisor in unmask(). This needs + * to be aligned with the count, and also not zero. Multi-msi + * is powers of 2 up to 32, so 32 will always work here. + */ + vector = 32; + vector_count = msi_desc->nvec_used; + } else { + vector = hv_msi_get_int_vector(data); + vector_count = 1; + } + memset(&ctxt, 0, sizeof(ctxt)); init_completion(&comp.comp_pkt.host_event); ctxt.pci_pkt.completion_func = hv_pci_compose_compl; @@ -1733,7 +1770,8 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) size = hv_compose_msi_req_v1(&ctxt.int_pkts.v1, dest, hpdev->desc.win_slot.slot, - hv_msi_get_int_vector(data)); + vector, + vector_count); break; case PCI_PROTOCOL_VERSION_1_2: @@ -1741,14 +1779,16 @@ static void hv_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) size = hv_compose_msi_req_v2(&ctxt.int_pkts.v2, dest, hpdev->desc.win_slot.slot, - hv_msi_get_int_vector(data)); + vector, + vector_count); break; case PCI_PROTOCOL_VERSION_1_4: size = hv_compose_msi_req_v3(&ctxt.int_pkts.v3, dest, hpdev->desc.win_slot.slot, - hv_msi_get_int_vector(data)); + vector, + vector_count); break; default: