From patchwork Fri Jun 22 11:22:33 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gersner X-Patchwork-Id: 933285 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ZkmlD5k0"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BxRy4yHZz9s3C for ; Fri, 22 Jun 2018 21:39:26 +1000 (AEST) Received: from localhost ([::1]:32902 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKPU-0002bC-6e for incoming@patchwork.ozlabs.org; Fri, 22 Jun 2018 07:39:24 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58687) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKAO-0007Wf-1W for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWKAM-0005a1-6T for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:48 -0400 Received: from mail-wr0-x241.google.com ([2a00:1450:400c:c0c::241]:35589) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fWKAE-0005Wb-G9; Fri, 22 Jun 2018 07:23:38 -0400 Received: by mail-wr0-x241.google.com with SMTP id y12-v6so1362933wrs.2; Fri, 22 Jun 2018 04:23:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=5+jc6NgjAfw/po2D+5g8kiEqCncQlejbIVO/1oTyEIo=; b=ZkmlD5k0FnBhbnOQ5VXYo00AJ+0en2geMD9Rps98NKtag/WfkozModltxotvo0F1Xp w0oMixIJJkSbf6cJ52FYsqiMoSVdkXBwzQjEFwMbVQDkBFNi5Ov4LHv45B2gq0Skk5bm W8Kt2ItgsL6Mim57nEPAMOLdIgZduQ+ID9L+rC1oS7VEZEPassK6U/lCC6tqhl2Su5aU szNe3HcYD4DvNmkOwZzT7WBWc60Nj3LErEsdhzm+K4I9Dcd0maEgh85gYTv50CfKLtqo oTG9JklYbeo6pNWsgNzPJIuxm1neThwVnrjeaXmmFy4BXD/u89q7faGhScAHkpBj3dl6 y6iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=5+jc6NgjAfw/po2D+5g8kiEqCncQlejbIVO/1oTyEIo=; b=N3jAoLDsLGdVEIUYPlQW+6oLthV/hKhE6PPX+RqcaRBU22berPsNOQf+V7QbOEgvti 42fD8djGUu0ve7tYQTpmA9gwFCDBzRGiGVneHD+K3dBpAybs3V79UUiQI0vzfIvod44E z6a9TJv5AOwXj52ajgOVEY4HmUmIJPrz5MVNXs1zc/BaTw2QEhBgSxumf73UtqBwdScI K9hzTsOUOjbcLCBq262IOOKhOkw+HPLYKoWzMqSu7cuW+o+mI9axfhpVku8Mgxhj7Qm2 AXdjCDkMC79qb0MamGdC/hS2QocPYmOB4l2DXIg7b7B2Zfdl3hePQgmezWgQMztwlcQU i6XQ== X-Gm-Message-State: APt69E1uaQ5yCDe3WLEtX5xc0mzyopIxI/obmsuuYV+Ja84cQ35yGsG4 g3dnPOg/k8Tmzi2/cUolWChye1Glsrg= X-Google-Smtp-Source: AAOMgpdzf8qqHMtYCDCYxEhROZP/PJ7Gsjx8RUL6fu8htiYYZuBkzkT3oX7fFjo9Bb5EE45NZ9yE/g== X-Received: by 2002:adf:dc52:: with SMTP id m18-v6mr1336944wrj.84.1529666617185; Fri, 22 Jun 2018 04:23:37 -0700 (PDT) Received: from localhost.localdomain (bzq-109-64-22-141.red.bezeqint.net. [109.64.22.141]) by smtp.gmail.com with ESMTPSA id e81-v6sm1758833wmi.28.2018.06.22.04.23.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Jun 2018 04:23:36 -0700 (PDT) From: Shimi Gersner To: qemu-block@nongnu.org, qemu-devel@nongnu.org Date: Fri, 22 Jun 2018 11:22:33 +0000 Message-Id: <20180622112237.2131-1-gersner@gmail.com> X-Mailer: git-send-email 2.17.1 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::241 Subject: [Qemu-devel] [PATCH 1/5] nvme: PCI/e configuration from specification X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Keith Busch , Kevin Wolf , David Sariel , Shimi Gersner , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" PCI/e configuration currently does not meets specifications. Patch includes various configuration changes to support specifications - BAR2 to return zero when read and CMD.IOSE is not set. - Expose NVME configuration through IO space (Optional). - PCI Power Management v1.2. - PCIe Function Level Reset. - Disable QEMUs default use of PCIe Link Status (DLLLA). - PCIe missing AOC compliance flag. - Mask PCIe End-to-End TLP as RO (Unspecified by specification). Change-Id: I9057f56266db16b013fa194730c998d4779cede3 Signed-off-by: Shimi Gersner --- hw/block/nvme.c | 59 +++++++++++++++++++++++++++++++++++++-- include/hw/pci/pci_regs.h | 1 + 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index d5bf95b79b..9d5414c80f 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -25,6 +25,7 @@ */ #include "qemu/osdep.h" +#include "qemu/range.h" #include "hw/block/block.h" #include "hw/hw.h" #include "hw/pci/msix.h" @@ -39,6 +40,9 @@ #include "trace.h" #include "nvme.h" +#define PCI_EXP_LNKCAP_AOC 0x00400000 /* ASPM Optionality Compliance (AOC) */ +#define PCI_EXP_DEVCAP2_CTDS 0x10 /* Completion Timeout Disable Supported (CTDS) */ + #define NVME_GUEST_ERR(trace, fmt, ...) \ do { \ (trace_##trace)(__VA_ARGS__); \ @@ -1195,14 +1199,31 @@ static const MemoryRegionOps nvme_cmb_ops = { }, }; +static uint32_t nvme_pci_read_config(PCIDevice *pci_dev, uint32_t addr, int len) +{ + uint32_t val; /* Value to be returned */ + + val = pci_default_read_config(pci_dev, addr, len); + if (ranges_overlap(addr, len, PCI_BASE_ADDRESS_2, 4) && (!(pci_dev->config[PCI_COMMAND] & PCI_COMMAND_IO))) { + /* When CMD.IOSE is not set */ + val = 0 ; + } + + return val; +} + static void nvme_realize(PCIDevice *pci_dev, Error **errp) { + static const uint16_t nvme_pm_offset = 0x80; + static const uint16_t nvme_pcie_offset = nvme_pm_offset + PCI_PM_SIZEOF; + NvmeCtrl *n = NVME(pci_dev); NvmeIdCtrl *id = &n->id_ctrl; int i; int64_t bs_size; uint8_t *pci_conf; + uint8_t *pci_wmask; if (!n->conf.blk) { error_setg(errp, "drive property not set"); @@ -1226,14 +1247,43 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp) } pci_conf = pci_dev->config; + pci_wmask = pci_dev->wmask; pci_conf[PCI_INTERRUPT_PIN] = 1; pci_config_set_prog_interface(pci_dev->config, 0x2); pci_config_set_class(pci_dev->config, PCI_CLASS_STORAGE_EXPRESS); - pcie_endpoint_cap_init(&n->parent_obj, 0x80); + + // Configure the PMC capability + (void)pci_add_capability(pci_dev, PCI_CAP_ID_PM, nvme_pm_offset, PCI_PM_SIZEOF, errp); + if (NULL != *errp) { + return; + } + + // - PCI Power Management v1.2, No PME support, No Soft Reset, Make state writeable + pci_set_word(pci_conf + nvme_pm_offset + PCI_PM_PMC, PCI_PM_CAP_VER_1_2); + pci_set_word(pci_conf + nvme_pm_offset + PCI_PM_CTRL, PCI_PM_CTRL_NO_SOFT_RESET); + pci_set_word(pci_wmask + nvme_pm_offset + PCI_PM_CTRL, PCI_PM_CTRL_STATE_MASK); + + // Disable QEMU default QEMU_PCIE_LNKSTA_DLLLA to disabled active flag in the Link Status Register of PCIE + pci_dev->cap_present &= ~(QEMU_PCIE_LNKSTA_DLLLA); + + // PCIE Capability + pcie_endpoint_cap_init(&n->parent_obj, nvme_pcie_offset); + + // PCIE Function Level Reset (FLRC) as required by 1.2 spec + pcie_cap_flr_init(&n->parent_obj); + + // PCIE Configured with L0s by default by QEMU, configure missing AOC flag required by compliance + pci_long_test_and_set_mask(pci_conf + pci_dev->exp.exp_cap + PCI_EXP_LNKCAP, PCI_EXP_LNKCAP_AOC); + + // Compliance requires Completion Timeout Disable Supported (CTDS). + pci_long_test_and_set_mask(pci_conf + pci_dev->exp.exp_cap + PCI_EXP_DEVCAP2, PCI_EXP_DEVCAP2_CTDS); + + // Make the End-End TLP Prefix readonly as NVME spec doesnt acknowledge this field + pci_word_test_and_clear_mask(pci_wmask + pci_dev->exp.exp_cap + PCI_EXP_DEVCTL2, PCI_EXP_DEVCTL2_EETLPPB); n->num_namespaces = 1; n->num_queues = 64; - n->reg_size = pow2ceil(0x1004 + 2 * (n->num_queues + 1) * 4); + n->reg_size = pow2ceil(0x2000 + 2 * (n->num_queues + 1) * 4); n->ns_size = bs_size / (uint64_t)n->num_namespaces; n->namespaces = g_new0(NvmeNamespace, n->num_namespaces); @@ -1245,6 +1295,10 @@ static void nvme_realize(PCIDevice *pci_dev, Error **errp) pci_register_bar(&n->parent_obj, 0, PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, &n->iomem); + + // Expose the NVME memory through Address Space IO (Optional by spec) + pci_register_bar(&n->parent_obj, 2, PCI_BASE_ADDRESS_SPACE_IO, &n->iomem); + msix_init_exclusive_bar(&n->parent_obj, n->num_queues, 4, NULL); id->vid = cpu_to_le16(pci_get_word(pci_conf + PCI_VENDOR_ID)); @@ -1355,6 +1409,7 @@ static void nvme_class_init(ObjectClass *oc, void *data) PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc); pc->realize = nvme_realize; + pc->config_read = nvme_pci_read_config; pc->exit = nvme_exit; pc->class_id = PCI_CLASS_STORAGE_EXPRESS; pc->vendor_id = PCI_VENDOR_ID_INTEL; diff --git a/include/hw/pci/pci_regs.h b/include/hw/pci/pci_regs.h index 7a83142578..54dc918f54 100644 --- a/include/hw/pci/pci_regs.h +++ b/include/hw/pci/pci_regs.h @@ -1,3 +1,4 @@ #include "standard-headers/linux/pci_regs.h" #define PCI_PM_CAP_VER_1_1 0x0002 /* PCI PM spec ver. 1.1 */ +#define PCI_PM_CAP_VER_1_2 0x0003 /* PCI PM spec ver. 1.2 */ From patchwork Fri Jun 22 11:22:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gersner X-Patchwork-Id: 933279 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="kVclkpYh"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BxK409CCz9s2L for ; Fri, 22 Jun 2018 21:33:28 +1000 (AEST) Received: from localhost ([::1]:32827 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKJd-0006EW-Mk for incoming@patchwork.ozlabs.org; Fri, 22 Jun 2018 07:33:21 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58678) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKAN-0007Vw-CB for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWKAL-0005ZW-R3 for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:47 -0400 Received: from mail-wr0-x244.google.com ([2a00:1450:400c:c0c::244]:34288) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fWKAF-0005Wp-Oi; Fri, 22 Jun 2018 07:23:39 -0400 Received: by mail-wr0-x244.google.com with SMTP id a12-v6so6371774wro.1; Fri, 22 Jun 2018 04:23:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pxObkUa+AeuKXsmIIwf3LGRtxb+bgORhcIvR5buH7Fs=; b=kVclkpYhPC0kiIpuyHQ7vL33aZkAer+suesLN+Uld2vyGbBdt5pMiH+W4EdB9fjG2l XJC2SrGBQc4+rNfL9DgLj38zuhl90BhSCq/pckZ6tEfBtl+tLxm0iLSGEAKmkMlInkWX 51MM6Kt8EH6KzXirvghwYi+8KB2dn3NvR4RsLgVMOkpbjQKEi+fpqRW4PXQU9cgbV0Lw 8bvt6ZC2qPygQk7ywJghA702+zj+U54OZAaQeuqbIGC5tb6nQFYCZkxAI1PTsvLsN0wQ gJm1fpzOa5ORINOktsWE3sfwE0de+ukKdBhby76HJUETPqUW3jMANEXUbyZ3BVHn/57m 5c7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pxObkUa+AeuKXsmIIwf3LGRtxb+bgORhcIvR5buH7Fs=; b=IuWBGUgfhcRRlA1pRfqS/8zJYgeupj+3VVq67Piqfc5giw1jVVCBMM9NekaF/+zFzd Szv2UGes129utkdZkTHejKeYiiqWwFrJD/foceURE2vetb68DZW84NupY2nRJ2Yg5sSp m5E+6NJW8VzHPvbjNZMfcobtE/3+KbJlhHbSVGjma5n7DbR5rAabzb8XmbNp2A5l0yD+ /U6+YQOYcDlqee6ebsQXiyKg8nnAnNJPOz86wRCuuGh0J3rvMKq8QeDXgGyV7ApaAyj7 eOSPRMKmYPXPDAzk9ytPn6kpDJt+VLQt1i6D/vId5zN3UYIMsajyZgql2Rpo3yqAPe7b WOVQ== X-Gm-Message-State: APt69E2tOwv9A5WhCO4/7p5Y+TB8hsY4DXRAqkiVoA7ZLpTSAtkuJoBA 91jni9wIjsIH5VXvk1OHwChSCQsF5jg= X-Google-Smtp-Source: AAOMgpcNwuCMPHUUfIZS0DF+7VFOS44DwD7jXEzYyIZ+QwMOwoZCUvJ8KTAdo+aS1SluDNvMGCAiqg== X-Received: by 2002:adf:aef3:: with SMTP id y106-v6mr1334192wrc.53.1529666618685; Fri, 22 Jun 2018 04:23:38 -0700 (PDT) Received: from localhost.localdomain (bzq-109-64-22-141.red.bezeqint.net. [109.64.22.141]) by smtp.gmail.com with ESMTPSA id e81-v6sm1758833wmi.28.2018.06.22.04.23.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Jun 2018 04:23:38 -0700 (PDT) From: Shimi Gersner To: qemu-block@nongnu.org, qemu-devel@nongnu.org Date: Fri, 22 Jun 2018 11:22:34 +0000 Message-Id: <20180622112237.2131-2-gersner@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180622112237.2131-1-gersner@gmail.com> References: <20180622112237.2131-1-gersner@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::244 Subject: [Qemu-devel] [PATCH 2/5] nvme: CQ/SQ proper validation & status code X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Keith Busch , Kevin Wolf , David Sariel , Shimi Gersner , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Device fails to properly comply CQ/SQ id validation. nvme_check_[cs]id was used for both validation of the id and to check if the id is used. Function was split and into two seperate functions and used properly on CQ/SQ creation/deletion. When id check is failed a proper error should be returned as defined by the sepecification. Additionally, CQ creation failed to properly check irq vector number. Change-Id: I3b6d8179ce567be4cd064c0be0ed69a740708096 Signed-off-by: Shimi Gersner --- hw/block/nvme.c | 40 +++++++++++++++++++++++++--------------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index 9d5414c80f..24a51d33ea 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -62,14 +62,24 @@ static void nvme_addr_read(NvmeCtrl *n, hwaddr addr, void *buf, int size) } } -static int nvme_check_sqid(NvmeCtrl *n, uint16_t sqid) +static int nvme_valid_sqid(NvmeCtrl *n, uint16_t sqid) { - return sqid < n->num_queues && n->sq[sqid] != NULL ? 0 : -1; + return sqid < n->num_queues; } -static int nvme_check_cqid(NvmeCtrl *n, uint16_t cqid) +static int nvme_used_sqid(NvmeCtrl *n, uint16_t sqid) { - return cqid < n->num_queues && n->cq[cqid] != NULL ? 0 : -1; + return sqid < n->num_queues && n->sq[sqid] != NULL ? 1 : 0; +} + +static int nvme_valid_cqid(NvmeCtrl *n, uint16_t cqid) +{ + return cqid < n->num_queues; +} + +static int nvme_used_cqid(NvmeCtrl *n, uint16_t cqid) +{ + return cqid < n->num_queues && n->cq[cqid] != NULL ? 1 : 0; } static void nvme_inc_cq_tail(NvmeCQueue *cq) @@ -433,7 +443,7 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeCmd *cmd) NvmeCQueue *cq; uint16_t qid = le16_to_cpu(c->qid); - if (unlikely(!qid || nvme_check_sqid(n, qid))) { + if (unlikely(!qid || !nvme_used_sqid(n, qid))) { trace_nvme_err_invalid_del_sq(qid); return NVME_INVALID_QID | NVME_DNR; } @@ -446,7 +456,7 @@ static uint16_t nvme_del_sq(NvmeCtrl *n, NvmeCmd *cmd) assert(req->aiocb); blk_aio_cancel(req->aiocb); } - if (!nvme_check_cqid(n, sq->cqid)) { + if (nvme_used_cqid(n, sq->cqid)) { cq = n->cq[sq->cqid]; QTAILQ_REMOVE(&cq->sq_list, sq, entry); @@ -504,11 +514,11 @@ static uint16_t nvme_create_sq(NvmeCtrl *n, NvmeCmd *cmd) trace_nvme_create_sq(prp1, sqid, cqid, qsize, qflags); - if (unlikely(!cqid || nvme_check_cqid(n, cqid))) { + if (unlikely(!cqid || !nvme_used_cqid(n, cqid))) { trace_nvme_err_invalid_create_sq_cqid(cqid); return NVME_INVALID_CQID | NVME_DNR; } - if (unlikely(!sqid || !nvme_check_sqid(n, sqid))) { + if (unlikely(!sqid || !nvme_valid_sqid(n, sqid) || nvme_used_sqid(n, sqid))) { trace_nvme_err_invalid_create_sq_sqid(sqid); return NVME_INVALID_QID | NVME_DNR; } @@ -546,9 +556,9 @@ static uint16_t nvme_del_cq(NvmeCtrl *n, NvmeCmd *cmd) NvmeCQueue *cq; uint16_t qid = le16_to_cpu(c->qid); - if (unlikely(!qid || nvme_check_cqid(n, qid))) { + if (unlikely(!qid || !nvme_used_cqid(n, qid))) { trace_nvme_err_invalid_del_cq_cqid(qid); - return NVME_INVALID_CQID | NVME_DNR; + return NVME_INVALID_QID | NVME_DNR; } cq = n->cq[qid]; @@ -592,9 +602,9 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd) trace_nvme_create_cq(prp1, cqid, vector, qsize, qflags, NVME_CQ_FLAGS_IEN(qflags) != 0); - if (unlikely(!cqid || !nvme_check_cqid(n, cqid))) { + if (unlikely(!cqid || !nvme_valid_cqid(n, cqid) || nvme_used_cqid(n, cqid))) { trace_nvme_err_invalid_create_cq_cqid(cqid); - return NVME_INVALID_CQID | NVME_DNR; + return NVME_INVALID_QID | NVME_DNR; } if (unlikely(!qsize || qsize > NVME_CAP_MQES(n->bar.cap))) { trace_nvme_err_invalid_create_cq_size(qsize); @@ -604,7 +614,7 @@ static uint16_t nvme_create_cq(NvmeCtrl *n, NvmeCmd *cmd) trace_nvme_err_invalid_create_cq_addr(prp1); return NVME_INVALID_FIELD | NVME_DNR; } - if (unlikely(vector > n->num_queues)) { + if (unlikely(vector >= n->num_queues)) { trace_nvme_err_invalid_create_cq_vector(vector); return NVME_INVALID_IRQ_VECTOR | NVME_DNR; } @@ -1091,7 +1101,7 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val) NvmeCQueue *cq; qid = (addr - (0x1000 + (1 << 2))) >> 3; - if (unlikely(nvme_check_cqid(n, qid))) { + if (unlikely(!nvme_used_cqid(n, qid))) { NVME_GUEST_ERR(nvme_ub_db_wr_invalid_cq, "completion queue doorbell write" " for nonexistent queue," @@ -1129,7 +1139,7 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val) NvmeSQueue *sq; qid = (addr - 0x1000) >> 3; - if (unlikely(nvme_check_sqid(n, qid))) { + if (unlikely(!nvme_used_sqid(n, qid))) { NVME_GUEST_ERR(nvme_ub_db_wr_invalid_sq, "submission queue doorbell write" " for nonexistent queue," From patchwork Fri Jun 22 11:22:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gersner X-Patchwork-Id: 933282 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="sEUA4+LZ"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BxPS6v47z9s2L for ; Fri, 22 Jun 2018 21:37:16 +1000 (AEST) Received: from localhost ([::1]:32865 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKNO-0000r7-J9 for incoming@patchwork.ozlabs.org; Fri, 22 Jun 2018 07:37:14 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58683) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKAN-0007WM-Nc for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWKAM-0005Zo-3A for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:47 -0400 Received: from mail-wm0-x242.google.com ([2a00:1450:400c:c09::242]:50792) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fWKAH-0005X9-83; Fri, 22 Jun 2018 07:23:41 -0400 Received: by mail-wm0-x242.google.com with SMTP id e16-v6so1909812wmd.0; Fri, 22 Jun 2018 04:23:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Uj+DXpMP1wpbvpX/i+9MA4AsoO05AZ13ThPQ6sdnlHo=; b=sEUA4+LZ+n/4zNtSytnIy9ATINW1fmcbLJH8+PLt/tHZo3m7t2MQ5U1uiv2z06BTQr linuOzVNtaVAB1UsnFvAU8EN4Y6MLgsvyEM6+mrD1KPsMAbEEW9fhEyPB8dVzmn7IuhJ x5oy1dDkijJ1Avv8tKc/qgE+FERktHq2bu7Mz7PaquG86sYq9Lt9y5NemzQp2Tc0SWF2 F7hWXEbx0IO8rxwwXbqTsshvoW0TTNGTQklsmQ9+dkjoG0pjzB6S6Gq11TmulW99ZfRK wnCf/eNmGvEGCwTOvWRI6nu3NCpggsXGmqwM6xms8EJMOlDTbDbzcmS7txQgM0u97Uh7 I3mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Uj+DXpMP1wpbvpX/i+9MA4AsoO05AZ13ThPQ6sdnlHo=; b=sXBCncCpF+9j95wpRukcAVB8fjdIOKMyVppxj3/EVDrDl5/fD0ID4pScFc9Uq2N+Ng yKmpPfKkz53gnJSiVLXnjSCdfOu7e7Ig8murjXNr53qrz1XN9EZ/cH9ZxvdmTKnKT0pt dg6NeXlcjO41SJqYCzawO94nIJvUPH/fDcZL4fuLeVz+E3/Rkj0tDAdb/UReRa5TUKKk oiyFP35WWL6HoooMhPw+IV3ZKPmz/xMCGfDdxtKqnytN+nvEqakFMXhrxrQBfvanOb+9 EIUbjuwA8jI7hzz6MTGEde8g4vVNFV0mQYXR+8kzAgTC7AZvK0wyFcY5QWA/X+UpGcKy Sq5g== X-Gm-Message-State: APt69E0wxX7csHQO9g29RlUFupzraPEW4O/E6LXq96OyZiTA71Wu/jzY m1XQlbJmJoeaOYdqYRP/0vj/JYLVhp4= X-Google-Smtp-Source: ADUXVKJEKrIs06HkTOuavqwAn4GgalbaWIdtll1N1sD92i5Jdc2924M3Y0wK217JqIY0yL34suL8vg== X-Received: by 2002:a1c:ae8b:: with SMTP id x133-v6mr1476652wme.125.1529666620158; Fri, 22 Jun 2018 04:23:40 -0700 (PDT) Received: from localhost.localdomain (bzq-109-64-22-141.red.bezeqint.net. [109.64.22.141]) by smtp.gmail.com with ESMTPSA id e81-v6sm1758833wmi.28.2018.06.22.04.23.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Jun 2018 04:23:39 -0700 (PDT) From: Shimi Gersner To: qemu-block@nongnu.org, qemu-devel@nongnu.org Date: Fri, 22 Jun 2018 11:22:35 +0000 Message-Id: <20180622112237.2131-3-gersner@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180622112237.2131-1-gersner@gmail.com> References: <20180622112237.2131-1-gersner@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c09::242 Subject: [Qemu-devel] [PATCH 3/5] nvme: Proper state handling on enable/disable X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Keith Busch , Kevin Wolf , David Sariel , Shimi Gersner , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" When device transitions from CC.EN=1 to CC.EN=0 (Controller Reset) it should properly reset its state, currently device only partially clears its state. Following actions were added to controller reset phase - Clear CC and CSTS completely - Clear and deassert irqs - On shutdown, leave state as is and only notify on shutdown completion. Change-Id: Ia0bc29775b12586c3aab2ca217603051062c3efe Signed-off-by: Shimi Gersner --- hw/block/nvme.c | 67 ++++++++++++++++++++++++++------------------ include/block/nvme.h | 5 ++++ 2 files changed, 44 insertions(+), 28 deletions(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index 24a51d33ea..206d8428fd 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -805,7 +805,7 @@ static void nvme_process_sq(void *opaque) } } -static void nvme_clear_ctrl(NvmeCtrl *n) +static void nvme_clear_ctrl(NvmeCtrl *n, bool reset) { int i; @@ -820,8 +820,18 @@ static void nvme_clear_ctrl(NvmeCtrl *n) } } + if (reset) { + // Reset clears all except for AWA, ASW, ACQ + n->bar.cc = 0; + n->bar.csts = 0; + } + + // Update the IRQ status + n->bar.intmc = n->bar.intms = 0; + n->irq_status = 0; + nvme_irq_check(n); + blk_flush(n->conf.blk); - n->bar.cc = 0; } static int nvme_start_ctrl(NvmeCtrl *n) @@ -963,16 +973,14 @@ static void nvme_write_bar(NvmeCtrl *n, hwaddr offset, uint64_t data, nvme_irq_check(n); break; case 0x14: /* CC */ - trace_nvme_mmio_cfg(data & 0xffffffff); - /* Windows first sends data, then sends enable bit */ - if (!NVME_CC_EN(data) && !NVME_CC_EN(n->bar.cc) && - !NVME_CC_SHN(data) && !NVME_CC_SHN(n->bar.cc)) - { - n->bar.cc = data; - } + trace_nvme_mmio_cfg(data & NVME_CC_WR_MASK); + + uint32_t previous_cc = n->bar.cc; + + // CC is all writeable + n->bar.cc = data & NVME_CC_WR_MASK; - if (NVME_CC_EN(data) && !NVME_CC_EN(n->bar.cc)) { - n->bar.cc = data; + if (NVME_CC_EN(data) && !NVME_CC_EN(previous_cc)) { if (unlikely(nvme_start_ctrl(n))) { trace_nvme_err_startfail(); n->bar.csts = NVME_CSTS_FAILED; @@ -980,21 +988,24 @@ static void nvme_write_bar(NvmeCtrl *n, hwaddr offset, uint64_t data, trace_nvme_mmio_start_success(); n->bar.csts = NVME_CSTS_READY; } - } else if (!NVME_CC_EN(data) && NVME_CC_EN(n->bar.cc)) { + + } else if (!NVME_CC_EN(data) && NVME_CC_EN(previous_cc)) { trace_nvme_mmio_stopped(); - nvme_clear_ctrl(n); - n->bar.csts &= ~NVME_CSTS_READY; + nvme_clear_ctrl(n, true); + } - if (NVME_CC_SHN(data) && !(NVME_CC_SHN(n->bar.cc))) { + + if (NVME_CC_SHN(data) && !(NVME_CC_SHN(previous_cc))) { trace_nvme_mmio_shutdown_set(); - nvme_clear_ctrl(n); - n->bar.cc = data; + nvme_clear_ctrl(n, false); n->bar.csts |= NVME_CSTS_SHST_COMPLETE; - } else if (!NVME_CC_SHN(data) && NVME_CC_SHN(n->bar.cc)) { + + } else if (!NVME_CC_SHN(data) && NVME_CC_SHN(previous_cc)) { trace_nvme_mmio_shutdown_cleared(); n->bar.csts &= ~NVME_CSTS_SHST_COMPLETE; - n->bar.cc = data; + } + break; case 0x1C: /* CSTS */ if (data & (1 << 4)) { @@ -1016,23 +1027,23 @@ static void nvme_write_bar(NvmeCtrl *n, hwaddr offset, uint64_t data, } break; case 0x24: /* AQA */ - n->bar.aqa = data & 0xffffffff; - trace_nvme_mmio_aqattr(data & 0xffffffff); + n->bar.aqa = data & NVME_AQA_WR_MASK; + trace_nvme_mmio_aqattr(n->bar.aqa); break; case 0x28: /* ASQ */ - n->bar.asq = data; - trace_nvme_mmio_asqaddr(data); + n->bar.asq = data & NVME_ASQ_WR_MASK; + trace_nvme_mmio_asqaddr(n->bar.asq); break; case 0x2c: /* ASQ hi */ - n->bar.asq |= data << 32; + n->bar.asq |= (data << 32) & NVME_ASQ_WR_MASK; trace_nvme_mmio_asqaddr_hi(data, n->bar.asq); break; case 0x30: /* ACQ */ - trace_nvme_mmio_acqaddr(data); - n->bar.acq = data; + n->bar.acq = data & NVME_ACQ_WR_MASK; + trace_nvme_mmio_acqaddr(n->bar.acq); break; case 0x34: /* ACQ hi */ - n->bar.acq |= data << 32; + n->bar.acq |= (data << 32) & NVME_ACQ_WR_MASK; trace_nvme_mmio_acqaddr_hi(data, n->bar.acq); break; case 0x38: /* CMBLOC */ @@ -1390,7 +1401,7 @@ static void nvme_exit(PCIDevice *pci_dev) { NvmeCtrl *n = NVME(pci_dev); - nvme_clear_ctrl(n); + nvme_clear_ctrl(n, true); g_free(n->namespaces); g_free(n->cq); g_free(n->sq); diff --git a/include/block/nvme.h b/include/block/nvme.h index 849a6f3fa3..4da6740543 100644 --- a/include/block/nvme.h +++ b/include/block/nvme.h @@ -90,6 +90,11 @@ enum NvmeCcMask { CC_IOCQES_MASK = 0xf, }; +#define NVME_CC_WR_MASK (0x00fffff1) +#define NVME_AQA_WR_MASK (0x0fff0fff) +#define NVME_ASQ_WR_MASK (0xfffffffffffff000) +#define NVME_ACQ_WR_MASK (0xfffffffffffff000) + #define NVME_CC_EN(cc) ((cc >> CC_EN_SHIFT) & CC_EN_MASK) #define NVME_CC_CSS(cc) ((cc >> CC_CSS_SHIFT) & CC_CSS_MASK) #define NVME_CC_MPS(cc) ((cc >> CC_MPS_SHIFT) & CC_MPS_MASK) From patchwork Fri Jun 22 11:22:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gersner X-Patchwork-Id: 933280 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ua0bVSyo"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41BxLW5rZ0z9s1B for ; Fri, 22 Jun 2018 21:34:43 +1000 (AEST) Received: from localhost ([::1]:32843 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKKv-00079x-Bs for incoming@patchwork.ozlabs.org; Fri, 22 Jun 2018 07:34:41 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58674) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKAN-0007Vt-Ak for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:48 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWKAL-0005Zb-Td for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:47 -0400 Received: from mail-wr0-x241.google.com ([2a00:1450:400c:c0c::241]:41045) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fWKAI-0005Xa-KE; Fri, 22 Jun 2018 07:23:42 -0400 Received: by mail-wr0-x241.google.com with SMTP id h10-v6so6336531wrq.8; Fri, 22 Jun 2018 04:23:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bCARgajCYfDTHITTvsIiJfEs0NMSqQ47fieTqivgiL8=; b=ua0bVSyoiQDR4zbQ0zUeN0A7X5JzXx5a1l+5E6enY4lrsnVVfo4nj2+cfLkN3EDTbd /l9yHpaA8VIE0SKhFczWcBylDAJ8xpUSsmX2g5W2hnQVcLEujBiPCQzXpBVFGPeHFLTN gY1eKseZvzhChQL/RhObGdI3XOx3mDSNmzyfm6G7s9RBvy71OzbHhBhfcXEgegpDBNS7 tiVIQhJEVH8lRH4XXhT/ouu3DZuWUZhN1rkfEGBRMUnUVTtePzxyeWHf5yx1mnG9U+R2 anC/LF+xHhINsVfP66FfKowcbZ6Ce9lO8DH5gPrc0wsDqxIcN0Ka3MCE9JbykIAcj5Wx sKeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bCARgajCYfDTHITTvsIiJfEs0NMSqQ47fieTqivgiL8=; b=MgCT8jvgk25OpCGTNm6TIv+800+0qfcTj6SdT3o1INlbQ6++zKa+HeCBjskNZlZdlw TSXHiBSO2zynK9k4K8vAIYUosqCASqMyOj8alip7X+lqS0/dGnlrCAWGam9Eq/wJSsHY 0v2KShD3rzFEaNUWl05YSFxpcDSIM0/KiV7Vz2cB883C1ihyM8U3w7dJJ+XBkqYyu8bY /hgfa517HslYOpcNgcEkVxwmonCuhTfNeIyRTinMT8NDKS+wwcrEjEieXPlAMTO/UFkT ZWRznzCg0uP3syYF31/wJHoADyLZ7HuMjpBH1geRhZr7RsJfgtViXp4F+CIwfrAyHMJP Vomw== X-Gm-Message-State: APt69E32ALnCPAjQ5BVGx4gnIw8RkC/I9Ln1qfycGjYLudMyQEsymNX2 Eb30Dnbcx9pRmbDtws9JQibLZguUrHY= X-Google-Smtp-Source: AAOMgpevKhj0Rd7zxLLso6SjWdctRj/Rwflx716C+x4hCHobMFODIfChk1327MQL244aRRvO2KGssQ== X-Received: by 2002:adf:b782:: with SMTP id s2-v6mr1211044wre.247.1529666621624; Fri, 22 Jun 2018 04:23:41 -0700 (PDT) Received: from localhost.localdomain (bzq-109-64-22-141.red.bezeqint.net. [109.64.22.141]) by smtp.gmail.com with ESMTPSA id e81-v6sm1758833wmi.28.2018.06.22.04.23.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Jun 2018 04:23:41 -0700 (PDT) From: Shimi Gersner To: qemu-block@nongnu.org, qemu-devel@nongnu.org Date: Fri, 22 Jun 2018 11:22:36 +0000 Message-Id: <20180622112237.2131-4-gersner@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180622112237.2131-1-gersner@gmail.com> References: <20180622112237.2131-1-gersner@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::241 Subject: [Qemu-devel] [PATCH 4/5] nvme: Fix phantom irq raise X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Keith Busch , Kevin Wolf , David Sariel , Shimi Gersner , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" CQ irq is asserted when requested are ready for the guest to collect. On edge cases, irq is raised when there is no actual request pending on the queue. Irq is raised for a CQ from nvme_post_cqes uncondtionally if there are processes requests or not. nvme_post_cqes is triggered through timer in two cases - Completion of sync or async request. - CQ was emptied by the guest and pending responses may be queued. Consider the following flow - Async request is made while CQ is full. - Guest reads entire CQ and triggers a DB. nvme_post_cqes is scheduled and executed through timer. - Async request is complete and nvme_post_cqes is scheduled and executed through timer. As nvme_post_cqes raises the irq unconditionally, it leads to a raise of the irq while no pending CQ request is ready. Issue is fixed by adding a fast path exit to nvme_post_cqes when it has no pending processing hence no need to raise the irq. Change-Id: Ib7af6c1bcb63d03022d9b57e079fdb2cf954e7dc Signed-off-by: Shimi Gersner --- hw/block/nvme.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index 206d8428fd..f639d7ae73 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -265,6 +265,11 @@ static void nvme_post_cqes(void *opaque) NvmeCtrl *n = cq->ctrl; NvmeRequest *req, *next; + // Fast path if nothing to be processed + if (QTAILQ_EMPTY(&cq->req_list)) { + return; + } + QTAILQ_FOREACH_SAFE(req, &cq->req_list, entry, next) { NvmeSQueue *sq; hwaddr addr; @@ -1130,14 +1135,21 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val) return; } + /* + CLARIFICATION + When CQ was full *before* the db write, nvme_post_cqes skipped processing of responses and + as a side effect SQ was unable to process new requests (As they are limited by size). + When CQ is cleared, spawn both processing of all CQs and SQs. As call to timer_mod + is serial, first handle the CQ to clear any pending requests and then clear the associated SQs. + */ start_sqs = nvme_cq_full(cq) ? 1 : 0; cq->head = new_head; if (start_sqs) { NvmeSQueue *sq; + timer_mod(cq->timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500); QTAILQ_FOREACH(sq, &cq->sq_list, entry) { timer_mod(sq->timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500); } - timer_mod(cq->timer, qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL) + 500); } if (cq->tail == cq->head) { From patchwork Fri Jun 22 11:22:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gersner X-Patchwork-Id: 933290 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="XmYdU+Wk"; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41Bxfx5khqz9s4V for ; Fri, 22 Jun 2018 21:48:57 +1000 (AEST) Received: from localhost ([::1]:32984 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKYg-0000ew-Ey for incoming@patchwork.ozlabs.org; Fri, 22 Jun 2018 07:48:54 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58705) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWKAO-0007XP-QV for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWKAN-0005aa-PA for qemu-devel@nongnu.org; Fri, 22 Jun 2018 07:23:48 -0400 Received: from mail-wr0-x241.google.com ([2a00:1450:400c:c0c::241]:38321) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1fWKAK-0005Xq-2D; Fri, 22 Jun 2018 07:23:44 -0400 Received: by mail-wr0-x241.google.com with SMTP id e18-v6so6357479wrs.5; Fri, 22 Jun 2018 04:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=j0tNx98jcDgo7JsdUKNiTsIOXNgYhqs7KN4HOPBLqNQ=; b=XmYdU+WkuFuUaEuyUxqIvtbd+B8qHCuyLtlLElZZAvg090juAt9le0/waWyiR+DoZY HNc6M7Dga2ELZwscPBvvKoPemDmLAkN+CCtuikllTCrbHh3c0lQ3zXvgVlI5ruB43+SH L3ymlBzNFQM3jwbu2+O1qjmPk0WvIqfcidT0r13YXkVIuxpPJzX3iBYx2N/gh7rY9LSh HlOUIdgM02wFR6TvecPQXSs2rNKfXAqqS5ZqzJGDaWIB0DKR2/LdTIitcj1g8XfHVlx1 nwgBReh00L7txrvfPM3xFHCyJ18zmo4JEzVv4zkShTNd50t7X/ogzpm6nJpoyzs/NEfj EKmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=j0tNx98jcDgo7JsdUKNiTsIOXNgYhqs7KN4HOPBLqNQ=; b=pDWBDf3x6O59KnoY1orkzaqcoy/2NVbk6G1WNUzRiZfI5Lm8DDHwHAPvJVd3PEuSCo X/mciUQgRkrXHmYpej/6+12m0X1m1plgSUrjqtXOpbQvwk5ZzZBcK20gf5JwxTUHgR+F O8jW9mX8Q66XUaG7PY9R1xdEW1olFkcCGuziRA+pT3XH2FFec8gkGdPBxRDsvzzbeQvU s477fXmiw4sdbLkAxFJ2rBAsAFgwkVK0Hk7irHFP5UkENFS5Ome+ZHQlo49Z364DjHCL w1PcT3gQ2HoDRX8HYtLqequFNNFNQ2Yqa/97KWG58O3OjVnCmIDLSX62TOOGc9BNbCrh 7ajQ== X-Gm-Message-State: APt69E1ukmVxZR0pXiHIXCbVGzpkld+oKZcoK4kh1I0Hp6KH2Xmtxl0x flBU3Dsi3T7C0IK8w7L5rwk22S8pQWE= X-Google-Smtp-Source: AAOMgpfiYP17OoVwm9TmfKvNa5/A7fDKwf1AI95Mhdrpbs5S3HzlxvjFdqhPd1i2/SUl2GhYXGVAUw== X-Received: by 2002:adf:e48e:: with SMTP id i14-v6mr1378010wrm.8.1529666623087; Fri, 22 Jun 2018 04:23:43 -0700 (PDT) Received: from localhost.localdomain (bzq-109-64-22-141.red.bezeqint.net. [109.64.22.141]) by smtp.gmail.com with ESMTPSA id e81-v6sm1758833wmi.28.2018.06.22.04.23.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 22 Jun 2018 04:23:42 -0700 (PDT) From: Shimi Gersner To: qemu-block@nongnu.org, qemu-devel@nongnu.org Date: Fri, 22 Jun 2018 11:22:37 +0000 Message-Id: <20180622112237.2131-5-gersner@gmail.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180622112237.2131-1-gersner@gmail.com> References: <20180622112237.2131-1-gersner@gmail.com> X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:400c:c0c::241 Subject: [Qemu-devel] [PATCH 5/5] nvme: Missing MSI message upon partial CQ read X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Keith Busch , Kevin Wolf , David Sariel , Shimi Gersner , Max Reitz Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" When MSIx interrupts are enabled, each CQ is associated with a vector which is expected to be called upon ready data in the CQ buffer. After guest reads from the buffer, it updates the processed buffer position using a CQ dedicated DB. Upon reading only partial data from the CQ buffer, device fails to re-send a new MSI message stating buffer still has unread data. The fix ensures a new message will be sent. Change-Id: I10190be127b8dcbd89732cfb95ea37faf8c5779b Signed-off-by: Shimi Gersner --- hw/block/nvme.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/hw/block/nvme.c b/hw/block/nvme.c index f639d7ae73..e2dca6e57f 100644 --- a/hw/block/nvme.c +++ b/hw/block/nvme.c @@ -1152,8 +1152,12 @@ static void nvme_process_db(NvmeCtrl *n, hwaddr addr, int val) } } + // When we have more messages, we should make sure irq is asserted. When MSIx is used + // this will make sure another notification is sent to the guest. if (cq->tail == cq->head) { nvme_irq_deassert(n, cq); + } else { + nvme_irq_assert(n, cq); } } else { /* Submission queue doorbell write */