From patchwork Thu May 1 17:40:55 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yinghai Lu X-Patchwork-Id: 344623 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id B70EB140106 for ; Fri, 2 May 2014 03:41:12 +1000 (EST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751203AbaEARlK (ORCPT ); Thu, 1 May 2014 13:41:10 -0400 Received: from userp1040.oracle.com ([156.151.31.81]:41093 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750922AbaEARlJ (ORCPT ); Thu, 1 May 2014 13:41:09 -0400 Received: from ucsinet21.oracle.com (ucsinet21.oracle.com [156.151.31.93]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id s41Hf5b2026202 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 1 May 2014 17:41:05 GMT Received: from userz7022.oracle.com (userz7022.oracle.com [156.151.31.86]) by ucsinet21.oracle.com (8.14.4+Sun/8.14.4) with ESMTP id s41Hf4eu025278 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 1 May 2014 17:41:04 GMT Received: from abhmp0016.oracle.com (abhmp0016.oracle.com [141.146.116.22]) by userz7022.oracle.com (8.14.5+Sun/8.14.4) with ESMTP id s41Hf4DC010509; Thu, 1 May 2014 17:41:04 GMT Received: from linux-siqj.site (/10.132.126.31) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 01 May 2014 10:41:03 -0700 From: Yinghai Lu To: Bjorn Helgaas Cc: Rajat Jain , linux-pci@vger.kernel.org, Yinghai Lu Subject: [PATCH v4] pciehp: only wait command complete for real hotplug control Date: Thu, 1 May 2014 10:40:55 -0700 Message-Id: <1398966055-19325-1-git-send-email-yinghai@kernel.org> X-Mailer: git-send-email 1.8.4.5 X-Source-IP: ucsinet21.oracle.com [156.151.31.93] Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On system with 16 PCI express hotplug slots, customer complain every slot will report "Command not completed in 1000 msec" during initialization. Intel says that we should only wait command complete only for Electromechanical Interlock Control Power Controller Control Power Indicator Control Attention Indicator Control For detail please check CF118 on http://www.intel.com/content/www/us/en/processors/xeon/xeon-e7-v2-spec-update.html We saw same problems on system with AMD/Nvidia chipsets. Two ways to address the problem: 1. change to only wait when (EIC, PCC, PIC, AIC) bits get changed. 2. or check CMD_COMPLETE and wait before write command. Only wait when CC is set and cmd_busy is true. For chipset from intel will only have CC set for real hotplug control, so we could skip the wait for others. This patch is using second way. With that we could save 16 seconds during booting, later with 32 sockets system with 64 pcie hotplug slots we could save 64 seconds. This patch address spurious "cmd completed" event problem that that Rajat met. -v2: use second way that is suggested by Bjorn. -v3: rebase after PCI: pciehp: Acknowledge spurious "cmd completed" event -v4: add comments according to Bjorn. also add URL for Intel doc about the CC set on intel chipset. Signed-off-by: Yinghai Lu Tested-by: Rajat Jain --- drivers/pci/hotplug/pciehp_hpc.c | 48 ++++++++++++++++++++++++--------------- 1 file changed, 30 insertions(+), 18 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Index: linux-2.6/drivers/pci/hotplug/pciehp_hpc.c =================================================================== --- linux-2.6.orig/drivers/pci/hotplug/pciehp_hpc.c +++ linux-2.6/drivers/pci/hotplug/pciehp_hpc.c @@ -155,20 +155,30 @@ static void pcie_write_cmd(struct contro u16 slot_status; u16 slot_ctrl; + /* + * We basically have two strategies for dealing with Command Complete. + * Both are legal according to the PCIe spec: + * A: Write command at first and wait for Command Complete is set. + * B: Wait at first iff Command Complete and cmd_busy are set, + * that means previous command is not complete yet. + * Then write command. + * + * Second way also cover "Command not completed in 1000 msec" + * Problem on system with chipsets that only set Command Complete + * for some "real" hotplug events like: + * Electromechanical Interlock Control + * Power Controller Control + * Power Indicator Control + * Attention Indicator Control + * like CF118 in + * http://www.intel.com/content/www/us/en/processors/xeon/xeon-e7-v2-spec-update.html + */ + mutex_lock(&ctrl->ctrl_lock); pcie_capability_read_word(pdev, PCI_EXP_SLTSTA, &slot_status); - if (slot_status & PCI_EXP_SLTSTA_CC) { - pcie_capability_write_word(pdev, PCI_EXP_SLTSTA, - PCI_EXP_SLTSTA_CC); - if (!ctrl->no_cmd_complete) { - /* - * After 1 sec and CMD_COMPLETED still not set, just - * proceed forward to issue the next command according - * to spec. Just print out the error message. - */ - ctrl_dbg(ctrl, "CMD_COMPLETED not clear after 1 sec\n"); - } else if (!NO_CMD_CMPL(ctrl)) { + if (ctrl->no_cmd_complete && slot_status & PCI_EXP_SLTSTA_CC) { + if (!NO_CMD_CMPL(ctrl)) { /* * This controller seems to notify of command completed * event even though it supports none of power @@ -184,16 +194,11 @@ static void pcie_write_cmd(struct contro } pcie_capability_read_word(pdev, PCI_EXP_SLTCTL, &slot_ctrl); - slot_ctrl &= ~mask; - slot_ctrl |= (cmd & mask); - ctrl->cmd_busy = 1; - smp_mb(); - pcie_capability_write_word(pdev, PCI_EXP_SLTCTL, slot_ctrl); - /* * Wait for command completion. */ - if (!ctrl->no_cmd_complete) { + if (!ctrl->no_cmd_complete && (slot_status & PCI_EXP_SLTSTA_CC) && + ctrl->cmd_busy) { int poll = 0; /* * if hotplug interrupt is not enabled or command @@ -205,6 +210,13 @@ static void pcie_write_cmd(struct contro poll = 1; pcie_wait_cmd(ctrl, poll); } + + slot_ctrl &= ~mask; + slot_ctrl |= (cmd & mask); + ctrl->cmd_busy = 1; + smp_mb(); + pcie_capability_write_word(pdev, PCI_EXP_SLTCTL, slot_ctrl); + mutex_unlock(&ctrl->ctrl_lock); }