From patchwork Thu May 25 08:49:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chen Yu X-Patchwork-Id: 766896 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3wYNGq5JwZz9s0m for ; Thu, 25 May 2017 18:49:07 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757049AbdEYIsq (ORCPT ); Thu, 25 May 2017 04:48:46 -0400 Received: from mga06.intel.com ([134.134.136.31]:46510 "EHLO mga06.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757030AbdEYIsm (ORCPT ); Thu, 25 May 2017 04:48:42 -0400 Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga104.jf.intel.com with ESMTP; 25 May 2017 01:48:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.38,390,1491289200"; d="scan'208";a="1134593436" Received: from yu-desktop-1.sh.intel.com ([10.239.14.123]) by orsmga001.jf.intel.com with ESMTP; 25 May 2017 01:48:38 -0700 From: Chen Yu To: Yu Chen Cc: "Rafael J . Wysocki" , Bjorn Helgaas , Len Brown , Dan Williams , Rui Zhang , Ying Huang , linux-pci@vger.kernel.org, linux-pm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH] PCI / PM: Restore the status of PCI devices across hibernation Date: Thu, 25 May 2017 16:49:07 +0800 Message-Id: <1495702147-31427-1-git-send-email-yu.c.chen@intel.com> X-Mailer: git-send-email 2.7.4 Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org Currently we saw a lot of "No irq handler" errors during hibernation, which caused the system hang finally: [ 710.141581] ata4.00: qc timeout (cmd 0xec) [ 710.147135] ata4.00: failed to IDENTIFY (I/O error, err_mask=0x4) [ 710.154593] ata4.00: revalidation failed (errno=-5) [ 710.468124] ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [ 710.477746] do_IRQ: 31.151 No irq handler for vector According to above logs, there is an interrupt triggered and it is dispatched to CPU31 with a vector number 151, but there is no handler for it, thus this irq will not get acked and caused irq flood which kill the system. To be more specific, the 31.151 is an interrupt from the ahci host controller. After some investigation, the reason why this issue is triggered is because the thaw_noirq() function does not restore the MSI/MSIX settings across hibernation. The scenario is illustrated below: 1. Before the hibernation starts, the irq 34 is the handler for the ahci device, which is binded on cpu31. 2. Hibernation starts, the ahci device is put into low power state. 3. All the nonboot CPUs are put offline, so the irq 34 has to be migrated to the last alive one - CPU0. 4. After the snapshot has been created, all the nonboot CPUs are brought up again, the CPU affinity for IRQ 34 remains to be 0. 5. ahci device are put into D0. 6. The snapshot is written to the disk. The issue is triggered in step 6, in theory the ahci interrupt should be delivered to CPU0, however the actually result is that this interrupt is delivered to the original CPU31 instead, which cause the "No irq handler" issue. Ying Huang has has provided a clue that, in step 3 it is possible that the writing to the register might not take effect as the PCI devices have been put suspended. Actually it is true: In step 3, the irq 34 affinity is supposed to be modified from 31 to 0, but actually it did not. In __pci_write_msi_msg(), if the device is already in low power state, the low level msi message entry will not be updated but cached. So in theory during the device restore process, the cached msi modification information should be written back to the hardware, and this is what pci_restore_msi_state() do during normal suspend-resume. But this is not the case for hibernation, pci_restore_msi_state() is not invoked currently, to be more specific, pci_restore_state() is not invoked in pci_pm_thaw_noirq(), although pci_save_state() has saved the necessary pci cached information in pci_pm_freeze_noirq(). This patch tries to restore the pci status for the device during hibernation, otherwise the status might be lost across hibernation(for example, the MSI/MSIX message settings), which might cause problems during hibernation. Suggested-by: Ying Huang Suggested-by: Rafael J. Wysocki Cc: Rafael J. Wysocki Cc: Bjorn Helgaas Cc: Len Brown Cc: Dan Williams Cc: Rui Zhang Cc: Ying Huang Cc: linux-pci@vger.kernel.org Cc: linux-pm@vger.kernel.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Chen Yu Reviewed-by: Rafael J. Wysocki --- drivers/pci/pci-driver.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index 192e7b6..b399fa3 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -964,6 +964,7 @@ static int pci_pm_thaw_noirq(struct device *dev) return pci_legacy_resume_early(dev); pci_update_current_state(pci_dev, PCI_D0); + pci_restore_state(pci_dev); if (drv && drv->pm && drv->pm->thaw_noirq) error = drv->pm->thaw_noirq(dev);