From patchwork Thu Apr 10 08:25:31 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Li Zhong X-Patchwork-Id: 338043 X-Patchwork-Delegate: benh@kernel.crashing.org Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from ozlabs.org (localhost [127.0.0.1]) by ozlabs.org (Postfix) with ESMTP id 9ABB4140149 for ; Thu, 10 Apr 2014 18:26:12 +1000 (EST) Received: from e28smtp05.in.ibm.com (e28smtp05.in.ibm.com [122.248.162.5]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 527C914008D for ; Thu, 10 Apr 2014 18:25:40 +1000 (EST) Received: from /spool/local by e28smtp05.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 10 Apr 2014 13:55:37 +0530 Received: from d28dlp03.in.ibm.com (9.184.220.128) by e28smtp05.in.ibm.com (192.168.1.135) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 10 Apr 2014 13:55:35 +0530 Received: from d28relay01.in.ibm.com (d28relay01.in.ibm.com [9.184.220.58]) by d28dlp03.in.ibm.com (Postfix) with ESMTP id E1F421258048 for ; Thu, 10 Apr 2014 13:58:09 +0530 (IST) Received: from d28av02.in.ibm.com (d28av02.in.ibm.com [9.184.220.64]) by d28relay01.in.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id s3A8PQHF4325878 for ; Thu, 10 Apr 2014 13:55:27 +0530 Received: from d28av02.in.ibm.com (localhost [127.0.0.1]) by d28av02.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id s3A8PXjm012026 for ; Thu, 10 Apr 2014 13:55:34 +0530 Received: from [9.111.29.30] (thinkpad-t5421.cn.ibm.com [9.111.29.30]) by d28av02.in.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id s3A8PV2a011914 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Thu, 10 Apr 2014 13:55:32 +0530 Message-ID: <1397118331.25199.89.camel@ThinkPad-T5421.cn.ibm.com> Subject: [RFC PATCH v2 powerpc] Protect remove_memory() with device hotplug lock From: Li Zhong To: Nathan Fontenot Date: Thu, 10 Apr 2014 16:25:31 +0800 In-Reply-To: <534571D0.2030700@linux.vnet.ibm.com> References: <1397033686.25199.33.camel@ThinkPad-T5421.cn.ibm.com> <534571D0.2030700@linux.vnet.ibm.com> X-Mailer: Evolution 3.2.3-0ubuntu6 Mime-Version: 1.0 X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 14041008-8256-0000-0000-00000C7FCDA0 Cc: PowerPC email list , Paul Mackerras X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" While testing memory hot-remove, I found following dead lock: Process #1141 is drmgr, trying to remove some memory, i.e. memory499. It holds the memory_hotplug_mutex, and blocks when trying to remove file "online" under dir memory499, in kernfs_drain(), at wait_event(root->deactivate_waitq, atomic_read(&kn->active) == KN_DEACTIVATED_BIAS); Process #1120 is trying to online memory499 by echo 1 > memory499/online In .kernfs_fop_write, it uses kernfs_get_active() to increase &kn->active, thus blocking process #1141. While itself is blocked later when trying to acquire memory_hotplug_mutex, which is held by process #1141. The backtrace of both processes are shown below: # cat /proc/1120/stack [] 0xc000000001b18600 [] .__switch_to+0x144/0x200 [] .online_pages+0x74/0x7b0 [] .memory_subsys_online+0x9c/0x150 [] .device_online+0xb8/0x120 [] .online_store+0xb4/0xc0 [] .dev_attr_store+0x64/0xa0 [] .sysfs_kf_write+0x7c/0xb0 [] .kernfs_fop_write+0x154/0x1e0 [] .vfs_write+0xe0/0x260 [] .SyS_write+0x64/0x110 [] syscall_exit+0x0/0x7c # cat /proc/1141/stack [] 0xc000000001b18600 [] .__switch_to+0x144/0x200 [] .__kernfs_remove+0x204/0x300 [] .kernfs_remove_by_name_ns+0x68/0xf0 [] .sysfs_remove_file_ns+0x38/0x60 [] .device_remove_attrs+0x54/0xc0 [] .device_del+0x158/0x250 [] .device_unregister+0x34/0xa0 [] .unregister_memory_section+0x164/0x170 [] .__remove_pages+0x108/0x4c0 [] .arch_remove_memory+0x60/0xc0 [] .remove_memory+0x8c/0xe0 [] .pseries_remove_memblock+0xd4/0x160 [] .pseries_memory_notifier+0x27c/0x290 [] .notifier_call_chain+0x8c/0x100 [] .__blocking_notifier_call_chain+0x6c/0xe0 [] .of_property_notify+0x7c/0xc0 [] .of_update_property+0x3c/0x1b0 [] .ofdt_write+0x3dc/0x740 [] .proc_reg_write+0xac/0x110 [] .vfs_write+0xe0/0x260 [] .SyS_write+0x64/0x110 [] syscall_exit+0x0/0x7c This patch uses lock_device_hotplug() to protect remove_memory() called in pseries_remove_memblock(), which is also stated before function remove_memory(): * NOTE: The caller must call lock_device_hotplug() to serialize hotplug * and online/offline operations before this call, as required by * try_offline_node(). */ void __ref remove_memory(int nid, u64 start, u64 size) With this lock held, the other process(#1120 above) trying to online the memory block will retry the system call when calling lock_device_hotplug_sysfs(), and finally find No such device error. Signed-off-by: Li Zhong --- arch/powerpc/platforms/pseries/hotplug-memory.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c b/arch/powerpc/platforms/pseries/hotplug-memory.c index 573b488..7f75c94 100644 --- a/arch/powerpc/platforms/pseries/hotplug-memory.c +++ b/arch/powerpc/platforms/pseries/hotplug-memory.c @@ -100,10 +100,10 @@ static int pseries_remove_memblock(unsigned long base, unsigned int memblock_siz start_pfn = base >> PAGE_SHIFT; - if (!pfn_valid(start_pfn)) { - memblock_remove(base, memblock_size); - return 0; - } + lock_device_hotplug(); + + if (!pfn_valid(start_pfn)) + goto out; block_sz = memory_block_size_bytes(); sections_per_block = block_sz / MIN_MEMORY_BLOCK_SIZE; @@ -114,8 +114,10 @@ static int pseries_remove_memblock(unsigned long base, unsigned int memblock_siz base += MIN_MEMORY_BLOCK_SIZE; } +out: /* Update memory regions for memory remove */ memblock_remove(base, memblock_size); + unlock_device_hotplug(); return 0; }