From patchwork Fri Jun 23 04:35:06 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Brown X-Patchwork-Id: 779761 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wv5HB5Tthz9s3T for ; Fri, 23 Jun 2017 14:35:50 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ALe7cmwe"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3wv5HB4C4pzDqn6 for ; Fri, 23 Jun 2017 14:35:50 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ALe7cmwe"; dkim-atps=neutral X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from mail-pg0-x243.google.com (mail-pg0-x243.google.com [IPv6:2607:f8b0:400e:c05::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3wv5Gs398XzDqjF for ; Fri, 23 Jun 2017 14:35:32 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ALe7cmwe"; dkim-atps=neutral Received: by mail-pg0-x243.google.com with SMTP id j186so4871435pge.1 for ; Thu, 22 Jun 2017 21:35:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id; bh=v28hM+zZfD8XBeRP4jZBLaAnkv4RInatG6beIJh20OI=; b=ALe7cmweS0b+/cyBPiapFDytWC0/GP9UEpCIWpvZZcdFyV0YDL7RDqDUecbPihsnzL Y6rvV5c/Zcq1HgiqKu2E0EUEl0BkAOO+HNKiRNeZD1YcplUxJ0++dvb9RzjnDK83e2ii yW+Uakcxu1DEqT0y7nf6yvKIxD44Aj79pH3+tXJrsz5N0sCs1TQRpd0NmHVmZnQCsUrM cOEJk8ItR4aPTYYvfQwc83mQK83NUb/RzBMfh8dBOjuFGahj9sJLgJxGpw+Edzcccojp ei07+xP0Y60nqYwm0FOVuEyt+qwX0eIUMwOH//8b8gBSlvCUkg16o6C3XD82zCQEKyNR qE8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=v28hM+zZfD8XBeRP4jZBLaAnkv4RInatG6beIJh20OI=; b=BQx3z645Ne+OxWBNGGueWbDn62a2tFuDij5idUJdfimtScqBt4NFNCg7D5UOwzMkFw m4RNan13+GZWY1eLTawyFkEU9Vh9yWZNvIVggpNi5ks3e5LCSxSXRofXeE9UeFOw8q7w Dy5FzMMCIfOtcnqCUDP8kct91aUWVPlqzURp710yIqbpRFCD5pRj2fFOKWnQMUDz8tjI Cou6ZdMkl3gES+wRKGEDJ/VplwhbcScFX9BDZex6K99P207IGSWzdRAp9WLp9PO/zGUg K19AEhScEE73GwCvQ34yTZK3c5nIqtWQw5klzdRU8C2v2Hn5L4ScdNFmJHCzcH3dH4M2 vpZQ== X-Gm-Message-State: AKS2vOykubofc7biM/gqbkCAUQPPkMjh2T/+FVOWn1/czL4BV4ZrfRA9 AqMRadRUC/F2JqbU X-Received: by 10.84.209.228 with SMTP id y91mr6812121plh.210.1498192530775; Thu, 22 Jun 2017 21:35:30 -0700 (PDT) Received: from matt.ozlabs.ibm.com ([122.99.82.10]) by smtp.gmail.com with ESMTPSA id i9sm7369001pgr.7.2017.06.22.21.35.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 22 Jun 2017 21:35:30 -0700 (PDT) From: Matt Brown To: skiboot@lists.ozlabs.org Date: Fri, 23 Jun 2017 14:35:06 +1000 Message-Id: <20170623043506.9653-1-matthew.brown.dev@gmail.com> X-Mailer: git-send-email 2.9.3 Subject: [Skiboot] [PATCH] core/lock: Add deadlock detection X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" This adds simple deadlock detection. The detection looks for circular dependencies in the lock requests. It will abort and display a stack trace when a deadlock occurs. The detection will reduce performance so only use for debugging. To use, enable DEBUG_DEADLOCKS in include/config.h (disabled by default). Signed-off-by: Matt Brown --- core/lock.c | 83 +++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/config.h | 3 ++ include/lock.h | 5 ++++ 3 files changed, 90 insertions(+), 1 deletion(-) diff --git a/core/lock.c b/core/lock.c index e82048b..0379fb0 100644 --- a/core/lock.c +++ b/core/lock.c @@ -1,4 +1,4 @@ -/* Copyright 2013-2014 IBM Corp. +/* Copyright 2013-2017 IBM Corp. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -66,6 +66,73 @@ static inline void lock_check(struct lock *l) { }; static inline void unlock_check(struct lock *l) { }; #endif /* DEBUG_LOCKS */ +#ifdef DEBUG_DEADLOCKS + +#define MAX_THREADS 2048 +static struct lock *lock_table[MAX_THREADS]; + +/* Find circular dependencies in the lock requests. */ +static bool check_deadlock(void) +{ + int lock_owner, i, start; + struct lock *next; + + start = this_cpu()->pir; + next = lock_table[start]; + i = 0; + + while (i < MAX_THREADS) { + + if (!next) + return false; + + if (!(next->lock_val & 1) || next->in_con_path) + return false; + + lock_owner = next->lock_val >> 32; + + if (lock_owner >= MAX_THREADS) + return false; + + if (lock_owner == start && i > 0) + return true; + + next = lock_table[lock_owner]; + i++; + } + + return false; +} + +void add_lock_request(struct lock *l) +{ + if (this_cpu()->state == cpu_state_active || + this_cpu()->state == cpu_state_os) { + + if (this_cpu()->pir >= MAX_THREADS) + return; + + lock_table[this_cpu()->pir] = l; + + if (check_deadlock() && check_deadlock()) { +#ifdef DEBUG_LOCKS + lock_error(l, "Deadlock detected", 0); +#else + prlog(PR_EMERG, "LOCK: Deadlock detected\n"); + backtrace(); + abort(); +#endif /* DEBUG_LOCKS */ + } + } +} + +void remove_lock_request(void) +{ + if (this_cpu()->pir < MAX_THREADS) + lock_table[this_cpu()->pir] = NULL; +} +#endif /* DEBUG_DEADLOCKS */ + bool lock_held_by_me(struct lock *l) { uint64_t pir64 = this_cpu()->pir; @@ -86,15 +153,29 @@ bool try_lock(struct lock *l) void lock(struct lock *l) { + if (bust_locks) return; lock_check(l); + +#ifdef DEBUG_DEADLOCKS + if (try_lock(l)) + return; + + add_lock_request(l); +#endif + for (;;) { if (try_lock(l)) break; cpu_relax(); } + +#ifdef DEBUG_DEADLOCKS + remove_lock_request(); +#endif + } void unlock(struct lock *l) diff --git a/include/config.h b/include/config.h index cd8a0a6..c2f80e9 100644 --- a/include/config.h +++ b/include/config.h @@ -39,6 +39,9 @@ /* Enable lock debugging */ #define DEBUG_LOCKS 1 +/* Enable deadlock detection */ +//#define DEBUG_DEADLOCKS 1 + /* Enable malloc debugging */ #define DEBUG_MALLOC 1 diff --git a/include/lock.h b/include/lock.h index 0ac943d..27bedc7 100644 --- a/include/lock.h +++ b/include/lock.h @@ -81,4 +81,9 @@ extern bool lock_recursive(struct lock *l); /* Called after per-cpu data structures are available */ extern void init_locks(void); +#ifdef DEBUG_DEADLOCKS +extern void add_lock_request(struct lock *l); +extern void remove_lock_request(void); +#endif /* DEBUG_DEADLOCKS*/ + #endif /* __LOCK_H */