From patchwork Mon Apr 16 11:48:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vasant Hegde X-Patchwork-Id: 898604 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Pmzb3DWGz9s08 for ; Mon, 16 Apr 2018 21:55:39 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 40Pmzb212zzF201 for ; Mon, 16 Apr 2018 21:55:39 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (mailfrom) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=hegdevasant@linux.vnet.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40Pmr45VByzF1Rv for ; Mon, 16 Apr 2018 21:49:08 +1000 (AEST) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w3GBm6lS001205 for ; Mon, 16 Apr 2018 07:49:05 -0400 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0b-001b2d01.pphosted.com with ESMTP id 2hcs5upae7-1 (version=TLSv1.2 cipher=AES256-SHA256 bits=256 verify=NOT) for ; Mon, 16 Apr 2018 07:49:05 -0400 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Mon, 16 Apr 2018 12:49:03 +0100 Received: from b06cxnps3074.portsmouth.uk.ibm.com (9.149.109.194) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Mon, 16 Apr 2018 12:49:02 +0100 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w3GBn1YL11338106; Mon, 16 Apr 2018 11:49:01 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BE36CAE04D; Mon, 16 Apr 2018 12:38:52 +0100 (BST) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 72097AE056; Mon, 16 Apr 2018 12:38:51 +0100 (BST) Received: from hegdevasant.in.ibm.com (unknown [9.109.198.237]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 16 Apr 2018 12:38:51 +0100 (BST) From: Vasant Hegde To: skiboot@lists.ozlabs.org Date: Mon, 16 Apr 2018 17:18:48 +0530 X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180416114848.19618-1-hegdevasant@linux.vnet.ibm.com> References: <20180416114848.19618-1-hegdevasant@linux.vnet.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 18041611-0020-0000-0000-000004126D37 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18041611-0021-0000-0000-000042A6A73B Message-Id: <20180416114848.19618-3-hegdevasant@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-04-16_06:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1804160112 Subject: [Skiboot] [PATCH v7 2/2] SBE: Add timer support X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" SBE on P9 provides one shot programmable timer facility. We can use this to implement OPAL timers and hence limit the reliance on the Linux heartbeat (similar to HW timer facility provided by SLW on P8). Design: - We will continue to run Linux heartbeat. - Each chip has SBE. This patch always schedules timer on SBE on master chip. - Start timer option starts new timer or modifies an active timer for the specified timeout. - SBE expects timeout value in microseconds. We track timeout value in TB. Hence we convert tb to microseconds before sending request to SBE. - We are requesting ack from SBE for timer message. It gaurantees that SBE has scheduled timer. - Disabling SBE timer We expect SBE to send timer expiry interrupt whenever timer expires. We wait for 10 more ms before disabling timer. In future we can consider below alternative approaches: - Presently SBE timer disable is permanent (until we reboot system). SBE sends "I'm back" interrupt after reset. We can consider restarting timer after SBE reset. - Reset SBE and start timer again. - Each chip has SBE. On multi chip system we can try to schedule timer on different chip. Signed-off-by: Vasant Hegde CC: Benjamin Herrenschmidt --- core/interrupts.c | 3 +- core/test/run-timer.c | 8 ++ core/timer.c | 17 ++++- hw/sbe-p9.c | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++ include/sbe-p9.h | 6 ++ 5 files changed, 235 insertions(+), 4 deletions(-) diff --git a/core/interrupts.c b/core/interrupts.c index 4452511f0..5d7a68cd5 100644 --- a/core/interrupts.c +++ b/core/interrupts.c @@ -26,6 +26,7 @@ #include #include #include +#include /* ICP registers */ #define ICP_XIRR 0x4 /* 32-bit access */ @@ -489,7 +490,7 @@ static int64_t opal_handle_interrupt(uint32_t isn, __be64 *outstanding_event_mas is->ops->interrupt(is, isn); /* Check timers if SBE timer isn't working */ - if (!p8_sbe_timer_ok()) + if (!p8_sbe_timer_ok() && !p9_sbe_timer_ok()) check_timers(true); /* Update output events */ diff --git a/core/test/run-timer.c b/core/test/run-timer.c index 986af28d8..159e007ac 100644 --- a/core/test/run-timer.c +++ b/core/test/run-timer.c @@ -4,12 +4,15 @@ #define __TEST__ #include +#include #define mftb() (stamp) #define sync() #define smt_lowest() #define smt_medium() +enum proc_gen proc_gen = proc_gen_p9; + static uint64_t stamp, last; struct lock; static inline void lock_caller(struct lock *l, const char *caller) @@ -53,6 +56,11 @@ void p8_sbe_update_timer_expiry(uint64_t new_target) /* FIXME: do intersting SLW timer sim */ } +void p9_sbe_update_timer_expiry(uint64_t new_target) +{ + (void)new_target; +} + int main(void) { unsigned int i; diff --git a/core/timer.c b/core/timer.c index 21f62a492..8eefb74be 100644 --- a/core/timer.c +++ b/core/timer.c @@ -5,6 +5,7 @@ #include #include #include +#include #ifdef __TEST__ #define this_cpu() ((void *)-1) @@ -109,8 +110,12 @@ static void __schedule_timer_at(struct timer *t, uint64_t when) bail: /* Pick up the next timer and upddate the SBE HW timer */ lt = list_top(&timer_list, struct timer, link); - if (lt) - p8_sbe_update_timer_expiry(lt->target); + if (lt) { + if (proc_gen < proc_gen_p9) + p8_sbe_update_timer_expiry(lt->target); + else + p9_sbe_update_timer_expiry(lt->target); + } } void schedule_timer_at(struct timer *t, uint64_t when) @@ -167,7 +172,11 @@ static void __check_poll_timers(uint64_t now) * arbitrarily 1us. */ if (t->running) { - p8_sbe_update_timer_expiry(now + usecs_to_tb(1)); + if (proc_gen < proc_gen_p9) + p8_sbe_update_timer_expiry(now + usecs_to_tb(1)); + else + p9_sbe_update_timer_expiry(now + usecs_to_tb(1)); + break; } @@ -266,6 +275,8 @@ void late_init_timers(void) */ if (platform.heartbeat_time) { heartbeat = platform.heartbeat_time(); + } else if (p9_sbe_timer_ok()) { + heartbeat = HEARTBEAT_DEFAULT_MS * 10; } else if (p8_sbe_timer_ok() || fsp_present()) { heartbeat = HEARTBEAT_DEFAULT_MS * 10; } diff --git a/hw/sbe-p9.c b/hw/sbe-p9.c index 9dc487ba2..a5b0706ef 100644 --- a/hw/sbe-p9.c +++ b/hw/sbe-p9.c @@ -53,6 +53,7 @@ #include #include #include +#include #include #include @@ -80,11 +81,36 @@ struct p9_sbe { /* Default SBE chip ID */ static int sbe_default_chip_id = -1; +/* Is SBE timer running? */ +static bool sbe_has_timer = false; +static bool sbe_timer_in_progress = false; + +/* Inflight and next timer in TB */ +static uint64_t sbe_last_gen_stamp; +static uint64_t sbe_timer_target; + +static bool has_new_target = false; +static bool got_timer_interrupt = false; + +/* Timer lock */ +struct lock sbe_timer_lock; + +/* + * Minimum timeout value for P9 is 500 microseconds. After that + * SBE timer can handle granularity of 1 microsecond. + */ +#define SBE_TIMER_DEFAULT_US 500 +static uint64_t sbe_timer_def_tb; + +/* Timer control message */ +static struct p9_sbe_msg *timer_ctrl_msg; + #define SBE_STATUS_PRI_SHIFT 0x30 #define SBE_STATUS_SEC_SHIFT 0x20 /* Forward declaration */ static void p9_sbe_timeout_poll_one(struct p9_sbe *sbe); +static void p9_sbe_timer_schedule(void); /* bit 0-15 : Primary status code */ static inline u16 p9_sbe_get_primary_rc(struct p9_sbe_msg *resp) @@ -562,6 +588,14 @@ static void __p9_sbe_interrupt(struct p9_sbe *sbe) p9_sbe_msg_complete(sbe, msg); } + /* Timer expired */ + if (data & SBE_HOST_TIMER_EXPIRY) { + if (sbe->chip_id == sbe_default_chip_id) { + sbe_timer_in_progress = false; + got_timer_interrupt = true; + } + } + next_msg: p9_sbe_process_queue(sbe); } @@ -578,6 +612,15 @@ void p9_sbe_interrupt(uint32_t chip_id) sbe = chip->sbe; lock(&sbe->lock); __p9_sbe_interrupt(sbe); + + if (got_timer_interrupt) { + got_timer_interrupt = false; + /* Drop lock and call timers */ + unlock(&sbe->lock); + check_timers(true); + return; + } + unlock(&sbe->lock); } @@ -634,6 +677,55 @@ out: unlock(&sbe->lock); } +/* + * Check if the timer is working. If at least 10ms elapsed since + * last scheduled timer expiry. + */ +static void p9_sbe_timer_poll(struct p9_sbe *sbe) +{ + if (!sbe_has_timer || !sbe_timer_in_progress) + return; + + lock(&sbe->lock); + if (tb_compare(mftb(), sbe_last_gen_stamp + msecs_to_tb(10)) + != TB_AAFTERB) + goto out; + + /* + * In some cases there will be a delay in calling OPAL interrupt + * handler routine (opal_handle_interrupt). In such cases its + * possible that SBE has responded, but OPAL didn't act on that. + * Hence check for SBE response before disabling timer. + */ + __p9_sbe_interrupt(sbe); + if (got_timer_interrupt) + goto call_timer; + + if (!sbe_timer_in_progress) + goto out; + + if (tb_compare(mftb(), sbe_last_gen_stamp + msecs_to_tb(10)) + != TB_AAFTERB) + goto out; + + prlog(PR_ERR, "Timer stuck, falling back to OPAL pollers.\n"); + prlog(PR_ERR, "You will likely have slower I2C and may have " + "experienced increased jitter.\n"); + p9_sbe_reg_dump(sbe_default_chip_id); + sbe_has_timer = false; + sbe_timer_in_progress = false; + +out: + unlock(&sbe->lock); + return; + +call_timer: + got_timer_interrupt = false; + /* Drop lock and call timers */ + unlock(&sbe->lock); + check_timers(true); +} + static void p9_sbe_timeout_poll(void *user_data __unused) { struct p9_sbe *sbe; @@ -644,9 +736,119 @@ static void p9_sbe_timeout_poll(void *user_data __unused) continue; sbe = chip->sbe; p9_sbe_timeout_poll_one(sbe); + if (sbe->chip_id == sbe_default_chip_id) + p9_sbe_timer_poll(sbe); } } +static void p9_sbe_timer_resp(struct p9_sbe_msg *msg) +{ + if (msg->state != sbe_msg_done) { + prlog(PR_DEBUG, "Failed to schedule timer [chip id %x]\n", + sbe_default_chip_id); + + if (!has_new_target) + return; + } else { + /* Update last scheduled timer value */ + sbe_last_gen_stamp = mftb() + + usecs_to_tb(timer_ctrl_msg->reg[1]); + sbe_timer_in_progress = true; + } + + lock(&sbe_timer_lock); + if (has_new_target) { + has_new_target = false; + p9_sbe_timer_schedule(); + } + unlock(&sbe_timer_lock); +} + +static void p9_sbe_timer_schedule(void) +{ + int rc; + u32 tick_us = SBE_TIMER_DEFAULT_US; + u64 tb_cnt, now = mftb(); + + if (sbe_timer_in_progress) { + /* Remaining time of inflight timer <= sbe_timer_def_tb */ + if ((sbe_last_gen_stamp - now) <= sbe_timer_def_tb) + return; + } + + if (now < sbe_timer_target) { + /* Calculate how many microseconds from now, rounded up */ + if ((sbe_timer_target - now) > sbe_timer_def_tb) { + tb_cnt = sbe_timer_target - now + usecs_to_tb(1) - 1; + tick_us = tb_to_usecs(tb_cnt); + } + } + + /* Clear sequence number. p9_sbe_queue_msg will add new sequene ID */ + timer_ctrl_msg->reg[0] &= ~(PPC_BITMASK(32, 47)); + /* Update timeout value */ + timer_ctrl_msg->reg[1] = tick_us; + rc = p9_sbe_queue_msg(sbe_default_chip_id, timer_ctrl_msg, + p9_sbe_timer_resp); + if (rc != OPAL_SUCCESS) { + prlog(PR_ERR, "Failed to start timer [chip id = %x]\n", + sbe_default_chip_id); + return; + } +} + +/* + * This is called with the timer lock held, so there is no + * issue with re-entrancy or concurrence + */ +void p9_sbe_update_timer_expiry(uint64_t new_target) +{ + u64 now = mftb(); + + if (!sbe_has_timer || new_target == sbe_timer_target) + return; + + lock(&sbe_timer_lock); + /* Timer message is in flight. Record new timer and schedule later */ + if (p9_sbe_msg_busy(timer_ctrl_msg) || has_new_target) { + if (new_target < now) + goto out; + + if (new_target > sbe_timer_target) + goto out; + + sbe_timer_target = new_target; + has_new_target = true; + goto out; + } + + sbe_timer_target = new_target; + p9_sbe_timer_schedule(); + +out: + unlock(&sbe_timer_lock); +} + +/* Initialize SBE timer */ +static void p9_sbe_timer_init(void) +{ + timer_ctrl_msg = p9_sbe_mkmsg(SBE_CMD_CONTROL_TIMER, + CONTROL_TIMER_START, 0, 0, 0); + assert(timer_ctrl_msg); + + init_lock(&sbe_timer_lock); + sbe_has_timer = true; + sbe_timer_target = mftb(); + sbe_last_gen_stamp = ~0ull; + sbe_timer_def_tb = usecs_to_tb(SBE_TIMER_DEFAULT_US); + prlog(PR_INFO, "Timer facility on chip %x\n", sbe_default_chip_id); +} + +bool p9_sbe_timer_ok(void) +{ + return sbe_has_timer; +} + void p9_sbe_init(void) { struct dt_node *xn; @@ -680,6 +882,9 @@ void p9_sbe_init(void) return; } + /* Initiate SBE timer */ + p9_sbe_timer_init(); + /* Initiate SBE timeout poller */ opal_add_poller(p9_sbe_timeout_poll, NULL); } diff --git a/include/sbe-p9.h b/include/sbe-p9.h index 329afff80..4b839d8ba 100644 --- a/include/sbe-p9.h +++ b/include/sbe-p9.h @@ -231,4 +231,10 @@ extern void p9_sbe_init(void); /* SBE interrupt */ extern void p9_sbe_interrupt(uint32_t chip_id); +/* Is SBE timer available ? */ +extern bool p9_sbe_timer_ok(void); + +/* Update SBE timer expiry */ +extern void p9_sbe_update_timer_expiry(uint64_t new_target); + #endif /* __SBE_P9_H */