| Message ID | a9bc6a76329778aadf7b7188c8d4b71342e16e24.1513579137.git-series.andrew.donnellan@au1.ibm.com |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | Initial OpenCAPI 3.0 Support for P9 | expand |
Le 18/12/2017 à 08:07, Andrew Donnellan a écrit : > The PowerBus Hotplug Mode Control register is used to control various > things about the PowerBus. > > Due to hardware timing constraints, updating this register requires > more than just a simple SCOM write. There are two sets of pb_hp_mode > registers, current (B) and next (A). Updating is done by a SCOM write > to the next registers, followed by a SWITCH_AB pulse from the > Alter/Display Unit, which triggers a hardware sequence to swap > between the current and next registers. > > The sequence used in this code was derived from the hardware > procedures used by Hostboot. > > This will be used in a subsequent patch as part of the > OpenCAPI/NVLink init sequence. > > Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> > --- > hw/Makefile.inc | 2 +- > hw/npu2.c | 1 +- > hw/p9-adu.c | 182 +++++++++++++++++++++++++++++++++++++++++++++- > include/npu2-regs.h | 2 +- > include/p9-adu.h | 58 ++++++++++++++- > 5 files changed, 242 insertions(+), 3 deletions(-) > create mode 100644 hw/p9-adu.c > create mode 100644 include/p9-adu.h > > diff --git a/hw/Makefile.inc b/hw/Makefile.inc > index cf8649d..27d8a38 100644 > --- a/hw/Makefile.inc > +++ b/hw/Makefile.inc > @@ -7,7 +7,7 @@ HW_OBJS += p7ioc.o p7ioc-inits.o p7ioc-phb.o > HW_OBJS += phb3.o sfc-ctrl.o fake-rtc.o bt.o p8-i2c.o prd.o > HW_OBJS += dts.o lpc-rtc.o npu.o npu-hw-procedures.o xive.o phb4.o > HW_OBJS += fake-nvram.o lpc-mbox.o npu2.o npu2-hw-procedures.o > -HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o > +HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o p9-adu.o > HW=hw/built-in.o > > # FIXME hack this for now > diff --git a/hw/npu2.c b/hw/npu2.c > index 1d7b936..12d239a 100644 > --- a/hw/npu2.c > +++ b/hw/npu2.c > @@ -37,6 +37,7 @@ > #include <phys-map.h> > #include <nvram.h> > #include <xive.h> > +#include <p9-adu.h> > > #define NPU2_IRQ_BASE_SHIFT 13 > #define NPU2_N_DL_IRQS 23 > diff --git a/hw/p9-adu.c b/hw/p9-adu.c > new file mode 100644 > index 0000000..b2b6287 > --- /dev/null > +++ b/hw/p9-adu.c > @@ -0,0 +1,182 @@ > +/* Copyright 2013-2017 IBM Corp. > + * > + * Licensed under the Apache License, Version 2.0 (the "License"); > + * you may not use this file except in compliance with the License. > + * You may obtain a copy of the License at > + * > + * http://www.apache.org/licenses/LICENSE-2.0 > + * > + * Unless required by applicable law or agreed to in writing, software > + * distributed under the License is distributed on an "AS IS" BASIS, > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > + * implied. > + * See the License for the specific language governing permissions and > + * limitations under the License. > + */ > + > +/* > + * Support code for POWER9 Fabric Alter/Display Unit (ADU) hotplug functionality > + * > + * The ADU acts as a bridge between the PowerBus and the Pervasive Interconnect > + * Bus (PIB). > + * > + * Among other things, the ADU is used to support PowerBus Hotplug. Skiboot > + * needs to set the PowerBus Hotplug Mode Control register as part of OpenCAPI > + * and NVLink initialisation. > + */ > + > +#include <skiboot.h> > +#include <timebase.h> > +#include <xscom.h> > +#include <chip.h> > +#include <p9-adu.h> > + > +/* > + * Lock or unlock ADU > + */ > +static int p9_adu_manage_lock(bool lock) > +{ > + int rc; > + uint64_t val = 0; > + struct proc_chip *chip = NULL; > + if (lock) { > + val |= PU_ALTD_CMD_REG_FBC_LOCKED; > + val |= PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM; > + val |= PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS; > + } > + > + while ((chip = next_chip(chip))) { > + rc = xscom_write(chip->id, PU_ALTD_CMD_REG, val); > + if (rc != OPAL_SUCCESS) { > + /* TODO: Lock picking support */ > + prlog(PR_ERR, > + "ADU: Error %d writing command (chip %d)\n", > + rc, chip->id); > + return OPAL_HARDWARE; > + } > + } > + return OPAL_SUCCESS; > +} > + > +static uint32_t find_master_chip(void) > +{ > + uint64_t reg; > + struct proc_chip *chip = NULL; > + while ((chip = next_chip(chip))) { > + xscom_read(chip->id, PB_CENT_HP_MODE_CURR, ®); > + if (reg & PB_CFG_MASTER_CHIP) > + break; > + } > + return chip->id; We may want to assert if we don't find the master chip. > +} > + > +/* > + * Trigger a SWITCH_AB pulse to switch the current PowerBus Hotplug Mode Control > + * register set. > + * > + * Overview of sequence: > + * > + * - acquire lock and reset all ADUs > + * - configure all ADUs for AB switch > + * - configure one ADU (on the fabric master chip) to issue a > + * quiesce/switch/reinit > + * - check status > + * - clear switch selectors > + * - reset all ADUs > + * - unlock > + */ > +static void p9_adu_switch_ab(void) > +{ > + uint32_t gcid = find_master_chip(); > + struct proc_chip *chip = NULL; > + uint64_t reg; > + uint64_t val; > + int rc = OPAL_SUCCESS; > + > + /* > + * There's a performance issue on P9DD1 that requires a workaround: > + * see IBM defect HW397129. However, this code isn't expected to be > + * used on DD1 machines. > + */ > + > + rc = p9_adu_manage_lock(true); > + if (rc) > + goto err; > + > + /* Set PB_SWITCH_AB on all ADUs */ > + while ((chip = next_chip(chip))) { > + xscom_read(chip->id, PU_SND_MODE_REG, ®); > + reg |= PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; I found the cronus sequence we're supposed to mimic in an old email and this should match: # configure all chips for AB switch putscom pu 090021 30 2 10 -ib -all However, here we are not clearing bit 31? Hostboot also seems to always define both bits (to 0b01 in our case), so it should be safer (see p9_build_smp_adu_set_switch_action()) The meaning of the bits is in the big scom list, but doesn't help me much. Same comment applies on the reset a few lines below. The rest of the function looks good. Fred > + rc = xscom_write(chip->id, PU_SND_MODE_REG, reg); > + if (rc) > + goto err_switch; > + } > + > + /* Set address 0 */ > + rc = xscom_write(gcid, PU_ALTD_ADDR_REG, 0); > + if (rc) > + goto err_switch; > + > + /* Configure ADU to issue quiesce + switch + reinit */ > + val = PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE; > + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT, > + val, QUIESCE_SWITCH_WAIT_COUNT); > + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT; > + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT, > + val, INIT_SWITCH_WAIT_COUNT); > + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH; /* see HW397129, DD2 */ > + rc = xscom_write(gcid, PU_ALTD_OPTION_REG, val); > + if (rc) > + goto err_switch; > + > + /* Set up command */ > + val = PU_ALTD_CMD_REG_FBC_LOCKED; > + val |= PU_ALTD_CMD_REG_FBC_ALTD_START_OP; > + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_SCOPE, val, > + PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM); > + val |= PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY; > + val |= PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE; > + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TTYPE, val, > + PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER); > + val |= PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE; > + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TSIZE, val, > + PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1); > + xscom_write(gcid, PU_ALTD_CMD_REG, val); > + > + /* > + * TODO: check ADU status is consistent, see > + * p9_build_smp_adu_check_status() in hostboot > + */ > + > +err_switch: > + /* Reset switch controls */ > + chip = NULL; > + while ((chip = next_chip(chip))) { > + xscom_read(chip->id, PU_SND_MODE_REG, ®); > + reg &= ~PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; > + xscom_write(chip->id, PU_SND_MODE_REG, reg); > + } > + > + /* Reset ADUs */ > + p9_adu_manage_lock(true); > + > +err: > + /* Unlock */ > + p9_adu_manage_lock(false); > +} > + > +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val) > +{ > + /* Write next value */ > + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); > + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); > + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); > + > + /* Send switch pulse */ > + p9_adu_switch_ab(); > + > + /* Now that switch is complete, overwrite old value */ > + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); > + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); > + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); > +} > diff --git a/include/npu2-regs.h b/include/npu2-regs.h > index cf74431..27956f2 100644 > --- a/include/npu2-regs.h > +++ b/include/npu2-regs.h > @@ -36,8 +36,6 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, > #define MCD_BANK_CN_VALID PPC_BIT(0) > #define MCD_BANK_CN_SIZE PPC_BITMASK(13,29) > #define MCD_BANK_CN_ADDR PPC_BITMASK(33,63) > -#define PB_CENT_HP_MODE_CURR 0x5011c0c > -#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) > #define PB_CENT_MODE 0x5011c0a > #define PB_CFG_CHIP_ADDR_EXTENSION_MASK_CENT PPC_BITMASK(42,48) > > diff --git a/include/p9-adu.h b/include/p9-adu.h > new file mode 100644 > index 0000000..1505704 > --- /dev/null > +++ b/include/p9-adu.h > @@ -0,0 +1,58 @@ > +/* Copyright 2013-2017 IBM Corp. > + * > + * Licensed under the Apache License, Version 2.0 (the "License"); > + * you may not use this file except in compliance with the License. > + * You may obtain a copy of the License at > + * > + * http://www.apache.org/licenses/LICENSE-2.0 > + * > + * Unless required by applicable law or agreed to in writing, software > + * distributed under the License is distributed on an "AS IS" BASIS, > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > + * implied. > + * See the License for the specific language governing permissions and > + * limitations under the License. > + */ > + > +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val); > + > +#define PU_ALTD_ADDR_REG 0x0090000 > + > +#define PU_ALTD_CMD_REG 0x0090001 > +#define PU_ALTD_CMD_REG_FBC_ALTD_START_OP PPC_BIT(2) > +#define PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS PPC_BIT(3) > +#define PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM PPC_BIT(4) > +#define PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE PPC_BIT(6) > +#define PU_ALTD_CMD_REG_FBC_LOCKED PPC_BIT(11) > +#define PU_ALTD_CMD_REG_FBC_LOCK_ID PPC_BITMASK(12, 15) > +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE PPC_BITMASK(16, 18) > +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM 0b101 > +#define PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY PPC_BIT(20) > +#define PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE PPC_BIT(24) > +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE PPC_BITMASK(25, 31) > +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER 0b0110001 > +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE PPC_BITMASK(32, 39) > +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1 0b00000010 > + > +#define PU_ALTD_OPTION_REG 0x0090002 > +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE PPC_BIT(23) > +#define PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT PPC_BITMASK(28, 47) > +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT PPC_BIT(51) > +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH PPC_BIT(52) > +#define PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT PPC_BITMASK(54, 63) > + > +#define QUIESCE_SWITCH_WAIT_COUNT 128 > +#define INIT_SWITCH_WAIT_COUNT 128 > + > +#define PU_SND_MODE_REG 0x0090021 > +#define PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB PPC_BIT(30) > + > +/* Hotplug mode registers */ > +#define PB_WEST_HP_MODE_NEXT 0x501180B > +#define PB_CENT_HP_MODE_NEXT 0x5011C0B > +#define PB_EAST_HP_MODE_NEXT 0x501200B > +#define PB_WEST_HP_MODE_CURR 0x501180C > +#define PB_CENT_HP_MODE_CURR 0x5011C0C > +#define PB_EAST_HP_MODE_CURR 0x501200C > +#define PB_CFG_MASTER_CHIP PPC_BIT(0) > +#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) >
On 11/01/18 05:39, Frederic Barrat wrote: > > Le 18/12/2017 à 08:07, Andrew Donnellan a écrit : >> The PowerBus Hotplug Mode Control register is used to control various >> things about the PowerBus. >> >> Due to hardware timing constraints, updating this register requires >> more than just a simple SCOM write. There are two sets of pb_hp_mode >> registers, current (B) and next (A). Updating is done by a SCOM write >> to the next registers, followed by a SWITCH_AB pulse from the >> Alter/Display Unit, which triggers a hardware sequence to swap >> between the current and next registers. >> >> The sequence used in this code was derived from the hardware >> procedures used by Hostboot. >> >> This will be used in a subsequent patch as part of the >> OpenCAPI/NVLink init sequence. >> >> Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> >> --- >> hw/Makefile.inc | 2 +- >> hw/npu2.c | 1 +- >> hw/p9-adu.c | 182 >> +++++++++++++++++++++++++++++++++++++++++++++- >> include/npu2-regs.h | 2 +- >> include/p9-adu.h | 58 ++++++++++++++- >> 5 files changed, 242 insertions(+), 3 deletions(-) >> create mode 100644 hw/p9-adu.c >> create mode 100644 include/p9-adu.h >> >> diff --git a/hw/Makefile.inc b/hw/Makefile.inc >> index cf8649d..27d8a38 100644 >> --- a/hw/Makefile.inc >> +++ b/hw/Makefile.inc >> @@ -7,7 +7,7 @@ HW_OBJS += p7ioc.o p7ioc-inits.o p7ioc-phb.o >> HW_OBJS += phb3.o sfc-ctrl.o fake-rtc.o bt.o p8-i2c.o prd.o >> HW_OBJS += dts.o lpc-rtc.o npu.o npu-hw-procedures.o xive.o phb4.o >> HW_OBJS += fake-nvram.o lpc-mbox.o npu2.o npu2-hw-procedures.o >> -HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o >> +HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o >> vas.o p9-adu.o >> HW=hw/built-in.o >> >> # FIXME hack this for now >> diff --git a/hw/npu2.c b/hw/npu2.c >> index 1d7b936..12d239a 100644 >> --- a/hw/npu2.c >> +++ b/hw/npu2.c >> @@ -37,6 +37,7 @@ >> #include <phys-map.h> >> #include <nvram.h> >> #include <xive.h> >> +#include <p9-adu.h> >> >> #define NPU2_IRQ_BASE_SHIFT 13 >> #define NPU2_N_DL_IRQS 23 >> diff --git a/hw/p9-adu.c b/hw/p9-adu.c >> new file mode 100644 >> index 0000000..b2b6287 >> --- /dev/null >> +++ b/hw/p9-adu.c >> @@ -0,0 +1,182 @@ >> +/* Copyright 2013-2017 IBM Corp. >> + * >> + * Licensed under the Apache License, Version 2.0 (the "License"); >> + * you may not use this file except in compliance with the License. >> + * You may obtain a copy of the License at >> + * >> + * http://www.apache.org/licenses/LICENSE-2.0 >> + * >> + * Unless required by applicable law or agreed to in writing, software >> + * distributed under the License is distributed on an "AS IS" BASIS, >> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> + * implied. >> + * See the License for the specific language governing permissions and >> + * limitations under the License. >> + */ >> + >> +/* >> + * Support code for POWER9 Fabric Alter/Display Unit (ADU) hotplug >> functionality >> + * >> + * The ADU acts as a bridge between the PowerBus and the Pervasive >> Interconnect >> + * Bus (PIB). >> + * >> + * Among other things, the ADU is used to support PowerBus Hotplug. >> Skiboot >> + * needs to set the PowerBus Hotplug Mode Control register as part of >> OpenCAPI >> + * and NVLink initialisation. >> + */ >> + >> +#include <skiboot.h> >> +#include <timebase.h> >> +#include <xscom.h> >> +#include <chip.h> >> +#include <p9-adu.h> >> + >> +/* >> + * Lock or unlock ADU >> + */ >> +static int p9_adu_manage_lock(bool lock) >> +{ >> + int rc; >> + uint64_t val = 0; >> + struct proc_chip *chip = NULL; >> + if (lock) { >> + val |= PU_ALTD_CMD_REG_FBC_LOCKED; >> + val |= PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM; >> + val |= PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS; >> + } >> + >> + while ((chip = next_chip(chip))) { >> + rc = xscom_write(chip->id, PU_ALTD_CMD_REG, val); >> + if (rc != OPAL_SUCCESS) { >> + /* TODO: Lock picking support */ >> + prlog(PR_ERR, >> + "ADU: Error %d writing command (chip %d)\n", >> + rc, chip->id); >> + return OPAL_HARDWARE; >> + } >> + } >> + return OPAL_SUCCESS; >> +} >> + >> +static uint32_t find_master_chip(void) >> +{ >> + uint64_t reg; >> + struct proc_chip *chip = NULL; >> + while ((chip = next_chip(chip))) { >> + xscom_read(chip->id, PB_CENT_HP_MODE_CURR, ®); >> + if (reg & PB_CFG_MASTER_CHIP) >> + break; >> + } >> + return chip->id; > > > We may want to assert if we don't find the master chip. Good point > > >> +} >> + >> +/* >> + * Trigger a SWITCH_AB pulse to switch the current PowerBus Hotplug >> Mode Control >> + * register set. >> + * >> + * Overview of sequence: >> + * >> + * - acquire lock and reset all ADUs >> + * - configure all ADUs for AB switch >> + * - configure one ADU (on the fabric master chip) to issue a >> + * quiesce/switch/reinit >> + * - check status >> + * - clear switch selectors >> + * - reset all ADUs >> + * - unlock >> + */ >> +static void p9_adu_switch_ab(void) >> +{ >> + uint32_t gcid = find_master_chip(); >> + struct proc_chip *chip = NULL; >> + uint64_t reg; >> + uint64_t val; >> + int rc = OPAL_SUCCESS; >> + >> + /* >> + * There's a performance issue on P9DD1 that requires a workaround: >> + * see IBM defect HW397129. However, this code isn't expected to be >> + * used on DD1 machines. >> + */ >> + >> + rc = p9_adu_manage_lock(true); >> + if (rc) >> + goto err; >> + >> + /* Set PB_SWITCH_AB on all ADUs */ >> + while ((chip = next_chip(chip))) { >> + xscom_read(chip->id, PU_SND_MODE_REG, ®); >> + reg |= PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; > > I found the cronus sequence we're supposed to mimic in an old email and > this should match: > > # configure all chips for AB switch > putscom pu 090021 30 2 10 -ib -all > > However, here we are not clearing bit 31? > > Hostboot also seems to always define both bits (to 0b01 in our case), so > it should be safer (see p9_build_smp_adu_set_switch_action()) > > The meaning of the bits is in the big scom list, but doesn't help me much. > > Same comment applies on the reset a few lines below. The rest of the > function looks good. I don't think we'll ever run into a problem here but I'll clear that just in case. We only need a SWITCH_AB, no idea what a SWITCH_CD does... Andrew > > Fred > > >> + rc = xscom_write(chip->id, PU_SND_MODE_REG, reg); >> + if (rc) >> + goto err_switch; >> + } >> + >> + /* Set address 0 */ >> + rc = xscom_write(gcid, PU_ALTD_ADDR_REG, 0); >> + if (rc) >> + goto err_switch; >> + >> + /* Configure ADU to issue quiesce + switch + reinit */ >> + val = PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE; >> + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT, >> + val, QUIESCE_SWITCH_WAIT_COUNT); >> + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT; >> + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT, >> + val, INIT_SWITCH_WAIT_COUNT); >> + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH; /* see >> HW397129, DD2 */ >> + rc = xscom_write(gcid, PU_ALTD_OPTION_REG, val); >> + if (rc) >> + goto err_switch; >> + >> + /* Set up command */ >> + val = PU_ALTD_CMD_REG_FBC_LOCKED; >> + val |= PU_ALTD_CMD_REG_FBC_ALTD_START_OP; >> + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_SCOPE, val, >> + PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM); >> + val |= PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY; >> + val |= PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE; >> + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TTYPE, val, >> + PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER); >> + val |= PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE; >> + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TSIZE, val, >> + PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1); >> + xscom_write(gcid, PU_ALTD_CMD_REG, val); >> + >> + /* >> + * TODO: check ADU status is consistent, see >> + * p9_build_smp_adu_check_status() in hostboot >> + */ >> + >> +err_switch: >> + /* Reset switch controls */ >> + chip = NULL; >> + while ((chip = next_chip(chip))) { >> + xscom_read(chip->id, PU_SND_MODE_REG, ®); >> + reg &= ~PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; >> + xscom_write(chip->id, PU_SND_MODE_REG, reg); >> + } >> + >> + /* Reset ADUs */ >> + p9_adu_manage_lock(true); >> + >> +err: >> + /* Unlock */ >> + p9_adu_manage_lock(false); >> +} >> + >> +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val) >> +{ >> + /* Write next value */ >> + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); >> + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); >> + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); >> + >> + /* Send switch pulse */ >> + p9_adu_switch_ab(); >> + >> + /* Now that switch is complete, overwrite old value */ >> + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); >> + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); >> + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); >> +} >> diff --git a/include/npu2-regs.h b/include/npu2-regs.h >> index cf74431..27956f2 100644 >> --- a/include/npu2-regs.h >> +++ b/include/npu2-regs.h >> @@ -36,8 +36,6 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, >> #define MCD_BANK_CN_VALID PPC_BIT(0) >> #define MCD_BANK_CN_SIZE PPC_BITMASK(13,29) >> #define MCD_BANK_CN_ADDR PPC_BITMASK(33,63) >> -#define PB_CENT_HP_MODE_CURR 0x5011c0c >> -#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) >> #define PB_CENT_MODE 0x5011c0a >> #define PB_CFG_CHIP_ADDR_EXTENSION_MASK_CENT PPC_BITMASK(42,48) >> >> diff --git a/include/p9-adu.h b/include/p9-adu.h >> new file mode 100644 >> index 0000000..1505704 >> --- /dev/null >> +++ b/include/p9-adu.h >> @@ -0,0 +1,58 @@ >> +/* Copyright 2013-2017 IBM Corp. >> + * >> + * Licensed under the Apache License, Version 2.0 (the "License"); >> + * you may not use this file except in compliance with the License. >> + * You may obtain a copy of the License at >> + * >> + * http://www.apache.org/licenses/LICENSE-2.0 >> + * >> + * Unless required by applicable law or agreed to in writing, software >> + * distributed under the License is distributed on an "AS IS" BASIS, >> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or >> + * implied. >> + * See the License for the specific language governing permissions and >> + * limitations under the License. >> + */ >> + >> +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val); >> + >> +#define PU_ALTD_ADDR_REG 0x0090000 >> + >> +#define PU_ALTD_CMD_REG 0x0090001 >> +#define PU_ALTD_CMD_REG_FBC_ALTD_START_OP PPC_BIT(2) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS PPC_BIT(3) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM PPC_BIT(4) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE PPC_BIT(6) >> +#define PU_ALTD_CMD_REG_FBC_LOCKED PPC_BIT(11) >> +#define PU_ALTD_CMD_REG_FBC_LOCK_ID PPC_BITMASK(12, 15) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE PPC_BITMASK(16, 18) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM 0b101 >> +#define PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY PPC_BIT(20) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE PPC_BIT(24) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE PPC_BITMASK(25, 31) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER 0b0110001 >> +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE PPC_BITMASK(32, 39) >> +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1 0b00000010 >> + >> +#define PU_ALTD_OPTION_REG 0x0090002 >> +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE PPC_BIT(23) >> +#define PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT >> PPC_BITMASK(28, 47) >> +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT PPC_BIT(51) >> +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH PPC_BIT(52) >> +#define PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT >> PPC_BITMASK(54, 63) >> + >> +#define QUIESCE_SWITCH_WAIT_COUNT 128 >> +#define INIT_SWITCH_WAIT_COUNT 128 >> + >> +#define PU_SND_MODE_REG 0x0090021 >> +#define PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB PPC_BIT(30) >> + >> +/* Hotplug mode registers */ >> +#define PB_WEST_HP_MODE_NEXT 0x501180B >> +#define PB_CENT_HP_MODE_NEXT 0x5011C0B >> +#define PB_EAST_HP_MODE_NEXT 0x501200B >> +#define PB_WEST_HP_MODE_CURR 0x501180C >> +#define PB_CENT_HP_MODE_CURR 0x5011C0C >> +#define PB_EAST_HP_MODE_CURR 0x501200C >> +#define PB_CFG_MASTER_CHIP PPC_BIT(0) >> +#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) >>
diff --git a/hw/Makefile.inc b/hw/Makefile.inc index cf8649d..27d8a38 100644 --- a/hw/Makefile.inc +++ b/hw/Makefile.inc @@ -7,7 +7,7 @@ HW_OBJS += p7ioc.o p7ioc-inits.o p7ioc-phb.o HW_OBJS += phb3.o sfc-ctrl.o fake-rtc.o bt.o p8-i2c.o prd.o HW_OBJS += dts.o lpc-rtc.o npu.o npu-hw-procedures.o xive.o phb4.o HW_OBJS += fake-nvram.o lpc-mbox.o npu2.o npu2-hw-procedures.o -HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o +HW_OBJS += npu2-common.o phys-map.o sbe-p9.o capp.o occ-sensor.o vas.o p9-adu.o HW=hw/built-in.o # FIXME hack this for now diff --git a/hw/npu2.c b/hw/npu2.c index 1d7b936..12d239a 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -37,6 +37,7 @@ #include <phys-map.h> #include <nvram.h> #include <xive.h> +#include <p9-adu.h> #define NPU2_IRQ_BASE_SHIFT 13 #define NPU2_N_DL_IRQS 23 diff --git a/hw/p9-adu.c b/hw/p9-adu.c new file mode 100644 index 0000000..b2b6287 --- /dev/null +++ b/hw/p9-adu.c @@ -0,0 +1,182 @@ +/* Copyright 2013-2017 IBM Corp. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + * implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * Support code for POWER9 Fabric Alter/Display Unit (ADU) hotplug functionality + * + * The ADU acts as a bridge between the PowerBus and the Pervasive Interconnect + * Bus (PIB). + * + * Among other things, the ADU is used to support PowerBus Hotplug. Skiboot + * needs to set the PowerBus Hotplug Mode Control register as part of OpenCAPI + * and NVLink initialisation. + */ + +#include <skiboot.h> +#include <timebase.h> +#include <xscom.h> +#include <chip.h> +#include <p9-adu.h> + +/* + * Lock or unlock ADU + */ +static int p9_adu_manage_lock(bool lock) +{ + int rc; + uint64_t val = 0; + struct proc_chip *chip = NULL; + if (lock) { + val |= PU_ALTD_CMD_REG_FBC_LOCKED; + val |= PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM; + val |= PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS; + } + + while ((chip = next_chip(chip))) { + rc = xscom_write(chip->id, PU_ALTD_CMD_REG, val); + if (rc != OPAL_SUCCESS) { + /* TODO: Lock picking support */ + prlog(PR_ERR, + "ADU: Error %d writing command (chip %d)\n", + rc, chip->id); + return OPAL_HARDWARE; + } + } + return OPAL_SUCCESS; +} + +static uint32_t find_master_chip(void) +{ + uint64_t reg; + struct proc_chip *chip = NULL; + while ((chip = next_chip(chip))) { + xscom_read(chip->id, PB_CENT_HP_MODE_CURR, ®); + if (reg & PB_CFG_MASTER_CHIP) + break; + } + return chip->id; +} + +/* + * Trigger a SWITCH_AB pulse to switch the current PowerBus Hotplug Mode Control + * register set. + * + * Overview of sequence: + * + * - acquire lock and reset all ADUs + * - configure all ADUs for AB switch + * - configure one ADU (on the fabric master chip) to issue a + * quiesce/switch/reinit + * - check status + * - clear switch selectors + * - reset all ADUs + * - unlock + */ +static void p9_adu_switch_ab(void) +{ + uint32_t gcid = find_master_chip(); + struct proc_chip *chip = NULL; + uint64_t reg; + uint64_t val; + int rc = OPAL_SUCCESS; + + /* + * There's a performance issue on P9DD1 that requires a workaround: + * see IBM defect HW397129. However, this code isn't expected to be + * used on DD1 machines. + */ + + rc = p9_adu_manage_lock(true); + if (rc) + goto err; + + /* Set PB_SWITCH_AB on all ADUs */ + while ((chip = next_chip(chip))) { + xscom_read(chip->id, PU_SND_MODE_REG, ®); + reg |= PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; + rc = xscom_write(chip->id, PU_SND_MODE_REG, reg); + if (rc) + goto err_switch; + } + + /* Set address 0 */ + rc = xscom_write(gcid, PU_ALTD_ADDR_REG, 0); + if (rc) + goto err_switch; + + /* Configure ADU to issue quiesce + switch + reinit */ + val = PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE; + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT, + val, QUIESCE_SWITCH_WAIT_COUNT); + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT; + val = SETFIELD(PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT, + val, INIT_SWITCH_WAIT_COUNT); + val |= PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH; /* see HW397129, DD2 */ + rc = xscom_write(gcid, PU_ALTD_OPTION_REG, val); + if (rc) + goto err_switch; + + /* Set up command */ + val = PU_ALTD_CMD_REG_FBC_LOCKED; + val |= PU_ALTD_CMD_REG_FBC_ALTD_START_OP; + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_SCOPE, val, + PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM); + val |= PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY; + val |= PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE; + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TTYPE, val, + PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER); + val |= PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE; + val = SETFIELD(PU_ALTD_CMD_REG_FBC_ALTD_TSIZE, val, + PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1); + xscom_write(gcid, PU_ALTD_CMD_REG, val); + + /* + * TODO: check ADU status is consistent, see + * p9_build_smp_adu_check_status() in hostboot + */ + +err_switch: + /* Reset switch controls */ + chip = NULL; + while ((chip = next_chip(chip))) { + xscom_read(chip->id, PU_SND_MODE_REG, ®); + reg &= ~PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB; + xscom_write(chip->id, PU_SND_MODE_REG, reg); + } + + /* Reset ADUs */ + p9_adu_manage_lock(true); + +err: + /* Unlock */ + p9_adu_manage_lock(false); +} + +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val) +{ + /* Write next value */ + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); + + /* Send switch pulse */ + p9_adu_switch_ab(); + + /* Now that switch is complete, overwrite old value */ + xscom_write(gcid, PB_WEST_HP_MODE_NEXT, val); + xscom_write(gcid, PB_CENT_HP_MODE_NEXT, val); + xscom_write(gcid, PB_EAST_HP_MODE_NEXT, val); +} diff --git a/include/npu2-regs.h b/include/npu2-regs.h index cf74431..27956f2 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -36,8 +36,6 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base, #define MCD_BANK_CN_VALID PPC_BIT(0) #define MCD_BANK_CN_SIZE PPC_BITMASK(13,29) #define MCD_BANK_CN_ADDR PPC_BITMASK(33,63) -#define PB_CENT_HP_MODE_CURR 0x5011c0c -#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2) #define PB_CENT_MODE 0x5011c0a #define PB_CFG_CHIP_ADDR_EXTENSION_MASK_CENT PPC_BITMASK(42,48) diff --git a/include/p9-adu.h b/include/p9-adu.h new file mode 100644 index 0000000..1505704 --- /dev/null +++ b/include/p9-adu.h @@ -0,0 +1,58 @@ +/* Copyright 2013-2017 IBM Corp. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + * implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +void p9_adu_set_pb_hp_mode(uint32_t gcid, uint64_t val); + +#define PU_ALTD_ADDR_REG 0x0090000 + +#define PU_ALTD_CMD_REG 0x0090001 +#define PU_ALTD_CMD_REG_FBC_ALTD_START_OP PPC_BIT(2) +#define PU_ALTD_CMD_REG_FBC_ALTD_CLEAR_STATUS PPC_BIT(3) +#define PU_ALTD_CMD_REG_FBC_ALTD_RESET_FSM PPC_BIT(4) +#define PU_ALTD_CMD_REG_FBC_ALTD_AXTYPE PPC_BIT(6) +#define PU_ALTD_CMD_REG_FBC_LOCKED PPC_BIT(11) +#define PU_ALTD_CMD_REG_FBC_LOCK_ID PPC_BITMASK(12, 15) +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE PPC_BITMASK(16, 18) +#define PU_ALTD_CMD_REG_FBC_ALTD_SCOPE_SYSTEM 0b101 +#define PU_ALTD_CMD_REG_FBC_ALTD_DROP_PRIORITY PPC_BIT(20) +#define PU_ALTD_CMD_REG_FBC_ALTD_WITH_TM_QUIESCE PPC_BIT(24) +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE PPC_BITMASK(25, 31) +#define PU_ALTD_CMD_REG_FBC_ALTD_TTYPE_PMISC_OPER 0b0110001 +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE PPC_BITMASK(32, 39) +#define PU_ALTD_CMD_REG_FBC_ALTD_TSIZE_PMISC_1 0b00000010 + +#define PU_ALTD_OPTION_REG 0x0090002 +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_PRE_QUIESCE PPC_BIT(23) +#define PU_ALTD_OPTION_REG_FBC_ALTD_AFTER_QUIESCE_WAIT_COUNT PPC_BITMASK(28, 47) +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_POST_INIT PPC_BIT(51) +#define PU_ALTD_OPTION_REG_FBC_ALTD_WITH_FAST_PATH PPC_BIT(52) +#define PU_ALTD_OPTION_REG_FBC_ALTD_BEFORE_INIT_WAIT_COUNT PPC_BITMASK(54, 63) + +#define QUIESCE_SWITCH_WAIT_COUNT 128 +#define INIT_SWITCH_WAIT_COUNT 128 + +#define PU_SND_MODE_REG 0x0090021 +#define PU_SND_MODE_REG_ENABLE_PB_SWITCH_AB PPC_BIT(30) + +/* Hotplug mode registers */ +#define PB_WEST_HP_MODE_NEXT 0x501180B +#define PB_CENT_HP_MODE_NEXT 0x5011C0B +#define PB_EAST_HP_MODE_NEXT 0x501200B +#define PB_WEST_HP_MODE_CURR 0x501180C +#define PB_CENT_HP_MODE_CURR 0x5011C0C +#define PB_EAST_HP_MODE_CURR 0x501200C +#define PB_CFG_MASTER_CHIP PPC_BIT(0) +#define PB_CFG_CHG_RATE_GP_MASTER PPC_BIT(2)
The PowerBus Hotplug Mode Control register is used to control various things about the PowerBus. Due to hardware timing constraints, updating this register requires more than just a simple SCOM write. There are two sets of pb_hp_mode registers, current (B) and next (A). Updating is done by a SCOM write to the next registers, followed by a SWITCH_AB pulse from the Alter/Display Unit, which triggers a hardware sequence to swap between the current and next registers. The sequence used in this code was derived from the hardware procedures used by Hostboot. This will be used in a subsequent patch as part of the OpenCAPI/NVLink init sequence. Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com> --- hw/Makefile.inc | 2 +- hw/npu2.c | 1 +- hw/p9-adu.c | 182 +++++++++++++++++++++++++++++++++++++++++++++- include/npu2-regs.h | 2 +- include/p9-adu.h | 58 ++++++++++++++- 5 files changed, 242 insertions(+), 3 deletions(-) create mode 100644 hw/p9-adu.c create mode 100644 include/p9-adu.h