| Message ID | 20190909123151.21944-15-fbarrat@linux.ibm.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | opencapi: enable card reset and link retraining | expand |
| Context | Check | Description |
|---|---|---|
| snowpatch_ozlabs/apply_patch | success | Successfully applied on branch master (470ffb5f29d741c3bed600f7bb7bf0cbb270e05a) |
| snowpatch_ozlabs/snowpatch_job_snowpatch-skiboot | success | Test snowpatch/job/snowpatch-skiboot on branch master |
| snowpatch_ozlabs/snowpatch_job_snowpatch-skiboot-dco | success | Signed-off-by present |
On 09/09/2019 14:31, Frederic Barrat wrote: > Implement the get_power_state() and set_power_state() callbacks for > the opencapi slot and add properties in the device tree to mark the > opencapi slot as hot-pluggable. > > We don't really power off/on the opencapi adapter. The slot at play > here is the virtual slot associated to the virtual opencapi PHB. The > real PCIe slot where the card is drawing its power from is > untouched (skiboot is not even aware which PCIe slot the card is > seated on). So the 'fake' power off is fencing the card and set it in > reset so that the FPGA image can be updated. The 'fake' power on is > not doing much, as the unfencing happens on the subsequent link > training. > > Opencapi slots are named 'OPENCAPI-xxxx' where xxxx is the opal ID of > the PHB/slot. This is meant to easily identify the slot used by an AFU > device, as the AFU device names are also built around that ID. > For example, the device /dev/ocxl/AFP3.0006:00:00.1.0 uses the slot > OPENCAPI-0006. > > Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> > --- > hw/npu2-opencapi.c | 69 +++++++++++++++++++++++++++++++++++++++++++--- > 1 file changed, 65 insertions(+), 4 deletions(-) > > diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c > index c8bc64d1..af309362 100644 > --- a/hw/npu2-opencapi.c > +++ b/hw/npu2-opencapi.c > @@ -1032,9 +1032,10 @@ static int64_t npu2_opencapi_get_presence_state(struct pci_slot __unused *slot, > * As such we will never be asked to get the presence of a slot that's > * empty. > * > - * This may change if we ever support hotplug down the track. > + * This may change if we ever support surprise hotplug down > + * the track. > */ > - *val = true; > + *val = OPAL_PCI_SLOT_PRESENT; > return OPAL_SUCCESS; > } > > @@ -1092,6 +1093,38 @@ static int64_t npu2_opencapi_get_link_state(struct pci_slot *slot, uint8_t *val) > return OPAL_SUCCESS; > } > > +static int64_t npu2_opencapi_get_power_state(struct pci_slot *slot, > + uint8_t *val) > +{ > + *val = slot->power_state; > + return OPAL_SUCCESS; > +} > + > +static int64_t npu2_opencapi_set_power_state(struct pci_slot *slot, uint8_t val) > +{ > + struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); > + > + switch (val) { > + case PCI_SLOT_POWER_OFF: > + OCAPIDBG(dev, "Fake power off\n"); > + fence_brick(dev); > + assert_adapter_reset(dev); > + slot->power_state = PCI_SLOT_POWER_OFF; > + return OPAL_SUCCESS; > + > + case PCI_SLOT_POWER_ON: > + if (slot->power_state != PCI_SLOT_POWER_OFF) > + return OPAL_SUCCESS; > + OCAPIDBG(dev, "Fake power on\n"); > + slot->power_state = PCI_SLOT_POWER_ON; > + slot->state = OCAPI_SLOT_NORMAL; > + return OPAL_SUCCESS; > + > + default: > + return OPAL_UNSUPPORTED; > + } > +} > + > static void check_trained_link(struct npu2_dev *dev, uint64_t odl_status) > { > if (get_link_width(odl_status) != OPAL_SHPC_LINK_UP_x8) { > @@ -1132,6 +1165,14 @@ static int64_t npu2_opencapi_retry_state(struct pci_slot *slot, > return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); > } > > +static void npu2_opencapi_prepare_link_change(struct pci_slot *slot __unused, > + bool up __unused) > +{ > + /* > + * PCI hotplug wants it defined, but we don't need to do anything > + */ > +} > + > static int64_t npu2_opencapi_poll_link(struct pci_slot *slot) > { > struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); > @@ -1259,6 +1300,24 @@ static int64_t npu2_opencapi_hreset(struct pci_slot *slot __unused) > return OPAL_UNSUPPORTED; > } > > +static void make_slot_hotpluggable(struct pci_slot *slot, struct phb *phb) > +{ > + char label[40]; > + > + /* > + * Add a few definitions to the DT so that the linux PCI > + * hotplug framework can find the slot and identify it as > + * hot-pluggable. > + * > + * The "ibm,slot-label" property is used by linux as the slot name > + */ > + slot->pluggable = 1; > + pci_slot_add_dt_properties(slot, phb->dt_node); > + snprintf(label, sizeof(label), "OPENCAPI-%04x", > + (int)PCI_SLOT_PHB_INDEX(slot->id)); > + dt_add_property_string(phb->dt_node, "ibm,slot-label", label); most of dt definitions were added in setup_device(). May be you should move this one ? Otherwise Reviewed-by: Christophe Lombard <clombard@linux.vnet.ibm.com> > +} > + > static struct pci_slot *npu2_opencapi_slot_create(struct phb *phb) > { > struct pci_slot *slot; > @@ -1270,17 +1329,19 @@ static struct pci_slot *npu2_opencapi_slot_create(struct phb *phb) > /* TODO: Figure out other slot functions */ > slot->ops.get_presence_state = npu2_opencapi_get_presence_state; > slot->ops.get_link_state = npu2_opencapi_get_link_state; > - slot->ops.get_power_state = NULL; > + slot->ops.get_power_state = npu2_opencapi_get_power_state; > slot->ops.get_attention_state = NULL; > slot->ops.get_latch_state = NULL; > - slot->ops.set_power_state = NULL; > + slot->ops.set_power_state = npu2_opencapi_set_power_state; > slot->ops.set_attention_state = NULL; > > + slot->ops.prepare_link_change = npu2_opencapi_prepare_link_change; > slot->ops.poll_link = npu2_opencapi_poll_link; > slot->ops.creset = npu2_opencapi_creset; > slot->ops.freset = npu2_opencapi_freset; > slot->ops.hreset = npu2_opencapi_hreset; > > + make_slot_hotpluggable(slot, phb); > return slot; > } > >
Le 17/09/2019 à 15:37, christophe lombard a écrit : > On 09/09/2019 14:31, Frederic Barrat wrote: >> Implement the get_power_state() and set_power_state() callbacks for >> the opencapi slot and add properties in the device tree to mark the >> opencapi slot as hot-pluggable. >> >> We don't really power off/on the opencapi adapter. The slot at play >> here is the virtual slot associated to the virtual opencapi PHB. The >> real PCIe slot where the card is drawing its power from is >> untouched (skiboot is not even aware which PCIe slot the card is >> seated on). So the 'fake' power off is fencing the card and set it in >> reset so that the FPGA image can be updated. The 'fake' power on is >> not doing much, as the unfencing happens on the subsequent link >> training. >> >> Opencapi slots are named 'OPENCAPI-xxxx' where xxxx is the opal ID of >> the PHB/slot. This is meant to easily identify the slot used by an AFU >> device, as the AFU device names are also built around that ID. >> For example, the device /dev/ocxl/AFP3.0006:00:00.1.0 uses the slot >> OPENCAPI-0006. >> >> Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> >> --- >> hw/npu2-opencapi.c | 69 +++++++++++++++++++++++++++++++++++++++++++--- >> 1 file changed, 65 insertions(+), 4 deletions(-) >> >> diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c >> index c8bc64d1..af309362 100644 >> --- a/hw/npu2-opencapi.c >> +++ b/hw/npu2-opencapi.c >> @@ -1032,9 +1032,10 @@ static int64_t >> npu2_opencapi_get_presence_state(struct pci_slot __unused *slot, >> * As such we will never be asked to get the presence of a slot >> that's >> * empty. >> * >> - * This may change if we ever support hotplug down the track. >> + * This may change if we ever support surprise hotplug down >> + * the track. >> */ >> - *val = true; >> + *val = OPAL_PCI_SLOT_PRESENT; >> return OPAL_SUCCESS; >> } >> @@ -1092,6 +1093,38 @@ static int64_t >> npu2_opencapi_get_link_state(struct pci_slot *slot, uint8_t *val) >> return OPAL_SUCCESS; >> } >> +static int64_t npu2_opencapi_get_power_state(struct pci_slot *slot, >> + uint8_t *val) >> +{ >> + *val = slot->power_state; >> + return OPAL_SUCCESS; >> +} >> + >> +static int64_t npu2_opencapi_set_power_state(struct pci_slot *slot, >> uint8_t val) >> +{ >> + struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); >> + >> + switch (val) { >> + case PCI_SLOT_POWER_OFF: >> + OCAPIDBG(dev, "Fake power off\n"); >> + fence_brick(dev); >> + assert_adapter_reset(dev); >> + slot->power_state = PCI_SLOT_POWER_OFF; >> + return OPAL_SUCCESS; >> + >> + case PCI_SLOT_POWER_ON: >> + if (slot->power_state != PCI_SLOT_POWER_OFF) >> + return OPAL_SUCCESS; >> + OCAPIDBG(dev, "Fake power on\n"); >> + slot->power_state = PCI_SLOT_POWER_ON; >> + slot->state = OCAPI_SLOT_NORMAL; >> + return OPAL_SUCCESS; >> + >> + default: >> + return OPAL_UNSUPPORTED; >> + } >> +} >> + >> static void check_trained_link(struct npu2_dev *dev, uint64_t >> odl_status) >> { >> if (get_link_width(odl_status) != OPAL_SHPC_LINK_UP_x8) { >> @@ -1132,6 +1165,14 @@ static int64_t npu2_opencapi_retry_state(struct >> pci_slot *slot, >> return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); >> } >> +static void npu2_opencapi_prepare_link_change(struct pci_slot *slot >> __unused, >> + bool up __unused) >> +{ >> + /* >> + * PCI hotplug wants it defined, but we don't need to do anything >> + */ >> +} >> + >> static int64_t npu2_opencapi_poll_link(struct pci_slot *slot) >> { >> struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); >> @@ -1259,6 +1300,24 @@ static int64_t npu2_opencapi_hreset(struct >> pci_slot *slot __unused) >> return OPAL_UNSUPPORTED; >> } >> +static void make_slot_hotpluggable(struct pci_slot *slot, struct phb >> *phb) >> +{ >> + char label[40]; >> + >> + /* >> + * Add a few definitions to the DT so that the linux PCI >> + * hotplug framework can find the slot and identify it as >> + * hot-pluggable. >> + * >> + * The "ibm,slot-label" property is used by linux as the slot name >> + */ >> + slot->pluggable = 1; >> + pci_slot_add_dt_properties(slot, phb->dt_node); >> + snprintf(label, sizeof(label), "OPENCAPI-%04x", >> + (int)PCI_SLOT_PHB_INDEX(slot->id)); >> + dt_add_property_string(phb->dt_node, "ibm,slot-label", label); > > most of dt definitions were added in setup_device(). May be you should > move this one ? > Otherwise mmm, ok. I need the slot to be created first, but we do that at the end of setup_device(). Maybe I should call make_slot_hotpluggable() one call higher, in setup_device() instead of within npu2_opencapi_slot_create(). Fred > Reviewed-by: Christophe Lombard <clombard@linux.vnet.ibm.com> > >> +} >> + >> static struct pci_slot *npu2_opencapi_slot_create(struct phb *phb) >> { >> struct pci_slot *slot; >> @@ -1270,17 +1329,19 @@ static struct pci_slot >> *npu2_opencapi_slot_create(struct phb *phb) >> /* TODO: Figure out other slot functions */ >> slot->ops.get_presence_state = npu2_opencapi_get_presence_state; >> slot->ops.get_link_state = npu2_opencapi_get_link_state; >> - slot->ops.get_power_state = NULL; >> + slot->ops.get_power_state = npu2_opencapi_get_power_state; >> slot->ops.get_attention_state = NULL; >> slot->ops.get_latch_state = NULL; >> - slot->ops.set_power_state = NULL; >> + slot->ops.set_power_state = npu2_opencapi_set_power_state; >> slot->ops.set_attention_state = NULL; >> + slot->ops.prepare_link_change = npu2_opencapi_prepare_link_change; >> slot->ops.poll_link = npu2_opencapi_poll_link; >> slot->ops.creset = npu2_opencapi_creset; >> slot->ops.freset = npu2_opencapi_freset; >> slot->ops.hreset = npu2_opencapi_hreset; >> + make_slot_hotpluggable(slot, phb); >> return slot; >> } >> >
diff --git a/hw/npu2-opencapi.c b/hw/npu2-opencapi.c index c8bc64d1..af309362 100644 --- a/hw/npu2-opencapi.c +++ b/hw/npu2-opencapi.c @@ -1032,9 +1032,10 @@ static int64_t npu2_opencapi_get_presence_state(struct pci_slot __unused *slot, * As such we will never be asked to get the presence of a slot that's * empty. * - * This may change if we ever support hotplug down the track. + * This may change if we ever support surprise hotplug down + * the track. */ - *val = true; + *val = OPAL_PCI_SLOT_PRESENT; return OPAL_SUCCESS; } @@ -1092,6 +1093,38 @@ static int64_t npu2_opencapi_get_link_state(struct pci_slot *slot, uint8_t *val) return OPAL_SUCCESS; } +static int64_t npu2_opencapi_get_power_state(struct pci_slot *slot, + uint8_t *val) +{ + *val = slot->power_state; + return OPAL_SUCCESS; +} + +static int64_t npu2_opencapi_set_power_state(struct pci_slot *slot, uint8_t val) +{ + struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); + + switch (val) { + case PCI_SLOT_POWER_OFF: + OCAPIDBG(dev, "Fake power off\n"); + fence_brick(dev); + assert_adapter_reset(dev); + slot->power_state = PCI_SLOT_POWER_OFF; + return OPAL_SUCCESS; + + case PCI_SLOT_POWER_ON: + if (slot->power_state != PCI_SLOT_POWER_OFF) + return OPAL_SUCCESS; + OCAPIDBG(dev, "Fake power on\n"); + slot->power_state = PCI_SLOT_POWER_ON; + slot->state = OCAPI_SLOT_NORMAL; + return OPAL_SUCCESS; + + default: + return OPAL_UNSUPPORTED; + } +} + static void check_trained_link(struct npu2_dev *dev, uint64_t odl_status) { if (get_link_width(odl_status) != OPAL_SHPC_LINK_UP_x8) { @@ -1132,6 +1165,14 @@ static int64_t npu2_opencapi_retry_state(struct pci_slot *slot, return pci_slot_set_sm_timeout(slot, msecs_to_tb(1)); } +static void npu2_opencapi_prepare_link_change(struct pci_slot *slot __unused, + bool up __unused) +{ + /* + * PCI hotplug wants it defined, but we don't need to do anything + */ +} + static int64_t npu2_opencapi_poll_link(struct pci_slot *slot) { struct npu2_dev *dev = phb_to_npu2_dev_ocapi(slot->phb); @@ -1259,6 +1300,24 @@ static int64_t npu2_opencapi_hreset(struct pci_slot *slot __unused) return OPAL_UNSUPPORTED; } +static void make_slot_hotpluggable(struct pci_slot *slot, struct phb *phb) +{ + char label[40]; + + /* + * Add a few definitions to the DT so that the linux PCI + * hotplug framework can find the slot and identify it as + * hot-pluggable. + * + * The "ibm,slot-label" property is used by linux as the slot name + */ + slot->pluggable = 1; + pci_slot_add_dt_properties(slot, phb->dt_node); + snprintf(label, sizeof(label), "OPENCAPI-%04x", + (int)PCI_SLOT_PHB_INDEX(slot->id)); + dt_add_property_string(phb->dt_node, "ibm,slot-label", label); +} + static struct pci_slot *npu2_opencapi_slot_create(struct phb *phb) { struct pci_slot *slot; @@ -1270,17 +1329,19 @@ static struct pci_slot *npu2_opencapi_slot_create(struct phb *phb) /* TODO: Figure out other slot functions */ slot->ops.get_presence_state = npu2_opencapi_get_presence_state; slot->ops.get_link_state = npu2_opencapi_get_link_state; - slot->ops.get_power_state = NULL; + slot->ops.get_power_state = npu2_opencapi_get_power_state; slot->ops.get_attention_state = NULL; slot->ops.get_latch_state = NULL; - slot->ops.set_power_state = NULL; + slot->ops.set_power_state = npu2_opencapi_set_power_state; slot->ops.set_attention_state = NULL; + slot->ops.prepare_link_change = npu2_opencapi_prepare_link_change; slot->ops.poll_link = npu2_opencapi_poll_link; slot->ops.creset = npu2_opencapi_creset; slot->ops.freset = npu2_opencapi_freset; slot->ops.hreset = npu2_opencapi_hreset; + make_slot_hotpluggable(slot, phb); return slot; }
Implement the get_power_state() and set_power_state() callbacks for the opencapi slot and add properties in the device tree to mark the opencapi slot as hot-pluggable. We don't really power off/on the opencapi adapter. The slot at play here is the virtual slot associated to the virtual opencapi PHB. The real PCIe slot where the card is drawing its power from is untouched (skiboot is not even aware which PCIe slot the card is seated on). So the 'fake' power off is fencing the card and set it in reset so that the FPGA image can be updated. The 'fake' power on is not doing much, as the unfencing happens on the subsequent link training. Opencapi slots are named 'OPENCAPI-xxxx' where xxxx is the opal ID of the PHB/slot. This is meant to easily identify the slot used by an AFU device, as the AFU device names are also built around that ID. For example, the device /dev/ocxl/AFP3.0006:00:00.1.0 uses the slot OPENCAPI-0006. Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com> --- hw/npu2-opencapi.c | 69 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 65 insertions(+), 4 deletions(-)