[v5,07/18] vfio-user: set qdev bus callbacks for remote machine

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
---
 hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

On Wed, Jan 19, 2022 at 04:41:56PM -0500, Jagannathan Raman wrote:
> Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
> ---
>  hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 57 insertions(+)
> 
> diff --git a/hw/remote/machine.c b/hw/remote/machine.c
> index 220ff01aa9..221a8430c1 100644
> --- a/hw/remote/machine.c
> +++ b/hw/remote/machine.c
> @@ -22,6 +22,60 @@
>  #include "hw/pci/pci_host.h"
>  #include "hw/remote/iohub.h"
>  
> +static bool remote_machine_get_bus(const char *type, BusState **bus,
> +                                   Error **errp)
> +{
> +    ERRP_GUARD();
> +    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
> +    BusState *root_bus = NULL;
> +    PCIBus *new_pci_bus = NULL;
> +
> +    if (!bus) {
> +        error_setg(errp, "Invalid argument");
> +        return false;
> +    }
> +
> +    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
> +        return true;
> +    }
> +
> +    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
> +    if (!root_bus) {
> +        error_setg(errp, "Unable to find root PCI device");
> +        return false;
> +    }
> +
> +    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
> +    if (!new_pci_bus) {
> +        return false;
> +    }
> +
> +    *bus = BUS(new_pci_bus);
> +
> +    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
> +                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
> +
> +    return true;
> +}

Can the user create the same PCI bus via QMP commands? If so, then this
is just a convenience that saves the extra step. Or is there some magic
that cannot be done via QMP device_add?

I'm asking because there are 3 objects involved and I'd like to
understand the lifecycle/dependencies:
1. The PCIDevice we wish to export.
2. The PCIBus with isolated address spaces that contains the PCIDevice.
3. The vfio-user server that exports a given PCIDevice.

Users can already create the PCIDevice via hotplug and the vfio-user
server via object-add. So if there's no magic they could also create the
PCI bus:
1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
3. object-add x-vfio-user-server,device=mydev

Unplug would work in the reverse order.

It may be more convenient to automatically create a PCIBus when the
PCIDevice is hotplugged, but this kind of magic also has drawbacks
(hidden devices, naming collisions, etc).

> +
> +static bool remote_machine_put_bus(BusState *bus, Error **errp)
> +{
> +    PCIBus *pci_bus = NULL;
> +
> +    if (!bus) {
> +        error_setg(errp, "Invalid argument");
> +        return false;
> +    }
> +
> +    if (!object_dynamic_cast(OBJECT(bus), TYPE_PCI_BUS)) {
> +        return true;
> +    }
> +
> +    pci_bus = PCI_BUS(bus);
> +
> +    return pci_isol_bus_free(pci_bus, errp);
> +}
> +
>  static void remote_machine_init(MachineState *machine)
>  {
>      MemoryRegion *system_memory, *system_io, *pci_memory;
> @@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine)
>                   &s->iohub, REMOTE_IOHUB_NB_PIRQS);
>  
>      qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
> +
> +    qdev_set_bus_cbs(remote_machine_get_bus, remote_machine_put_bus,
> +                     &error_fatal);
>  }
>  
>  static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
> -- 
> 2.20.1
>

> On Jan 25, 2022, at 5:44 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> On Wed, Jan 19, 2022 at 04:41:56PM -0500, Jagannathan Raman wrote:
>> Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
>> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
>> Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
>> ---
>> hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 57 insertions(+)
>> 
>> diff --git a/hw/remote/machine.c b/hw/remote/machine.c
>> index 220ff01aa9..221a8430c1 100644
>> --- a/hw/remote/machine.c
>> +++ b/hw/remote/machine.c
>> @@ -22,6 +22,60 @@
>> #include "hw/pci/pci_host.h"
>> #include "hw/remote/iohub.h"
>> 
>> +static bool remote_machine_get_bus(const char *type, BusState **bus,
>> +                                   Error **errp)
>> +{
>> +    ERRP_GUARD();
>> +    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
>> +    BusState *root_bus = NULL;
>> +    PCIBus *new_pci_bus = NULL;
>> +
>> +    if (!bus) {
>> +        error_setg(errp, "Invalid argument");
>> +        return false;
>> +    }
>> +
>> +    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
>> +        return true;
>> +    }
>> +
>> +    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
>> +    if (!root_bus) {
>> +        error_setg(errp, "Unable to find root PCI device");
>> +        return false;
>> +    }
>> +
>> +    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
>> +    if (!new_pci_bus) {
>> +        return false;
>> +    }
>> +
>> +    *bus = BUS(new_pci_bus);
>> +
>> +    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
>> +                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
>> +
>> +    return true;
>> +}
> 
> Can the user create the same PCI bus via QMP commands? If so, then this

I think there is a way we could achieve it.

When I looked around, both the command line and the QMP didn’t have a direct
way to create a bus. However, there are some indirect ways. For example, the
TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly,
there are some special PCI devices like TYPE_PCI_BRIDGE which create a
secondary PCI bus.

Similarly, we could implement a PCI device that creates a PCI bus with
isolated address spaces.

> is just a convenience that saves the extra step. Or is there some magic
> that cannot be done via QMP device_add?
> 
> I'm asking because there are 3 objects involved and I'd like to
> understand the lifecycle/dependencies:
> 1. The PCIDevice we wish to export.
> 2. The PCIBus with isolated address spaces that contains the PCIDevice.
> 3. The vfio-user server that exports a given PCIDevice.
> 
> Users can already create the PCIDevice via hotplug and the vfio-user
> server via object-add. So if there's no magic they could also create the
> PCI bus:
> 1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
> 2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
> 3. object-add x-vfio-user-server,device=mydev

We are able to do 2 & 3 already. We could introduce a PCI device that
creates an isolated PCI bus. That would cover step 1 outlined above.

> 
> Unplug would work in the reverse order.
> 
> It may be more convenient to automatically create a PCIBus when the
> PCIDevice is hotplugged, but this kind of magic also has drawbacks
> (hidden devices, naming collisions, etc).

OK, makes sense.

--
Jag

> 
>> +
>> +static bool remote_machine_put_bus(BusState *bus, Error **errp)
>> +{
>> +    PCIBus *pci_bus = NULL;
>> +
>> +    if (!bus) {
>> +        error_setg(errp, "Invalid argument");
>> +        return false;
>> +    }
>> +
>> +    if (!object_dynamic_cast(OBJECT(bus), TYPE_PCI_BUS)) {
>> +        return true;
>> +    }
>> +
>> +    pci_bus = PCI_BUS(bus);
>> +
>> +    return pci_isol_bus_free(pci_bus, errp);
>> +}
>> +
>> static void remote_machine_init(MachineState *machine)
>> {
>>     MemoryRegion *system_memory, *system_io, *pci_memory;
>> @@ -56,6 +110,9 @@ static void remote_machine_init(MachineState *machine)
>>                  &s->iohub, REMOTE_IOHUB_NB_PIRQS);
>> 
>>     qbus_set_hotplug_handler(BUS(pci_host->bus), OBJECT(s));
>> +
>> +    qdev_set_bus_cbs(remote_machine_get_bus, remote_machine_put_bus,
>> +                     &error_fatal);
>> }
>> 
>> static void remote_machine_pre_plug_cb(HotplugHandler *hotplug_dev,
>> -- 
>> 2.20.1
>>

On Tue, Jan 25, 2022 at 09:12:28PM +0000, Jag Raman wrote:
> 
> 
> > On Jan 25, 2022, at 5:44 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > 
> > On Wed, Jan 19, 2022 at 04:41:56PM -0500, Jagannathan Raman wrote:
> >> Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
> >> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
> >> Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
> >> ---
> >> hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
> >> 1 file changed, 57 insertions(+)
> >> 
> >> diff --git a/hw/remote/machine.c b/hw/remote/machine.c
> >> index 220ff01aa9..221a8430c1 100644
> >> --- a/hw/remote/machine.c
> >> +++ b/hw/remote/machine.c
> >> @@ -22,6 +22,60 @@
> >> #include "hw/pci/pci_host.h"
> >> #include "hw/remote/iohub.h"
> >> 
> >> +static bool remote_machine_get_bus(const char *type, BusState **bus,
> >> +                                   Error **errp)
> >> +{
> >> +    ERRP_GUARD();
> >> +    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
> >> +    BusState *root_bus = NULL;
> >> +    PCIBus *new_pci_bus = NULL;
> >> +
> >> +    if (!bus) {
> >> +        error_setg(errp, "Invalid argument");
> >> +        return false;
> >> +    }
> >> +
> >> +    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
> >> +        return true;
> >> +    }
> >> +
> >> +    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
> >> +    if (!root_bus) {
> >> +        error_setg(errp, "Unable to find root PCI device");
> >> +        return false;
> >> +    }
> >> +
> >> +    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
> >> +    if (!new_pci_bus) {
> >> +        return false;
> >> +    }
> >> +
> >> +    *bus = BUS(new_pci_bus);
> >> +
> >> +    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
> >> +                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
> >> +
> >> +    return true;
> >> +}
> > 
> > Can the user create the same PCI bus via QMP commands? If so, then this
> 
> I think there is a way we could achieve it.
> 
> When I looked around, both the command line and the QMP didn’t have a direct
> way to create a bus. However, there are some indirect ways. For example, the
> TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly,
> there are some special PCI devices like TYPE_PCI_BRIDGE which create a
> secondary PCI bus.
> 
> Similarly, we could implement a PCI device that creates a PCI bus with
> isolated address spaces.

Exactly. device_add instantiates DeviceStates, not busses, so there
needs to be a parent device like a SCSI controller, a PCI bridge, etc
that owns and creates the bus.

> > is just a convenience that saves the extra step. Or is there some magic
> > that cannot be done via QMP device_add?
> > 
> > I'm asking because there are 3 objects involved and I'd like to
> > understand the lifecycle/dependencies:
> > 1. The PCIDevice we wish to export.
> > 2. The PCIBus with isolated address spaces that contains the PCIDevice.
> > 3. The vfio-user server that exports a given PCIDevice.
> > 
> > Users can already create the PCIDevice via hotplug and the vfio-user
> > server via object-add. So if there's no magic they could also create the
> > PCI bus:
> > 1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
> > 2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
> > 3. object-add x-vfio-user-server,device=mydev
> 
> We are able to do 2 & 3 already. We could introduce a PCI device that
> creates an isolated PCI bus. That would cover step 1 outlined above.

I wonder if a new device is needed or whether it's possible to add an
isol_as=on|off (default: off) option to an existing PCI bridge/expander?
Hopefully most of the code is already there.

Stefan

> On Jan 26, 2022, at 4:37 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> 
> On Tue, Jan 25, 2022 at 09:12:28PM +0000, Jag Raman wrote:
>> 
>> 
>>> On Jan 25, 2022, at 5:44 AM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>>> 
>>> On Wed, Jan 19, 2022 at 04:41:56PM -0500, Jagannathan Raman wrote:
>>>> Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
>>>> Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
>>>> Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
>>>> ---
>>>> hw/remote/machine.c | 57 +++++++++++++++++++++++++++++++++++++++++++++
>>>> 1 file changed, 57 insertions(+)
>>>> 
>>>> diff --git a/hw/remote/machine.c b/hw/remote/machine.c
>>>> index 220ff01aa9..221a8430c1 100644
>>>> --- a/hw/remote/machine.c
>>>> +++ b/hw/remote/machine.c
>>>> @@ -22,6 +22,60 @@
>>>> #include "hw/pci/pci_host.h"
>>>> #include "hw/remote/iohub.h"
>>>> 
>>>> +static bool remote_machine_get_bus(const char *type, BusState **bus,
>>>> +                                   Error **errp)
>>>> +{
>>>> +    ERRP_GUARD();
>>>> +    RemoteMachineState *s = REMOTE_MACHINE(current_machine);
>>>> +    BusState *root_bus = NULL;
>>>> +    PCIBus *new_pci_bus = NULL;
>>>> +
>>>> +    if (!bus) {
>>>> +        error_setg(errp, "Invalid argument");
>>>> +        return false;
>>>> +    }
>>>> +
>>>> +    if (strcmp(type, TYPE_PCI_BUS) && strcmp(type, TYPE_PCI_BUS)) {
>>>> +        return true;
>>>> +    }
>>>> +
>>>> +    root_bus = qbus_find_recursive(sysbus_get_default(), NULL, TYPE_PCIE_BUS);
>>>> +    if (!root_bus) {
>>>> +        error_setg(errp, "Unable to find root PCI device");
>>>> +        return false;
>>>> +    }
>>>> +
>>>> +    new_pci_bus = pci_isol_bus_new(root_bus, type, errp);
>>>> +    if (!new_pci_bus) {
>>>> +        return false;
>>>> +    }
>>>> +
>>>> +    *bus = BUS(new_pci_bus);
>>>> +
>>>> +    pci_bus_irqs(new_pci_bus, remote_iohub_set_irq, remote_iohub_map_irq,
>>>> +                 &s->iohub, REMOTE_IOHUB_NB_PIRQS);
>>>> +
>>>> +    return true;
>>>> +}
>>> 
>>> Can the user create the same PCI bus via QMP commands? If so, then this
>> 
>> I think there is a way we could achieve it.
>> 
>> When I looked around, both the command line and the QMP didn’t have a direct
>> way to create a bus. However, there are some indirect ways. For example, the
>> TYPE_LSI53C895A device creates a SCSI bus to attach SCSI devices. Similarly,
>> there are some special PCI devices like TYPE_PCI_BRIDGE which create a
>> secondary PCI bus.
>> 
>> Similarly, we could implement a PCI device that creates a PCI bus with
>> isolated address spaces.
> 
> Exactly. device_add instantiates DeviceStates, not busses, so there
> needs to be a parent device like a SCSI controller, a PCI bridge, etc
> that owns and creates the bus.
> 
>>> is just a convenience that saves the extra step. Or is there some magic
>>> that cannot be done via QMP device_add?
>>> 
>>> I'm asking because there are 3 objects involved and I'd like to
>>> understand the lifecycle/dependencies:
>>> 1. The PCIDevice we wish to export.
>>> 2. The PCIBus with isolated address spaces that contains the PCIDevice.
>>> 3. The vfio-user server that exports a given PCIDevice.
>>> 
>>> Users can already create the PCIDevice via hotplug and the vfio-user
>>> server via object-add. So if there's no magic they could also create the
>>> PCI bus:
>>> 1. device_add ...some PCI bus stuff here...,id=isol-pci-bus0
>>> 2. device_add ...the PCIDevice...,bus=isol-pci-bus0,id=mydev
>>> 3. object-add x-vfio-user-server,device=mydev
>> 
>> We are able to do 2 & 3 already. We could introduce a PCI device that
>> creates an isolated PCI bus. That would cover step 1 outlined above.
> 
> I wonder if a new device is needed or whether it's possible to add an
> isol_as=on|off (default: off) option to an existing PCI bridge/expander?
> Hopefully most of the code is already there.

OK, it makes sense to add isol_as=on|off (default: off) option to an existing
PCI bridge/expander. Will shortly confirm with you the device that seems
interesting.

--
Jag

> 
> Stefan