Patchwork [v2,08/27] pci: implement an emulated PCI-to-PCI bridge

login
register
mail settings
Submitter Thomas Petazzoni
Date Jan. 28, 2013, 6:56 p.m.
Message ID <1359399397-29729-9-git-send-email-thomas.petazzoni@free-electrons.com>
Download mbox | patch
Permalink /patch/216310/
State Not Applicable
Headers show

Comments

Thomas Petazzoni - Jan. 28, 2013, 6:56 p.m.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 drivers/pci/Kconfig             |    3 +
 drivers/pci/Makefile            |    1 +
 drivers/pci/sw-pci-pci-bridge.c |  185 +++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h             |   43 +++++++++
 4 files changed, 232 insertions(+)
 create mode 100644 drivers/pci/sw-pci-pci-bridge.c
Jason Gunthorpe - Jan. 28, 2013, 7:35 p.m.
On Mon, Jan 28, 2013 at 07:56:17PM +0100, Thomas Petazzoni wrote:

> +int pci_sw_pci_bridge_read(struct pci_sw_pci_bridge *bridge,
> +			   unsigned int where, int size, u32 *value)
> +{
> +	switch (where & ~3) {

It is not essential, but desirable, to report an Express Root Port
capability for PCI-E bridges:

        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #1, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <1us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
                        Slot #0, PowerLimit 25.000W; Interlock- NoCompl+
                SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
                        Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
                SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
                        Changed: MRL- PresDet- LinkState-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+ ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- ARIFwd-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB

In the Marvell case, this capability can be constructed by pulling
data from the the Express End Point capability of the PCI-E port:

        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB

This lets user space see the width/speed/etc state of the PCI-E link
itself...

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Thomas Petazzoni - Jan. 28, 2013, 7:39 p.m.
Dear Jason Gunthorpe,

Thanks a lot for your quick feedback!

On Mon, 28 Jan 2013 12:35:16 -0700, Jason Gunthorpe wrote:

> It is not essential, but desirable, to report an Express Root Port
> capability for PCI-E bridges:

[...]

> In the Marvell case, this capability can be constructed by pulling
> data from the the Express End Point capability of the PCI-E port:

I am not sure what you mean by "pulling". Do you mean that I should get
informations from the real PCIe interface, from within the emulated
PCI-to-PCI bridge implementation? This would unfortunately not be
really nice, because until now, the PCI-to-PCI bridge emulation is
clearly separated from the Marvell PCIe driver itself. Of course, it
could register a hook or something like that, so that the emulated
PCI-to-PCI bridge could potentially call back into the Marvell PCIe
driver.

I'll have to dig a little bit more about this capability to see how it
works exactly.

Thanks again for the feedback,

Thomas
Jason Gunthorpe - Jan. 28, 2013, 7:55 p.m.
On Mon, Jan 28, 2013 at 08:39:47PM +0100, Thomas Petazzoni wrote:

> > In the Marvell case, this capability can be constructed by pulling
> > data from the the Express End Point capability of the PCI-E port:
> 
> I am not sure what you mean by "pulling". Do you mean that I should get
> informations from the real PCIe interface, from within the emulated
> PCI-to-PCI bridge implementation? This would unfortunately not be
> really nice, because until now, the PCI-to-PCI bridge emulation is
> clearly separated from the Marvell PCIe driver itself. Of course, it
> could register a hook or something like that, so that the emulated
> PCI-to-PCI bridge could potentially call back into the Marvell PCIe
> driver.

Yes, a callback would be needed to the main driver and IIRC the driver
can read/write the end port link info config regsiters via MMIO. They
probably need a bit of massaging to be in root port format, but
otherwise it should be straightforward..

> I'll have to dig a little bit more about this capability to see how it
> works exactly.

All ports have registers to report and control the link, but the root
port and end port versions are a bit different, so the goal is to read
the end port formatted registers and map them into the root port
format so that userspace can properly see the link state and
configuration.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Stephen Warren - Jan. 28, 2013, 10:06 p.m.
On 01/28/2013 12:55 PM, Jason Gunthorpe wrote:
> On Mon, Jan 28, 2013 at 08:39:47PM +0100, Thomas Petazzoni wrote:
> 
>>> In the Marvell case, this capability can be constructed by pulling
>>> data from the the Express End Point capability of the PCI-E port:
>>
>> I am not sure what you mean by "pulling". Do you mean that I should get
>> informations from the real PCIe interface, from within the emulated
>> PCI-to-PCI bridge implementation? This would unfortunately not be
>> really nice, because until now, the PCI-to-PCI bridge emulation is
>> clearly separated from the Marvell PCIe driver itself. Of course, it
>> could register a hook or something like that, so that the emulated
>> PCI-to-PCI bridge could potentially call back into the Marvell PCIe
>> driver.
> 
> Yes, a callback would be needed to the main driver and IIRC the driver
> can read/write the end port link info config regsiters via MMIO. They
> probably need a bit of massaging to be in root port format, but
> otherwise it should be straightforward..
> 
>> I'll have to dig a little bit more about this capability to see how it
>> works exactly.
> 
> All ports have registers to report and control the link, but the root
> port and end port versions are a bit different, so the goal is to read
> the end port formatted registers and map them into the root port
> format so that userspace can properly see the link state and
> configuration.

Isn't the thing being emulated here a host bridge, which "contains" the
PCIe root ports underneath, which in turn "contain" the PCIe devices
underneath? At least on Tegra, there is no host bridge device that
exposes PCIe config registers, but the PCIe root ports do exist and do
expose PCIe config registers...

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jason Gunthorpe - Jan. 28, 2013, 10:16 p.m.
On Mon, Jan 28, 2013 at 03:06:32PM -0700, Stephen Warren wrote:
> On 01/28/2013 12:55 PM, Jason Gunthorpe wrote:
> > On Mon, Jan 28, 2013 at 08:39:47PM +0100, Thomas Petazzoni wrote:
> > 
> >>> In the Marvell case, this capability can be constructed by pulling
> >>> data from the the Express End Point capability of the PCI-E port:
> >>
> >> I am not sure what you mean by "pulling". Do you mean that I should get
> >> informations from the real PCIe interface, from within the emulated
> >> PCI-to-PCI bridge implementation? This would unfortunately not be
> >> really nice, because until now, the PCI-to-PCI bridge emulation is
> >> clearly separated from the Marvell PCIe driver itself. Of course, it
> >> could register a hook or something like that, so that the emulated
> >> PCI-to-PCI bridge could potentially call back into the Marvell PCIe
> >> driver.
> > 
> > Yes, a callback would be needed to the main driver and IIRC the driver
> > can read/write the end port link info config regsiters via MMIO. They
> > probably need a bit of massaging to be in root port format, but
> > otherwise it should be straightforward..
> > 
> >> I'll have to dig a little bit more about this capability to see how it
> >> works exactly.
> > 
> > All ports have registers to report and control the link, but the root
> > port and end port versions are a bit different, so the goal is to read
> > the end port formatted registers and map them into the root port
> > format so that userspace can properly see the link state and
> > configuration.
> 
> Isn't the thing being emulated here a host bridge, which "contains" the
> PCIe root ports underneath, which in turn "contain" the PCIe devices
> underneath? At least on Tegra, there is no host bridge device that
> exposes PCIe config registers, but the PCIe root ports do exist and do
> expose PCIe config registers...

Patch #7 create a SW emulated host bridge, which tegra and marvell
lack in HW.

Patch #8 creates a SW emulated root port bridge, which tegra has
properly in HW, while Marvell doesn't.

Basically, on the Marvell chips, the PCI config space of the PCI
complex is useless when used as a root complex - the config space is
only usable when the device is configured as an end port.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Bjorn Helgaas - Jan. 29, 2013, 10:35 p.m.
On Mon, Jan 28, 2013 at 11:56 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  drivers/pci/Kconfig             |    3 +
>  drivers/pci/Makefile            |    1 +
>  drivers/pci/sw-pci-pci-bridge.c |  185 +++++++++++++++++++++++++++++++++++++++
>  include/linux/pci.h             |   43 +++++++++
>  4 files changed, 232 insertions(+)
>  create mode 100644 drivers/pci/sw-pci-pci-bridge.c

If you need this, it can be done in architecture code, can't it?  It's
true that there's nothing architecture-specific in this patch (other
than the fact that ARM is the only arch that needs it), but I'm not
sure there's anything useful for sharing here.

In fact, it seems like what you're after is not so much an *emulated*
bridge that has no corresponding hardware, as it is a wrapper that
presents a standard PCIe interface to hardware that exists but doesn't
conform to the PCIe spec.  If you really do need to ultimately connect
this pci_sw_pci_bridge to a piece of hardware, that will certainly be
arch-specific.

> diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
> index f7548e2..6ed3db1 100644
> --- a/drivers/pci/Kconfig
> +++ b/drivers/pci/Kconfig
> @@ -122,3 +122,6 @@ config PCI_LABEL
>
>  config PCI_SW_HOST_BRIDGE
>         bool
> +
> +config PCI_SW_PCI_PCI_BRIDGE
> +       bool
> diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
> index 44ce914..5b48961 100644
> --- a/drivers/pci/Makefile
> +++ b/drivers/pci/Makefile
> @@ -17,6 +17,7 @@ obj-$(CONFIG_PCI_IOAPIC) += ioapic.o
>
>  # Emulated PCI elements
>  obj-$(CONFIG_PCI_SW_HOST_BRIDGE) += sw-host-bridge.o
> +obj-$(CONFIG_PCI_SW_PCI_PCI_BRIDGE) += sw-pci-pci-bridge.o
>
>  # Build the PCI Hotplug drivers if we were asked to
>  obj-$(CONFIG_HOTPLUG_PCI) += hotplug/
> diff --git a/drivers/pci/sw-pci-pci-bridge.c b/drivers/pci/sw-pci-pci-bridge.c
> new file mode 100644
> index 0000000..25679cc
> --- /dev/null
> +++ b/drivers/pci/sw-pci-pci-bridge.c
> @@ -0,0 +1,185 @@
> +/*
> + * Implementation of a simple emulated PCI-to-PCI bridge.
> + *
> + * Thierry Reding <thierry.reding@avionic-design.de>
> + * Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> + *
> + * This file is licensed under the terms of the GNU General Public
> + * License version 2.  This program is licensed "as is" without any
> + * warranty of any kind, whether express or implied.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/pci.h>
> +#include <linux/module.h>
> +
> +int pci_sw_pci_bridge_init(struct pci_sw_pci_bridge *bridge)
> +{
> +       if (!bridge)
> +               return -EINVAL;
> +
> +       memset(bridge, 0, sizeof(struct pci_sw_pci_bridge));
> +
> +       bridge->status = PCI_STATUS_CAP_LIST;
> +       bridge->class = PCI_CLASS_BRIDGE_PCI;
> +       bridge->header_type = PCI_HEADER_TYPE_BRIDGE;
> +       bridge->cache_line_size = 0x10;
> +
> +       /* We support 32 bits I/O addressing */
> +       bridge->iobase = PCI_IO_RANGE_TYPE_32;
> +       bridge->iolimit = PCI_IO_RANGE_TYPE_32;
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(pci_sw_pci_bridge_init);
> +
> +int pci_sw_pci_bridge_read(struct pci_sw_pci_bridge *bridge,
> +                          unsigned int where, int size, u32 *value)
> +{
> +       switch (where & ~3) {
> +       case PCI_VENDOR_ID:
> +               *value = bridge->device << 16 | bridge->vendor;
> +               break;
> +
> +       case PCI_COMMAND:
> +               *value = bridge->status << 16 | bridge->command;
> +               break;
> +
> +       case PCI_CLASS_REVISION:
> +               *value = bridge->class << 16 | bridge->interface << 8 |
> +                        bridge->revision;
> +               break;
> +
> +       case PCI_CACHE_LINE_SIZE:
> +               *value = bridge->bist << 24 | bridge->header_type << 16 |
> +                        bridge->latency_timer << 8 | bridge->cache_line_size;
> +               break;
> +
> +       case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1:
> +               *value = bridge->bar[((where & ~3) - PCI_BASE_ADDRESS_0) / 4];
> +               break;
> +
> +       case PCI_PRIMARY_BUS:
> +               *value = (bridge->secondary_latency_timer << 24 |
> +                         bridge->subordinate_bus         << 16 |
> +                         bridge->secondary_bus           <<  8 |
> +                         bridge->primary_bus);
> +               break;
> +
> +       case PCI_IO_BASE:
> +               *value = (bridge->secondary_status << 16 |
> +                         bridge->iolimit          <<  8 |
> +                         bridge->iobase);
> +               break;
> +
> +       case PCI_MEMORY_BASE:
> +               *value = (bridge->memlimit << 16 | bridge->membase);
> +               break;
> +
> +       case PCI_PREF_MEMORY_BASE:
> +               *value = (bridge->prefmemlimit << 16 | bridge->prefmembase);
> +               break;
> +
> +       case PCI_PREF_BASE_UPPER32:
> +               *value = bridge->prefbaseupper;
> +               break;
> +
> +       case PCI_PREF_LIMIT_UPPER32:
> +               *value = bridge->preflimitupper;
> +               break;
> +
> +       case PCI_IO_BASE_UPPER16:
> +               *value = (bridge->iolimitupper << 16 | bridge->iobaseupper);
> +               break;
> +
> +       case PCI_ROM_ADDRESS1:
> +               *value = 0;
> +               break;
> +
> +       default:
> +               *value = 0xffffffff;
> +               return PCIBIOS_BAD_REGISTER_NUMBER;
> +       }
> +
> +       if (size == 2)
> +               *value = (*value >> (8 * (where & 3))) & 0xffff;
> +       else if (size == 1)
> +               *value = (*value >> (8 * (where & 3))) & 0xff;
> +
> +       return PCIBIOS_SUCCESSFUL;
> +}
> +EXPORT_SYMBOL_GPL(pci_sw_pci_bridge_read);
> +
> +int pci_sw_pci_bridge_write(struct pci_sw_pci_bridge *bridge,
> +                           unsigned int where, int size, u32 value)
> +{
> +       u32 mask, reg;
> +       int err;
> +
> +       if (size == 4)
> +               mask = 0x0;
> +       else if (size == 2)
> +               mask = ~(0xffff << ((where & 3) * 8));
> +       else if (size == 1)
> +               mask = ~(0xff << ((where & 3) * 8));
> +       else
> +               return PCIBIOS_BAD_REGISTER_NUMBER;
> +
> +       err = pci_sw_pci_bridge_read(bridge, where & ~3, 4, &reg);
> +       if (err)
> +               return err;
> +
> +       value = (reg & mask) | value << ((where & 3) * 8);
> +
> +       switch (where & ~3) {
> +       case PCI_COMMAND:
> +               bridge->command = value & 0xffff;
> +               bridge->status = value >> 16;
> +               break;
> +
> +       case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1:
> +               bridge->bar[((where & ~3) - PCI_BASE_ADDRESS_0) / 4] = value;
> +               break;
> +
> +       case PCI_IO_BASE:
> +               /*
> +                * We also keep bit 1 set, it is a read-only bit that
> +                * indicates we support 32 bits addressing for the
> +                * I/O
> +                */
> +               bridge->iobase = (value & 0xff) | PCI_IO_RANGE_TYPE_32;
> +               bridge->iolimit = ((value >> 8) & 0xff) | PCI_IO_RANGE_TYPE_32;
> +               bridge->secondary_status = value >> 16;
> +               break;
> +
> +       case PCI_MEMORY_BASE:
> +               bridge->membase = value & 0xffff;
> +               bridge->memlimit = value >> 16;
> +               break;
> +
> +       case PCI_PREF_MEMORY_BASE:
> +               bridge->prefmembase = value & 0xffff;
> +               bridge->prefmemlimit = value >> 16;
> +               break;
> +
> +       case PCI_PREF_BASE_UPPER32:
> +               bridge->prefbaseupper = value;
> +               break;
> +
> +       case PCI_PREF_LIMIT_UPPER32:
> +               bridge->preflimitupper = value;
> +               break;
> +
> +       case PCI_IO_BASE_UPPER16:
> +               bridge->iobaseupper = value & 0xffff;
> +               bridge->iolimitupper = value >> 16;
> +               break;
> +
> +       default:
> +               break;
> +       }
> +
> +       return PCIBIOS_SUCCESSFUL;
> +}
> +EXPORT_SYMBOL_GPL(pci_sw_pci_bridge_write);
> diff --git a/include/linux/pci.h b/include/linux/pci.h
> index c93e258..b83b4c8 100644
> --- a/include/linux/pci.h
> +++ b/include/linux/pci.h
> @@ -1864,4 +1864,47 @@ extern int pci_sw_host_bridge_read(struct pci_sw_host_bridge *bridge,
>  extern int pci_sw_host_bridge_write(struct pci_sw_host_bridge *bridge,
>                                     unsigned int where, int size, u32 value);
>
> +struct pci_sw_pci_bridge {
> +       u16 vendor;
> +       u16 device;
> +       u16 command;
> +       u16 status;
> +       u16 class;
> +       u8 interface;
> +       u8 revision;
> +       u8 bist;
> +       u8 header_type;
> +       u8 latency_timer;
> +       u8 cache_line_size;
> +       u32 bar[2];
> +       u8 primary_bus;
> +       u8 secondary_bus;
> +       u8 subordinate_bus;
> +       u8 secondary_latency_timer;
> +       u8 iobase;
> +       u8 iolimit;
> +       u16 secondary_status;
> +       u16 membase;
> +       u16 memlimit;
> +       u16 prefmembase;
> +       u16 prefmemlimit;
> +       u32 prefbaseupper;
> +       u32 preflimitupper;
> +       u16 iobaseupper;
> +       u16 iolimitupper;
> +       u8 cappointer;
> +       u8 reserved1;
> +       u16 reserved2;
> +       u32 romaddr;
> +       u8 intline;
> +       u8 intpin;
> +       u16 bridgectrl;
> +};
> +
> +extern int pci_sw_pci_bridge_init(struct pci_sw_pci_bridge *bridge);
> +extern int pci_sw_pci_bridge_read(struct pci_sw_pci_bridge *bridge,
> +                                 unsigned int where, int size, u32 *value);
> +extern int pci_sw_pci_bridge_write(struct pci_sw_pci_bridge *bridge,
> +                                  unsigned int where, int size, u32 value);
> +
>  #endif /* LINUX_PCI_H */
> --
> 1.7.9.5
>
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Arnd Bergmann - Jan. 29, 2013, 11:06 p.m.
On Tuesday 29 January 2013, Bjorn Helgaas wrote:
> If you need this, it can be done in architecture code, can't it?  It's
> true that there's nothing architecture-specific in this patch (other
> than the fact that ARM is the only arch that needs it), but I'm not
> sure there's anything useful for sharing here.

Since we're moving the host bridge code to drivers/pci/host now, I think
this code should live in the same place. It's entirely possible that
it will be shared between arch/arm and arch/arm64, although I would
hope that we can do away with the emulated bridge code entirely.

More generally speaking, this is infrastructure code, and for any
piece of infrastructure my rule is

* don't add platform specific infrastructure if it can be done at
  the architecture level
* don't add architecture specific infrastructure if it can be
  written in an architecture independent way
* don't add global infrastructure if you can use or extend the
  existing infrastructure.

> In fact, it seems like what you're after is not so much an emulated
> bridge that has no corresponding hardware, as it is a wrapper that
> presents a standard PCIe interface to hardware that exists but doesn't
> conform to the PCIe spec.  If you really do need to ultimately connect
> this pci_sw_pci_bridge to a piece of hardware, that will certainly be
> arch-specific.

As Jason Gunthorpe suggested, we might not need this at all, if the
Linux PCI code can be convinced not to need a configuration space
for the devices that in case of the Marvell hardware don't provide
one.

	Arnd
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jason Gunthorpe - Jan. 30, 2013, 4:12 a.m.
On Tue, Jan 29, 2013 at 11:06:13PM +0000, Arnd Bergmann wrote:
> On Tuesday 29 January 2013, Bjorn Helgaas wrote:
> > If you need this, it can be done in architecture code, can't it?  It's
> > true that there's nothing architecture-specific in this patch (other
> > than the fact that ARM is the only arch that needs it), but I'm not
> > sure there's anything useful for sharing here.
> 
> Since we're moving the host bridge code to drivers/pci/host now, I think
> this code should live in the same place. It's entirely possible that
> it will be shared between arch/arm and arch/arm64, although I would
> hope that we can do away with the emulated bridge code entirely.

This sounds right to me, this is part of the host bridge driver for
various Marvell SOCs, so these days it should live in the
drivers/pci/host or related, not arch/arm.
 
> > In fact, it seems like what you're after is not so much an emulated
> > bridge that has no corresponding hardware, as it is a wrapper that
> > presents a standard PCIe interface to hardware that exists but doesn't
> > conform to the PCIe spec.  If you really do need to ultimately connect
> > this pci_sw_pci_bridge to a piece of hardware, that will certainly be
> > arch-specific.
> 
> As Jason Gunthorpe suggested, we might not need this at all, if the
> Linux PCI code can be convinced not to need a configuration space
> for the devices that in case of the Marvell hardware don't provide
> one.

To be clear, that isn't what I was talking about.. Just to clarify a
few things in the last couple emails:

The PCI 'host bridge configuration space' software emulation code in
patch #7 is not necessary. Bjorn and Thierry both confirm this.

In several places when Bjorn/Arnd talked about a 'host bridge' this is
referring to (more or less) the PCI host *driver* and its attachment
to the kernel interfaces. Specifically a configuration access
mechanism and the resource ranges to allocate against. It has nothing
to do with the bus 0, device 0, function 0 host bridge config space.

Arnd's suggestion to use multiple domains would be broadly equivilent
to the first iteration of this driver - essentially the driver would
manage one link and there would be multiple instances. This gets us
back to where Thomas started - there is currently no code to do cross
domain resource allocation, and static allocation is not possible with
so many links on the chip.

Bjorn is quite right, the purpose of the PCI-PCI SW layer is to bind
the non-standard registers in the Marvell SOC to the standard PCI-E
config interface so the kernel can control it normally. This corrects
what is, IMHO, a defect in the Marvell hardware.

The alternative is to add some kind of cross-domain resource
allocation (or similar) to the PCI core code - however this would
*only* be required to support hardware broken in the same way as
Marvell, so I feel a bit leery about doing that kind of work before we
know if other chips require this. (early on in the discussion there
was some thought that Tegra might also be similary broken, but it
turned out to be pretty much fine, with a bit of driver work)

So, I still think using a SW layer to provide a compliant PCI-PCI
bridge configuration space for the Marvell hardware is the best way
forward..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/drivers/pci/Kconfig b/drivers/pci/Kconfig
index f7548e2..6ed3db1 100644
--- a/drivers/pci/Kconfig
+++ b/drivers/pci/Kconfig
@@ -122,3 +122,6 @@  config PCI_LABEL
 
 config PCI_SW_HOST_BRIDGE
 	bool
+
+config PCI_SW_PCI_PCI_BRIDGE
+	bool
diff --git a/drivers/pci/Makefile b/drivers/pci/Makefile
index 44ce914..5b48961 100644
--- a/drivers/pci/Makefile
+++ b/drivers/pci/Makefile
@@ -17,6 +17,7 @@  obj-$(CONFIG_PCI_IOAPIC) += ioapic.o
 
 # Emulated PCI elements
 obj-$(CONFIG_PCI_SW_HOST_BRIDGE) += sw-host-bridge.o
+obj-$(CONFIG_PCI_SW_PCI_PCI_BRIDGE) += sw-pci-pci-bridge.o
 
 # Build the PCI Hotplug drivers if we were asked to
 obj-$(CONFIG_HOTPLUG_PCI) += hotplug/
diff --git a/drivers/pci/sw-pci-pci-bridge.c b/drivers/pci/sw-pci-pci-bridge.c
new file mode 100644
index 0000000..25679cc
--- /dev/null
+++ b/drivers/pci/sw-pci-pci-bridge.c
@@ -0,0 +1,185 @@ 
+/*
+ * Implementation of a simple emulated PCI-to-PCI bridge.
+ *
+ * Thierry Reding <thierry.reding@avionic-design.de>
+ * Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
+ *
+ * This file is licensed under the terms of the GNU General Public
+ * License version 2.  This program is licensed "as is" without any
+ * warranty of any kind, whether express or implied.
+ */
+
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/pci.h>
+#include <linux/module.h>
+
+int pci_sw_pci_bridge_init(struct pci_sw_pci_bridge *bridge)
+{
+	if (!bridge)
+		return -EINVAL;
+
+	memset(bridge, 0, sizeof(struct pci_sw_pci_bridge));
+
+	bridge->status = PCI_STATUS_CAP_LIST;
+	bridge->class = PCI_CLASS_BRIDGE_PCI;
+	bridge->header_type = PCI_HEADER_TYPE_BRIDGE;
+	bridge->cache_line_size = 0x10;
+
+	/* We support 32 bits I/O addressing */
+	bridge->iobase = PCI_IO_RANGE_TYPE_32;
+	bridge->iolimit = PCI_IO_RANGE_TYPE_32;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(pci_sw_pci_bridge_init);
+
+int pci_sw_pci_bridge_read(struct pci_sw_pci_bridge *bridge,
+			   unsigned int where, int size, u32 *value)
+{
+	switch (where & ~3) {
+	case PCI_VENDOR_ID:
+		*value = bridge->device << 16 | bridge->vendor;
+		break;
+
+	case PCI_COMMAND:
+		*value = bridge->status << 16 | bridge->command;
+		break;
+
+	case PCI_CLASS_REVISION:
+		*value = bridge->class << 16 | bridge->interface << 8 |
+			 bridge->revision;
+		break;
+
+	case PCI_CACHE_LINE_SIZE:
+		*value = bridge->bist << 24 | bridge->header_type << 16 |
+			 bridge->latency_timer << 8 | bridge->cache_line_size;
+		break;
+
+	case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1:
+		*value = bridge->bar[((where & ~3) - PCI_BASE_ADDRESS_0) / 4];
+		break;
+
+	case PCI_PRIMARY_BUS:
+		*value = (bridge->secondary_latency_timer << 24 |
+			  bridge->subordinate_bus         << 16 |
+			  bridge->secondary_bus           <<  8 |
+			  bridge->primary_bus);
+		break;
+
+	case PCI_IO_BASE:
+		*value = (bridge->secondary_status << 16 |
+			  bridge->iolimit          <<  8 |
+			  bridge->iobase);
+		break;
+
+	case PCI_MEMORY_BASE:
+		*value = (bridge->memlimit << 16 | bridge->membase);
+		break;
+
+	case PCI_PREF_MEMORY_BASE:
+		*value = (bridge->prefmemlimit << 16 | bridge->prefmembase);
+		break;
+
+	case PCI_PREF_BASE_UPPER32:
+		*value = bridge->prefbaseupper;
+		break;
+
+	case PCI_PREF_LIMIT_UPPER32:
+		*value = bridge->preflimitupper;
+		break;
+
+	case PCI_IO_BASE_UPPER16:
+		*value = (bridge->iolimitupper << 16 | bridge->iobaseupper);
+		break;
+
+	case PCI_ROM_ADDRESS1:
+		*value = 0;
+		break;
+
+	default:
+		*value = 0xffffffff;
+		return PCIBIOS_BAD_REGISTER_NUMBER;
+	}
+
+	if (size == 2)
+		*value = (*value >> (8 * (where & 3))) & 0xffff;
+	else if (size == 1)
+		*value = (*value >> (8 * (where & 3))) & 0xff;
+
+	return PCIBIOS_SUCCESSFUL;
+}
+EXPORT_SYMBOL_GPL(pci_sw_pci_bridge_read);
+
+int pci_sw_pci_bridge_write(struct pci_sw_pci_bridge *bridge,
+			    unsigned int where, int size, u32 value)
+{
+	u32 mask, reg;
+	int err;
+
+	if (size == 4)
+		mask = 0x0;
+	else if (size == 2)
+		mask = ~(0xffff << ((where & 3) * 8));
+	else if (size == 1)
+		mask = ~(0xff << ((where & 3) * 8));
+	else
+		return PCIBIOS_BAD_REGISTER_NUMBER;
+
+	err = pci_sw_pci_bridge_read(bridge, where & ~3, 4, &reg);
+	if (err)
+		return err;
+
+	value = (reg & mask) | value << ((where & 3) * 8);
+
+	switch (where & ~3) {
+	case PCI_COMMAND:
+		bridge->command = value & 0xffff;
+		bridge->status = value >> 16;
+		break;
+
+	case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1:
+		bridge->bar[((where & ~3) - PCI_BASE_ADDRESS_0) / 4] = value;
+		break;
+
+	case PCI_IO_BASE:
+		/*
+		 * We also keep bit 1 set, it is a read-only bit that
+		 * indicates we support 32 bits addressing for the
+		 * I/O
+		 */
+		bridge->iobase = (value & 0xff) | PCI_IO_RANGE_TYPE_32;
+		bridge->iolimit = ((value >> 8) & 0xff) | PCI_IO_RANGE_TYPE_32;
+		bridge->secondary_status = value >> 16;
+		break;
+
+	case PCI_MEMORY_BASE:
+		bridge->membase = value & 0xffff;
+		bridge->memlimit = value >> 16;
+		break;
+
+	case PCI_PREF_MEMORY_BASE:
+		bridge->prefmembase = value & 0xffff;
+		bridge->prefmemlimit = value >> 16;
+		break;
+
+	case PCI_PREF_BASE_UPPER32:
+		bridge->prefbaseupper = value;
+		break;
+
+	case PCI_PREF_LIMIT_UPPER32:
+		bridge->preflimitupper = value;
+		break;
+
+	case PCI_IO_BASE_UPPER16:
+		bridge->iobaseupper = value & 0xffff;
+		bridge->iolimitupper = value >> 16;
+		break;
+
+	default:
+		break;
+	}
+
+	return PCIBIOS_SUCCESSFUL;
+}
+EXPORT_SYMBOL_GPL(pci_sw_pci_bridge_write);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index c93e258..b83b4c8 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1864,4 +1864,47 @@  extern int pci_sw_host_bridge_read(struct pci_sw_host_bridge *bridge,
 extern int pci_sw_host_bridge_write(struct pci_sw_host_bridge *bridge,
 				    unsigned int where, int size, u32 value);
 
+struct pci_sw_pci_bridge {
+	u16 vendor;
+	u16 device;
+	u16 command;
+	u16 status;
+	u16 class;
+	u8 interface;
+	u8 revision;
+	u8 bist;
+	u8 header_type;
+	u8 latency_timer;
+	u8 cache_line_size;
+	u32 bar[2];
+	u8 primary_bus;
+	u8 secondary_bus;
+	u8 subordinate_bus;
+	u8 secondary_latency_timer;
+	u8 iobase;
+	u8 iolimit;
+	u16 secondary_status;
+	u16 membase;
+	u16 memlimit;
+	u16 prefmembase;
+	u16 prefmemlimit;
+	u32 prefbaseupper;
+	u32 preflimitupper;
+	u16 iobaseupper;
+	u16 iolimitupper;
+	u8 cappointer;
+	u8 reserved1;
+	u16 reserved2;
+	u32 romaddr;
+	u8 intline;
+	u8 intpin;
+	u16 bridgectrl;
+};
+
+extern int pci_sw_pci_bridge_init(struct pci_sw_pci_bridge *bridge);
+extern int pci_sw_pci_bridge_read(struct pci_sw_pci_bridge *bridge,
+				  unsigned int where, int size, u32 *value);
+extern int pci_sw_pci_bridge_write(struct pci_sw_pci_bridge *bridge,
+				   unsigned int where, int size, u32 value);
+
 #endif /* LINUX_PCI_H */