[RFC,00/21] Implement NTB Controller using multiple PCI
mbox series

Message ID 20190926112933.8922-1-kishon@ti.com
Headers show
Series
  • Implement NTB Controller using multiple PCI
Related show

Message

Kishon Vijay Abraham I Sept. 26, 2019, 11:29 a.m. UTC
This series is sent as RFC since this series is dependent on
[1] (cannot be merged before that series) and to get early review
comments.

I'll also split this series into smaller chunk when I post the
next revision.

This series is about implementing SW defined NTB using
multiple endpoint instances. This series has been tested using
2 endpoint instances in J7 connected to two DRA7 boards.

This was presented in Linux Plumbers Conference. The presentation
can be found @ [2]

This series:
  *) Add support to define endpoint function using device tree
  *) Add a specification for implementing NTB controller using
     multiple endpoint instances.
  *) Add a NTB endpoint function driver and a NTB host side PCI
     driver that follows the specification.
  *) Add support in PCIe endpoint core to support secondary
     interface.
  *) Add a device tree overlay file to configure J7 as NTB

The test setup is something like below
   +-------------+                                   +-------------+
   |             |                                   |             |
   |    DRA72    |                                   |    DRA76    |
   |             |                                   |             |
   +------^------+                                   +------^------+
          |                                                 |
          |                                                 |
+---------|-------------------------------------------------|---------+
|  +------v------+                                   +------v------+  |
|  |             |                                   |             |  |
|  |     EP      |                                   |     EP      |  |
|  | CONTROLLER1 |                                   | CONTROLLER2 |  |
|  |             <----------------------------------->             |  |
|  |             |                                   |             |  |
|  |             |                                   |             |  |
|  |             |                 J7                |             |  |
|  |             |  (Configured using NTB Function)  |             |  |
|  +-------------+                                   +-------------+  |
+---------------------------------------------------------------------+

Here DRA72 and DRA76 could be replaced with *any* PCI host.

EP side (J7):
=============

In the kernel:
        cd /sys/kernel/config/pci_ep/
        echo 1 > controllers/d800000.pcie-ep/start
        echo 1 > controllers/d000000.pcie-ep/start

RC side (DRA7):
===============
        echo 0000:01:00.0 > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind
        echo 0000:01:00.0 > /sys/bus/pci/drivers/ntb_hw_epf/bind
        modprobe ntb_transport
        modprobe ntb_netdev

On each of the hosts Ethernet Interface will be created.

Provide an IP address to each of the hosts:
HOST1 (dra72):
ifconfig eth2 192.168.1.2 up

HOST2 (dra76):
ifconfig eth2 192.168.1.1 up

Once this is done standard network utilities like ping or iperf can be
used.

root@dra7xx-evm:~# iperf -c 192.168.1.2
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 2.50 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 60814 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   705 MBytes   591 Mbits/sec

[1] -> http://lore.kernel.org/r/20190604131516.13596-1-kishon@ti.com
[2] -> https://www.linuxplumbersconf.org/event/4/contributions/395/attachments/284/481/Implementing_NTB_Controller_Using_PCIe_Endpoint_-_final.pdf

Kishon Vijay Abraham I (21):
  dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Bus
  dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Device
  dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF NTB Device
  Documentation: PCI: Add specification for the *PCI NTB* function
    device
  PCI: endpoint: Add API to get reference to EPC from device-tree
  PCI: endpoint: Add API to create EPF device from device tree
  PCI: endpoint: Add "pci-epf-bus" driver
  PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit
    BAR
  PCI: endpoint: Add helper API to get the 'next' unreserved BAR
  PCI: endpoint: Make pci_epf_driver ops optional
  PCI: endpoint: Add helper API to populate header with values from DT
  PCI: endpoint: Add support to associate secondary EPC with EPF
  PCI: endpoint: Add pci_epc_ops to map MSI irq
  PCI: cadence: Implement ->msi_map_irq() ops
  PCI: endpoint: Remove unused pci_epf_match_device()
  PCI: endpoint: Fix missing mutex_unlock in error case
  PCI: endpoint: *_free_bar() to return error codes on failure
  PCI: endpoint: Add EP function driver to provide NTB functionality
  PCI: Add TI J721E device to pci ids
  NTB: Add support for EPF PCI-Express Non-Transparent Bridge
  NTB: tool: Enable the NTB/PCIe link on the local or remote side of
    bridge

 Documentation/PCI/endpoint/pci-test-ntb.txt   |  315 +++++
 .../bindings/pci/endpoint/pci-epf-bus.txt     |   27 +
 .../bindings/pci/endpoint/pci-epf-ntb.txt     |   31 +
 .../bindings/pci/endpoint/pci-epf.txt         |   28 +
 drivers/ntb/hw/Kconfig                        |    1 +
 drivers/ntb/hw/Makefile                       |    1 +
 drivers/ntb/hw/epf/Kconfig                    |    5 +
 drivers/ntb/hw/epf/Makefile                   |    1 +
 drivers/ntb/hw/epf/ntb_hw_epf.c               |  648 ++++++++++
 drivers/ntb/test/ntb_tool.c                   |    1 +
 drivers/pci/controller/pcie-cadence-ep.c      |   59 +
 drivers/pci/endpoint/Makefile                 |    3 +-
 drivers/pci/endpoint/functions/Kconfig        |   12 +
 drivers/pci/endpoint/functions/Makefile       |    1 +
 drivers/pci/endpoint/functions/pci-epf-ntb.c  | 1143 +++++++++++++++++
 drivers/pci/endpoint/functions/pci-epf-test.c |   12 +-
 drivers/pci/endpoint/pci-ep-cfs.c             |    6 +-
 drivers/pci/endpoint/pci-epc-core.c           |  221 +++-
 drivers/pci/endpoint/pci-epf-bus.c            |   54 +
 drivers/pci/endpoint/pci-epf-core.c           |  133 +-
 include/linux/pci-epc.h                       |   42 +-
 include/linux/pci-epf.h                       |   35 +-
 include/linux/pci_ids.h                       |    1 +
 23 files changed, 2715 insertions(+), 65 deletions(-)
 create mode 100644 Documentation/PCI/endpoint/pci-test-ntb.txt
 create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-bus.txt
 create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-ntb.txt
 create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf.txt
 create mode 100644 drivers/ntb/hw/epf/Kconfig
 create mode 100644 drivers/ntb/hw/epf/Makefile
 create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c
 create mode 100644 drivers/pci/endpoint/functions/pci-epf-ntb.c
 create mode 100644 drivers/pci/endpoint/pci-epf-bus.c

Comments

Jon Mason Nov. 17, 2019, 11:43 p.m. UTC | #1
On Thu, Sep 26, 2019 at 7:30 AM 'Kishon Vijay Abraham I' via linux-ntb
<linux-ntb@googlegroups.com> wrote:
>
> This series is sent as RFC since this series is dependent on
> [1] (cannot be merged before that series) and to get early review
> comments.
>
> I'll also split this series into smaller chunk when I post the
> next revision.
>
> This series is about implementing SW defined NTB using
> multiple endpoint instances. This series has been tested using
> 2 endpoint instances in J7 connected to two DRA7 boards.
>
> This was presented in Linux Plumbers Conference. The presentation
> can be found @ [2]
>
> This series:
>   *) Add support to define endpoint function using device tree
>   *) Add a specification for implementing NTB controller using
>      multiple endpoint instances.
>   *) Add a NTB endpoint function driver and a NTB host side PCI
>      driver that follows the specification.
>   *) Add support in PCIe endpoint core to support secondary
>      interface.
>   *) Add a device tree overlay file to configure J7 as NTB
>
> The test setup is something like below
>    +-------------+                                   +-------------+
>    |             |                                   |             |
>    |    DRA72    |                                   |    DRA76    |
>    |             |                                   |             |
>    +------^------+                                   +------^------+
>           |                                                 |
>           |                                                 |
> +---------|-------------------------------------------------|---------+
> |  +------v------+                                   +------v------+  |
> |  |             |                                   |             |  |
> |  |     EP      |                                   |     EP      |  |
> |  | CONTROLLER1 |                                   | CONTROLLER2 |  |
> |  |             <----------------------------------->             |  |
> |  |             |                                   |             |  |
> |  |             |                                   |             |  |
> |  |             |                 J7                |             |  |
> |  |             |  (Configured using NTB Function)  |             |  |
> |  +-------------+                                   +-------------+  |
> +---------------------------------------------------------------------+
>
> Here DRA72 and DRA76 could be replaced with *any* PCI host.
>
> EP side (J7):
> =============
>
> In the kernel:
>         cd /sys/kernel/config/pci_ep/
>         echo 1 > controllers/d800000.pcie-ep/start
>         echo 1 > controllers/d000000.pcie-ep/start
>
> RC side (DRA7):
> ===============
>         echo 0000:01:00.0 > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind
>         echo 0000:01:00.0 > /sys/bus/pci/drivers/ntb_hw_epf/bind
>         modprobe ntb_transport
>         modprobe ntb_netdev
>
> On each of the hosts Ethernet Interface will be created.
>
> Provide an IP address to each of the hosts:
> HOST1 (dra72):
> ifconfig eth2 192.168.1.2 up
>
> HOST2 (dra76):
> ifconfig eth2 192.168.1.1 up
>
> Once this is done standard network utilities like ping or iperf can be
> used.
>
> root@dra7xx-evm:~# iperf -c 192.168.1.2
> ------------------------------------------------------------
> Client connecting to 192.168.1.2, TCP port 5001
> TCP window size: 2.50 MByte (default)
> ------------------------------------------------------------
> [  3] local 192.168.1.1 port 60814 connected with 192.168.1.2 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec   705 MBytes   591 Mbits/sec
>
> [1] -> http://lore.kernel.org/r/20190604131516.13596-1-kishon@ti.com
> [2] -> https://www.linuxplumbersconf.org/event/4/contributions/395/attachments/284/481/Implementing_NTB_Controller_Using_PCIe_Endpoint_-_final.pdf


I had a few nits, but I think this series looks good enough to be sent
out for inclusion.

Thanks,
Jon

> Kishon Vijay Abraham I (21):
>   dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Bus
>   dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Device
>   dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF NTB Device
>   Documentation: PCI: Add specification for the *PCI NTB* function
>     device
>   PCI: endpoint: Add API to get reference to EPC from device-tree
>   PCI: endpoint: Add API to create EPF device from device tree
>   PCI: endpoint: Add "pci-epf-bus" driver
>   PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit
>     BAR
>   PCI: endpoint: Add helper API to get the 'next' unreserved BAR
>   PCI: endpoint: Make pci_epf_driver ops optional
>   PCI: endpoint: Add helper API to populate header with values from DT
>   PCI: endpoint: Add support to associate secondary EPC with EPF
>   PCI: endpoint: Add pci_epc_ops to map MSI irq
>   PCI: cadence: Implement ->msi_map_irq() ops
>   PCI: endpoint: Remove unused pci_epf_match_device()
>   PCI: endpoint: Fix missing mutex_unlock in error case
>   PCI: endpoint: *_free_bar() to return error codes on failure
>   PCI: endpoint: Add EP function driver to provide NTB functionality
>   PCI: Add TI J721E device to pci ids
>   NTB: Add support for EPF PCI-Express Non-Transparent Bridge
>   NTB: tool: Enable the NTB/PCIe link on the local or remote side of
>     bridge
>
>  Documentation/PCI/endpoint/pci-test-ntb.txt   |  315 +++++
>  .../bindings/pci/endpoint/pci-epf-bus.txt     |   27 +
>  .../bindings/pci/endpoint/pci-epf-ntb.txt     |   31 +
>  .../bindings/pci/endpoint/pci-epf.txt         |   28 +
>  drivers/ntb/hw/Kconfig                        |    1 +
>  drivers/ntb/hw/Makefile                       |    1 +
>  drivers/ntb/hw/epf/Kconfig                    |    5 +
>  drivers/ntb/hw/epf/Makefile                   |    1 +
>  drivers/ntb/hw/epf/ntb_hw_epf.c               |  648 ++++++++++
>  drivers/ntb/test/ntb_tool.c                   |    1 +
>  drivers/pci/controller/pcie-cadence-ep.c      |   59 +
>  drivers/pci/endpoint/Makefile                 |    3 +-
>  drivers/pci/endpoint/functions/Kconfig        |   12 +
>  drivers/pci/endpoint/functions/Makefile       |    1 +
>  drivers/pci/endpoint/functions/pci-epf-ntb.c  | 1143 +++++++++++++++++
>  drivers/pci/endpoint/functions/pci-epf-test.c |   12 +-
>  drivers/pci/endpoint/pci-ep-cfs.c             |    6 +-
>  drivers/pci/endpoint/pci-epc-core.c           |  221 +++-
>  drivers/pci/endpoint/pci-epf-bus.c            |   54 +
>  drivers/pci/endpoint/pci-epf-core.c           |  133 +-
>  include/linux/pci-epc.h                       |   42 +-
>  include/linux/pci-epf.h                       |   35 +-
>  include/linux/pci_ids.h                       |    1 +
>  23 files changed, 2715 insertions(+), 65 deletions(-)
>  create mode 100644 Documentation/PCI/endpoint/pci-test-ntb.txt
>  create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-bus.txt
>  create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-ntb.txt
>  create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf.txt
>  create mode 100644 drivers/ntb/hw/epf/Kconfig
>  create mode 100644 drivers/ntb/hw/epf/Makefile
>  create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c
>  create mode 100644 drivers/pci/endpoint/functions/pci-epf-ntb.c
>  create mode 100644 drivers/pci/endpoint/pci-epf-bus.c
>
> --
> 2.17.1
>
> --
> You received this message because you are subscribed to the Google Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/20190926112933.8922-1-kishon%40ti.com.