| Message ID | 20190926112933.8922-1-kishon@ti.com |
|---|---|
| Headers | show |
| Series | Implement NTB Controller using multiple PCI | expand |
On Thu, Sep 26, 2019 at 7:30 AM 'Kishon Vijay Abraham I' via linux-ntb <linux-ntb@googlegroups.com> wrote: > > Add specification for the *PCI NTB* function device. The endpoint function > driver and the host PCI driver should be created based on this > specification. > > Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > --- > Documentation/PCI/endpoint/pci-test-ntb.txt | 315 ++++++++++++++++++++ > 1 file changed, 315 insertions(+) > create mode 100644 Documentation/PCI/endpoint/pci-test-ntb.txt > > diff --git a/Documentation/PCI/endpoint/pci-test-ntb.txt b/Documentation/PCI/endpoint/pci-test-ntb.txt > new file mode 100644 > index 000000000000..c8bfe9dbfd8b > --- /dev/null > +++ b/Documentation/PCI/endpoint/pci-test-ntb.txt > @@ -0,0 +1,315 @@ > + PCI NTB FUNCTION > + Kishon Vijay Abraham I <kishon@ti.com> > + > +PCI NTB Function allows two different systems (or hosts) to communicate > +with each other by configurig the endpoint instances in such a way that > +transactions from one system is routed to the other system. > + > +In the below diagram, PCI NTB function configures the SoC with multiple > +PCIe Endpoint (EP) instances in such a way that transaction from one EP > +controller is routed to the other EP controller. Once PCI NTB function > +configures the SoC with multiple EP instances, HOST1 and HOST2 can > +communicate with each other using SoC as a bridge. > + > + +-------------+ +-------------+ > + | | | | > + | HOST1 | | HOST2 | > + | | | | > + +------^------+ +------^------+ > + | | > + | | > ++---------|-------------------------------------------------|---------+ > +| +------v------+ +------v------+ | > +| | | | | | > +| | EP | | EP | | > +| | CONTROLLER1 | | CONTROLLER2 | | > +| | <-----------------------------------> | | > +| | | | | | > +| | | | | | > +| | | SoC With Multiple EP Instances | | | > +| | | (Configured using NTB Function) | | | > +| +-------------+ +-------------+ | > ++---------------------------------------------------------------------+ > + > +Constructs used for Implementing NTB: > + > + *) Config Region > + *) Self Scratchpad Registers > + *) Peer Scratchpad Registers > + *) Doorbell Registers > + *) Memory Window > + > +Modeling Constructs: > + > + There are 5 or more distinct regions (config, self scratchpad, peer > +scratchpad, doorbell, one or more memory windows) to be modeled to achieve > +NTB functionality. Atleast one memory window is required while more than > +one is permitted. All these regions should be mapped to BAR for hosts to > +access these regions. > + > +If one 32-bit BAR is allocated for each of these regions, the scheme would > +look like > + BAR0 -> Config Region > + BAR1 -> Self Scratchpad > + BAR2 -> Peer Scratchpad > + BAR3 -> Doorbell > + BAR4 -> Memory Window 1 > + BAR5 -> Memory Window 2 > + > +However if we allocate a separate BAR for each of the region, there would not > +be enough BARs for all the regions in a platform that supports only 64-bit > +BAR. > + > +In order to be be supported by most of the platforms, the regions should be > +packed and mapped to BARs in a way that provides NTB functionality and > +also making sure the hosts doesn't access any region that it is not supposed > +to. > + > +The following scheme is used in EPF NTB Function > + > + BAR0 -> Config Region + Self Scratchpad > + BAR1 -> Peer Scratchpad > + BAR2 -> Doorbell + Memory Window 1 > + BAR3 -> Memory Window 2 > + BAR4 -> Memory Window 3 > + BAR4 -> Memory Window 4 > + > +With this scheme, for the basic NTB functionality 3 BARs should be sufficient. > + > +Modeling Config/Scratchpad Region: > + > ++-----------------+------->+------------------+ +-----------------+ > +| BAR0 | | CONFIG REGION | | BAR0 | > ++-----------------+----+ +------------------+<-------+-----------------+ > +| BAR1 | | |SCRATCHPAD REGION | | BAR1 | > ++-----------------+ +-->+------------------+<-------+-----------------+ > +| BAR2 | Local Memory | BAR2 | > ++-----------------+ +-----------------+ > +| BAR3 | | BAR3 | > ++-----------------+ +-----------------+ > +| BAR4 | | BAR4 | > ++-----------------+ +-----------------+ > +| BAR5 | | BAR5 | > ++-----------------+ +-----------------+ > + EP CONTROLLER 1 EP CONTROLLER 2 > + > +Above diagram shows Config region + Scratchpad region for HOST1 (connected to > +EP controller 1) allocated in local memory. The HOST1 can access the config > +region and scratchpad region (self scratchpad) using BAR0 of EP controller 1. > +The peer host (HOST2 connected to EP controller 2) can also access this > +scratchpad region (peer scratchpad) using BAR1 of EP controller 2. This > +diagram shows the case where Config region and Scratchpad region is allocated > +for HOST1, however the same is applicable for HOST2. > + > +Modeling Doorbell/Memory Window 1: > + > ++-----------------+ +----->+----------------+-----------+-----------------+ > +| BAR0 | | | Doorbell 1 +-----------> MSI|X ADDRESS 1 | > ++-----------------+ | +----------------+ +-----------------+ > +| BAR1 | | | Doorbell 2 +---------+ | | > ++-----------------+ | +----------------+ | | | > +| BAR2 | | | Doorbell 3 +-------+ | +-----------------+ > ++-----------------+ | +----------------+ | +-> MSI|X ADDRESS 2 | > +| BAR3 | | | Doorbell 4 +-----+ | +-----------------+ > ++----------------------+ +----------------+ | | | | > +| BAR4 | | | | | +-----------------+ > ++----------------------+ | MW1 +---+ | +-->+ MSI|X ADDRESS 3|| > +| BAR5 | | | | | | +-----------------+ > ++-----------------+ +----->-----------------+ | | | | > + EP CONTROLLER 1 | | | | +-----------------+ > + | | | +---->+ MSI|X ADDRESS 4 | > + +----------------+ | +-----------------+ > + EP CONTROLLER 2 | | | > + (OB SPACE) | | | > + +-------> MW1 | > + | | > + | | > + +-----------------+ > + | | > + | | > + | | > + | | > + | | > + +-----------------+ > + PCI Address Space > + (Managed by HOST2) > + > +Above diagram shows how the doorbell and memory window 1 is mapped so that > +HOST1 can raise doorbell interrupt on HOST2 and also how HOST1 can access > +buffers exposed by HOST2 using memory window1 (MW1). Here doorbell and > +memory window 1 regions are allocated in EP controller 2 outbound (OB) address > +space. Allocating and configuring BARs for doorbell and memory window1 > +is done during the initialization phase of NTB endpoint function driver. > +Mapping from EP controller 2 OB space to PCI address space is done when HOST2 > +sends CMD_CONFIGURE_MW/CMD_CONFIGURE_DOORBELL. The commands are explained > +below. > + > +Modeling Optional Memory Windows: > + > +This is modeled the same was as MW1 but each of the additional memory windows > +is mapped to separate BARs. > + > +Config Region: > + > +Config Region is a construct that is specific to NTB implemented using NTB > +Endpoint Function Driver. The host and endpoint side NTB function driver will > +exchange informatio with each other using this region. Config Region has s/informatio/information > +Control/Status Registers for configuring the Endpoint Controller. Host can > +write into this region for configuring the outbound ATU and to indicate the > +link status. Endpoint can indicate the status of commands issued be host in > +this region. Endpoint can also indicate the scratchpad offset, number of > +memory windows to the host using this region. > + > +The format of Config Region is given below. Each of the fields here are 32 > +bits. > + > + +------------------------+ > + | COMMAND | > + +------------------------+ > + | ARGUMENT | > + +------------------------+ > + | STATUS | > + +------------------------+ > + | TOPOLOGY | > + +------------------------+ > + | ADDRESS (LOWER 32) | > + +------------------------+ > + | ADDRESS (UPPER 32) | > + +------------------------+ > + | SIZE | > + +------------------------+ > + | MEMORY WINDOW1 OFFSET | > + +------------------------+ > + | NO OF MEMORY WINDOW | > + +------------------------+ > + | SPAD OFFSET | > + +------------------------+ > + | SPAD COUNT | > + +------------------------+ > + | DB ENTRY SIZE | > + +------------------------+ > + | DB DATA | > + +------------------------+ > + | : | > + +------------------------+ > + | : | > + +------------------------+ > + | DB DATA | > + +------------------------+ The number should probably come before the MW1 offset. Also, this assumes that they are all contiguous and of the same size. Optimally, there should be a tuple for each MW with the start and size, and you could parse this with the num of mw you mention above to find out where the SPAD information starts. Worst case, I think you might be able to get away with them all being the same size, but you'll need to say what size that is in another field. Thanks, Jon > + > + > + COMMAND: > + > + NTB function supports three commands: > + > + CMD_CONFIGURE_DOORBELL (0x1): Command to configure doorbell. Before > + invoking this command, the host should allocate and initialize > + MSI/MSI-X vectors (i.e initialize the MSI/MSI-X capability in the > + Endpoint). The endpoint on receiving this command will configure > + the outbound ATU such that transaction to DB BAR will be routed > + to the MSI/MSI-X address programmed by the host. The ARGUMENT > + register should be populated with number of DBs to configure (in the > + lower 16 bits) and if MSI or MSI-X should be configured (BIT 16). > + (TODO: Add support for MSI-X). > + > + CMD_CONFIGURE_MW (0x2): Command to configure memory window. The > + host invokes this command after allocating a buffer that can be > + accessed by remote host. The allocated address should be programmed > + in the ADDRESS register (64 bit), the size should be programmed in > + the SIZE register and the memory window index should be programmed > + in the ARGUMENT register. The endpoint on receiving this command > + will configure the outbound ATU such that trasaction to MW BAR > + will be routed to the address provided by the host. > + > + CMD_LINK_UP (0x3): Command to indicate an NTB application is > + bound to the EP device on the host side. Once the endpoint > + receives this command from both the hosts, the endpoint will > + raise an LINK_UP event to both the hosts to indicate the hosts > + can start communicating with each other. > + > + ARGUMENT: > + > + The value of this register is based on the commands issued in > + command register. See COMMAND section for more information. > + > + configuring memory window and to indicate the host side NTB application > + has initialized. > + > + TOPOLOGY: > + > + Set to NTB_TOPO_B2B_USD for Primary interface > + Set to NTB_TOPO_B2B_DSD for Secondary interface > + > + ADDRESS/SIZE: > + > + Address and Size to be used while configuring the memory window. > + See "CMD_CONFIGURE_MW" for more info. > + > + MEMORY WINDOW1 OFFSET: > + > + Memory Window 1 and Doorbell registers are packed together in the > + same BAR. The initial portion of the region will have doorbell > + registers and the latter portion of the region is for memory window 1. > + This register will specify the offset of the memory window 1. > + > + NO OF MEMORY WINDOW: > + > + Specifies the number of memory windows supported by the NTB device. > + > + SPAD OFFSET: > + > + Self scratchpad region and config region are packed together in the > + same BAR. The initial portion of the will have config region and > + the latter portion of the region is for self scratchpad. This > + register will specify the offset of the self scratchpad registers. > + > + SPAD COUNT: > + > + Specifies the number of scratchpad registers supported by the NTB > + device. > + > + DB ENTRY SIZE: > + > + Used to determine the offset within the DB BAR that should be written > + in order to raise doorbell. EPF NTB can use either MSI/MSI-X to > + ring doorbell (MSI-X support will be added later). MSI uses same > + address for all the interrupts and MSI-X can provide different > + addresses for different interrupts. The MSI/MSI-X address is provided > + by the host and the address it gives is based on the MSI/MSI-X > + implementation supported by the host. For instance, ARM platform > + using GIC ITS will have same MSI-X address for all the interrupts. > + In order to support all the combinations and use the same mechanism > + for both MSI and MSI-X, EPF NTB allocates separate region in the > + Outbound Address Space for each of the interrupts. This region will > + be mapped to the MSI/MSI-X address provided by the host. If a host > + provides the same address for all the interrupts, all the regions > + will be translated to the same address. If a host provides different > + address, the regions will be translated to different address. This > + will ensure there is no difference while raising the doorbell. > + > + DB DATA: > + > + EPF NTB supports 32 interrupts. So there are 32 DB DATA registers. > + This holds the MSI/MSI-X data that has to be written to MSI address > + for raising doorbell interrupt. This will be populated by EPF NTB > + while invoking CMD_CONFIGURE_MW. > + > +Scratchpad Registers: > + > + Each host has it's own register space allocated in the memory of NTB EPC. > + They are both readable and writable from both sides of the bridge. They > + are used by applications built over NTB and can be used to pass control > + and status information between both sides of a device. > + > + Scratchpad registers has 2 parts > + 1) Self Scratchpad: Host's own register space > + 2) Peer Scratchpad: Remote host's register space. > + > +Doorbell Registers: > + > + Registers using which one host can interrupt the other host. > + > +Memory Window: > + > + Actual transfer of data between the two hosts will happen using the > + memory window. > -- > 2.17.1 > > -- > You received this message because you are subscribed to the Google Groups "linux-ntb" group. > To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/20190926112933.8922-5-kishon%40ti.com.
On Thu, Sep 26, 2019 at 7:31 AM 'Kishon Vijay Abraham I' via linux-ntb <linux-ntb@googlegroups.com> wrote: > > Add of_pci_epc_get() and of_pci_epc_get_by_name() to get reference > to EPC from device-tree. This is added in preparation to define > an endpoint function from device tree. I can't get this patch to apply cleanly to my git tree (for the current or any of the previous kernels I tried). Please rebase this series when you send it out as a patch. Thanks, Jon > Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > --- > drivers/pci/endpoint/pci-epc-core.c | 61 +++++++++++++++++++++++++++++ > include/linux/pci-epc.h | 4 +- > 2 files changed, 64 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c > index 5bc094093a47..0c2fdd39090c 100644 > --- a/drivers/pci/endpoint/pci-epc-core.c > +++ b/drivers/pci/endpoint/pci-epc-core.c > @@ -83,6 +83,66 @@ struct pci_epc *pci_epc_get(const char *epc_name) > } > EXPORT_SYMBOL_GPL(pci_epc_get); > > +/** > + * of_pci_epc_get() - get PCI endpoint controller from device node and index > + * @node: device node which contains the phandle to endpoint controller > + * @index: index of the endpoint controller in "epcs" property > + * > + * Returns the EPC corresponding to the _index_ entry in "epcs" property > + * present in device node, after getting a refcount to it or -ENODEV if > + * there is no such EPC or -EPROBE_DEFER if there is a phandle to the phy, > + * but the device is not yet loaded. > + */ > +struct pci_epc *of_pci_epc_get(struct device_node *node, int index) > +{ > + struct device_node *epc_node; > + struct class_dev_iter iter; > + struct pci_epc *epc; > + struct device *dev; > + > + epc_node = of_parse_phandle(node, "epcs", index); > + if (!epc_node) > + return ERR_PTR(-ENODEV); > + > + class_dev_iter_init(&iter, pci_epc_class, NULL, NULL); > + while ((dev = class_dev_iter_next(&iter))) { > + epc = to_pci_epc(dev); > + if (epc_node != epc->dev.of_node) > + continue; > + > + of_node_put(epc_node); > + class_dev_iter_exit(&iter); > + get_device(&epc->dev); > + return epc; > + } > + > + of_node_put(node); > + class_dev_iter_exit(&iter); > + return ERR_PTR(-EPROBE_DEFER); > +} > +EXPORT_SYMBOL_GPL(of_pci_epc_get); > + > +/** > + * of_pci_epc_get_by_name() - get PCI endpoint controller from device node > + * and string > + * @node: device node which contains the phandle to endpoint controller > + * @epc_name: name of endpoint controller as present in "epc-names" property > + * > + * Returns the EPC corresponding to the epc_name in "epc-names" property > + * present in device node. > + */ > +struct pci_epc *of_pci_epc_get_by_name(struct device_node *node, > + const char *epc_name) > +{ > + int index = 0; > + > + if (epc_name) > + index = of_property_match_string(node, "epc-names", epc_name); > + > + return of_pci_epc_get(node, index); > +} > +EXPORT_SYMBOL_GPL(of_pci_epc_get_by_name); > + > /** > * pci_epc_get_first_free_bar() - helper to get first unreserved BAR > * @epc_features: pci_epc_features structure that holds the reserved bar bitmap > @@ -661,6 +721,7 @@ __pci_epc_create(struct device *dev, const struct pci_epc_ops *ops, > device_initialize(&epc->dev); > epc->dev.class = pci_epc_class; > epc->dev.parent = dev; > + epc->dev.of_node = dev->of_node; > epc->ops = ops; > > ret = dev_set_name(&epc->dev, "%s", dev_name(dev)); > diff --git a/include/linux/pci-epc.h b/include/linux/pci-epc.h > index 0fff52675a6b..ef6531af6ed2 100644 > --- a/include/linux/pci-epc.h > +++ b/include/linux/pci-epc.h > @@ -202,7 +202,9 @@ unsigned int pci_epc_get_first_free_bar(const struct pci_epc_features > *epc_features); > struct pci_epc *pci_epc_get(const char *epc_name); > void pci_epc_put(struct pci_epc *epc); > - > +struct pci_epc *of_pci_epc_get(struct device_node *node, int index); > +struct pci_epc *of_pci_epc_get_by_name(struct device_node *node, > + const char *epc_name); > int __pci_epc_mem_init(struct pci_epc *epc, phys_addr_t phys_addr, size_t size, > size_t page_size); > void pci_epc_mem_exit(struct pci_epc *epc); > -- > 2.17.1 > > -- > You received this message because you are subscribed to the Google Groups "linux-ntb" group. > To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/20190926112933.8922-6-kishon%40ti.com.
On Thu, Sep 26, 2019 at 7:32 AM 'Kishon Vijay Abraham I' via linux-ntb <linux-ntb@googlegroups.com> wrote: > > Add support for EPF PCI-Express Non-Transparent Bridge (NTB) device. > This driver is platform independent and could be used by any platform > which have multiple PCIe endpoint instances configured using the > pci-epf-ntb driver. The driver connnects to the standard NTB sub-system > interface. The EPF NTB device has configurable number of memory windows > (Max 4), configurable number of doorbell (Max 32), and configurable > number of scratch-pad registers. > > Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > --- > drivers/ntb/hw/Kconfig | 1 + > drivers/ntb/hw/Makefile | 1 + > drivers/ntb/hw/epf/Kconfig | 5 + > drivers/ntb/hw/epf/Makefile | 1 + > drivers/ntb/hw/epf/ntb_hw_epf.c | 648 ++++++++++++++++++++++++++++++++ > 5 files changed, 656 insertions(+) > create mode 100644 drivers/ntb/hw/epf/Kconfig > create mode 100644 drivers/ntb/hw/epf/Makefile > create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c > > diff --git a/drivers/ntb/hw/Kconfig b/drivers/ntb/hw/Kconfig > index e51b581fd102..623d43a2bec0 100644 > --- a/drivers/ntb/hw/Kconfig > +++ b/drivers/ntb/hw/Kconfig > @@ -1,4 +1,5 @@ > source "drivers/ntb/hw/amd/Kconfig" > source "drivers/ntb/hw/idt/Kconfig" > source "drivers/ntb/hw/intel/Kconfig" > +source "drivers/ntb/hw/epf/Kconfig" > source "drivers/ntb/hw/mscc/Kconfig" > diff --git a/drivers/ntb/hw/Makefile b/drivers/ntb/hw/Makefile > index 923c442db750..48f672ca857a 100644 > --- a/drivers/ntb/hw/Makefile > +++ b/drivers/ntb/hw/Makefile > @@ -1,4 +1,5 @@ > obj-$(CONFIG_NTB_AMD) += amd/ > obj-$(CONFIG_NTB_IDT) += idt/ > obj-$(CONFIG_NTB_INTEL) += intel/ > +obj-$(CONFIG_NTB_EPF) += epf/ > obj-$(CONFIG_NTB_SWITCHTEC) += mscc/ > diff --git a/drivers/ntb/hw/epf/Kconfig b/drivers/ntb/hw/epf/Kconfig > new file mode 100644 > index 000000000000..314485574bf8 > --- /dev/null > +++ b/drivers/ntb/hw/epf/Kconfig > @@ -0,0 +1,5 @@ > +config NTB_EPF > + tristate "Generic EPF Non-Transparent Bridge support" > + help > + This driver supports EPF NTB on configurable endpoint. > + If unsure, say N. > diff --git a/drivers/ntb/hw/epf/Makefile b/drivers/ntb/hw/epf/Makefile > new file mode 100644 > index 000000000000..2f560a422bc6 > --- /dev/null > +++ b/drivers/ntb/hw/epf/Makefile > @@ -0,0 +1 @@ > +obj-$(CONFIG_NTB_EPF) += ntb_hw_epf.o > diff --git a/drivers/ntb/hw/epf/ntb_hw_epf.c b/drivers/ntb/hw/epf/ntb_hw_epf.c > new file mode 100644 > index 000000000000..fba3d88886f2 > --- /dev/null > +++ b/drivers/ntb/hw/epf/ntb_hw_epf.c > @@ -0,0 +1,648 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/** > + * Host side endpoint driver to implement Non-Transparent Bridge functionality > + * > + * Copyright (C) 2019 Texas Instruments > + * Author: Kishon Vijay Abraham I <kishon@ti.com> > + */ > + > +#include <linux/delay.h> > +#include <linux/module.h> > +#include <linux/pci.h> > +#include <linux/slab.h> > +#include <linux/ntb.h> > + > +#define NTB_EPF_COMMAND 0x0 > +#define CMD_CONFIGURE_DOORBELL 1 > +#define CMD_CONFIGURE_MW 2 > +#define CMD_LINK_UP 3 > + > +#define NTB_EPF_ARGUMENT 0x4 > + > +#define NTB_EPF_STATUS 0x8 > +#define COMMAND_STATUS_OK BIT(0) > +#define LINK_STATUS_UP BIT(1) > + > +#define NTB_EPF_TOPOLOGY 0xc > +#define NTB_EPF_ADDR 0x10 > +#define NTB_EPF_SIZE 0x18 > +#define NTB_EPF_MW1_OFFSET 0x1c > +#define NTB_EPF_MW_COUNT 0x20 > +#define NTB_EPF_SPAD_OFFSET 0x24 > +#define NTB_EPF_SPAD_COUNT 0x28 > +#define NTB_EPF_DB_ENTRY_SIZE 0x2c > +#define NTB_EPF_DB_DATA(n) (0x30 + (n) * 4) > + > +#define NTB_MIN_DB_COUNT 2 > +#define NTB_MAX_DB_COUNT 32 > +#define NTB_MW_OFFSET 2 > + > +enum pci_barno { > + BAR_0, > + BAR_1, > + BAR_2, > + BAR_3, > + BAR_4, > + BAR_5, > +}; > + > +struct ntb_epf_dev { > + struct ntb_dev ntb; > + > + enum pci_barno ctrl_reg_bar; > + enum pci_barno peer_spad_reg_bar; > + enum pci_barno db_reg_bar; > + > + unsigned int mw_count; > + unsigned int spad_count; > + unsigned int db_count; > + > + void __iomem *ctrl_reg; > + void __iomem *db_reg; > + void __iomem *peer_spad_reg; > + > + unsigned int self_spad; > + unsigned int peer_spad; > + > + int db_val; > + u64 db_valid_mask; > +}; > + > +#define ntb_ndev(__ntb) container_of(__ntb, struct ntb_epf_dev, ntb) > + > +struct ntb_epf_data { > + /* BAR that contains both control region and self spad region */ > + enum pci_barno ctrl_reg_bar; > + /* BAR that contains peer spad region */ > + enum pci_barno peer_spad_reg_bar; > + /* BAR that contains Doorbell region and Memory window '1' */ > + enum pci_barno db_reg_bar; > +}; > + > +static inline u32 ntb_epf_ctrl_readl(struct ntb_epf_dev *ndev, u32 offset) > +{ > + return readl(ndev->ctrl_reg + offset); > +} > + > +static inline void ntb_epf_ctrl_writel(struct ntb_epf_dev *ndev, u32 offset, > + u32 val) > +{ > + return writel(val, ndev->ctrl_reg + offset); > +} Please no wrappers that hide standard functions. > + > +static int ntb_epf_send_command(struct ntb_epf_dev *ndev, u32 command, > + u32 argument) > +{ > + ktime_t timeout; > + bool timedout; > + u32 status; > + > + ntb_epf_ctrl_writel(ndev, NTB_EPF_ARGUMENT, argument); > + ntb_epf_ctrl_writel(ndev, NTB_EPF_COMMAND, command); > + > + /* wait 50ms */ > + timeout = ktime_add_ms(ktime_get(), 50); Why 50? Seems arbitrary, and like a good thing to be a #define > + while (1) { > + timedout = ktime_after(ktime_get(), timeout); > + status = ntb_epf_ctrl_readl(ndev, NTB_EPF_STATUS); > + > + if (status & COMMAND_STATUS_OK) > + break; > + > + if (WARN_ON(timedout)) > + return -ETIMEDOUT; > + > + usleep_range(5, 10); > + } > + > + ntb_epf_ctrl_writel(ndev, NTB_EPF_STATUS, status & ~COMMAND_STATUS_OK); > + > + return 0; > +} > + > +static int ntb_epf_mw_to_bar(struct ntb_epf_dev *ndev, int idx) > +{ > + if (idx < 0 || idx > ndev->mw_count) > + return -EINVAL; > + > + return idx + 2; > +} > + > +static int ntb_epf_mw_count(struct ntb_dev *ntb, int pidx) > +{ > + if (pidx != NTB_DEF_PEER_IDX) > + return -EINVAL; > + > + return ntb_ndev(ntb)->mw_count; > +} > + > +static int ntb_epf_mw_get_align(struct ntb_dev *ntb, int pidx, int idx, > + resource_size_t *addr_align, > + resource_size_t *size_align, > + resource_size_t *size_max) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + int bar; > + > + if (pidx != NTB_DEF_PEER_IDX) > + return -EINVAL; > + > + bar = ntb_epf_mw_to_bar(ndev, idx); > + if (bar < 0) > + return bar; > + > + if (addr_align) > + *addr_align = SZ_4K; > + > + if (size_align) > + *size_align = 1; > + > + if (size_max) > + *size_max = pci_resource_len(ndev->ntb.pdev, bar); > + > + return 0; > +} > + > +static u64 ntb_epf_link_is_up(struct ntb_dev *ntb, > + enum ntb_speed *speed, > + enum ntb_width *width) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + u32 status; > + > + status = ntb_epf_ctrl_readl(ndev, NTB_EPF_STATUS); > + > + return !!(status & LINK_STATUS_UP); > +} > + > +static u32 ntb_epf_spad_read(struct ntb_dev *ntb, int idx) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + u32 offset; > + > + if (idx < 0 || idx >= ndev->spad_count) > + return 0; > + > + offset = ntb_epf_ctrl_readl(ndev, NTB_EPF_SPAD_OFFSET); > + offset += (idx << 2); > + > + return ntb_epf_ctrl_readl(ndev, offset); > +} > + > +static int ntb_epf_spad_write(struct ntb_dev *ntb, > + int idx, u32 val) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + u32 offset; > + > + if (idx < 0 || idx >= ndev->spad_count) > + return -EINVAL; > + > + offset = ntb_epf_ctrl_readl(ndev, NTB_EPF_SPAD_OFFSET); > + offset += (idx << 2); > + ntb_epf_ctrl_writel(ndev, offset, val); > + > + return 0; > +} > + > +static u32 ntb_epf_peer_spad_read(struct ntb_dev *ntb, int pidx, int idx) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + u32 offset; > + > + if (idx < 0 || idx >= ndev->spad_count) > + return -EINVAL; > + > + offset = (idx << 2); > + return readl(ndev->peer_spad_reg + offset); > +} > + > +static int ntb_epf_peer_spad_write(struct ntb_dev *ntb, int pidx, > + int idx, u32 val) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + u32 offset; > + > + if (idx < 0 || idx >= ndev->spad_count) > + return -EINVAL; > + > + offset = (idx << 2); > + writel(val, ndev->peer_spad_reg + offset); > + > + return 0; > +} > + > +static int ntb_epf_link_enable(struct ntb_dev *ntb, > + enum ntb_speed max_speed, > + enum ntb_width max_width) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + struct device *dev = &ntb->pdev->dev; > + int ret; > + > + ret = ntb_epf_send_command(ndev, CMD_LINK_UP, 0); > + if (ret) { > + dev_err(dev, "Fail to enable link\n"); > + return ret; > + } > + > + return 0; > +} > + > +static int ntb_epf_link_disable(struct ntb_dev *ntb) > +{ > + return 0; > +} > + > +static irqreturn_t ndev_vec_isr(int irq, void *dev) > +{ > + struct ntb_epf_dev *ndev = dev; > + int irq_no; > + > + irq_no = irq - ndev->ntb.pdev->irq; > + ndev->db_val = irq_no + 1; > + > + if (irq_no == 0) > + ntb_link_event(&ndev->ntb); > + else > + ntb_db_event(&ndev->ntb, irq_no); > + > + return IRQ_HANDLED; > +} > + > +static int ntb_epf_init_isr(struct ntb_epf_dev *ndev, int msi_min, int msi_max) > +{ > + struct pci_dev *pdev = ndev->ntb.pdev; > + struct device *dev = &pdev->dev; > + int irq; > + int ret; > + int i; > + > + irq = pci_alloc_irq_vectors(pdev, msi_min, msi_max, PCI_IRQ_MSI); > + if (irq < 0) { > + dev_err(dev, "Failed to get MSI interrupts\n"); > + return irq; > + } > + > + for (i = 0; i < irq; i++) { > + ret = devm_request_irq(&pdev->dev, pci_irq_vector(pdev, i), > + ndev_vec_isr, IRQF_SHARED, "ntb_epf", > + ndev); > + if (ret) { > + dev_err(dev, "Failed to request irq\n"); > + goto err_request_irq; > + } > + } > + > + ndev->db_count = irq; > + > + ret = ntb_epf_send_command(ndev, CMD_CONFIGURE_DOORBELL, irq); > + if (ret) { > + dev_err(dev, "Failed to configure doorbell\n"); > + goto err_request_irq; > + } > + > + return 0; > + > +err_request_irq: > + pci_free_irq_vectors(pdev); > + > + return ret; > +} > + > +static int ntb_epf_peer_mw_count(struct ntb_dev *ntb) > +{ > + return ntb_ndev(ntb)->mw_count; > +} > + > +static int ntb_epf_spad_count(struct ntb_dev *ntb) > +{ > + return ntb_ndev(ntb)->spad_count; > +} > + > +static u64 ntb_epf_db_valid_mask(struct ntb_dev *ntb) > +{ > + return ntb_ndev(ntb)->db_valid_mask; > +} > + > +static int ntb_epf_db_set_mask(struct ntb_dev *ntb, u64 db_bits) > +{ > + return 0; > +} > + > +static int ntb_epf_mw_set_trans(struct ntb_dev *ntb, int pidx, int idx, > + dma_addr_t addr, resource_size_t size) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + struct device *dev = &ntb->pdev->dev; > + resource_size_t mw_size; > + int bar; > + > + if (pidx != NTB_DEF_PEER_IDX) > + return -EINVAL; > + > + bar = idx + NTB_MW_OFFSET; > + > + mw_size = pci_resource_len(ntb->pdev, bar); > + > + if (size > mw_size) { > + dev_err(dev, "Size is greater than the MW size\n"); > + return -EINVAL; > + } > + > + ntb_epf_ctrl_writel(ndev, NTB_EPF_ADDR, addr); > + ntb_epf_ctrl_writel(ndev, NTB_EPF_SIZE, size); > + ntb_epf_send_command(ndev, CMD_CONFIGURE_MW, idx); > + > + return 0; > +} > + > +static int ntb_epf_peer_mw_get_addr(struct ntb_dev *ntb, int idx, > + phys_addr_t *base, resource_size_t *size) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + u32 offset = 0; > + int bar; > + > + if (idx == 0) > + offset = ntb_epf_ctrl_readl(ndev, NTB_EPF_MW1_OFFSET); > + > + bar = idx + NTB_MW_OFFSET; > + > + if (base) > + *base = pci_resource_start(ndev->ntb.pdev, bar) + offset; > + > + if (size) > + *size = pci_resource_len(ndev->ntb.pdev, bar) - offset; > + > + return 0; > +} > + > +static int ntb_epf_peer_db_set(struct ntb_dev *ntb, u64 db_bits) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + u32 interrupt_num = ffs(db_bits) + 1; > + u32 db_entry_size; > + u32 db_data; > + > + if (interrupt_num > ndev->db_count) > + return -EINVAL; > + > + db_entry_size = ntb_epf_ctrl_readl(ndev, NTB_EPF_DB_ENTRY_SIZE); > + > + db_data = readl(ndev->ctrl_reg + NTB_EPF_DB_DATA(interrupt_num)); > + writel(db_data, ndev->db_reg + (db_entry_size * interrupt_num)); > + > + return 0; > +} > + > +static u64 ntb_epf_db_read(struct ntb_dev *ntb) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + > + return ndev->db_val; > +} > + > +static int ntb_epf_db_clear_mask(struct ntb_dev *ntb, u64 db_bits) > +{ > + return 0; > +} > + > +static int ntb_epf_db_clear(struct ntb_dev *ntb, u64 db_bits) > +{ > + struct ntb_epf_dev *ndev = ntb_ndev(ntb); > + > + ndev->db_val = 0; > + > + return 0; > +} > + > +static const struct ntb_dev_ops ntb_epf_ops = { > + .mw_count = ntb_epf_mw_count, > + .spad_count = ntb_epf_spad_count, > + .peer_mw_count = ntb_epf_peer_mw_count, > + .db_valid_mask = ntb_epf_db_valid_mask, > + .db_set_mask = ntb_epf_db_set_mask, > + .mw_set_trans = ntb_epf_mw_set_trans, > + .peer_mw_get_addr = ntb_epf_peer_mw_get_addr, > + .link_enable = ntb_epf_link_enable, > + .spad_read = ntb_epf_spad_read, > + .spad_write = ntb_epf_spad_write, > + .peer_spad_read = ntb_epf_peer_spad_read, > + .peer_spad_write = ntb_epf_peer_spad_write, > + .peer_db_set = ntb_epf_peer_db_set, > + .db_read = ntb_epf_db_read, > + .mw_get_align = ntb_epf_mw_get_align, > + .link_is_up = ntb_epf_link_is_up, > + .db_clear_mask = ntb_epf_db_clear_mask, > + .db_clear = ntb_epf_db_clear, > + .link_disable = ntb_epf_link_disable, > +}; > + > +static inline void ntb_epf_init_struct(struct ntb_epf_dev *ndev, > + struct pci_dev *pdev) > +{ > + ndev->ntb.pdev = pdev; > + ndev->ntb.topo = NTB_TOPO_NONE; > + ndev->ntb.ops = &ntb_epf_ops; > +} > + > +static int ntb_epf_init_dev(struct ntb_epf_dev *ndev) > +{ > + struct pci_dev *pdev = ndev->ntb.pdev; > + struct device *dev = &pdev->dev; > + int ret; > + > + ret = ntb_epf_init_isr(ndev, NTB_MIN_DB_COUNT, NTB_MAX_DB_COUNT); > + if (ret) { > + dev_err(dev, "Failed to init ISR\n"); > + return ret; > + } > + > + ndev->db_valid_mask = BIT_ULL(ndev->db_count) - 1; > + ndev->mw_count = ntb_epf_ctrl_readl(ndev, NTB_EPF_MW_COUNT); > + ndev->spad_count = ntb_epf_ctrl_readl(ndev, NTB_EPF_SPAD_COUNT); > + > + return 0; > +} > + > +static int ntb_epf_init_pci(struct ntb_epf_dev *ndev, > + struct pci_dev *pdev) > +{ > + struct device *dev = &pdev->dev; > + int ret; > + > + pci_set_drvdata(pdev, ndev); > + > + ret = pci_enable_device(pdev); > + if (ret) { > + dev_err(dev, "Cannot enable PCI device\n"); > + goto err_pci_enable; > + } > + > + ret = pci_request_regions(pdev, "ntb"); > + if (ret) { > + dev_err(dev, "Cannot obtain PCI resources\n"); > + goto err_pci_regions; > + } > + > + pci_set_master(pdev); > + > + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(64)); > + if (ret) { > + ret = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); > + if (ret) { > + dev_err(dev, "Cannot set DMA mask\n"); > + goto err_dma_mask; > + } > + dev_warn(&pdev->dev, "Cannot DMA highmem\n"); > + } > + > + ret = dma_coerce_mask_and_coherent(&ndev->ntb.dev, > + dma_get_mask(&pdev->dev)); > + if (ret) { > + dev_err(dev, "Cannot set DMA mask\n"); > + goto err_dma_mask; > + } > + > + ndev->ctrl_reg = pci_iomap(pdev, 0, 0); > + if (!ndev->ctrl_reg) { > + ret = -EIO; > + goto err_dma_mask; > + } > + > + ndev->peer_spad_reg = pci_iomap(pdev, 1, 0); > + if (!ndev->peer_spad_reg) { > + ret = -EIO; > + goto err_dma_mask; > + } > + > + ndev->db_reg = pci_iomap(pdev, 2, 0); > + if (!ndev->db_reg) { > + ret = -EIO; > + goto err_dma_mask; > + } > + > + return 0; > + > +err_dma_mask: > + pci_clear_master(pdev); > + > +err_pci_regions: > + pci_disable_device(pdev); > + > +err_pci_enable: > + pci_set_drvdata(pdev, NULL); > + > + return ret; > +} > + > +static void ntb_epf_deinit_pci(struct ntb_epf_dev *ndev) > +{ > + struct pci_dev *pdev = ndev->ntb.pdev; > + > + pci_iounmap(pdev, ndev->ctrl_reg); > + pci_iounmap(pdev, ndev->peer_spad_reg); > + pci_iounmap(pdev, ndev->db_reg); > + > + pci_clear_master(pdev); > + pci_release_regions(pdev); > + pci_disable_device(pdev); > + pci_set_drvdata(pdev, NULL); > +} > + > +static int ntb_epf_pci_probe(struct pci_dev *pdev, > + const struct pci_device_id *id) > +{ > + enum pci_barno peer_spad_reg_bar = BAR_1; > + enum pci_barno ctrl_reg_bar = BAR_0; > + enum pci_barno db_reg_bar = BAR_2; > + struct device *dev = &pdev->dev; > + struct ntb_epf_data *data; > + struct ntb_epf_dev *ndev; > + int ret; > + > + ndev = devm_kzalloc(dev, sizeof(*ndev), GFP_KERNEL); > + if (!ndev) > + return -ENOMEM; > + > + data = (struct ntb_epf_data *)id->driver_data; > + if (data) { > + if (data->peer_spad_reg_bar) > + peer_spad_reg_bar = data->peer_spad_reg_bar; > + if (data->ctrl_reg_bar) > + ctrl_reg_bar = data->ctrl_reg_bar; > + if (data->db_reg_bar) > + db_reg_bar = data->db_reg_bar; > + } > + > + ndev->peer_spad_reg_bar = peer_spad_reg_bar; > + ndev->ctrl_reg_bar = ctrl_reg_bar; > + ndev->db_reg_bar = db_reg_bar; > + > + ntb_epf_init_struct(ndev, pdev); > + > + ret = ntb_epf_init_pci(ndev, pdev); > + if (ret) { > + dev_err(dev, "Failed to init PCI\n"); > + return ret; > + } > + > + ret = ntb_epf_init_dev(ndev); > + if (ret) { > + dev_err(dev, "Failed to init device\n"); > + goto err_init_dev; > + } > + > + ret = ntb_register_device(&ndev->ntb); > + if (ret) { > + dev_err(dev, "Failed to register NTB device\n"); > + goto err_register_dev; > + } > + > + return 0; > + > +err_register_dev: > + pci_free_irq_vectors(pdev); > + > +err_init_dev: > + ntb_epf_deinit_pci(ndev); > + > + return ret; > +} > + > +static void ntb_epf_pci_remove(struct pci_dev *pdev) > +{ > + struct ntb_epf_dev *ndev = pci_get_drvdata(pdev); > + > + ntb_unregister_device(&ndev->ntb); > + pci_free_irq_vectors(pdev); > + kfree(ndev); > +} > + > +static const struct ntb_epf_data j721e_data = { > + .ctrl_reg_bar = BAR_0, > + .peer_spad_reg_bar = BAR_1, > + .db_reg_bar = BAR_2, > +}; > + > +static const struct pci_device_id ntb_epf_pci_tbl[] = { > + { > + PCI_DEVICE(PCI_VENDOR_ID_TI, PCI_DEVICE_ID_TI_J721E), > + .driver_data = (kernel_ulong_t)&j721e_data, > + }, > + { }, > +}; > +MODULE_DEVICE_TABLE(pci, ntb_epf_pci_tbl); > + > +static struct pci_driver ntb_epf_pci_driver = { > + .name = KBUILD_MODNAME, > + .id_table = ntb_epf_pci_tbl, > + .probe = ntb_epf_pci_probe, > + .remove = ntb_epf_pci_remove, > +}; > +module_pci_driver(ntb_epf_pci_driver); > + > +MODULE_DESCRIPTION("PCI ENDPOINT NTB HOST DRIVER"); > +MODULE_AUTHOR("Kishon Vijay Abraham I <kishon@ti.com>"); > +MODULE_LICENSE("GPL v2"); > -- > 2.17.1 > > -- > You received this message because you are subscribed to the Google Groups "linux-ntb" group. > To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/20190926112933.8922-21-kishon%40ti.com.
On Thu, Sep 26, 2019 at 7:30 AM 'Kishon Vijay Abraham I' via linux-ntb <linux-ntb@googlegroups.com> wrote: > > This series is sent as RFC since this series is dependent on > [1] (cannot be merged before that series) and to get early review > comments. > > I'll also split this series into smaller chunk when I post the > next revision. > > This series is about implementing SW defined NTB using > multiple endpoint instances. This series has been tested using > 2 endpoint instances in J7 connected to two DRA7 boards. > > This was presented in Linux Plumbers Conference. The presentation > can be found @ [2] > > This series: > *) Add support to define endpoint function using device tree > *) Add a specification for implementing NTB controller using > multiple endpoint instances. > *) Add a NTB endpoint function driver and a NTB host side PCI > driver that follows the specification. > *) Add support in PCIe endpoint core to support secondary > interface. > *) Add a device tree overlay file to configure J7 as NTB > > The test setup is something like below > +-------------+ +-------------+ > | | | | > | DRA72 | | DRA76 | > | | | | > +------^------+ +------^------+ > | | > | | > +---------|-------------------------------------------------|---------+ > | +------v------+ +------v------+ | > | | | | | | > | | EP | | EP | | > | | CONTROLLER1 | | CONTROLLER2 | | > | | <-----------------------------------> | | > | | | | | | > | | | | | | > | | | J7 | | | > | | | (Configured using NTB Function) | | | > | +-------------+ +-------------+ | > +---------------------------------------------------------------------+ > > Here DRA72 and DRA76 could be replaced with *any* PCI host. > > EP side (J7): > ============= > > In the kernel: > cd /sys/kernel/config/pci_ep/ > echo 1 > controllers/d800000.pcie-ep/start > echo 1 > controllers/d000000.pcie-ep/start > > RC side (DRA7): > =============== > echo 0000:01:00.0 > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind > echo 0000:01:00.0 > /sys/bus/pci/drivers/ntb_hw_epf/bind > modprobe ntb_transport > modprobe ntb_netdev > > On each of the hosts Ethernet Interface will be created. > > Provide an IP address to each of the hosts: > HOST1 (dra72): > ifconfig eth2 192.168.1.2 up > > HOST2 (dra76): > ifconfig eth2 192.168.1.1 up > > Once this is done standard network utilities like ping or iperf can be > used. > > root@dra7xx-evm:~# iperf -c 192.168.1.2 > ------------------------------------------------------------ > Client connecting to 192.168.1.2, TCP port 5001 > TCP window size: 2.50 MByte (default) > ------------------------------------------------------------ > [ 3] local 192.168.1.1 port 60814 connected with 192.168.1.2 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 705 MBytes 591 Mbits/sec > > [1] -> http://lore.kernel.org/r/20190604131516.13596-1-kishon@ti.com > [2] -> https://www.linuxplumbersconf.org/event/4/contributions/395/attachments/284/481/Implementing_NTB_Controller_Using_PCIe_Endpoint_-_final.pdf I had a few nits, but I think this series looks good enough to be sent out for inclusion. Thanks, Jon > Kishon Vijay Abraham I (21): > dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Bus > dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Device > dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF NTB Device > Documentation: PCI: Add specification for the *PCI NTB* function > device > PCI: endpoint: Add API to get reference to EPC from device-tree > PCI: endpoint: Add API to create EPF device from device tree > PCI: endpoint: Add "pci-epf-bus" driver > PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit > BAR > PCI: endpoint: Add helper API to get the 'next' unreserved BAR > PCI: endpoint: Make pci_epf_driver ops optional > PCI: endpoint: Add helper API to populate header with values from DT > PCI: endpoint: Add support to associate secondary EPC with EPF > PCI: endpoint: Add pci_epc_ops to map MSI irq > PCI: cadence: Implement ->msi_map_irq() ops > PCI: endpoint: Remove unused pci_epf_match_device() > PCI: endpoint: Fix missing mutex_unlock in error case > PCI: endpoint: *_free_bar() to return error codes on failure > PCI: endpoint: Add EP function driver to provide NTB functionality > PCI: Add TI J721E device to pci ids > NTB: Add support for EPF PCI-Express Non-Transparent Bridge > NTB: tool: Enable the NTB/PCIe link on the local or remote side of > bridge > > Documentation/PCI/endpoint/pci-test-ntb.txt | 315 +++++ > .../bindings/pci/endpoint/pci-epf-bus.txt | 27 + > .../bindings/pci/endpoint/pci-epf-ntb.txt | 31 + > .../bindings/pci/endpoint/pci-epf.txt | 28 + > drivers/ntb/hw/Kconfig | 1 + > drivers/ntb/hw/Makefile | 1 + > drivers/ntb/hw/epf/Kconfig | 5 + > drivers/ntb/hw/epf/Makefile | 1 + > drivers/ntb/hw/epf/ntb_hw_epf.c | 648 ++++++++++ > drivers/ntb/test/ntb_tool.c | 1 + > drivers/pci/controller/pcie-cadence-ep.c | 59 + > drivers/pci/endpoint/Makefile | 3 +- > drivers/pci/endpoint/functions/Kconfig | 12 + > drivers/pci/endpoint/functions/Makefile | 1 + > drivers/pci/endpoint/functions/pci-epf-ntb.c | 1143 +++++++++++++++++ > drivers/pci/endpoint/functions/pci-epf-test.c | 12 +- > drivers/pci/endpoint/pci-ep-cfs.c | 6 +- > drivers/pci/endpoint/pci-epc-core.c | 221 +++- > drivers/pci/endpoint/pci-epf-bus.c | 54 + > drivers/pci/endpoint/pci-epf-core.c | 133 +- > include/linux/pci-epc.h | 42 +- > include/linux/pci-epf.h | 35 +- > include/linux/pci_ids.h | 1 + > 23 files changed, 2715 insertions(+), 65 deletions(-) > create mode 100644 Documentation/PCI/endpoint/pci-test-ntb.txt > create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-bus.txt > create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-ntb.txt > create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf.txt > create mode 100644 drivers/ntb/hw/epf/Kconfig > create mode 100644 drivers/ntb/hw/epf/Makefile > create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c > create mode 100644 drivers/pci/endpoint/functions/pci-epf-ntb.c > create mode 100644 drivers/pci/endpoint/pci-epf-bus.c > > -- > 2.17.1 > > -- > You received this message because you are subscribed to the Google Groups "linux-ntb" group. > To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/20190926112933.8922-1-kishon%40ti.com.
This series is sent as RFC since this series is dependent on [1] (cannot be merged before that series) and to get early review comments. I'll also split this series into smaller chunk when I post the next revision. This series is about implementing SW defined NTB using multiple endpoint instances. This series has been tested using 2 endpoint instances in J7 connected to two DRA7 boards. This was presented in Linux Plumbers Conference. The presentation can be found @ [2] This series: *) Add support to define endpoint function using device tree *) Add a specification for implementing NTB controller using multiple endpoint instances. *) Add a NTB endpoint function driver and a NTB host side PCI driver that follows the specification. *) Add support in PCIe endpoint core to support secondary interface. *) Add a device tree overlay file to configure J7 as NTB The test setup is something like below +-------------+ +-------------+ | | | | | DRA72 | | DRA76 | | | | | +------^------+ +------^------+ | | | | +---------|-------------------------------------------------|---------+ | +------v------+ +------v------+ | | | | | | | | | EP | | EP | | | | CONTROLLER1 | | CONTROLLER2 | | | | <-----------------------------------> | | | | | | | | | | | | | | | | | J7 | | | | | | (Configured using NTB Function) | | | | +-------------+ +-------------+ | +---------------------------------------------------------------------+ Here DRA72 and DRA76 could be replaced with *any* PCI host. EP side (J7): ============= In the kernel: cd /sys/kernel/config/pci_ep/ echo 1 > controllers/d800000.pcie-ep/start echo 1 > controllers/d000000.pcie-ep/start RC side (DRA7): =============== echo 0000:01:00.0 > /sys/bus/pci/devices/0000\:01\:00.0/driver/unbind echo 0000:01:00.0 > /sys/bus/pci/drivers/ntb_hw_epf/bind modprobe ntb_transport modprobe ntb_netdev On each of the hosts Ethernet Interface will be created. Provide an IP address to each of the hosts: HOST1 (dra72): ifconfig eth2 192.168.1.2 up HOST2 (dra76): ifconfig eth2 192.168.1.1 up Once this is done standard network utilities like ping or iperf can be used. root@dra7xx-evm:~# iperf -c 192.168.1.2 ------------------------------------------------------------ Client connecting to 192.168.1.2, TCP port 5001 TCP window size: 2.50 MByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.1 port 60814 connected with 192.168.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 705 MBytes 591 Mbits/sec [1] -> http://lore.kernel.org/r/20190604131516.13596-1-kishon@ti.com [2] -> https://www.linuxplumbersconf.org/event/4/contributions/395/attachments/284/481/Implementing_NTB_Controller_Using_PCIe_Endpoint_-_final.pdf Kishon Vijay Abraham I (21): dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Bus dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF Device dt-bindings: PCI: Endpoint: Add DT bindings for PCI EPF NTB Device Documentation: PCI: Add specification for the *PCI NTB* function device PCI: endpoint: Add API to get reference to EPC from device-tree PCI: endpoint: Add API to create EPF device from device tree PCI: endpoint: Add "pci-epf-bus" driver PCI: endpoint: Make *_get_first_free_bar() take into account 64 bit BAR PCI: endpoint: Add helper API to get the 'next' unreserved BAR PCI: endpoint: Make pci_epf_driver ops optional PCI: endpoint: Add helper API to populate header with values from DT PCI: endpoint: Add support to associate secondary EPC with EPF PCI: endpoint: Add pci_epc_ops to map MSI irq PCI: cadence: Implement ->msi_map_irq() ops PCI: endpoint: Remove unused pci_epf_match_device() PCI: endpoint: Fix missing mutex_unlock in error case PCI: endpoint: *_free_bar() to return error codes on failure PCI: endpoint: Add EP function driver to provide NTB functionality PCI: Add TI J721E device to pci ids NTB: Add support for EPF PCI-Express Non-Transparent Bridge NTB: tool: Enable the NTB/PCIe link on the local or remote side of bridge Documentation/PCI/endpoint/pci-test-ntb.txt | 315 +++++ .../bindings/pci/endpoint/pci-epf-bus.txt | 27 + .../bindings/pci/endpoint/pci-epf-ntb.txt | 31 + .../bindings/pci/endpoint/pci-epf.txt | 28 + drivers/ntb/hw/Kconfig | 1 + drivers/ntb/hw/Makefile | 1 + drivers/ntb/hw/epf/Kconfig | 5 + drivers/ntb/hw/epf/Makefile | 1 + drivers/ntb/hw/epf/ntb_hw_epf.c | 648 ++++++++++ drivers/ntb/test/ntb_tool.c | 1 + drivers/pci/controller/pcie-cadence-ep.c | 59 + drivers/pci/endpoint/Makefile | 3 +- drivers/pci/endpoint/functions/Kconfig | 12 + drivers/pci/endpoint/functions/Makefile | 1 + drivers/pci/endpoint/functions/pci-epf-ntb.c | 1143 +++++++++++++++++ drivers/pci/endpoint/functions/pci-epf-test.c | 12 +- drivers/pci/endpoint/pci-ep-cfs.c | 6 +- drivers/pci/endpoint/pci-epc-core.c | 221 +++- drivers/pci/endpoint/pci-epf-bus.c | 54 + drivers/pci/endpoint/pci-epf-core.c | 133 +- include/linux/pci-epc.h | 42 +- include/linux/pci-epf.h | 35 +- include/linux/pci_ids.h | 1 + 23 files changed, 2715 insertions(+), 65 deletions(-) create mode 100644 Documentation/PCI/endpoint/pci-test-ntb.txt create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-bus.txt create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf-ntb.txt create mode 100644 Documentation/devicetree/bindings/pci/endpoint/pci-epf.txt create mode 100644 drivers/ntb/hw/epf/Kconfig create mode 100644 drivers/ntb/hw/epf/Makefile create mode 100644 drivers/ntb/hw/epf/ntb_hw_epf.c create mode 100644 drivers/pci/endpoint/functions/pci-epf-ntb.c create mode 100644 drivers/pci/endpoint/pci-epf-bus.c