[net-next,v5] devlink: Add devlink port documentation

Added documentation for devlink port and port function related commands.

Signed-off-by: Parav Pandit <parav@nvidia.com>
Reviewed-by: Jiri Pirko <jiri@nvidia.com>
Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
---
Changelog:
v4->v5:
 - described logically ingress and egress point of devlink port
 - removed networking from devlink port description
 - rephrased port type description
 - introdue PCI controller section and description
 - rephrased controller, device, function description
 - removed confusing eswitch to system wording
 - rephrased port function description
 - added example of mac address in port function attribute description
v3->v4:
 - changed 'exist' to 'exists'
 - added 'an' eswitch
 - changed 'can have one' to 'consists of'
 - changed 'who intents' to 'that intends'
 - removed unnecessary comma
 - rewrote description for the example diagram
 - changed 'controller consist of' to 'controller consists of'
v2->v3:
 - rephrased many lines
 - first paragraph now describe devlink port
 - instead of saying PCI device/function, using PCI function every
   where
 - changed 'physical link layer' to 'link layer'
 - made devlink port type description more clear
 - made devlink port flavour description more clear
 - moved devlink port type table after port flavour
 - added description for the example diagram
 - describe CPU port that its linked to DSA
 - made devlink port description for eswitch port more clear
v1->v2:
 - Removed duplicate table entries for DEVLINK_PORT_FLAVOUR_VIRTUAL.
 - replaced 'consist of' to 'consisting'
 - changed 'can be' to 'can be of'
---
 .../networking/devlink/devlink-port.rst       | 116 ++++++++++++++++++
 Documentation/networking/devlink/index.rst    |   1 +
 2 files changed, 117 insertions(+)
 create mode 100644 Documentation/networking/devlink/devlink-port.rst

Hi--

On 12/7/20 2:13 PM, Parav Pandit wrote:
> Added documentation for devlink port and port function related commands.
> 
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> Reviewed-by: Jiri Pirko <jiri@nvidia.com>
> Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
> ---
> Changelog:
> v4->v5:
>  - described logically ingress and egress point of devlink port
>  - removed networking from devlink port description
>  - rephrased port type description
>  - introdue PCI controller section and description

     introduce

>  - rephrased controller, device, function description
>  - removed confusing eswitch to system wording
>  - rephrased port function description
>  - added example of mac address in port function attribute description

> ---
>  .../networking/devlink/devlink-port.rst       | 116 ++++++++++++++++++
>  Documentation/networking/devlink/index.rst    |   1 +
>  2 files changed, 117 insertions(+)
>  create mode 100644 Documentation/networking/devlink/devlink-port.rst
> 
> diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst
> new file mode 100644
> index 000000000000..dce87d2c07ac
> --- /dev/null
> +++ b/Documentation/networking/devlink/devlink-port.rst
> @@ -0,0 +1,116 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +============
> +Devlink Port
> +============
> +
> +``devlink-port`` is a port that exists on the device. It has a logically
> +separate ingress/egress point of the device. A devlink port can be of one

                                                          port can be any one
   of many flavours.

> +among many flavours. A devlink port flavour along with port attributes
> +describe what a port represents.
> +
> +A device driver that intends to publish a devlink port sets the
> +devlink port attributes and registers the devlink port.
> +
> +Devlink port flavours are described below.
> +
> +.. list-table:: List of devlink port flavours
> +   :widths: 33 90
> +
> +   * - Flavour
> +     - Description
> +   * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL``
> +     - Any kind of physical networking port. This can be an eswitch physical
> +       port or any other physical port on the device.
> +   * - ``DEVLINK_PORT_FLAVOUR_DSA``
> +     - This indicates a DSA interconnect port.
> +   * - ``DEVLINK_PORT_FLAVOUR_CPU``
> +     - This indicates a CPU port applicable only to DSA.
> +   * - ``DEVLINK_PORT_FLAVOUR_PCI_PF``
> +     - This indicates an eswitch port representing a networking port of
> +       PCI physical function (PF).
> +   * - ``DEVLINK_PORT_FLAVOUR_PCI_VF``
> +     - This indicates an eswitch port representing a networking port of
> +       PCI virtual function (VF).
> +   * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL``
> +     - This indicates a virtual port for the virtual PCI device such as PCI VF.
> +
> +Devlink port can have a different type based on the link layer described below.
> +
> +.. list-table:: List of devlink port types
> +   :widths: 23 90
> +
> +   * - Type
> +     - Description
> +   * - ``DEVLINK_PORT_TYPE_ETH``
> +     - Driver should set this port type when a link layer of the port is
> +       Ethernet.
> +   * - ``DEVLINK_PORT_TYPE_IB``
> +     - Driver should set this port type when a link layer of the port is
> +       InfiniBand.
> +   * - ``DEVLINK_PORT_TYPE_AUTO``
> +     - This type is indicated by the user when driver should detect the port
> +       type automatically.
> +
> +PCI controllers
> +---------------
> +In most cases PCI device has only one controller. A controller consists of

either           PCI devices have
or               a PCI device has

> +potentially multiple physical and virtual functions. Such PCI function consists
> +of one or more ports. This port of the function is represented by the devlink
> +eswitch port.
> +
> +A PCI Device connected to multiple CPUs or multiple PCI root complex or

                                                                complexes

> +SmartNIC, however, may have multiple controllers. For a device with multiple
> +controllers, each controller is distinguished by a unique controller number.
> +An eswitch on the PCI device may suppport ports of multiple controllers. 

                                    support

> +
> +An example view of two controller systems::

I think that this is

  An example view of a two-controller system::

instead of 2 controller systems.  ?

> +
> +In this example, external controller (identified by controller number = 1)
> +doesn't have eswitch. Local controller (identified by controller number = 0)
> +has the eswitch. Devlink instance on local controller has eswitch devlink
> +ports representing ports for both the controllers.
> +
> +                 ---------------------------------------------------------
> +                 |                                                       |
> +                 |           --------- ---------         ------- ------- |
> +    -----------  |           | vf(s) | | sf(s) |         |vf(s)| |sf(s)| |
> +    | server  |  | -------   ----/---- ---/----- ------- ---/--- ---/--- |
> +    | pci rc  |=== | pf0 |______/________/       | pf1 |___/_______/     |
> +    | connect |  | -------                       -------                 |
> +    -----------  |     | controller_num=1 (no eswitch)                   |
> +                 ------|--------------------------------------------------
> +                 (internal wire)
> +                       |
> +                 ---------------------------------------------------------
> +                 | devlink eswitch ports and reps                        |
> +                 | ----------------------------------------------------- |
> +                 | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | |
> +                 | |pf0    | pf0vfN | pf0sfN | pf1    | pf1vfN |pf1sfN | |
> +                 | ----------------------------------------------------- |
> +                 | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | |
> +                 | |pf0    | pf0vfN | pf0sfN | pf1    | pf1vfN |pf1sfN | |
> +                 | ----------------------------------------------------- |
> +                 |                                                       |
> +                 |                                                       |
> +    -----------  |           --------- ---------         ------- ------- |
> +    | smartNIC|  |           | vf(s) | | sf(s) |         |vf(s)| |sf(s)| |
> +    | pci rc  |==| -------   ----/---- ---/----- ------- ---/--- ---/--- |
> +    | connect |  | | pf0 |______/________/       | pf1 |___/_______/     |
> +    -----------  | -------                       -------                 |
> +                 |                                                       |
> +                 |  local controller_num=0 (eswitch)                     |
> +                 ---------------------------------------------------------
> +
> +Port function configuration
> +===========================
> +
> +A user can configure the port function attribute before enumerating the
> +PCI function. Usually it means, user should configure port function attribute
> +before a bus specific device for the function is created. However, when
> +SRIOV is enabled, virtual function devices are created on the PCI bus.
> +Hence, function attribute should be configured before binding virtual
> +function device to the driver.
> +
> +User may set the hardware address of the function represented by the devlink
> +port function. For Ethernet port function this means a MAC address.

Hi Randy,

> From: Randy Dunlap <rdunlap@infradead.org>
> Sent: Tuesday, December 8, 2020 6:45 AM
> 
> > +An eswitch on the PCI device may suppport ports of multiple controllers.
> 
>                                     support
> 
> > +
> > +An example view of two controller systems::
> 
> I think that this is
> 
>   An example view of a two-controller system::
> 
> instead of 2 controller systems.  ?

Rephrasing it as "An example view of a system with two controllers" along with other comments.
Since Saeed is sending the sf patchset which also needs port documentation, I am sending v6 of this patch along with the change log in the subfunction patchset.

> From: Randy Dunlap <rdunlap@infradead.org>
> Sent: Tuesday, December 8, 2020 6:45 AM
> 
> Hi--
> 
> On 12/7/20 2:13 PM, Parav Pandit wrote:
> > Added documentation for devlink port and port function related
> commands.
> >
> > Signed-off-by: Parav Pandit <parav@nvidia.com>
> > Reviewed-by: Jiri Pirko <jiri@nvidia.com>
> > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
> > ---
> > Changelog:
> > v4->v5:
> >  - described logically ingress and egress point of devlink port
> >  - removed networking from devlink port description
> >  - rephrased port type description
> >  - introdue PCI controller section and description
> 
>      introduce
> 
> >  - rephrased controller, device, function description
> >  - removed confusing eswitch to system wording
> >  - rephrased port function description
> >  - added example of mac address in port function attribute description
> 
> > ---
> >  .../networking/devlink/devlink-port.rst       | 116 ++++++++++++++++++
> >  Documentation/networking/devlink/index.rst    |   1 +
> >  2 files changed, 117 insertions(+)
> >  create mode 100644 Documentation/networking/devlink/devlink-port.rst
> >
> > diff --git a/Documentation/networking/devlink/devlink-port.rst
> > b/Documentation/networking/devlink/devlink-port.rst
> > new file mode 100644
> > index 000000000000..dce87d2c07ac
> > --- /dev/null
> > +++ b/Documentation/networking/devlink/devlink-port.rst
> > @@ -0,0 +1,116 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +============
> > +Devlink Port
> > +============
> > +
> > +``devlink-port`` is a port that exists on the device. It has a
> > +logically separate ingress/egress point of the device. A devlink port
> > +can be of one
> 
>                                                           port can be any one
>    of many flavours.
> 
> > +among many flavours. A devlink port flavour along with port
> > +attributes describe what a port represents.
> > +
> > +A device driver that intends to publish a devlink port sets the
> > +devlink port attributes and registers the devlink port.
> > +
> > +Devlink port flavours are described below.
> > +
> > +.. list-table:: List of devlink port flavours
> > +   :widths: 33 90
> > +
> > +   * - Flavour
> > +     - Description
> > +   * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL``
> > +     - Any kind of physical networking port. This can be an eswitch physical
> > +       port or any other physical port on the device.
> > +   * - ``DEVLINK_PORT_FLAVOUR_DSA``
> > +     - This indicates a DSA interconnect port.
> > +   * - ``DEVLINK_PORT_FLAVOUR_CPU``
> > +     - This indicates a CPU port applicable only to DSA.
> > +   * - ``DEVLINK_PORT_FLAVOUR_PCI_PF``
> > +     - This indicates an eswitch port representing a networking port of
> > +       PCI physical function (PF).
I forgot to remove 'networking' from the networking port here.
Will include this fix this in v6 in Saeed's subfunction patchset along with addressing Randy's comments.