| Message ID | 20201203180255.5253-1-parav@nvidia.com |
|---|---|
| State | Not Applicable |
| Headers | show |
| Series | [net-next,v4] devlink: Add devlink port documentation | expand |
Hi-- On 12/3/20 10:02 AM, Parav Pandit wrote: > Added documentation for devlink port and port function related commands. > > Signed-off-by: Parav Pandit <parav@nvidia.com> > Reviewed-by: Jiri Pirko <jiri@nvidia.com> > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > --- > Changelog: > v3->v4: > - changed 'exist' to 'exists' > - added 'an' eswitch > - changed 'can have one' to 'consists of' > - changed 'who intents' to 'that intends' > - removed unnecessary comma > - rewrote description for the example diagram > - changed 'controller consist of' to 'controller consists of' > v2->v3: > - rephrased many lines > - first paragraph now describe devlink port > - instead of saying PCI device/function, using PCI function every > where > - changed 'physical link layer' to 'link layer' > - made devlink port type description more clear > - made devlink port flavour description more clear > - moved devlink port type table after port flavour > - added description for the example diagram > - describe CPU port that its linked to DSA > - made devlink port description for eswitch port more clear > v1->v2: > - Removed duplicate table entries for DEVLINK_PORT_FLAVOUR_VIRTUAL. > - replaced 'consist of' to 'consisting' > - changed 'can be' to 'can be of' > --- > .../networking/devlink/devlink-port.rst | 111 ++++++++++++++++++ > Documentation/networking/devlink/index.rst | 1 + > 2 files changed, 112 insertions(+) > create mode 100644 Documentation/networking/devlink/devlink-port.rst > > diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst > new file mode 100644 > index 000000000000..ac18cb8041dc > --- /dev/null > +++ b/Documentation/networking/devlink/devlink-port.rst > @@ -0,0 +1,111 @@ I find that this doc is now readable. :) Other people can comment on the technical details. thanks.
On Thu, 3 Dec 2020 20:02:55 +0200 Parav Pandit wrote: > Added documentation for devlink port and port function related commands. > > Signed-off-by: Parav Pandit <parav@nvidia.com> > Reviewed-by: Jiri Pirko <jiri@nvidia.com> > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > +============ > +Devlink Port > +============ > + > +``devlink-port`` is a port that exists on the device. Can we add something like: Each port is a logically separate ingress/egress point of the device. ? > A devlink port can > +be of one among many flavours. A devlink port flavour along with port > +attributes describe what a port represents. > + > +A device driver that intends to publish a devlink port sets the > +devlink port attributes and registers the devlink port. > + > +Devlink port flavours are described below. > + > +.. list-table:: List of devlink port flavours > + :widths: 33 90 > + > + * - Flavour > + - Description > + * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` > + - Any kind of physical networking port. This can be an eswitch physical > + port or any other physical port on the device. > + * - ``DEVLINK_PORT_FLAVOUR_DSA`` > + - This indicates a DSA interconnect port. > + * - ``DEVLINK_PORT_FLAVOUR_CPU`` > + - This indicates a CPU port applicable only to DSA. > + * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` > + - This indicates an eswitch port representing a networking port of > + PCI physical function (PF). > + * - ``DEVLINK_PORT_FLAVOUR_PCI_VF`` > + - This indicates an eswitch port representing a networking port of > + PCI virtual function (VF). > + * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL`` > + - This indicates a virtual port for the virtual PCI device such as PCI VF. > + > +Devlink port types are described below. How about: Physical ports can have different types based on the link layer: > +.. list-table:: List of devlink port types > + :widths: 23 90 > + > + * - Type > + - Description > + * - ``DEVLINK_PORT_TYPE_ETH`` > + - Driver should set this port type when a link layer of the port is Ethernet. > + * - ``DEVLINK_PORT_TYPE_IB`` > + - Driver should set this port type when a link layer of the port is InfiniBand. Please wrap at 80 chars. > + * - ``DEVLINK_PORT_TYPE_AUTO`` > + - This type is indicated by the user when user prefers to set the port type > + to be automatically detected by the device driver. How about: This type is indicated by the user when driver should detect the port type automatically. > +A controller consists of one or more PCI functions. This need some intro. Like: PCI controllers --------------- In most cases PCI devices will have only one controller, with potentially multiple physical and virtual functions. Devices connected to multiple CPUs and SmartNICs, however, may have multiple controllers. > Such PCI function consists > +of one or more networking ports. PCI function consists of networking ports? What do you mean by a networking port? All devlink ports are networking ports. > A networking port of such PCI function is > +represented by the eswitch devlink port. What's eswitch devlink port? It was never defined. > A devlink instance holds ports of two > +types of controllers. For devices with multiple controllers we can distinguish... > +(1) controller discovered on same system where eswitch resides: > +This is the case where PCI PF/VF of a controller and devlink eswitch > +instance both are located on a single system. How is eswitch located on a system? Eswitch is in the NIC I think you should say refer to eswitch being controlled by a system. > +(2) controller located on external host system. > +This is the case where a controller is in one system and its devlink > +eswitch ports are in a different system. Such controller is called > +external controller. > +An example view of two controller systems:: > + > +In this example, external controller (identified by controller number = 1) > +doesn't have eswitch. Local controller (identified by controller number = 0) > +has the eswitch. Devlink instance on local Controller has eswitch devlink > +ports representing networking ports for both the controllers. > + > + --------------------------------------------------------- > + | | > + | --------- --------- ------- ------- | > + ----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | > + | server | | ------- ----/---- ---/----- ------- ---/--- ---/--- | > + | pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ | > + | connect | | ------- ------- | > + ----------- | | controller_num=1 (no eswitch) | > + ------|-------------------------------------------------- > + (internal wire) > + | > + --------------------------------------------------------- > + | devlink eswitch ports and reps | > + | ----------------------------------------------------- | > + | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | | > + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | > + | ----------------------------------------------------- | > + | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | | > + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | > + | ----------------------------------------------------- | > + | | > + | | > + | --------- --------- ------- ------- | > + | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | > + | ------- ----/---- ---/----- ------- ---/--- ---/--- | > + | | pf0 |______/________/ | pf1 |___/_______/ | > + | ------- ------- | > + | | > + | local controller_num=0 (eswitch) | > + --------------------------------------------------------- > + > +Port function configuration > +=========================== > + > +When a port flavor is ``DEVLINK_PORT_FLAVOUR_PCI_PF`` or > +``DEVLINK_PORT_FLAVOUR_PCI_VF``, it represents the networking port of a > +PCI function. Networking port of a PCI function? > A user can configure the port function attributes port function attributes? > before > +enumerating the function. What does this mean? What does enumerate mean in this context? > For example user may set the hardware address of > +the function represented by the devlink port function. What's a hardware address? You mean MAC address?
> From: Jakub Kicinski <kuba@kernel.org> > Sent: Sunday, December 6, 2020 1:57 AM > > On Thu, 3 Dec 2020 20:02:55 +0200 Parav Pandit wrote: > > Added documentation for devlink port and port function related commands. > > > > Signed-off-by: Parav Pandit <parav@nvidia.com> > > Reviewed-by: Jiri Pirko <jiri@nvidia.com> > > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > > > +============ > > +Devlink Port > > +============ > > + > > +``devlink-port`` is a port that exists on the device. > > Can we add something like: > > Each port is a logically separate ingress/egress point of the device. > > ? This may not be true when both physical ports are under bond. > > > A devlink port can > > +be of one among many flavours. A devlink port flavour along with port > > +attributes describe what a port represents. > > + > > +A device driver that intends to publish a devlink port sets the > > +devlink port attributes and registers the devlink port. > > + > > +Devlink port types are described below. > > How about: > > Physical ports can have different types based on the link layer: > Ok. will add. > > +.. list-table:: List of devlink port types > > + :widths: 23 90 > > + > > + * - Type > > + - Description > > + * - ``DEVLINK_PORT_TYPE_ETH`` > > + - Driver should set this port type when a link layer of the port is Ethernet. > > + * - ``DEVLINK_PORT_TYPE_IB`` > > + - Driver should set this port type when a link layer of the port is InfiniBand. > > Please wrap at 80 chars. > Ack. > > + * - ``DEVLINK_PORT_TYPE_AUTO`` > > + - This type is indicated by the user when user prefers to set the port type > > + to be automatically detected by the device driver. > > How about: > > This type is indicated by the user when driver should detect the port type > automatically. > Will change. > > +A controller consists of one or more PCI functions. > > This need some intro. Like: > > PCI controllers > --------------- > > In most cases PCI devices will have only one controller, with potentially multiple > physical and virtual functions. Devices connected to multiple CPUs and > SmartNICs, however, may have multiple controllers. > > > Such PCI function consists > > +of one or more networking ports. > > PCI function consists of networking ports? What do you mean by a networking > port? All devlink ports are networking ports. > I am not sure this document should be a starting point to define such restriction. > > A networking port of such PCI function is > > +represented by the eswitch devlink port. > > What's eswitch devlink port? It was never defined. Eswitch devlink port is the port which sets eswitch attributes (id and length). > > > A devlink instance holds ports of two > > +types of controllers. > > For devices with multiple controllers we can distinguish... > Yes, will change. > > +(1) controller discovered on same system where eswitch resides: > > +This is the case where PCI PF/VF of a controller and devlink eswitch > > +instance both are located on a single system. > > How is eswitch located on a system? Eswitch is in the NIC > Yes, I meant eswitch devlink instance and controller devlink instance are same. Will rephase. > I think you should say refer to eswitch being controlled by a system. > > > +(2) controller located on external host system. > > +This is the case where a controller is in one system and its devlink > > +eswitch ports are in a different system. Such controller is called > > +external controller. > > > +An example view of two controller systems:: > > + > > +Port function configuration > > +=========================== > > + > > +When a port flavor is ``DEVLINK_PORT_FLAVOUR_PCI_PF`` or > > +``DEVLINK_PORT_FLAVOUR_PCI_VF``, it represents the networking port of > > +a PCI function. > > Networking port of a PCI function? > > > A user can configure the port function attributes > > port function attributes? > > > before > > +enumerating the function. > > What does this mean? What does enumerate mean in this context? > Enumerate means before creating the device of the function. However today due to SR-IOV limitation, it is before probing the function device. > > For example user may set the hardware address of > > +the function represented by the devlink port function. > > What's a hardware address? You mean MAC address? Yes, MAC address. Port function attribute is named as hardware address to be generic enough similar to other iproute2 tools.
On Mon, 7 Dec 2020 04:46:14 +0000 Parav Pandit wrote: > > From: Jakub Kicinski <kuba@kernel.org> > > Sent: Sunday, December 6, 2020 1:57 AM > > > > On Thu, 3 Dec 2020 20:02:55 +0200 Parav Pandit wrote: > > > Added documentation for devlink port and port function related commands. > > > > > > Signed-off-by: Parav Pandit <parav@nvidia.com> > > > Reviewed-by: Jiri Pirko <jiri@nvidia.com> > > > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > > > > > +============ > > > +Devlink Port > > > +============ > > > + > > > +``devlink-port`` is a port that exists on the device. > > > > Can we add something like: > > > > Each port is a logically separate ingress/egress point of the device. > > > > ? > This may not be true when both physical ports are under bond. Bonding changes forwarding logic, not what points of egress ASIC has. > > > Such PCI function consists > > > +of one or more networking ports. > > > > PCI function consists of networking ports? What do you mean by a networking > > port? All devlink ports are networking ports. > > > I am not sure this document should be a starting point to define such restriction. Well it's the reality today. Adding "networking" everywhere in this document is pointless. > > > A networking port of such PCI function is > > > +represented by the eswitch devlink port. > > > > What's eswitch devlink port? It was never defined. > Eswitch devlink port is the port which sets eswitch attributes (id and length). You mean phys_port_id? > > > before > > > +enumerating the function. > > > > What does this mean? What does enumerate mean in this context? > > > Enumerate means before creating the device of the function. > However today due to SR-IOV limitation, it is before probing the function device. Can you rephrase to make the point clearer? > > > For example user may set the hardware address of > > > +the function represented by the devlink port function. > > > > What's a hardware address? You mean MAC address? > Yes, MAC address. > Port function attribute is named as hardware address to be generic enough similar to other iproute2 tools. Right, but in iproute2 the context makes it clear. Here we could be talking about many things.
> From: Jakub Kicinski <kuba@kernel.org> > Sent: Monday, December 7, 2020 11:10 PM > > On Mon, 7 Dec 2020 04:46:14 +0000 Parav Pandit wrote: > > > From: Jakub Kicinski <kuba@kernel.org> > > > Sent: Sunday, December 6, 2020 1:57 AM > > > > > > On Thu, 3 Dec 2020 20:02:55 +0200 Parav Pandit wrote: > > > > Added documentation for devlink port and port function related > commands. > > > > > > > > Signed-off-by: Parav Pandit <parav@nvidia.com> > > > > Reviewed-by: Jiri Pirko <jiri@nvidia.com> > > > > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > > > > > > > +============ > > > > +Devlink Port > > > > +============ > > > > + > > > > +``devlink-port`` is a port that exists on the device. > > > > > > Can we add something like: > > > > > > Each port is a logically separate ingress/egress point of the device. > > > > > > ? > > This may not be true when both physical ports are under bond. > > Bonding changes forwarding logic, not what points of egress ASIC has. Ok. Do CPU and DSA port also follow same? > > > > > Such PCI function consists > > > > +of one or more networking ports. > > > > > > PCI function consists of networking ports? What do you mean by a > > > networking port? All devlink ports are networking ports. > > > > > I am not sure this document should be a starting point to define such > restriction. > > Well it's the reality today. Adding "networking" everywhere in this document > is pointless. > Ok. so I drop networking and just say PCI function port. > > > > A networking port of such PCI function is > > > > +represented by the eswitch devlink port. > > > > > > What's eswitch devlink port? It was never defined. > > Eswitch devlink port is the port which sets eswitch attributes (id and > length). > > You mean phys_port_id? No. I mean phys_switch_id. > > > > > before > > > > +enumerating the function. > > > > > > What does this mean? What does enumerate mean in this context? > > > > > Enumerate means before creating the device of the function. > > However today due to SR-IOV limitation, it is before probing the function > device. > > Can you rephrase to make the point clearer? Sure. Will do. > > > > > For example user may set the hardware address of > > > > +the function represented by the devlink port function. > > > > > > What's a hardware address? You mean MAC address? > > Yes, MAC address. > > Port function attribute is named as hardware address to be generic enough > similar to other iproute2 tools. > > Right, but in iproute2 the context makes it clear. Here we could be talking > about many things. Kernel interface is not restricted to mac address, nor the iproute2. So I will mention as hardware address to be precise what is does and additionally mention that it is mac address for Ethernet ports. Will send v5 addressing these comments.
On Mon, 7 Dec 2020 20:00:53 +0000 Parav Pandit wrote: > > > > Each port is a logically separate ingress/egress point of the device. > > > > > > > > ? > > > This may not be true when both physical ports are under bond. > > > > Bonding changes forwarding logic, not what points of egress ASIC has. > Ok. Do CPU and DSA port also follow same? Yes, DSA/CPU port types are points of egress to the devlink instance.
> From: Jakub Kicinski <kuba@kernel.org> > Sent: Tuesday, December 8, 2020 1:43 AM > > On Mon, 7 Dec 2020 20:00:53 +0000 Parav Pandit wrote: > > > > > Each port is a logically separate ingress/egress point of the device. > > > > > > > > > > ? > > > > This may not be true when both physical ports are under bond. > > > > > > Bonding changes forwarding logic, not what points of egress ASIC has. > > Ok. Do CPU and DSA port also follow same? > > Yes, DSA/CPU port types are points of egress to the devlink instance. Ok. got it. Thanks.
diff --git a/Documentation/networking/devlink/devlink-port.rst b/Documentation/networking/devlink/devlink-port.rst new file mode 100644 index 000000000000..ac18cb8041dc --- /dev/null +++ b/Documentation/networking/devlink/devlink-port.rst @@ -0,0 +1,111 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +Devlink Port +============ + +``devlink-port`` is a port that exists on the device. A devlink port can +be of one among many flavours. A devlink port flavour along with port +attributes describe what a port represents. + +A device driver that intends to publish a devlink port sets the +devlink port attributes and registers the devlink port. + +Devlink port flavours are described below. + +.. list-table:: List of devlink port flavours + :widths: 33 90 + + * - Flavour + - Description + * - ``DEVLINK_PORT_FLAVOUR_PHYSICAL`` + - Any kind of physical networking port. This can be an eswitch physical + port or any other physical port on the device. + * - ``DEVLINK_PORT_FLAVOUR_DSA`` + - This indicates a DSA interconnect port. + * - ``DEVLINK_PORT_FLAVOUR_CPU`` + - This indicates a CPU port applicable only to DSA. + * - ``DEVLINK_PORT_FLAVOUR_PCI_PF`` + - This indicates an eswitch port representing a networking port of + PCI physical function (PF). + * - ``DEVLINK_PORT_FLAVOUR_PCI_VF`` + - This indicates an eswitch port representing a networking port of + PCI virtual function (VF). + * - ``DEVLINK_PORT_FLAVOUR_VIRTUAL`` + - This indicates a virtual port for the virtual PCI device such as PCI VF. + +Devlink port types are described below. + +.. list-table:: List of devlink port types + :widths: 23 90 + + * - Type + - Description + * - ``DEVLINK_PORT_TYPE_ETH`` + - Driver should set this port type when a link layer of the port is Ethernet. + * - ``DEVLINK_PORT_TYPE_IB`` + - Driver should set this port type when a link layer of the port is InfiniBand. + * - ``DEVLINK_PORT_TYPE_AUTO`` + - This type is indicated by the user when user prefers to set the port type + to be automatically detected by the device driver. + +A controller consists of one or more PCI functions. Such PCI function consists +of one or more networking ports. A networking port of such PCI function is +represented by the eswitch devlink port. A devlink instance holds ports of two +types of controllers. + +(1) controller discovered on same system where eswitch resides: +This is the case where PCI PF/VF of a controller and devlink eswitch +instance both are located on a single system. + +(2) controller located on external host system. +This is the case where a controller is in one system and its devlink +eswitch ports are in a different system. Such controller is called +external controller. + +An example view of two controller systems:: + +In this example, external controller (identified by controller number = 1) +doesn't have eswitch. Local controller (identified by controller number = 0) +has the eswitch. Devlink instance on local Controller has eswitch devlink +ports representing networking ports for both the controllers. + + --------------------------------------------------------- + | | + | --------- --------- ------- ------- | + ----------- | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | + | server | | ------- ----/---- ---/----- ------- ---/--- ---/--- | + | pci rc |=== | pf0 |______/________/ | pf1 |___/_______/ | + | connect | | ------- ------- | + ----------- | | controller_num=1 (no eswitch) | + ------|-------------------------------------------------- + (internal wire) + | + --------------------------------------------------------- + | devlink eswitch ports and reps | + | ----------------------------------------------------- | + | |ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 | ctrl-0 |ctrl-0 | | + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | + | ----------------------------------------------------- | + | |ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 | ctrl-1 |ctrl-1 | | + | |pf0 | pf0vfN | pf0sfN | pf1 | pf1vfN |pf1sfN | | + | ----------------------------------------------------- | + | | + | | + | --------- --------- ------- ------- | + | | vf(s) | | sf(s) | |vf(s)| |sf(s)| | + | ------- ----/---- ---/----- ------- ---/--- ---/--- | + | | pf0 |______/________/ | pf1 |___/_______/ | + | ------- ------- | + | | + | local controller_num=0 (eswitch) | + --------------------------------------------------------- + +Port function configuration +=========================== + +When a port flavor is ``DEVLINK_PORT_FLAVOUR_PCI_PF`` or +``DEVLINK_PORT_FLAVOUR_PCI_VF``, it represents the networking port of a +PCI function. A user can configure the port function attributes before +enumerating the function. For example user may set the hardware address of +the function represented by the devlink port function. diff --git a/Documentation/networking/devlink/index.rst b/Documentation/networking/devlink/index.rst index d82874760ae2..aab79667f97b 100644 --- a/Documentation/networking/devlink/index.rst +++ b/Documentation/networking/devlink/index.rst @@ -18,6 +18,7 @@ general. devlink-info devlink-flash devlink-params + devlink-port devlink-region devlink-resource devlink-reload