| Message ID | 20190331164450.23618-1-leon@kernel.org |
|---|---|
| Headers | show |
| Series | User space steering | expand |
On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote: > From: Leon Romanovsky <leonro@mellanox.com> > > From Ariel, > > This series of patches adds user space managed steering infrastructure > to the mlx5_ib driver. > > User space managed steering requires the means to access a dedicated > memory space that is used by the device to store the packet steering > and header modification tables and rules in order to manage them directly > without the device's firmware involvement. This dedicated memory is part > of the ICM memory space. > > The changes are introducing the mlx5_ib API to allocate, deallocate and > register this dedicated SW ICM memory via the existing device memory API > using a private attribute which specifies the memory type. > > The allocated memory itself is not IO mapped and user can only access it > using remote RDMA operations. > > In addition, the series exposed the ICM address of the receive transport > interface (TIR) of Raw Packet and RSS QPs to user since they are required > to properly create and insert steering rules that direct flows to these QPs. Hi Leon! Out of curiosity what protocols are you guys steering here? Is this some form of RoCE or IB? Is there a good resource to read up on the DEVX stuff? (I presume this series is related?)
On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote: > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote: > > From: Leon Romanovsky <leonro@mellanox.com> > > > > From Ariel, > > > > This series of patches adds user space managed steering infrastructure > > to the mlx5_ib driver. > > > > User space managed steering requires the means to access a dedicated > > memory space that is used by the device to store the packet steering > > and header modification tables and rules in order to manage them directly > > without the device's firmware involvement. This dedicated memory is part > > of the ICM memory space. > > > > The changes are introducing the mlx5_ib API to allocate, deallocate and > > register this dedicated SW ICM memory via the existing device memory API > > using a private attribute which specifies the memory type. > > > > The allocated memory itself is not IO mapped and user can only access it > > using remote RDMA operations. > > > > In addition, the series exposed the ICM address of the receive transport > > interface (TIR) of Raw Packet and RSS QPs to user since they are required > > to properly create and insert steering rules that direct flows to these QPs. > > Hi Leon! Out of curiosity what protocols are you guys steering here? > Is this some form of RoCE or IB? Is there a good resource to read up on > the DEVX stuff? (I presume this series is related?) mlx5 devices steering operates on packets without relation to specific protocol. The decision to steer is based on HW capabilities and user configuration for matching specific fields. Such configuration is done through kernel and through DEVX. General description of DEVX can be found in manuals of libmlx5. https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/ Thanks
On Mon, 1 Apr 2019 09:28:16 +0300, Leon Romanovsky wrote: > On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote: > > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote: > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > From Ariel, > > > > > > This series of patches adds user space managed steering infrastructure > > > to the mlx5_ib driver. > > > > > > User space managed steering requires the means to access a dedicated > > > memory space that is used by the device to store the packet steering > > > and header modification tables and rules in order to manage them directly > > > without the device's firmware involvement. This dedicated memory is part > > > of the ICM memory space. > > > > > > The changes are introducing the mlx5_ib API to allocate, deallocate and > > > register this dedicated SW ICM memory via the existing device memory API > > > using a private attribute which specifies the memory type. > > > > > > The allocated memory itself is not IO mapped and user can only access it > > > using remote RDMA operations. > > > > > > In addition, the series exposed the ICM address of the receive transport > > > interface (TIR) of Raw Packet and RSS QPs to user since they are required > > > to properly create and insert steering rules that direct flows to these QPs. > > > > Hi Leon! Out of curiosity what protocols are you guys steering here? > > Is this some form of RoCE or IB? Is there a good resource to read up on > > the DEVX stuff? (I presume this series is related?) > > mlx5 devices steering operates on packets without relation to specific > protocol. The decision to steer is based on HW capabilities and user > configuration for matching specific fields. Such configuration is done > through kernel and through DEVX. > > General description of DEVX can be found in manuals of libmlx5. > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/ Thanks for those, looks quite powerful. What's your primary use case here? Is there some flow steering and encapsulation going on in RDMA world?
On Mon, Apr 01, 2019 at 11:32:21AM -0700, Jakub Kicinski wrote: > On Mon, 1 Apr 2019 09:28:16 +0300, Leon Romanovsky wrote: > > On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote: > > > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote: > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > From Ariel, > > > > > > > > This series of patches adds user space managed steering infrastructure > > > > to the mlx5_ib driver. > > > > > > > > User space managed steering requires the means to access a dedicated > > > > memory space that is used by the device to store the packet steering > > > > and header modification tables and rules in order to manage them directly > > > > without the device's firmware involvement. This dedicated memory is part > > > > of the ICM memory space. > > > > > > > > The changes are introducing the mlx5_ib API to allocate, deallocate and > > > > register this dedicated SW ICM memory via the existing device memory API > > > > using a private attribute which specifies the memory type. > > > > > > > > The allocated memory itself is not IO mapped and user can only access it > > > > using remote RDMA operations. > > > > > > > > In addition, the series exposed the ICM address of the receive transport > > > > interface (TIR) of Raw Packet and RSS QPs to user since they are required > > > > to properly create and insert steering rules that direct flows to these QPs. > > > > > > Hi Leon! Out of curiosity what protocols are you guys steering here? > > > Is this some form of RoCE or IB? Is there a good resource to read up on > > > the DEVX stuff? (I presume this series is related?) > > > > mlx5 devices steering operates on packets without relation to specific > > protocol. The decision to steer is based on HW capabilities and user > > configuration for matching specific fields. Such configuration is done > > through kernel and through DEVX. > > > > General description of DEVX can be found in manuals of libmlx5. > > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md > > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/ > > Thanks for those, looks quite powerful. What's your primary use case > here? Is there some flow steering and encapsulation going on in RDMA > world? Usual use case for that is in RoCE fabric, but as I said it is not limited to that. For example, flow steering is used in libpcap [1] and encapsulation is used for efficient VM to VM communication over different hosts (I assume exactly as in netdev world). In regarding to this patch set, the combination of this steering functionality with DEVX allows us to speedup greatly buildup of rules by doing it in powerful host CPU instead of internal to NIC CPU. [1] https://github.com/the-tcpdump-group/libpcap/commit/f5f1484a931851d2d3b67d6033aedea8da5acb42 Thanks
On Tue, 2 Apr 2019 11:14:59 +0300, Leon Romanovsky wrote: > On Mon, Apr 01, 2019 at 11:32:21AM -0700, Jakub Kicinski wrote: > > On Mon, 1 Apr 2019 09:28:16 +0300, Leon Romanovsky wrote: > > > On Sun, Mar 31, 2019 at 02:06:33PM -0700, Jakub Kicinski wrote: > > > > On Sun, 31 Mar 2019 19:44:42 +0300, Leon Romanovsky wrote: > > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > > > From Ariel, > > > > > > > > > > This series of patches adds user space managed steering infrastructure > > > > > to the mlx5_ib driver. > > > > > > > > > > User space managed steering requires the means to access a dedicated > > > > > memory space that is used by the device to store the packet steering > > > > > and header modification tables and rules in order to manage them directly > > > > > without the device's firmware involvement. This dedicated memory is part > > > > > of the ICM memory space. > > > > > > > > > > The changes are introducing the mlx5_ib API to allocate, deallocate and > > > > > register this dedicated SW ICM memory via the existing device memory API > > > > > using a private attribute which specifies the memory type. > > > > > > > > > > The allocated memory itself is not IO mapped and user can only access it > > > > > using remote RDMA operations. > > > > > > > > > > In addition, the series exposed the ICM address of the receive transport > > > > > interface (TIR) of Raw Packet and RSS QPs to user since they are required > > > > > to properly create and insert steering rules that direct flows to these QPs. > > > > > > > > Hi Leon! Out of curiosity what protocols are you guys steering here? > > > > Is this some form of RoCE or IB? Is there a good resource to read up on > > > > the DEVX stuff? (I presume this series is related?) > > > > > > mlx5 devices steering operates on packets without relation to specific > > > protocol. The decision to steer is based on HW capabilities and user > > > configuration for matching specific fields. Such configuration is done > > > through kernel and through DEVX. > > > > > > General description of DEVX can be found in manuals of libmlx5. > > > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/mlx5dv_devx_obj_create.3.md > > > https://github.com/linux-rdma/rdma-core/blob/master/providers/mlx5/man/ > > > > Thanks for those, looks quite powerful. What's your primary use case > > here? Is there some flow steering and encapsulation going on in RDMA > > world? > > Usual use case for that is in RoCE fabric, but as I said it is not > limited to that. For example, flow steering is used in libpcap [1] > and encapsulation is used for efficient VM to VM communication over > different hosts (I assume exactly as in netdev world). > > In regarding to this patch set, the combination of this steering > functionality with DEVX allows us to speedup greatly buildup of rules > by doing it in powerful host CPU instead of internal to NIC CPU. > > [1] > https://github.com/the-tcpdump-group/libpcap/commit/f5f1484a931851d2d3b67d6033aedea8da5acb42 Interesting, thanks for the info!
On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote: > From: Leon Romanovsky <leonro@mellanox.com> > > >From Ariel, > > This series of patches adds user space managed steering infrastructure > to the mlx5_ib driver. > > User space managed steering requires the means to access a dedicated > memory space that is used by the device to store the packet steering > and header modification tables and rules in order to manage them directly > without the device's firmware involvement. This dedicated memory is part > of the ICM memory space. > > The changes are introducing the mlx5_ib API to allocate, deallocate and > register this dedicated SW ICM memory via the existing device memory API > using a private attribute which specifies the memory type. > > The allocated memory itself is not IO mapped and user can only access it > using remote RDMA operations. > > In addition, the series exposed the ICM address of the receive transport > interface (TIR) of Raw Packet and RSS QPs to user since they are required > to properly create and insert steering rules that direct flows to these QPs. > > Thanks > > Ariel Levkovich (8): > net/mlx5: Expose SW ICM related device memory capabilities > IB/mlx5: Support device memory type attribute > IB/mlx5: Warn on allocated MEMIC buffers during cleanup > IB/mlx5: Add steering SW ICM device memory type > IB/mlx5: Device resource control for privileged DEVX user This doesn't apply, it conflicts with patches in mlx5-next, please resent > net/mlx5: Expose TIR ICM address in command outbox > net/mlx5: Introduce new TIR creation core API > IB/mlx5: Expose TIR ICM address to user space Is this actually two patch series? This one looks OK, lets apply it to mlx5-next please Jason
On Wed, 2019-04-24 at 10:07 -0300, Jason Gunthorpe wrote: > On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote: > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > From Ariel, > > > > This series of patches adds user space managed steering > > infrastructure > > to the mlx5_ib driver. > > > > User space managed steering requires the means to access a > > dedicated > > memory space that is used by the device to store the packet > > steering > > and header modification tables and rules in order to manage them > > directly > > without the device's firmware involvement. This dedicated memory is > > part > > of the ICM memory space. > > > > The changes are introducing the mlx5_ib API to allocate, deallocate > > and > > register this dedicated SW ICM memory via the existing device > > memory API > > using a private attribute which specifies the memory type. > > > > The allocated memory itself is not IO mapped and user can only > > access it > > using remote RDMA operations. > > > > In addition, the series exposed the ICM address of the receive > > transport > > interface (TIR) of Raw Packet and RSS QPs to user since they are > > required > > to properly create and insert steering rules that direct flows to > > these QPs. > > > > Thanks > > > > Ariel Levkovich (8): > > net/mlx5: Expose SW ICM related device memory capabilities > > IB/mlx5: Support device memory type attribute > > IB/mlx5: Warn on allocated MEMIC buffers during cleanup > > IB/mlx5: Add steering SW ICM device memory type > > IB/mlx5: Device resource control for privileged DEVX user > > This doesn't apply, it conflicts with patches in mlx5-next, please > resent > > > net/mlx5: Expose TIR ICM address in command outbox > > net/mlx5: Introduce new TIR creation core API > > IB/mlx5: Expose TIR ICM address to user space > it is a 3 patch series: net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API net/mlx5: Expose SW ICM related device memory capabilities LGTM, I can apply them if you want, or do you want to wait for leon ?
On Wed, Apr 24, 2019 at 04:40:54PM +0000, Saeed Mahameed wrote: > On Wed, 2019-04-24 at 10:07 -0300, Jason Gunthorpe wrote: > > On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote: > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > From Ariel, > > > > > > This series of patches adds user space managed steering > > > infrastructure > > > to the mlx5_ib driver. > > > > > > User space managed steering requires the means to access a > > > dedicated > > > memory space that is used by the device to store the packet > > > steering > > > and header modification tables and rules in order to manage them > > > directly > > > without the device's firmware involvement. This dedicated memory is > > > part > > > of the ICM memory space. > > > > > > The changes are introducing the mlx5_ib API to allocate, deallocate > > > and > > > register this dedicated SW ICM memory via the existing device > > > memory API > > > using a private attribute which specifies the memory type. > > > > > > The allocated memory itself is not IO mapped and user can only > > > access it > > > using remote RDMA operations. > > > > > > In addition, the series exposed the ICM address of the receive > > > transport > > > interface (TIR) of Raw Packet and RSS QPs to user since they are > > > required > > > to properly create and insert steering rules that direct flows to > > > these QPs. > > > > > > Thanks > > > > > > Ariel Levkovich (8): > > > net/mlx5: Expose SW ICM related device memory capabilities > > > IB/mlx5: Support device memory type attribute > > > IB/mlx5: Warn on allocated MEMIC buffers during cleanup > > > IB/mlx5: Add steering SW ICM device memory type > > > IB/mlx5: Device resource control for privileged DEVX user > > > > This doesn't apply, it conflicts with patches in mlx5-next, please > > resent > > > > > net/mlx5: Expose TIR ICM address in command outbox > > > net/mlx5: Introduce new TIR creation core API > > > IB/mlx5: Expose TIR ICM address to user space > > > > it is a 3 patch series: > net/mlx5: Expose TIR ICM address in command outbox > net/mlx5: Introduce new TIR creation core API > net/mlx5: Expose SW ICM related device memory capabilities > > LGTM, I can apply them if you want, or do you want to wait for leon ? Fine by me Jason
On Wed, 2019-04-24 at 16:03 -0300, Jason Gunthorpe wrote: > On Wed, Apr 24, 2019 at 04:40:54PM +0000, Saeed Mahameed wrote: > > On Wed, 2019-04-24 at 10:07 -0300, Jason Gunthorpe wrote: > > > On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote: > > > > From: Leon Romanovsky <leonro@mellanox.com> > > > > > > > > > From Ariel, > > > > > > > > This series of patches adds user space managed steering > > > > infrastructure > > > > to the mlx5_ib driver. > > > > > > > > User space managed steering requires the means to access a > > > > dedicated > > > > memory space that is used by the device to store the packet > > > > steering > > > > and header modification tables and rules in order to manage > > > > them > > > > directly > > > > without the device's firmware involvement. This dedicated > > > > memory is > > > > part > > > > of the ICM memory space. > > > > > > > > The changes are introducing the mlx5_ib API to allocate, > > > > deallocate > > > > and > > > > register this dedicated SW ICM memory via the existing device > > > > memory API > > > > using a private attribute which specifies the memory type. > > > > > > > > The allocated memory itself is not IO mapped and user can only > > > > access it > > > > using remote RDMA operations. > > > > > > > > In addition, the series exposed the ICM address of the receive > > > > transport > > > > interface (TIR) of Raw Packet and RSS QPs to user since they > > > > are > > > > required > > > > to properly create and insert steering rules that direct flows > > > > to > > > > these QPs. > > > > > > > > Thanks > > > > > > > > Ariel Levkovich (8): > > > > net/mlx5: Expose SW ICM related device memory capabilities > > > > IB/mlx5: Support device memory type attribute > > > > IB/mlx5: Warn on allocated MEMIC buffers during cleanup > > > > IB/mlx5: Add steering SW ICM device memory type > > > > IB/mlx5: Device resource control for privileged DEVX user > > > > > > This doesn't apply, it conflicts with patches in mlx5-next, > > > please > > > resent > > > > > > > net/mlx5: Expose TIR ICM address in command outbox > > > > net/mlx5: Introduce new TIR creation core API > > > > IB/mlx5: Expose TIR ICM address to user space > > > > it is a 3 patch series: > > net/mlx5: Expose TIR ICM address in command outbox > > net/mlx5: Introduce new TIR creation core API > > net/mlx5: Expose SW ICM related device memory capabilities > > > > LGTM, I can apply them if you want, or do you want to wait for leon > > ? > > Fine by me > the above 3 mlx5-next patches applied. Thanks, Saeed. > Jason
On Sun, Mar 31, 2019 at 07:44:42PM +0300, Leon Romanovsky wrote: > net/mlx5: Expose TIR ICM address in command outbox > net/mlx5: Introduce new TIR creation core API > IB/mlx5: Expose TIR ICM address to user space These are applied to for-next now Thanks, Jason
From: Leon Romanovsky <leonro@mellanox.com> From Ariel, This series of patches adds user space managed steering infrastructure to the mlx5_ib driver. User space managed steering requires the means to access a dedicated memory space that is used by the device to store the packet steering and header modification tables and rules in order to manage them directly without the device's firmware involvement. This dedicated memory is part of the ICM memory space. The changes are introducing the mlx5_ib API to allocate, deallocate and register this dedicated SW ICM memory via the existing device memory API using a private attribute which specifies the memory type. The allocated memory itself is not IO mapped and user can only access it using remote RDMA operations. In addition, the series exposed the ICM address of the receive transport interface (TIR) of Raw Packet and RSS QPs to user since they are required to properly create and insert steering rules that direct flows to these QPs. Thanks Ariel Levkovich (8): net/mlx5: Expose SW ICM related device memory capabilities IB/mlx5: Support device memory type attribute IB/mlx5: Warn on allocated MEMIC buffers during cleanup IB/mlx5: Add steering SW ICM device memory type IB/mlx5: Device resource control for privileged DEVX user net/mlx5: Expose TIR ICM address in command outbox net/mlx5: Introduce new TIR creation core API IB/mlx5: Expose TIR ICM address to user space drivers/infiniband/hw/mlx5/Kconfig | 2 +- drivers/infiniband/hw/mlx5/cmd.c | 155 +++++++++- drivers/infiniband/hw/mlx5/cmd.h | 8 +- drivers/infiniband/hw/mlx5/devx.c | 4 + drivers/infiniband/hw/mlx5/main.c | 264 +++++++++++++++--- drivers/infiniband/hw/mlx5/mlx5_ib.h | 40 ++- drivers/infiniband/hw/mlx5/mr.c | 40 ++- drivers/infiniband/hw/mlx5/qp.c | 46 ++- .../ethernet/mellanox/mlx5/core/transobj.c | 18 +- include/linux/mlx5/mlx5_ifc.h | 51 +++- include/linux/mlx5/transobj.h | 3 + include/uapi/rdma/mlx5-abi.h | 2 + include/uapi/rdma/mlx5_user_ioctl_cmds.h | 1 + include/uapi/rdma/mlx5_user_ioctl_verbs.h | 6 + 14 files changed, 541 insertions(+), 99 deletions(-) -- 2.20.1