| Message ID | 20190319092439.10701-4-leon@kernel.org |
|---|---|
| State | Awaiting Upstream |
| Delegated to: | David Miller |
| Headers | show |
| Series | Small set of mlx5 related fixes | expand |
On Tue, Mar 19, 2019 at 11:25 AM Leon Romanovsky <leon@kernel.org> wrote: > From: Artemy Kovalyov <artemyko@mellanox.com> > > Delete initialization of high order entries in mr cache to decrease initial > memory footprint. When required, the administrator can populate the > entries with memory keys via the /sys interface. Please add here: This approach is very helpful to reduce the per HW function memory footprint in environments such as VMs. Before the patch we see consumption of 0.9GB per function and after the patch about 0.1GB > Signed-off-by: Artemy Kovalyov <artemyko@mellanox.com> > Signed-off-by: Moni Shoua <monis@mellanox.com> > Signed-off-by: Leon Romanovsky <leonro@mellanox.com> Reported-by: Shalom Toledo <shalomt@mellanox.com> Acked-by: Or Gerlitz <ogerlitz@mellanox.com> Lets push it into stable kernels, a Fixes tag here will cause that to happen more easily, so please consider that. > --- > .../net/ethernet/mellanox/mlx5/core/main.c | 20 ------------------- > 1 file changed, 20 deletions(-) > > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c > index 70cc906a102b..76716419370d 100644 > --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c > +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c > @@ -164,26 +164,6 @@ static struct mlx5_profile profile[] = { > .size = 8, > .limit = 4 > }, > - .mr_cache[16] = { > - .size = 8, > - .limit = 4 > - }, > - .mr_cache[17] = { > - .size = 8, > - .limit = 4 > - }, > - .mr_cache[18] = { > - .size = 8, > - .limit = 4 > - }, > - .mr_cache[19] = { > - .size = 4, > - .limit = 2 > - }, > - .mr_cache[20] = { > - .size = 4, > - .limit = 2 > - }, > }, > }; > > -- > 2.20.1 >
On Wed, Mar 27, 2019 at 12:07:54PM +0200, Or Gerlitz wrote: > On Tue, Mar 19, 2019 at 11:25 AM Leon Romanovsky <leon@kernel.org> wrote: > > From: Artemy Kovalyov <artemyko@mellanox.com> > > > > Delete initialization of high order entries in mr cache to decrease initial > > memory footprint. When required, the administrator can populate the > > entries with memory keys via the /sys interface. > > Please add here: > > This approach is very helpful to reduce the per HW function memory > footprint in environments such as VMs. Before the patch we see > consumption of 0.9GB per function and after the patch about 0.1GB > > > Signed-off-by: Artemy Kovalyov <artemyko@mellanox.com> > > Signed-off-by: Moni Shoua <monis@mellanox.com> > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com> > > Reported-by: Shalom Toledo <shalomt@mellanox.com> > Acked-by: Or Gerlitz <ogerlitz@mellanox.com> > > Lets push it into stable kernels, a Fixes tag here will cause that > to happen more easily, so please consider that. Thanks, I'll add it at the "apply" stage.
On Wed, Mar 27, 2019 at 1:41 PM Leon Romanovsky <leon@kernel.org> wrote: > On Wed, Mar 27, 2019 at 12:07:54PM +0200, Or Gerlitz wrote: > > On Tue, Mar 19, 2019 at 11:25 AM Leon Romanovsky <leon@kernel.org> wrote: > > > From: Artemy Kovalyov <artemyko@mellanox.com> > > > > > > Delete initialization of high order entries in mr cache to decrease initial > > > memory footprint. When required, the administrator can populate the > > > entries with memory keys via the /sys interface. > > > > Please add here: > > > > This approach is very helpful to reduce the per HW function memory > > footprint in environments such as VMs. Before the patch we see > > consumption of 0.9GB per function and after the patch about 0.1GB > > > > > Signed-off-by: Artemy Kovalyov <artemyko@mellanox.com> > > > Signed-off-by: Moni Shoua <monis@mellanox.com> > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com> > > > > Reported-by: Shalom Toledo <shalomt@mellanox.com> > > Acked-by: Or Gerlitz <ogerlitz@mellanox.com> > > > > Lets push it into stable kernels, a Fixes tag here will cause that > > to happen more easily, so please consider that. > I'll add it at the "apply" stage. Just to make sure, by "add it" you mean the signatures, the text and the Fixes that? Also, is there any reason not to eliminate the mr cache pre-population all together? Currently we consume 100-200MB per function after the patch which is also problematic for some environments.
On Wed, Mar 27, 2019 at 01:58:17PM +0200, Or Gerlitz wrote: > On Wed, Mar 27, 2019 at 1:41 PM Leon Romanovsky <leon@kernel.org> wrote: > > On Wed, Mar 27, 2019 at 12:07:54PM +0200, Or Gerlitz wrote: > > > On Tue, Mar 19, 2019 at 11:25 AM Leon Romanovsky <leon@kernel.org> wrote: > > > > From: Artemy Kovalyov <artemyko@mellanox.com> > > > > > > > > Delete initialization of high order entries in mr cache to decrease initial > > > > memory footprint. When required, the administrator can populate the > > > > entries with memory keys via the /sys interface. > > > > > > Please add here: > > > > > > This approach is very helpful to reduce the per HW function memory > > > footprint in environments such as VMs. Before the patch we see > > > consumption of 0.9GB per function and after the patch about 0.1GB > > > > > > > Signed-off-by: Artemy Kovalyov <artemyko@mellanox.com> > > > > Signed-off-by: Moni Shoua <monis@mellanox.com> > > > > Signed-off-by: Leon Romanovsky <leonro@mellanox.com> > > > > > > Reported-by: Shalom Toledo <shalomt@mellanox.com> > > > Acked-by: Or Gerlitz <ogerlitz@mellanox.com> > > > > > > Lets push it into stable kernels, a Fixes tag here will cause that > > > to happen more easily, so please consider that. > > > I'll add it at the "apply" stage. > > Just to make sure, by "add it" you mean the signatures, the text and > the Fixes that? Yes, of course. > > Also, is there any reason not to eliminate the mr cache pre-population > all together? AFAIK, pre-populated cache helps MPI application to start faster and with out-of-box experience. Nobody seems to care enough to challenge this internal assumption. > > Currently we consume 100-200MB per function after the patch which is > also problematic for some > environments.
On Wed, Mar 27, 2019 at 3:36 PM Leon Romanovsky <leon@kernel.org> wrote: > On Wed, Mar 27, 2019 at 01:58:17PM +0200, Or Gerlitz wrote: > > On Wed, Mar 27, 2019 at 1:41 PM Leon Romanovsky <leon@kernel.org> wrote: > > > On Wed, Mar 27, 2019 at 12:07:54PM +0200, Or Gerlitz wrote: > > > > On Tue, Mar 19, 2019 at 11:25 AM Leon Romanovsky <leon@kernel.org> wrote: > > > > > From: Artemy Kovalyov <artemyko@mellanox.com> > > > > > > > > > > Delete initialization of high order entries in mr cache to decrease initial > > > > > memory footprint. When required, the administrator can populate the > > > > > entries with memory keys via the /sys interface. > > > > > > > > Please add here: > > > > > > > > This approach is very helpful to reduce the per HW function memory > > > > footprint in environments such as VMs. Before the patch we see > > > > consumption of 0.9GB per function and after the patch about 0.1GB here it needs to be "per physical function" and not "per function" > > > > Lets push it into stable kernels, a Fixes tag here will cause that > > > > to happen more easily, so please consider that. >>> I'll add it at the "apply" stage. >> Just to make sure, by "add it" you mean the signatures, the text and >> the Fixes that? > Yes, of course. thanks >> Also, is there any reason not to eliminate the mr cache pre-population >> all together? > AFAIK, pre-populated cache helps MPI application to start faster and > with out-of-box experience. Nobody seems to care enough to challenge > this internal assumption. I see, we are cutting here ~0.8GB so lets just do it and wonder on the remaining 0.1GB later.. It would be good to push it into 5.1-rc so the fallout into stables will start right away and not only by 5.2 time - can we do that?
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 70cc906a102b..76716419370d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -164,26 +164,6 @@ static struct mlx5_profile profile[] = { .size = 8, .limit = 4 }, - .mr_cache[16] = { - .size = 8, - .limit = 4 - }, - .mr_cache[17] = { - .size = 8, - .limit = 4 - }, - .mr_cache[18] = { - .size = 8, - .limit = 4 - }, - .mr_cache[19] = { - .size = 4, - .limit = 2 - }, - .mr_cache[20] = { - .size = 4, - .limit = 2 - }, }, };