| Message ID | 1490389127-23171-1-git-send-email-greearb@candelatech.com |
|---|---|
| State | Rejected, archived |
| Delegated to: | David Miller |
| Headers | show |
From: greearb@candelatech.com Date: Fri, 24 Mar 2017 13:58:47 -0700 > From: Ben Greear <greearb@candelatech.com> > > In systems where you may have a very large number of network > adapters, certain drivers may consume an unfair amount of > IRQ resources. So, allow a module param that will limit the > number of IRQs at driver load time. This way, other drivers > (40G Ethernet, for instance), which probably will need the > multiple IRQs more, will not be starved of IRQ resources. > > Signed-off-by: Ben Greear <greearb@candelatech.com> Sorry, no module params. Use generic run-time facilities such as ethtool to configure such things.
On 03/24/2017 02:12 PM, David Miller wrote: > From: greearb@candelatech.com > Date: Fri, 24 Mar 2017 13:58:47 -0700 > >> From: Ben Greear <greearb@candelatech.com> >> >> In systems where you may have a very large number of network >> adapters, certain drivers may consume an unfair amount of >> IRQ resources. So, allow a module param that will limit the >> number of IRQs at driver load time. This way, other drivers >> (40G Ethernet, for instance), which probably will need the >> multiple IRQs more, will not be starved of IRQ resources. >> >> Signed-off-by: Ben Greear <greearb@candelatech.com> > > Sorry, no module params. > > Use generic run-time facilities such as ethtool to configure > such things. You cannot call ethtool before module load time, and that is when the IRQs are first acquired. It may be way more useful to give each of 20 network adapters 2 irqs than have the first few grab 16 and the rest get lumped into legacy crap. Thanks, Ben
From: Ben Greear <greearb@candelatech.com> Date: Fri, 24 Mar 2017 14:20:56 -0700 > On 03/24/2017 02:12 PM, David Miller wrote: >> From: greearb@candelatech.com >> Date: Fri, 24 Mar 2017 13:58:47 -0700 >> >>> From: Ben Greear <greearb@candelatech.com> >>> >>> In systems where you may have a very large number of network >>> adapters, certain drivers may consume an unfair amount of >>> IRQ resources. So, allow a module param that will limit the >>> number of IRQs at driver load time. This way, other drivers >>> (40G Ethernet, for instance), which probably will need the >>> multiple IRQs more, will not be starved of IRQ resources. >>> >>> Signed-off-by: Ben Greear <greearb@candelatech.com> >> >> Sorry, no module params. >> >> Use generic run-time facilities such as ethtool to configure >> such things. > > You cannot call ethtool before module load time, and that is when > the IRQs are first acquired. It may be way more useful to give each > of 20 network adapters 2 irqs than have the first few grab 16 and the > rest > get lumped into legacy crap. Sorry, you'll have to find a way to fix this without a module parameter. I understand it might not be easy, but that's not a reason to add countless driver private module parameters all over the tree which is the worst user experience possible.
On Fri, 24 Mar 2017 14:20:56 -0700 Ben Greear <greearb@candelatech.com> wrote: > On 03/24/2017 02:12 PM, David Miller wrote: > > From: greearb@candelatech.com > > Date: Fri, 24 Mar 2017 13:58:47 -0700 > > > >> From: Ben Greear <greearb@candelatech.com> > >> > >> In systems where you may have a very large number of network > >> adapters, certain drivers may consume an unfair amount of > >> IRQ resources. So, allow a module param that will limit the > >> number of IRQs at driver load time. This way, other drivers > >> (40G Ethernet, for instance), which probably will need the > >> multiple IRQs more, will not be starved of IRQ resources. > >> > >> Signed-off-by: Ben Greear <greearb@candelatech.com> > > > > Sorry, no module params. > > > > Use generic run-time facilities such as ethtool to configure > > such things. > > You cannot call ethtool before module load time, and that is when > the IRQs are first acquired. It may be way more useful to give each > of 20 network adapters 2 irqs than have the first few grab 16 and the rest > get lumped into legacy crap. Almost all network devices do not acquire interrupts until device is brought up. I.e request_irq is called from open not probe. This is done so that configuration can be done and also so that unused ports don't consume interrupt space.
On 03/24/2017 04:14 PM, Stephen Hemminger wrote: > On Fri, 24 Mar 2017 14:20:56 -0700 > Ben Greear <greearb@candelatech.com> wrote: > >> On 03/24/2017 02:12 PM, David Miller wrote: >>> From: greearb@candelatech.com >>> Date: Fri, 24 Mar 2017 13:58:47 -0700 >>> >>>> From: Ben Greear <greearb@candelatech.com> >>>> >>>> In systems where you may have a very large number of network >>>> adapters, certain drivers may consume an unfair amount of >>>> IRQ resources. So, allow a module param that will limit the >>>> number of IRQs at driver load time. This way, other drivers >>>> (40G Ethernet, for instance), which probably will need the >>>> multiple IRQs more, will not be starved of IRQ resources. >>>> >>>> Signed-off-by: Ben Greear <greearb@candelatech.com> >>> >>> Sorry, no module params. >>> >>> Use generic run-time facilities such as ethtool to configure >>> such things. >> >> You cannot call ethtool before module load time, and that is when >> the IRQs are first acquired. It may be way more useful to give each >> of 20 network adapters 2 irqs than have the first few grab 16 and the rest >> get lumped into legacy crap. > > Almost all network devices do not acquire interrupts until device is brought up. > I.e request_irq is called from open not probe. This is done so that configuration > can be done and also so that unused ports don't consume interrupt space. If I ever have to deal with this on stock kernels again I'll keep that in mind. Thanks, Ben
From: greearb@candelatech.com > Sent: 24 March 2017 20:59 > From: Ben Greear <greearb@candelatech.com> > > In systems where you may have a very large number of network > adapters, certain drivers may consume an unfair amount of > IRQ resources. So, allow a module param that will limit the > number of IRQs at driver load time. This way, other drivers > (40G Ethernet, for instance), which probably will need the > multiple IRQs more, will not be starved of IRQ resources. ISTM that the functions for allocating MSI-X should allow a driver to request additional interrupts after the original probe. Then drivers than can use additional interrupts but get probed early won't use up all the vectors before other drivers get a chance. David
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index f9f2874..fdb12e0 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -218,6 +218,10 @@ module_param(max_vfs, uint, 0); MODULE_PARM_DESC(max_vfs, "Maximum number of virtual functions to allocate per physical function"); #endif /* CONFIG_PCI_IOV */ +static unsigned int max_rss_qs; +module_param(max_rss_qs, uint, 0); +MODULE_PARM_DESC(max_rss_qs, "Maximum number of RSS queues. Forcing lower will use less IRQ resources."); + static pci_ers_result_t igb_io_error_detected(struct pci_dev *, pci_channel_state_t); static pci_ers_result_t igb_io_slot_reset(struct pci_dev *); @@ -2993,6 +2997,9 @@ static void igb_init_queue_configuration(struct igb_adapter *adapter) break; } + if (max_rss_qs && max_rss_qs < max_rss_queues) + max_rss_queues = max_rss_qs; + adapter->rss_queues = min_t(u32, max_rss_queues, num_online_cpus()); igb_set_flag_queue_pairs(adapter, max_rss_queues);