| Message ID | 46a08278c2ba21737528eb4b77391a7e8bc88000.1292405004.git.fenghua.yu@intel.com |
|---|---|
| State | Rejected, archived |
| Delegated to: | David Miller |
| Headers | show |
Le mercredi 15 décembre 2010 à 12:02 -0800, Fenghua Yu a écrit : > From: Fenghua Yu <fenghua.yu@intel.com> > > Multiqueue and multicore provide packet parallel processing methodology. > Current kernel and network drivers place one queue on one core. But the higher > level socket doesn't know multiqueue. Current socket only can receive or send > packets through one network interfaces. In some cases e.g. multi bpf filter > tcpdump and snort, a lot of contentions come from socket operations like ring > buffer. Even if the application itself has been fully parallelized and run on > multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer > and NIC device driver assemble packets to a single, serialized queue. Thus the > application cannot actually run in parallel in high speed. > > To break the serialized packets assembling bottleneck in kernel, one way is to > allow socket to know multiqueue associated with a NIC interface. So each socket > can handle tx/rx in one queue in parallel. > > Kernel provides several interfaces by which sockets can be bound to rx/tx queues. > User applications can configure socket by providing several sockets that each > bound to a single queue, applications can get data from kernel in parallel. After > that, competitions mentioned above can be removed. > > With this patch, the user-space receiving speed on a Intel SR1690 server with > a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps > to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core > processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed > the performance penalty comes from NUMA memory allocation. > ??? please elaborate on these NUMA memory allocations. This should be OK after commit 564824b0c52c34692d (net: allocate skbs on local node) > This patch set provides kernel ioctl interfaces for user space. User space can > either directly call the interfaces or libpcap interfaces can be further provided > on the top of the kernel ioctl interfaces. So, say we have 8 queues, you want libpcap opens 8 sockets, and bind them to each queue. Add a bpf filter to each one of them. This seems not generic way, because it wont work for an UDP socket for example. And you already can do this using SKF_AD_QUEUE (added in commit d19742fb) Also your AF_PACKET patch only address mmaped sockets. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 12/15/2010 12:02 PM, Yu, Fenghua wrote: > From: Fenghua Yu <fenghua.yu@intel.com> > > Multiqueue and multicore provide packet parallel processing methodology. > Current kernel and network drivers place one queue on one core. But the higher > level socket doesn't know multiqueue. Current socket only can receive or send > packets through one network interfaces. In some cases e.g. multi bpf filter > tcpdump and snort, a lot of contentions come from socket operations like ring > buffer. Even if the application itself has been fully parallelized and run on > multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer > and NIC device driver assemble packets to a single, serialized queue. Thus the > application cannot actually run in parallel in high speed. > > To break the serialized packets assembling bottleneck in kernel, one way is to > allow socket to know multiqueue associated with a NIC interface. So each socket > can handle tx/rx in one queue in parallel. > > Kernel provides several interfaces by which sockets can be bound to rx/tx queues. > User applications can configure socket by providing several sockets that each > bound to a single queue, applications can get data from kernel in parallel. After > that, competitions mentioned above can be removed. > > With this patch, the user-space receiving speed on a Intel SR1690 server with > a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps > to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core > processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed > the performance penalty comes from NUMA memory allocation. > > This patch set provides kernel ioctl interfaces for user space. User space can > either directly call the interfaces or libpcap interfaces can be further provided > on the top of the kernel ioctl interfaces. > > The order of tx/rx packets is up to user application. In some cases, e.g. network > monitors, ordering is not a big problem because they more care how to receive and > analyze packets in highest performance in parallel. > > This patch set only implements multiqueue interfaces for AF_PACKET and Intel > ixgbe NIC. Other protocols and NIC's can be handled on the top of this patch set. > > Signed-off-by: Fenghua Yu <fenghua.yu@intel.com> > Signed-off-by: Junchang Wang <junchangwang@gmail.com> > Signed-off-by: Xinan Tang <xinan.tang@intel.com> > --- I think it would be easier to manipulate the sk_hash to accomplish this. Allowing this from user space doesn't seem so great to me. You don't really want to pick the tx/rx bindings for sockets I think what you actually want is to optimize the hashing for this case to avoid the bottleneck you observe. I'm not too familiar with the af_packet stuff but could you do this with a single flag that indicates the sk_hash should be set in {t}packet_snd(). Maybe I missed your point or there is a reason this wouldn't work. But, then you don't need to do funny stuff in select_queue and it works with rps/xps as well. --John. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Le mercredi 15 décembre 2010 à 21:48 +0100, Eric Dumazet a écrit : > Le mercredi 15 décembre 2010 à 12:02 -0800, Fenghua Yu a écrit : > > From: Fenghua Yu <fenghua.yu@intel.com> > > > > Multiqueue and multicore provide packet parallel processing methodology. > > Current kernel and network drivers place one queue on one core. But the higher > > level socket doesn't know multiqueue. Current socket only can receive or send > > packets through one network interfaces. In some cases e.g. multi bpf filter > > tcpdump and snort, a lot of contentions come from socket operations like ring > > buffer. Even if the application itself has been fully parallelized and run on > > multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer > > and NIC device driver assemble packets to a single, serialized queue. Thus the > > application cannot actually run in parallel in high speed. I forgot to say that your patches are not against net-next-2.6, and not apply anyway. Always use David trees for networking patches... -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Dec 15, 2010 at 12:48:38PM -0800, Eric Dumazet wrote: > Le mercredi 15 décembre 2010 à 12:02 -0800, Fenghua Yu a écrit : > > From: Fenghua Yu <fenghua.yu@intel.com> > > > > Multiqueue and multicore provide packet parallel processing methodology. > > Current kernel and network drivers place one queue on one core. But the higher > > level socket doesn't know multiqueue. Current socket only can receive or send > > packets through one network interfaces. In some cases e.g. multi bpf filter > > tcpdump and snort, a lot of contentions come from socket operations like ring > > buffer. Even if the application itself has been fully parallelized and run on > > multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer > > and NIC device driver assemble packets to a single, serialized queue. Thus the > > application cannot actually run in parallel in high speed. > > > > To break the serialized packets assembling bottleneck in kernel, one way is to > > allow socket to know multiqueue associated with a NIC interface. So each socket > > can handle tx/rx in one queue in parallel. > > > > Kernel provides several interfaces by which sockets can be bound to rx/tx queues. > > User applications can configure socket by providing several sockets that each > > bound to a single queue, applications can get data from kernel in parallel. After > > that, competitions mentioned above can be removed. > > > > With this patch, the user-space receiving speed on a Intel SR1690 server with > > a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps > > to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core > > processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed > > the performance penalty comes from NUMA memory allocation. > > > > ??? please elaborate on these NUMA memory allocations. This should be OK > after commit 564824b0c52c34692d (net: allocate skbs on local node) > > > This patch set provides kernel ioctl interfaces for user space. User space can > > either directly call the interfaces or libpcap interfaces can be further provided > > on the top of the kernel ioctl interfaces. > > So, say we have 8 queues, you want libpcap opens 8 sockets, and bind > them to each queue. Add a bpf filter to each one of them. This seems not > generic way, because it wont work for an UDP socket for example. This only works for AF_PACKET like this patch set shows. > And you already can do this using SKF_AD_QUEUE (added in commit > d19742fb) SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't specify right SKF_AD_QUEUE. SKF_AD_QUEUE only works for rx. There is no queue bound interfaces for tx. I can change the patch set to use SKF_AD_QUEUE by removing the set rx queue interface and still keep interfaces of #define SIOGNUMRXQUEUE 0x8939 /* Get number of rx queues. */ #define SIOGNUMTXQUEUE 0x893A /* Get number of tx queues. */ #define SIOSTXQUEUEMAPPING 0x893C /* Set tx queue mapping. */ #define SIOGRXQUEUEMAPPING 0x893D /* Get rx queue mapping. */ #define SIOGTXQUEUEMAPPING 0x893E /* Get tx queue mapping. */ > > Also your AF_PACKET patch only address mmaped sockets. > The new patch set will use SKF_AD_QUEUE for rx. So it won't be limited to mmaped sockets. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 15 Dec 2010 17:14:25 -0800 Fenghua Yu <fenghua.yu@intel.com> wrote: > On Wed, Dec 15, 2010 at 12:48:38PM -0800, Eric Dumazet wrote: > > Le mercredi 15 décembre 2010 à 12:02 -0800, Fenghua Yu a écrit : > > > From: Fenghua Yu <fenghua.yu@intel.com> > > > > > > Multiqueue and multicore provide packet parallel processing methodology. > > > Current kernel and network drivers place one queue on one core. But the higher > > > level socket doesn't know multiqueue. Current socket only can receive or send > > > packets through one network interfaces. In some cases e.g. multi bpf filter > > > tcpdump and snort, a lot of contentions come from socket operations like ring > > > buffer. Even if the application itself has been fully parallelized and run on > > > multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer > > > and NIC device driver assemble packets to a single, serialized queue. Thus the > > > application cannot actually run in parallel in high speed. > > > > > > To break the serialized packets assembling bottleneck in kernel, one way is to > > > allow socket to know multiqueue associated with a NIC interface. So each socket > > > can handle tx/rx in one queue in parallel. > > > > > > Kernel provides several interfaces by which sockets can be bound to rx/tx queues. > > > User applications can configure socket by providing several sockets that each > > > bound to a single queue, applications can get data from kernel in parallel. After > > > that, competitions mentioned above can be removed. > > > > > > With this patch, the user-space receiving speed on a Intel SR1690 server with > > > a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps > > > to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core > > > processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed > > > the performance penalty comes from NUMA memory allocation. > > > > > > > ??? please elaborate on these NUMA memory allocations. This should be OK > > after commit 564824b0c52c34692d (net: allocate skbs on local node) > > > > > This patch set provides kernel ioctl interfaces for user space. User space can > > > either directly call the interfaces or libpcap interfaces can be further provided > > > on the top of the kernel ioctl interfaces. > > > > So, say we have 8 queues, you want libpcap opens 8 sockets, and bind > > them to each queue. Add a bpf filter to each one of them. This seems not > > generic way, because it wont work for an UDP socket for example. > > This only works for AF_PACKET like this patch set shows. > > > And you already can do this using SKF_AD_QUEUE (added in commit > > d19742fb) > > SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't > specify right SKF_AD_QUEUE. > > SKF_AD_QUEUE only works for rx. There is no queue bound interfaces for tx. > > I can change the patch set to use SKF_AD_QUEUE by removing the set rx queue > interface and still keep interfaces of > #define SIOGNUMRXQUEUE 0x8939 /* Get number of rx queues. */ > #define SIOGNUMTXQUEUE 0x893A /* Get number of tx queues. */ > #define SIOSTXQUEUEMAPPING 0x893C /* Set tx queue mapping. */ > #define SIOGRXQUEUEMAPPING 0x893D /* Get rx queue mapping. */ > #define SIOGTXQUEUEMAPPING 0x893E /* Get tx queue mapping. */ > > > > > Also your AF_PACKET patch only address mmaped sockets. > > > The new patch set will use SKF_AD_QUEUE for rx. So it won't be limited to mmaped > sockets. Do we really want to expose this kind of internals to userspace? The problem is once exposed, it becomes a kernel ABI and can not ever change.
On Thu, Dec 16, 2010 at 9:14 AM, Fenghua Yu <fenghua.yu@intel.com> wrote: > > SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't > specify right SKF_AD_QUEUE. It is wrong. AFAIK, you can get the queue number through /sys/class/net/eth*/queues/ or /proc/interrupts > > SKF_AD_QUEUE only works for rx. There is no queue bound interfaces for tx. Do you really need queue number? The packets must be already spreaded among CPUs, I think you means the current CPU number. Please see SKF_AD_CPU added by Eric. > > I can change the patch set to use SKF_AD_QUEUE by removing the set rx queue > interface and still keep interfaces of > #define SIOGNUMRXQUEUE 0x8939 /* Get number of rx queues. */ > #define SIOGNUMTXQUEUE 0x893A /* Get number of tx queues. */ > #define SIOSTXQUEUEMAPPING 0x893C /* Set tx queue mapping. */ > #define SIOGRXQUEUEMAPPING 0x893D /* Get rx queue mapping. */ > #define SIOGTXQUEUEMAPPING 0x893E /* Get tx queue mapping. */ > >> >> Also your AF_PACKET patch only address mmaped sockets. >> > The new patch set will use SKF_AD_QUEUE for rx. So it won't be limited to mmaped > sockets. > If you turn to SKF_AD_QUEUE, I think no patch for kernel is needed.
On Thu, Dec 16, 2010 at 4:48 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: >> With this patch, the user-space receiving speed on a Intel SR1690 server with >> a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps >> to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core >> processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed >> the performance penalty comes from NUMA memory allocation. >> > > ??? please elaborate on these NUMA memory allocations. This should be OK > after commit 564824b0c52c34692d (net: allocate skbs on local node) > Hi Eric, Commit 564824b0c52c34692d had been used in the experiments, but the problem remained unsolved. SLUB was used, and both servers were equipped with 8G physical memory. Is there any additional information I can provide? Thanks.
Changli Gao wrote: > On Thu, Dec 16, 2010 at 9:14 AM, Fenghua Yu <fenghua.yu@intel.com> wrote: >> SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't >> specify right SKF_AD_QUEUE. > > It is wrong. AFAIK, you can get the queue number through > /sys/class/net/eth*/queues/ or /proc/interrupts The number of Rx queues is also available through ETHTOOL_GRXRINGS though few drivers support it currently. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Le mercredi 15 décembre 2010 à 17:14 -0800, Fenghua Yu a écrit : > On Wed, Dec 15, 2010 at 12:48:38PM -0800, Eric Dumazet wrote: > > Le mercredi 15 décembre 2010 à 12:02 -0800, Fenghua Yu a écrit : > > > From: Fenghua Yu <fenghua.yu@intel.com> > > > > > > Multiqueue and multicore provide packet parallel processing methodology. > > > Current kernel and network drivers place one queue on one core. But the higher > > > level socket doesn't know multiqueue. Current socket only can receive or send > > > packets through one network interfaces. In some cases e.g. multi bpf filter > > > tcpdump and snort, a lot of contentions come from socket operations like ring > > > buffer. Even if the application itself has been fully parallelized and run on > > > multi-core systems and NIC handlex tx/rx in multiqueue in parallel, network layer > > > and NIC device driver assemble packets to a single, serialized queue. Thus the > > > application cannot actually run in parallel in high speed. > > > > > > To break the serialized packets assembling bottleneck in kernel, one way is to > > > allow socket to know multiqueue associated with a NIC interface. So each socket > > > can handle tx/rx in one queue in parallel. > > > > > > Kernel provides several interfaces by which sockets can be bound to rx/tx queues. > > > User applications can configure socket by providing several sockets that each > > > bound to a single queue, applications can get data from kernel in parallel. After > > > that, competitions mentioned above can be removed. > > > > > > With this patch, the user-space receiving speed on a Intel SR1690 server with > > > a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps > > > to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core > > > processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed > > > the performance penalty comes from NUMA memory allocation. > > > > > > > ??? please elaborate on these NUMA memory allocations. This should be OK > > after commit 564824b0c52c34692d (net: allocate skbs on local node) > > No data for this NUMA problem ? We had to convince Andrew Morton for this patch to get in. > > > This patch set provides kernel ioctl interfaces for user space. User space can > > > either directly call the interfaces or libpcap interfaces can be further provided > > > on the top of the kernel ioctl interfaces. > > > > So, say we have 8 queues, you want libpcap opens 8 sockets, and bind > > them to each queue. Add a bpf filter to each one of them. This seems not > > generic way, because it wont work for an UDP socket for example. > > This only works for AF_PACKET like this patch set shows. > Yes, we also should address other sockets, with generic mechanisms. > > And you already can do this using SKF_AD_QUEUE (added in commit > > d19742fb) > > SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't > specify right SKF_AD_QUEUE. > > SKF_AD_QUEUE only works for rx. There is no queue bound interfaces for tx. > > I can change the patch set to use SKF_AD_QUEUE by removing the set rx queue > interface and still keep interfaces of > #define SIOGNUMRXQUEUE 0x8939 /* Get number of rx queues. */ > #define SIOGNUMTXQUEUE 0x893A /* Get number of tx queues. */ > #define SIOSTXQUEUEMAPPING 0x893C /* Set tx queue mapping. */ > #define SIOGRXQUEUEMAPPING 0x893D /* Get rx queue mapping. */ > #define SIOGTXQUEUEMAPPING 0x893E /* Get tx queue mapping. */ > > > > > Also your AF_PACKET patch only address mmaped sockets. > > > The new patch set will use SKF_AD_QUEUE for rx. So it won't be limited to mmaped > sockets. > We really need to be smarter than that, not adding raw API. Tom Herbert added RPS, RFS, XPS, in a way applications dont have to use special API, just run normal code. Please understand that using 8 AF_PACKET sockets bound to a given device is a total waste, because the way we loop on ptype_all before entering AF_PACKET code, and in 12% of the cases deliver the packet into a queue, and 77.5% of the case reject the packet. This is absolutely not scalable to say... 64 queues. I do believe we can handle that using one AF_PACKET socket for the RX side, in order to not slow down the loop we have in __netif_receive_skb() list_for_each_entry_rcu(ptype, &ptype_all, list) { ... deliver_skb(skb, pt_prev, orig_dev); } (Same problem with dev_queue_xmit_nit() by the way, even worse since we skb_clone() packet _before_ entering af_packet code) And we can change af_packet to split the load to N skb queues or N ring buffers, N not being necessarly number of NIC queues, but the number needed to handle the expected load. There is nothing preventing us changing af_packet/udp/tcp_listener to something more scalable in itself, using a set of receive queues, and NUMA friendly data set. We did multiqueue for a net_device like this, not adding N pseudo devices as we could have done. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Le jeudi 16 décembre 2010 à 09:52 +0800, Junchang Wang a écrit : > On Thu, Dec 16, 2010 at 4:48 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > >> With this patch, the user-space receiving speed on a Intel SR1690 server with > >> a single L5640 6-core processor and a single ixgbe-based NIC goes from 0.73Mpps > >> to 4.20Mpps, nearly a linear speedup. A Intel SR1625 server two E5530 4-core > >> processors and a single ixgbe-based NIC goes from 0.80Mpps to 4.6Mpps. We noticed > >> the performance penalty comes from NUMA memory allocation. > >> > > > > ??? please elaborate on these NUMA memory allocations. This should be OK > > after commit 564824b0c52c34692d (net: allocate skbs on local node) > > > Hi Eric, > Commit 564824b0c52c34692d had been used in the experiments, but the problem > remained unsolved. > > SLUB was used, and both servers were equipped with 8G physical memory. > Is there any > additional information I can provide? > Yes, sure, you could provide a description of the bench you used, and data you gathered to make the conclusion that NUMA was a problem. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Dec 16, 2010 at 12:44 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > > We really need to be smarter than that, not adding raw API. > > Tom Herbert added RPS, RFS, XPS, in a way applications dont have to use > special API, just run normal code. > > Please understand that using 8 AF_PACKET sockets bound to a given device > is a total waste, because the way we loop on ptype_all before entering > AF_PACKET code, and in 12% of the cases deliver the packet into a queue, > and 77.5% of the case reject the packet. > > This is absolutely not scalable to say... 64 queues. > > I do believe we can handle that using one AF_PACKET socket for the RX > side, in order to not slow down the loop we have in > __netif_receive_skb() > > list_for_each_entry_rcu(ptype, &ptype_all, list) { > ... > deliver_skb(skb, pt_prev, orig_dev); > } > > (Same problem with dev_queue_xmit_nit() by the way, even worse since we > skb_clone() packet _before_ entering af_packet code) > > And we can change af_packet to split the load to N skb queues or N ring > buffers, N not being necessarly number of NIC queues, but the number > needed to handle the expected load. > > There is nothing preventing us changing af_packet/udp/tcp_listener to > something more scalable in itself, using a set of receive queues, and > NUMA friendly data set. We did multiqueue for a net_device like this, > not adding N pseudo devices as we could have done. > Valuable comments. Thank you very much. We'll cook a new version and resubmit it.
On Thu, Dec 16, 2010 at 1:00 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote: > Le jeudi 16 décembre 2010 à 09:52 +0800, Junchang Wang a écrit : >> Commit 564824b0c52c34692d had been used in the experiments, but the problem >> remained unsolved. >> >> SLUB was used, and both servers were equipped with 8G physical memory. >> Is there any >> additional information I can provide? >> > > Yes, sure, you could provide a description of the bench you used, and > data you gathered to make the conclusion that NUMA was a problem. > Under the current circumstances (1Mpps), we can hardly see side effects from memory allocator. With higher speed (say, 5Mpps with this patch set), the problem emerged. I'll continue this work after the patch set is done. Thanks.
On Thu, Dec 16, 2010 at 9:28 AM, Changli Gao <xiaosuo@gmail.com> wrote: > On Thu, Dec 16, 2010 at 9:14 AM, Fenghua Yu <fenghua.yu@intel.com> wrote: >> >> SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't >> specify right SKF_AD_QUEUE. > > It is wrong. AFAIK, you can get the queue number through > /sys/class/net/eth*/queues/ or /proc/interrupts > Valuable comment. Thanks. > > If you turn to SKF_AD_QUEUE, I think no patch for kernel is needed. > This patch set is about parallelization of socket interfaces to gain performance boost (say, from 1Mpps to around 5Mpps), rather than simply bounding socket to cpu/queue. Therefore, it does worth having. Thanks.
Le vendredi 17 décembre 2010 à 14:22 +0800, Junchang Wang a écrit : > On Thu, Dec 16, 2010 at 9:28 AM, Changli Gao <xiaosuo@gmail.com> wrote: > > On Thu, Dec 16, 2010 at 9:14 AM, Fenghua Yu <fenghua.yu@intel.com> wrote: > >> > >> SKF_AD_QUEUE doesn't know number of rx queues. Thus user application can't > >> specify right SKF_AD_QUEUE. > > > > It is wrong. AFAIK, you can get the queue number through > > /sys/class/net/eth*/queues/ or /proc/interrupts > > > > Valuable comment. Thanks. > > > > > If you turn to SKF_AD_QUEUE, I think no patch for kernel is needed. > > > This patch set is about parallelization of socket interfaces to gain > performance boost (say, from 1Mpps to around 5Mpps), rather than > simply bounding socket to cpu/queue. Therefore, it does worth having. > Definitely, but this needs to be designed so that it can be used by even dumb applications :) -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/include/linux/sockios.h b/include/linux/sockios.h index 241f179..b121d9a 100644 --- a/include/linux/sockios.h +++ b/include/linux/sockios.h @@ -66,6 +66,13 @@ #define SIOCSIFHWBROADCAST 0x8937 /* set hardware broadcast addr */ #define SIOCGIFCOUNT 0x8938 /* get number of devices */ +#define SIOGNUMRXQUEUE 0x8939 /* Get number of rx queues. */ +#define SIOGNUMTXQUEUE 0x893A /* Get number of tx queues. */ +#define SIOSRXQUEUEMAPPING 0x893B /* Set rx queue mapping. */ +#define SIOSTXQUEUEMAPPING 0x893C /* Set tx queue mapping. */ +#define SIOGRXQUEUEMAPPING 0x893D /* Get rx queue mapping. */ +#define SIOGTXQUEUEMAPPING 0x893E /* Get tx queue mapping. */ + #define SIOCGIFBR 0x8940 /* Bridging support */ #define SIOCSIFBR 0x8941 /* Set bridging options */ diff --git a/include/net/sock.h b/include/net/sock.h index 659d968..d677bba 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -109,6 +109,7 @@ struct net; * @skc_nulls_node: main hash linkage for TCP/UDP/UDP-Lite protocol * @skc_refcnt: reference count * @skc_tx_queue_mapping: tx queue number for this connection + * @skc_rx_queue_mapping: rx queue number for this connection * @skc_hash: hash value used with various protocol lookup tables * @skc_u16hashes: two u16 hash values used by UDP lookup tables * @skc_family: network address family @@ -133,6 +134,7 @@ struct sock_common { }; atomic_t skc_refcnt; int skc_tx_queue_mapping; + int skc_rx_queue_mapping; union { unsigned int skc_hash; @@ -231,6 +233,7 @@ struct sock { #define sk_nulls_node __sk_common.skc_nulls_node #define sk_refcnt __sk_common.skc_refcnt #define sk_tx_queue_mapping __sk_common.skc_tx_queue_mapping +#define sk_rx_queue_mapping __sk_common.skc_rx_queue_mapping #define sk_copy_start __sk_common.skc_hash #define sk_hash __sk_common.skc_hash @@ -1234,6 +1237,21 @@ static inline int sk_tx_queue_get(const struct sock *sk) return sk ? sk->sk_tx_queue_mapping : -1; } +static inline void sk_rx_queue_set(struct sock *sk, int rx_queue) +{ + sk->sk_rx_queue_mapping = rx_queue; +} + +static inline int sk_rx_queue_get(const struct sock *sk) +{ + return sk ? sk->sk_rx_queue_mapping : -1; +} + +static inline void sk_rx_queue_clear(struct sock *sk) +{ + sk->sk_rx_queue_mapping = -1; +} + static inline void sk_set_socket(struct sock *sk, struct socket *sock) { sk_tx_queue_clear(sk); diff --git a/net/core/sock.c b/net/core/sock.c index fb60801..9ad92cb 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -1000,7 +1000,8 @@ static void sock_copy(struct sock *nsk, const struct sock *osk) #endif BUILD_BUG_ON(offsetof(struct sock, sk_copy_start) != sizeof(osk->sk_node) + sizeof(osk->sk_refcnt) + - sizeof(osk->sk_tx_queue_mapping)); + sizeof(osk->sk_tx_queue_mapping) + + sizeof(osk->sk_rx_queue_mapping)); memcpy(&nsk->sk_copy_start, &osk->sk_copy_start, osk->sk_prot->obj_size - offsetof(struct sock, sk_copy_start)); #ifdef CONFIG_SECURITY_NETWORK @@ -1045,6 +1046,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, if (!try_module_get(prot->owner)) goto out_free_sec; sk_tx_queue_clear(sk); + sk_rx_queue_clear(sk); } return sk;