[RFC,1/2] netmap: infrastructure (in staging)

Message ID	20130419120651.4e646976@nehalam.linuxnetplumber.net
State	RFC, archived
Delegated to:	David Miller
Headers	show Return-Path: <netdev-owner@vger.kernel.org> Date: Fri, 19 Apr 2013 12:06:51 -0700 From: Stephen Hemminger <stephen@networkplumber.org> To: David Miller <davem@davemloft.net>, Greg KH <greg@kroah.com> Cc: netdev@vger.kernel.org Subject: [RFC 1/2] netmap: infrastructure (in staging) Message-ID: <20130419120651.4e646976@nehalam.linuxnetplumber.net> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk

Hi all Various10GbE-NIC 64Byte Short Packet Processing Bechmark (RX) with netmap memo share for you. http://twitpic.com/clb97k FYI: Another netmap benchmark with DPDK result. Disruptive IP Networking with Intel DPDK on Linux http://slidesha.re/SeVFZn enjoy! -- Naoto On Fri, 19 Apr 2013 12:06:51 -0700 Stephen Hemminger <stephen@networkplumber.org> wrote: > Netmap is a framework for packet generation and capture from user > space. It allows for efficient packet handling (up to line rate on > 10Gb) with minimum system load. For more info see: > http://info.iet.unipi.it/~luigi/netmap/ > > This version is based on the latest version from Luigi Rizzo. It has > been modified to live to work with current net-next kernel. It still > has all the BSD ugliness, and that is why it needs to spend some time > in the staging penalty box. > > It builds and loads, but definitely needs more work. > > Signed-off-by: Stephen Hemminger <stephen@networkplumger.org> > > --- > Patch is against net-next > > drivers/staging/Kconfig | 2 > drivers/staging/Makefile | 1 > drivers/staging/netmap/Kconfig | 16 > drivers/staging/netmap/Makefile | 2 > drivers/staging/netmap/README | 127 + > drivers/staging/netmap/TODO | 16 > drivers/staging/netmap/netmap.c | 2514 +++++++++++++++++++++++++++++++++++ > drivers/staging/netmap/netmap_mem2.c | 974 +++++++++++++ > include/netmap/bsd_glue.h | 263 +++ > include/netmap/netmap_kern.h | 474 ++++++ > include/uapi/Kbuild | 1 > include/uapi/linux/if.h | 1 > include/uapi/netmap/Kbuild | 3 > include/uapi/netmap/netmap.h | 289 ++++ > include/uapi/netmap/netmap_user.h | 95 + > 15 files changed, 4778 insertions(+) > > --- a/drivers/staging/Kconfig 2013-02-26 10:19:35.000000000 -0800 > +++ b/drivers/staging/Kconfig 2013-03-10 10:08:20.323671490 -0700 > @@ -140,4 +140,6 @@ source "drivers/staging/zcache/Kconfig" > > source "drivers/staging/goldfish/Kconfig" > > +source "drivers/staging/netmap/Kconfig" > + > endif # STAGING > --- a/drivers/staging/Makefile 2013-02-26 10:19:35.000000000 -0800 > +++ b/drivers/staging/Makefile 2013-03-10 10:08:48.555305267 -0700 > @@ -62,3 +62,4 @@ obj-$(CONFIG_SB105X) += sb105x/ > obj-$(CONFIG_FIREWIRE_SERIAL) += fwserial/ > obj-$(CONFIG_ZCACHE) += zcache/ > obj-$(CONFIG_GOLDFISH) += goldfish/ > +obj-$(CONFIG_NETMAP) += netmap/ > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/drivers/staging/netmap/Kconfig 2013-03-10 10:08:20.323671490 -0700 > @@ -0,0 +1,16 @@ > +# > +# Netmap - user mode packet processing framework > +# > + > +config NETMAP > + tristate "Netmap - user mode networking" > + depends on NET && !IOMMU_API > + default n > + help > + If you say Y here, you will get experimental support for > + netmap a framework for fast packet I/O using memory mapped > + buffers. > + > + See <http:/nfo.iet.unipi.it/~luigi/netmap/> for more information. > + > + If unsure, say N. > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/drivers/staging/netmap/Makefile 2013-03-10 10:08:20.323671490 -0700 > @@ -0,0 +1,2 @@ > +EXTRA_CFLAGS := -DDEBUG > +obj-$(CONFIG_NETMAP) += netmap.o > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/drivers/staging/netmap/netmap.c 2013-03-10 16:52:34.160592316 -0700 > @@ -0,0 +1,2515 @@ > +/* > + * Copyright (C) 2011-2012 Matteo Landi, Luigi Rizzo. All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > + > +#define NM_BRIDGE > + > +/* > + * This module supports memory mapped access to network devices, > + * see netmap(4). > + * > + * The module uses a large, memory pool allocated by the kernel > + * and accessible as mmapped memory by multiple userspace threads/processes. > + * The memory pool contains packet buffers and "netmap rings", > + * i.e. user-accessible copies of the interface's queues. > + * > + * Access to the network card works like this: > + * 1. a process/thread issues one or more open() on /dev/netmap, to create > + * select()able file descriptor on which events are reported. > + * 2. on each descriptor, the process issues an ioctl() to identify > + * the interface that should report events to the file descriptor. > + * 3. on each descriptor, the process issues an mmap() request to > + * map the shared memory region within the process' address space. > + * The list of interesting queues is indicated by a location in > + * the shared memory region. > + * 4. using the functions in the netmap(4) userspace API, a process > + * can look up the occupation state of a queue, access memory buffers, > + * and retrieve received packets or enqueue packets to transmit. > + * 5. using some ioctl()s the process can synchronize the userspace view > + * of the queue with the actual status in the kernel. This includes both > + * receiving the notification of new packets, and transmitting new > + * packets on the output interface. > + * 6. select() or poll() can be used to wait for events on individual > + * transmit or receive queues (or all queues for a given interface). > + */ > + > +#ifdef linux > +#include <netmap/bsd_glue.h> > +static netdev_tx_t linux_netmap_start(struct sk_buff *skb, struct net_device *dev); > +#endif /* linux */ > + > +#ifdef __APPLE__ > +#include "osx_glue.h" > +#endif /* __APPLE__ */ > + > +#ifdef __FreeBSD__ > +#include <sys/cdefs.h> /* prerequisite */ > +__FBSDID("$FreeBSD: head/sys/dev/netmap/netmap.c 241723 2012-10-19 09:41:45Z glebius $"); > + > +#include <sys/types.h> > +#include <sys/module.h> > +#include <sys/errno.h> > +#include <sys/param.h> /* defines used in kernel.h */ > +#include <sys/jail.h> > +#include <sys/kernel.h> /* types used in module initialization */ > +#include <sys/conf.h> /* cdevsw struct */ > +#include <sys/uio.h> /* uio struct */ > +#include <sys/sockio.h> > +#include <sys/socketvar.h> /* struct socket */ > +#include <sys/malloc.h> > +#include <sys/mman.h> /* PROT_EXEC */ > +#include <sys/poll.h> > +#include <sys/proc.h> > +#include <vm/vm.h> /* vtophys */ > +#include <vm/pmap.h> /* vtophys */ > +#include <sys/socket.h> /* sockaddrs */ > +#include <machine/bus.h> > +#include <sys/selinfo.h> > +#include <sys/sysctl.h> > +#include <net/if.h> > +#include <net/bpf.h> /* BIOCIMMEDIATE */ > +#include <net/vnet.h> > +#include <machine/bus.h> /* bus_dmamap_* */ > + > +MALLOC_DEFINE(M_NETMAP, "netmap", "Network memory map"); > +#endif /* __FreeBSD__ */ > + > +#include <netmap/netmap.h> > +#include <netmap/netmap_kern.h> > + > +u_int netmap_total_buffers; > +u_int netmap_buf_size; > +char *netmap_buffer_base; /* address of an invalid buffer */ > + > +/* user-controlled variables */ > +int netmap_verbose = 1; > + > +static int netmap_no_timestamp; /* don't timestamp on rxsync */ > + > +SYSCTL_NODE(_dev, OID_AUTO, netmap, CTLFLAG_RW, 0, "Netmap args"); > +SYSCTL_INT(_dev_netmap, OID_AUTO, verbose, > + CTLFLAG_RW, &netmap_verbose, 0, "Verbose mode"); > +SYSCTL_INT(_dev_netmap, OID_AUTO, no_timestamp, > + CTLFLAG_RW, &netmap_no_timestamp, 0, "no_timestamp"); > +int netmap_mitigate = 1; > +SYSCTL_INT(_dev_netmap, OID_AUTO, mitigate, CTLFLAG_RW, &netmap_mitigate, 0, ""); > +int netmap_no_pendintr = 1; > +SYSCTL_INT(_dev_netmap, OID_AUTO, no_pendintr, > + CTLFLAG_RW, &netmap_no_pendintr, 0, "Always look for new received packets."); > + > +int netmap_drop = 0; /* debugging */ > +int netmap_flags = 0; /* debug flags */ > +int netmap_fwd = 0; /* force transparent mode */ > +int netmap_copy = 0; /* debugging, copy content */ > + > +SYSCTL_INT(_dev_netmap, OID_AUTO, drop, CTLFLAG_RW, &netmap_drop, 0 , ""); > +SYSCTL_INT(_dev_netmap, OID_AUTO, flags, CTLFLAG_RW, &netmap_flags, 0 , ""); > +SYSCTL_INT(_dev_netmap, OID_AUTO, fwd, CTLFLAG_RW, &netmap_fwd, 0 , ""); > +SYSCTL_INT(_dev_netmap, OID_AUTO, copy, CTLFLAG_RW, &netmap_copy, 0 , ""); > + > +#ifdef NM_BRIDGE /* support for netmap bridge */ > + > +/* > + * system parameters. > + * > + * All switched ports have prefix NM_NAME. > + * The switch has a max of NM_BDG_MAXPORTS ports (often stored in a bitmap, > + * so a practical upper bound is 64). > + * Each tx ring is read-write, whereas rx rings are readonly (XXX not done yet). > + * The virtual interfaces use per-queue lock instead of core lock. > + * In the tx loop, we aggregate traffic in batches to make all operations > + * faster. The batch size is NM_BDG_BATCH > + */ > +#define NM_NAME "vale" /* prefix for the interface */ > +#define NM_BDG_MAXPORTS 16 /* up to 64 ? */ > +#define NM_BRIDGE_RINGSIZE 1024 /* in the device */ > +#define NM_BDG_HASH 1024 /* forwarding table entries */ > +#define NM_BDG_BATCH 1024 /* entries in the forwarding buffer */ > +#define NM_BRIDGES 4 /* number of bridges */ > +int netmap_bridge = NM_BDG_BATCH; /* bridge batch size */ > +SYSCTL_INT(_dev_netmap, OID_AUTO, bridge, CTLFLAG_RW, &netmap_bridge, 0 , ""); > + > +#ifdef linux > +#define ADD_BDG_REF(ifp) (NA(ifp)->if_refcount++) > +#define DROP_BDG_REF(ifp) (NA(ifp)->if_refcount-- <= 1) > +#else /* !linux */ > +#define ADD_BDG_REF(ifp) (ifp)->if_refcount++ > +#define DROP_BDG_REF(ifp) refcount_release(&(ifp)->if_refcount) > +#ifdef __FreeBSD__ > +#include <sys/endian.h> > +#include <sys/refcount.h> > +#endif /* __FreeBSD__ */ > +#define prefetch(x) __builtin_prefetch(x) > +#endif /* !linux */ > + > +static void bdg_netmap_attach(struct ifnet *ifp); > +static int bdg_netmap_reg(struct ifnet *ifp, int onoff); > +/* per-tx-queue entry */ > +struct nm_bdg_fwd { /* forwarding entry for a bridge */ > + void *buf; > + uint64_t dst; /* dst mask */ > + uint32_t src; /* src index ? */ > + uint16_t len; /* src len */ > +}; > + > +struct nm_hash_ent { > + uint64_t mac; /* the top 2 bytes are the epoch */ > + uint64_t ports; > +}; > + > +/* > + * Interfaces for a bridge are all in ports[]. > + * The array has fixed size, an empty entry does not terminate > + * the search. > + */ > +struct nm_bridge { > + struct ifnet *bdg_ports[NM_BDG_MAXPORTS]; > + int n_ports; > + uint64_t act_ports; > + int freelist; /* first buffer index */ > + NM_SELINFO_T si; /* poll/select wait queue */ > + NM_LOCK_T bdg_lock; /* protect the selinfo ? */ > + > + /* the forwarding table, MAC+ports */ > + struct nm_hash_ent ht[NM_BDG_HASH]; > + > + int namelen; /* 0 means free */ > + char basename[IFNAMSIZ]; > +}; > + > +struct nm_bridge nm_bridges[NM_BRIDGES]; > + > +#define BDG_LOCK(b) mtx_lock(&(b)->bdg_lock) > +#define BDG_UNLOCK(b) mtx_unlock(&(b)->bdg_lock) > + > +/* > + * NA(ifp)->bdg_port port index > + */ > + > +// XXX only for multiples of 64 bytes, non overlapped. > +static inline void > +pkt_copy(void *_src, void *_dst, int l) > +{ > + uint64_t *src = _src; > + uint64_t *dst = _dst; > + if (unlikely(l >= 1024)) { > + bcopy(src, dst, l); > + return; > + } > + for (; likely(l > 0); l-=64) { > + *dst++ = *src++; > + *dst++ = *src++; > + *dst++ = *src++; > + *dst++ = *src++; > + *dst++ = *src++; > + *dst++ = *src++; > + *dst++ = *src++; > + *dst++ = *src++; > + } > +} > + > +/* > + * locate a bridge among the existing ones. > + * a ':' in the name terminates the bridge name. Otherwise, just NM_NAME. > + * We assume that this is called with a name of at least NM_NAME chars. > + */ > +static struct nm_bridge * > +nm_find_bridge(const char *name) > +{ > + int i, l, namelen, e; > + struct nm_bridge *b = NULL; > + > + namelen = strlen(NM_NAME); /* base length */ > + l = strlen(name); /* actual length */ > + for (i = namelen + 1; i < l; i++) { > + if (name[i] == ':') { > + namelen = i; > + break; > + } > + } > + if (namelen >= IFNAMSIZ) > + namelen = IFNAMSIZ; > + ND("--- prefix is '%.*s' ---", namelen, name); > + > + /* use the first entry for locking */ > + BDG_LOCK(nm_bridges); // XXX do better > + for (e = -1, i = 1; i < NM_BRIDGES; i++) { > + b = nm_bridges + i; > + if (b->namelen == 0) > + e = i; /* record empty slot */ > + else if (strncmp(name, b->basename, namelen) == 0) { > + ND("found '%.*s' at %d", namelen, name, i); > + break; > + } > + } > + if (i == NM_BRIDGES) { /* all full */ > + if (e == -1) { /* no empty slot */ > + b = NULL; > + } else { > + b = nm_bridges + e; > + strncpy(b->basename, name, namelen); > + b->namelen = namelen; > + } > + } > + BDG_UNLOCK(nm_bridges); > + return b; > +} > +#endif /* NM_BRIDGE */ > + > + > +/* > + * Fetch configuration from the device, to cope with dynamic > + * reconfigurations after loading the module. > + */ > +static int > +netmap_update_config(struct netmap_adapter *na) > +{ > + struct ifnet *ifp = na->ifp; > + u_int txr, txd, rxr, rxd; > + > + txr = txd = rxr = rxd = 0; > + if (na->nm_config) { > + na->nm_config(ifp, &txr, &txd, &rxr, &rxd); > + } else { > + /* take whatever we had at init time */ > + txr = na->num_tx_rings; > + txd = na->num_tx_desc; > + rxr = na->num_rx_rings; > + rxd = na->num_rx_desc; > + } > + > + if (na->num_tx_rings == txr && na->num_tx_desc == txd && > + na->num_rx_rings == rxr && na->num_rx_desc == rxd) > + return 0; /* nothing changed */ > + if (netmap_verbose || na->refcount > 0) { > + D("stored config %s: txring %d x %d, rxring %d x %d", > + ifp->if_xname, > + na->num_tx_rings, na->num_tx_desc, > + na->num_rx_rings, na->num_rx_desc); > + D("new config %s: txring %d x %d, rxring %d x %d", > + ifp->if_xname, txr, txd, rxr, rxd); > + } > + if (na->refcount == 0) { > + D("configuration changed (but fine)"); > + na->num_tx_rings = txr; > + na->num_tx_desc = txd; > + na->num_rx_rings = rxr; > + na->num_rx_desc = rxd; > + return 0; > + } > + D("configuration changed while active, this is bad..."); > + return 1; > +} > + > +/*------------- memory allocator -----------------*/ > +#ifdef NETMAP_MEM2 > +#include "netmap_mem2.c" > +#else /* !NETMAP_MEM2 */ > +#include "netmap_mem1.c" > +#endif /* !NETMAP_MEM2 */ > +/*------------ end of memory allocator ----------*/ > + > + > +/* Structure associated to each thread which registered an interface. > + * > + * The first 4 fields of this structure are written by NIOCREGIF and > + * read by poll() and NIOC?XSYNC. > + * There is low contention among writers (actually, a correct user program > + * should have no contention among writers) and among writers and readers, > + * so we use a single global lock to protect the structure initialization. > + * Since initialization involves the allocation of memory, we reuse the memory > + * allocator lock. > + * Read access to the structure is lock free. Readers must check that > + * np_nifp is not NULL before using the other fields. > + * If np_nifp is NULL initialization has not been performed, so they should > + * return an error to userlevel. > + * > + * The ref_done field is used to regulate access to the refcount in the > + * memory allocator. The refcount must be incremented at most once for > + * each open("/dev/netmap"). The increment is performed by the first > + * function that calls netmap_get_memory() (currently called by > + * mmap(), NIOCGINFO and NIOCREGIF). > + * If the refcount is incremented, it is then decremented when the > + * private structure is destroyed. > + */ > +struct netmap_priv_d { > + struct netmap_if * volatile np_nifp; /* netmap interface descriptor. */ > + > + struct ifnet *np_ifp; /* device for which we hold a reference */ > + int np_ringid; /* from the ioctl */ > + u_int np_qfirst, np_qlast; /* range of rings to scan */ > + uint16_t np_txpoll; > + > + unsigned long ref_done; /* use with NMA_LOCK held */ > +}; > + > + > +static int > +netmap_get_memory(struct netmap_priv_d* p) > +{ > + int error = 0; > + NMA_LOCK(); > + if (!p->ref_done) { > + error = netmap_memory_finalize(); > + if (!error) > + p->ref_done = 1; > + } > + NMA_UNLOCK(); > + return error; > +} > + > +/* > + * File descriptor's private data destructor. > + * > + * Call nm_register(ifp,0) to stop netmap mode on the interface and > + * revert to normal operation. We expect that np_ifp has not gone. > + */ > +/* call with NMA_LOCK held */ > +static void > +netmap_dtor_locked(void *data) > +{ > + struct netmap_priv_d *priv = data; > + struct ifnet *ifp = priv->np_ifp; > + struct netmap_adapter *na = NA(ifp); > + struct netmap_if *nifp = priv->np_nifp; > + > + na->refcount--; > + if (na->refcount <= 0) { /* last instance */ > + u_int i, j, lim; > + > + if (netmap_verbose) > + D("deleting last instance for %s", ifp->if_xname); > + /* > + * there is a race here with *_netmap_task() and > + * netmap_poll(), which don't run under NETMAP_REG_LOCK. > + * na->refcount == 0 && na->ifp->if_capenable & IFCAP_NETMAP > + * (aka NETMAP_DELETING(na)) are a unique marker that the > + * device is dying. > + * Before destroying stuff we sleep a bit, and then complete > + * the job. NIOCREG should realize the condition and > + * loop until they can continue; the other routines > + * should check the condition at entry and quit if > + * they cannot run. > + */ > + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); > + tsleep(na, 0, "NIOCUNREG", 4); > + na->nm_lock(ifp, NETMAP_REG_LOCK, 0); > + na->nm_register(ifp, 0); /* off, clear IFCAP_NETMAP */ > + /* Wake up any sleeping threads. netmap_poll will > + * then return POLLERR > + */ > + for (i = 0; i < na->num_tx_rings + 1; i++) > + selwakeuppri(&na->tx_rings[i].si, PI_NET); > + for (i = 0; i < na->num_rx_rings + 1; i++) > + selwakeuppri(&na->rx_rings[i].si, PI_NET); > + selwakeuppri(&na->tx_si, PI_NET); > + selwakeuppri(&na->rx_si, PI_NET); > + /* release all buffers */ > + for (i = 0; i < na->num_tx_rings + 1; i++) { > + struct netmap_ring *ring = na->tx_rings[i].ring; > + lim = na->tx_rings[i].nkr_num_slots; > + for (j = 0; j < lim; j++) > + netmap_free_buf(nifp, ring->slot[j].buf_idx); > + /* knlist_destroy(&na->tx_rings[i].si.si_note); */ > + mtx_destroy(&na->tx_rings[i].q_lock); > + } > + for (i = 0; i < na->num_rx_rings + 1; i++) { > + struct netmap_ring *ring = na->rx_rings[i].ring; > + lim = na->rx_rings[i].nkr_num_slots; > + for (j = 0; j < lim; j++) > + netmap_free_buf(nifp, ring->slot[j].buf_idx); > + /* knlist_destroy(&na->rx_rings[i].si.si_note); */ > + mtx_destroy(&na->rx_rings[i].q_lock); > + } > + /* XXX kqueue(9) needed; these will mirror knlist_init. */ > + /* knlist_destroy(&na->tx_si.si_note); */ > + /* knlist_destroy(&na->rx_si.si_note); */ > + netmap_free_rings(na); > + wakeup(na); > + } > + netmap_if_free(nifp); > +} > + > +static void > +nm_if_rele(struct ifnet *ifp) > +{ > +#ifndef NM_BRIDGE > + if_rele(ifp); > +#else /* NM_BRIDGE */ > + int i, full; > + struct nm_bridge *b; > + > + if (strncmp(ifp->if_xname, NM_NAME, sizeof(NM_NAME) - 1)) { > + if_rele(ifp); > + return; > + } > + if (!DROP_BDG_REF(ifp)) > + return; > + b = ifp->if_bridge; > + BDG_LOCK(nm_bridges); > + BDG_LOCK(b); > + ND("want to disconnect %s from the bridge", ifp->if_xname); > + full = 0; > + for (i = 0; i < NM_BDG_MAXPORTS; i++) { > + if (b->bdg_ports[i] == ifp) { > + b->bdg_ports[i] = NULL; > + bzero(ifp, sizeof(*ifp)); > + free(ifp, M_DEVBUF); > + break; > + } > + else if (b->bdg_ports[i] != NULL) > + full = 1; > + } > + BDG_UNLOCK(b); > + if (full == 0) { > + ND("freeing bridge %d", b - nm_bridges); > + b->namelen = 0; > + } > + BDG_UNLOCK(nm_bridges); > + if (i == NM_BDG_MAXPORTS) > + D("ouch, cannot find ifp to remove"); > +#endif /* NM_BRIDGE */ > +} > + > +static void > +netmap_dtor(void *data) > +{ > + struct netmap_priv_d *priv = data; > + struct ifnet *ifp = priv->np_ifp; > + struct netmap_adapter *na; > + > + NMA_LOCK(); > + if (ifp) { > + na = NA(ifp); > + na->nm_lock(ifp, NETMAP_REG_LOCK, 0); > + netmap_dtor_locked(data); > + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); > + > + nm_if_rele(ifp); > + } > + if (priv->ref_done) { > + netmap_memory_deref(); > + } > + NMA_UNLOCK(); > + bzero(priv, sizeof(*priv)); /* XXX for safety */ > + free(priv, M_DEVBUF); > +} > + > +#ifdef __FreeBSD__ > +#include <vm/vm.h> > +#include <vm/vm_param.h> > +#include <vm/vm_object.h> > +#include <vm/vm_page.h> > +#include <vm/vm_pager.h> > +#include <vm/uma.h> > + > +static struct cdev_pager_ops saved_cdev_pager_ops; > + > +static int > +netmap_dev_pager_ctor(void *handle, vm_ooffset_t size, vm_prot_t prot, > + vm_ooffset_t foff, struct ucred *cred, u_short *color) > +{ > + if (netmap_verbose) > + D("first mmap for %p", handle); > + return saved_cdev_pager_ops.cdev_pg_ctor(handle, > + size, prot, foff, cred, color); > +} > + > +static void > +netmap_dev_pager_dtor(void *handle) > +{ > + saved_cdev_pager_ops.cdev_pg_dtor(handle); > + ND("ready to release memory for %p", handle); > +} > + > + > +static struct cdev_pager_ops netmap_cdev_pager_ops = { > + .cdev_pg_ctor = netmap_dev_pager_ctor, > + .cdev_pg_dtor = netmap_dev_pager_dtor, > + .cdev_pg_fault = NULL, > +}; > + > +static int > +netmap_mmap_single(struct cdev *cdev, vm_ooffset_t *foff, > + vm_size_t objsize, vm_object_t *objp, int prot) > +{ > + vm_object_t obj; > + > + ND("cdev %p foff %jd size %jd objp %p prot %d", cdev, > + (intmax_t )*foff, (intmax_t )objsize, objp, prot); > + obj = vm_pager_allocate(OBJT_DEVICE, cdev, objsize, prot, *foff, > + curthread->td_ucred); > + ND("returns obj %p", obj); > + if (obj == NULL) > + return EINVAL; > + if (saved_cdev_pager_ops.cdev_pg_fault == NULL) { > + ND("initialize cdev_pager_ops"); > + saved_cdev_pager_ops = *(obj->un_pager.devp.ops); > + netmap_cdev_pager_ops.cdev_pg_fault = > + saved_cdev_pager_ops.cdev_pg_fault; > + }; > + obj->un_pager.devp.ops = &netmap_cdev_pager_ops; > + *objp = obj; > + return 0; > +} > +#endif /* __FreeBSD__ */ > + > + > +/* > + * mmap(2) support for the "netmap" device. > + * > + * Expose all the memory previously allocated by our custom memory > + * allocator: this way the user has only to issue a single mmap(2), and > + * can work on all the data structures flawlessly. > + * > + * Return 0 on success, -1 otherwise. > + */ > + > +#ifdef __FreeBSD__ > +static int > +netmap_mmap(__unused struct cdev *dev, > +#if __FreeBSD_version < 900000 > + vm_offset_t offset, vm_paddr_t *paddr, int nprot > +#else > + vm_ooffset_t offset, vm_paddr_t *paddr, int nprot, > + __unused vm_memattr_t *memattr > +#endif > + ) > +{ > + int error = 0; > + struct netmap_priv_d *priv; > + > + if (nprot & PROT_EXEC) > + return (-1); // XXX -1 or EINVAL ? > + > + error = devfs_get_cdevpriv((void **)&priv); > + if (error == EBADF) { /* called on fault, memory is initialized */ > + ND(5, "handling fault at ofs 0x%x", offset); > + error = 0; > + } else if (error == 0) /* make sure memory is set */ > + error = netmap_get_memory(priv); > + if (error) > + return (error); > + > + ND("request for offset 0x%x", (uint32_t)offset); > + *paddr = netmap_ofstophys(offset); > + > + return (*paddr ? 0 : ENOMEM); > +} > + > +static int > +netmap_close(struct cdev *dev, int fflag, int devtype, struct thread *td) > +{ > + if (netmap_verbose) > + D("dev %p fflag 0x%x devtype %d td %p", > + dev, fflag, devtype, td); > + return 0; > +} > + > +static int > +netmap_open(struct cdev *dev, int oflags, int devtype, struct thread *td) > +{ > + struct netmap_priv_d *priv; > + int error; > + > + priv = malloc(sizeof(struct netmap_priv_d), M_DEVBUF, > + M_NOWAIT | M_ZERO); > + if (priv == NULL) > + return ENOMEM; > + > + error = devfs_set_cdevpriv(priv, netmap_dtor); > + if (error) > + return error; > + > + return 0; > +} > +#endif /* __FreeBSD__ */ > + > + > +/* > + * Handlers for synchronization of the queues from/to the host. > + * Netmap has two operating modes: > + * - in the default mode, the rings connected to the host stack are > + * just another ring pair managed by userspace; > + * - in transparent mode (XXX to be defined) incoming packets > + * (from the host or the NIC) are marked as NS_FORWARD upon > + * arrival, and the user application has a chance to reset the > + * flag for packets that should be dropped. > + * On the RXSYNC or poll(), packets in RX rings between > + * kring->nr_kcur and ring->cur with NS_FORWARD still set are moved > + * to the other side. > + * The transfer NIC --> host is relatively easy, just encapsulate > + * into mbufs and we are done. The host --> NIC side is slightly > + * harder because there might not be room in the tx ring so it > + * might take a while before releasing the buffer. > + */ > + > +/* > + * pass a chain of buffers to the host stack as coming from 'dst' > + */ > +static void > +netmap_send_up(struct ifnet *dst, struct mbuf *head) > +{ > + struct mbuf *m; > + > + /* send packets up, outside the lock */ > + while ((m = head) != NULL) { > + head = head->m_nextpkt; > + m->m_nextpkt = NULL; > + if (netmap_verbose & NM_VERB_HOST) > + D("sending up pkt %p size %d", m, MBUF_LEN(m)); > + NM_SEND_UP(dst, m); > + } > +} > + > +struct mbq { > + struct mbuf *head; > + struct mbuf *tail; > + int count; > +}; > + > +/* > + * put a copy of the buffers marked NS_FORWARD into an mbuf chain. > + * Run from hwcur to cur - reserved > + */ > +static void > +netmap_grab_packets(struct netmap_kring *kring, struct mbq *q, int force) > +{ > + /* Take packets from hwcur to cur-reserved and pass them up. > + * In case of no buffers we give up. At the end of the loop, > + * the queue is drained in all cases. > + * XXX handle reserved > + */ > + int k = kring->ring->cur - kring->ring->reserved; > + u_int n, lim = kring->nkr_num_slots - 1; > + struct mbuf *m, *tail = q->tail; > + > + if (k < 0) > + k = k + kring->nkr_num_slots; > + for (n = kring->nr_hwcur; n != k;) { > + struct netmap_slot *slot = &kring->ring->slot[n]; > + > + n = (n == lim) ? 0 : n + 1; > + if ((slot->flags & NS_FORWARD) == 0 && !force) > + continue; > + if (slot->len < 14 || slot->len > NETMAP_BUF_SIZE) { > + D("bad pkt at %d len %d", n, slot->len); > + continue; > + } > + slot->flags &= ~NS_FORWARD; // XXX needed ? > + m = m_devget(NMB(slot), slot->len, 0, kring->na->ifp, NULL); > + > + if (m == NULL) > + break; > + if (tail) > + tail->m_nextpkt = m; > + else > + q->head = m; > + tail = m; > + q->count++; > + m->m_nextpkt = NULL; > + } > + q->tail = tail; > +} > + > +/* > + * called under main lock to send packets from the host to the NIC > + * The host ring has packets from nr_hwcur to (cur - reserved) > + * to be sent down. We scan the tx rings, which have just been > + * flushed so nr_hwcur == cur. Pushing packets down means > + * increment cur and decrement avail. > + * XXX to be verified > + */ > +static void > +netmap_sw_to_nic(struct netmap_adapter *na) > +{ > + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings]; > + struct netmap_kring *k1 = &na->tx_rings[0]; > + int i, howmany, src_lim, dst_lim; > + > + howmany = kring->nr_hwavail; /* XXX otherwise cur - reserved - nr_hwcur */ > + > + src_lim = kring->nkr_num_slots; > + for (i = 0; howmany > 0 && i < na->num_tx_rings; i++, k1++) { > + ND("%d packets left to ring %d (space %d)", howmany, i, k1->nr_hwavail); > + dst_lim = k1->nkr_num_slots; > + while (howmany > 0 && k1->ring->avail > 0) { > + struct netmap_slot *src, *dst, tmp; > + src = &kring->ring->slot[kring->nr_hwcur]; > + dst = &k1->ring->slot[k1->ring->cur]; > + tmp = *src; > + src->buf_idx = dst->buf_idx; > + src->flags = NS_BUF_CHANGED; > + > + dst->buf_idx = tmp.buf_idx; > + dst->len = tmp.len; > + dst->flags = NS_BUF_CHANGED; > + ND("out len %d buf %d from %d to %d", > + dst->len, dst->buf_idx, > + kring->nr_hwcur, k1->ring->cur); > + > + if (++kring->nr_hwcur >= src_lim) > + kring->nr_hwcur = 0; > + howmany--; > + kring->nr_hwavail--; > + if (++k1->ring->cur >= dst_lim) > + k1->ring->cur = 0; > + k1->ring->avail--; > + } > + kring->ring->cur = kring->nr_hwcur; // XXX > + k1++; > + } > +} > + > +/* > + * netmap_sync_to_host() passes packets up. We are called from a > + * system call in user process context, and the only contention > + * can be among multiple user threads erroneously calling > + * this routine concurrently. > + */ > +static void > +netmap_sync_to_host(struct netmap_adapter *na) > +{ > + struct netmap_kring *kring = &na->tx_rings[na->num_tx_rings]; > + struct netmap_ring *ring = kring->ring; > + u_int k, lim = kring->nkr_num_slots - 1; > + struct mbq q = { NULL, NULL }; > + > + k = ring->cur; > + if (k > lim) { > + netmap_ring_reinit(kring); > + return; > + } > + // na->nm_lock(na->ifp, NETMAP_CORE_LOCK, 0); > + > + /* Take packets from hwcur to cur and pass them up. > + * In case of no buffers we give up. At the end of the loop, > + * the queue is drained in all cases. > + */ > + netmap_grab_packets(kring, &q, 1); > + kring->nr_hwcur = k; > + kring->nr_hwavail = ring->avail = lim; > + // na->nm_lock(na->ifp, NETMAP_CORE_UNLOCK, 0); > + > + netmap_send_up(na->ifp, q.head); > +} > + > +/* > + * rxsync backend for packets coming from the host stack. > + * They have been put in the queue by netmap_start() so we > + * need to protect access to the kring using a lock. > + * > + * This routine also does the selrecord if called from the poll handler > + * (we know because td != NULL). > + * > + * NOTE: on linux, selrecord() is defined as a macro and uses pwait > + * as an additional hidden argument. > + */ > +static void > +netmap_sync_from_host(struct netmap_adapter *na, struct thread *td, void *pwait) > +{ > + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings]; > + struct netmap_ring *ring = kring->ring; > + u_int j, n, lim = kring->nkr_num_slots; > + u_int k = ring->cur, resvd = ring->reserved; > + > + (void)pwait; /* disable unused warnings */ > + na->nm_lock(na->ifp, NETMAP_CORE_LOCK, 0); > + if (k >= lim) { > + netmap_ring_reinit(kring); > + return; > + } > + /* new packets are already set in nr_hwavail */ > + /* skip past packets that userspace has released */ > + j = kring->nr_hwcur; > + if (resvd > 0) { > + if (resvd + ring->avail >= lim + 1) { > + D("XXX invalid reserve/avail %d %d", resvd, ring->avail); > + ring->reserved = resvd = 0; // XXX panic... > + } > + k = (k >= resvd) ? k - resvd : k + lim - resvd; > + } > + if (j != k) { > + n = k >= j ? k - j : k + lim - j; > + kring->nr_hwavail -= n; > + kring->nr_hwcur = k; > + } > + k = ring->avail = kring->nr_hwavail - resvd; > + if (k == 0 && td) > + selrecord(td, &kring->si); > + if (k && (netmap_verbose & NM_VERB_HOST)) > + D("%d pkts from stack", k); > + na->nm_lock(na->ifp, NETMAP_CORE_UNLOCK, 0); > +} > + > + > +/* > + * get a refcounted reference to an interface. > + * Return ENXIO if the interface does not exist, EINVAL if netmap > + * is not supported by the interface. > + * If successful, hold a reference. > + */ > +static int > +get_ifp(const char *name, struct ifnet **ifp) > +{ > +#ifdef NM_BRIDGE > + struct ifnet *iter = NULL; > + > + do { > + struct nm_bridge *b; > + int i, l, cand = -1; > + > + if (strncmp(name, NM_NAME, sizeof(NM_NAME) - 1)) > + break; > + b = nm_find_bridge(name); > + if (b == NULL) { > + D("no bridges available for '%s'", name); > + return (ENXIO); > + } > + /* XXX locking */ > + BDG_LOCK(b); > + /* lookup in the local list of ports */ > + for (i = 0; i < NM_BDG_MAXPORTS; i++) { > + iter = b->bdg_ports[i]; > + if (iter == NULL) { > + if (cand == -1) > + cand = i; /* potential insert point */ > + continue; > + } > + if (!strcmp(iter->if_xname, name)) { > + ADD_BDG_REF(iter); > + ND("found existing interface"); > + BDG_UNLOCK(b); > + break; > + } > + } > + if (i < NM_BDG_MAXPORTS) /* already unlocked */ > + break; > + if (cand == -1) { > + D("bridge full, cannot create new port"); > +no_port: > + BDG_UNLOCK(b); > + *ifp = NULL; > + return EINVAL; > + } > + ND("create new bridge port %s", name); > + /* space for forwarding list after the ifnet */ > + l = sizeof(*iter) + > + sizeof(struct nm_bdg_fwd)*NM_BDG_BATCH ; > + iter = malloc(l, M_DEVBUF, M_NOWAIT | M_ZERO); > + if (!iter) > + goto no_port; > + strcpy(iter->if_xname, name); > + bdg_netmap_attach(iter); > + b->bdg_ports[cand] = iter; > + iter->if_bridge = b; > + ADD_BDG_REF(iter); > + BDG_UNLOCK(b); > + ND("attaching virtual bridge %p", b); > + } while (0); > + *ifp = iter; > + if (! *ifp) > +#endif /* NM_BRIDGE */ > + *ifp = ifunit_ref(name); > + if (*ifp == NULL) > + return (ENXIO); > + /* can do this if the capability exists and if_pspare[0] > + * points to the netmap descriptor. > + */ > + if (NETMAP_CAPABLE(*ifp)) > + return 0; /* valid pointer, we hold the refcount */ > + nm_if_rele(*ifp); > + return EINVAL; // not NETMAP capable > +} > + > + > +/* > + * Error routine called when txsync/rxsync detects an error. > + * Can't do much more than resetting cur = hwcur, avail = hwavail. > + * Return 1 on reinit. > + * > + * This routine is only called by the upper half of the kernel. > + * It only reads hwcur (which is changed only by the upper half, too) > + * and hwavail (which may be changed by the lower half, but only on > + * a tx ring and only to increase it, so any error will be recovered > + * on the next call). For the above, we don't strictly need to call > + * it under lock. > + */ > +int > +netmap_ring_reinit(struct netmap_kring *kring) > +{ > + struct netmap_ring *ring = kring->ring; > + u_int i, lim = kring->nkr_num_slots - 1; > + int errors = 0; > + > + RD(10, "called for %s", kring->na->ifp->if_xname); > + if (ring->cur > lim) > + errors++; > + for (i = 0; i <= lim; i++) { > + u_int idx = ring->slot[i].buf_idx; > + u_int len = ring->slot[i].len; > + if (idx < 2 || idx >= netmap_total_buffers) { > + if (!errors++) > + D("bad buffer at slot %d idx %d len %d ", i, idx, len); > + ring->slot[i].buf_idx = 0; > + ring->slot[i].len = 0; > + } else if (len > NETMAP_BUF_SIZE) { > + ring->slot[i].len = 0; > + if (!errors++) > + D("bad len %d at slot %d idx %d", > + len, i, idx); > + } > + } > + if (errors) { > + int pos = kring - kring->na->tx_rings; > + int n = kring->na->num_tx_rings + 1; > + > + RD(10, "total %d errors", errors); > + errors++; > + RD(10, "%s %s[%d] reinit, cur %d -> %d avail %d -> %d", > + kring->na->ifp->if_xname, > + pos < n ? "TX" : "RX", pos < n ? pos : pos - n, > + ring->cur, kring->nr_hwcur, > + ring->avail, kring->nr_hwavail); > + ring->cur = kring->nr_hwcur; > + ring->avail = kring->nr_hwavail; > + } > + return (errors ? 1 : 0); > +} > + > + > +/* > + * Set the ring ID. For devices with a single queue, a request > + * for all rings is the same as a single ring. > + */ > +static int > +netmap_set_ringid(struct netmap_priv_d *priv, u_int ringid) > +{ > + struct ifnet *ifp = priv->np_ifp; > + struct netmap_adapter *na = NA(ifp); > + u_int i = ringid & NETMAP_RING_MASK; > + /* initially (np_qfirst == np_qlast) we don't want to lock */ > + int need_lock = (priv->np_qfirst != priv->np_qlast); > + int lim = na->num_rx_rings; > + > + if (na->num_tx_rings > lim) > + lim = na->num_tx_rings; > + if ( (ringid & NETMAP_HW_RING) && i >= lim) { > + D("invalid ring id %d", i); > + return (EINVAL); > + } > + if (need_lock) > + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); > + priv->np_ringid = ringid; > + if (ringid & NETMAP_SW_RING) { > + priv->np_qfirst = NETMAP_SW_RING; > + priv->np_qlast = 0; > + } else if (ringid & NETMAP_HW_RING) { > + priv->np_qfirst = i; > + priv->np_qlast = i + 1; > + } else { > + priv->np_qfirst = 0; > + priv->np_qlast = NETMAP_HW_RING ; > + } > + priv->np_txpoll = (ringid & NETMAP_NO_TX_POLL) ? 0 : 1; > + if (need_lock) > + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0); > + if (netmap_verbose) { > + if (ringid & NETMAP_SW_RING) > + D("ringid %s set to SW RING", ifp->if_xname); > + else if (ringid & NETMAP_HW_RING) > + D("ringid %s set to HW RING %d", ifp->if_xname, > + priv->np_qfirst); > + else > + D("ringid %s set to all %d HW RINGS", ifp->if_xname, lim); > + } > + return 0; > +} > + > +/* > + * ioctl(2) support for the "netmap" device. > + * > + * Following a list of accepted commands: > + * - NIOCGINFO > + * - SIOCGIFADDR just for convenience > + * - NIOCREGIF > + * - NIOCUNREGIF > + * - NIOCTXSYNC > + * - NIOCRXSYNC > + * > + * Return 0 on success, errno otherwise. > + */ > +static int > +netmap_ioctl(struct cdev *dev, u_long cmd, caddr_t data, > + int fflag, struct thread *td) > +{ > + struct netmap_priv_d *priv = NULL; > + struct ifnet *ifp; > + struct nmreq *nmr = (struct nmreq *) data; > + struct netmap_adapter *na; > + int error; > + u_int i, lim; > + struct netmap_if *nifp; > + > + (void)dev; /* UNUSED */ > + (void)fflag; /* UNUSED */ > +#ifdef linux > +#define devfs_get_cdevpriv(pp) \ > + ({ *(struct netmap_priv_d **)pp = ((struct file *)td)->private_data; \ > + (*pp ? 0 : ENOENT); }) > + > +/* devfs_set_cdevpriv cannot fail on linux */ > +#define devfs_set_cdevpriv(p, fn) \ > + ({ ((struct file *)td)->private_data = p; (p ? 0 : EINVAL); }) > + > + > +#define devfs_clear_cdevpriv() do { \ > + netmap_dtor(priv); ((struct file *)td)->private_data = 0; \ > + } while (0) > +#endif /* linux */ > + > + CURVNET_SET(TD_TO_VNET(td)); > + > + error = devfs_get_cdevpriv((void **)&priv); > + if (error) { > + CURVNET_RESTORE(); > + /* XXX ENOENT should be impossible, since the priv > + * is now created in the open */ > + return (error == ENOENT ? ENXIO : error); > + } > + > + nmr->nr_name[sizeof(nmr->nr_name) - 1] = '\0'; /* truncate name */ > + switch (cmd) { > + case NIOCGINFO: /* return capabilities etc */ > + if (nmr->nr_version != NETMAP_API) { > + D("API mismatch got %d have %d", > + nmr->nr_version, NETMAP_API); > + nmr->nr_version = NETMAP_API; > + error = EINVAL; > + break; > + } > + /* update configuration */ > + error = netmap_get_memory(priv); > + ND("get_memory returned %d", error); > + if (error) > + break; > + /* memsize is always valid */ > + nmr->nr_memsize = nm_mem.nm_totalsize; > + nmr->nr_offset = 0; > + nmr->nr_rx_rings = nmr->nr_tx_rings = 0; > + nmr->nr_rx_slots = nmr->nr_tx_slots = 0; > + if (nmr->nr_name[0] == '\0') /* just get memory info */ > + break; > + error = get_ifp(nmr->nr_name, &ifp); /* get a refcount */ > + if (error) > + break; > + na = NA(ifp); /* retrieve netmap_adapter */ > + netmap_update_config(na); > + nmr->nr_rx_rings = na->num_rx_rings; > + nmr->nr_tx_rings = na->num_tx_rings; > + nmr->nr_rx_slots = na->num_rx_desc; > + nmr->nr_tx_slots = na->num_tx_desc; > + nm_if_rele(ifp); /* return the refcount */ > + break; > + > + case NIOCREGIF: > + if (nmr->nr_version != NETMAP_API) { > + nmr->nr_version = NETMAP_API; > + error = EINVAL; > + break; > + } > + /* ensure allocators are ready */ > + error = netmap_get_memory(priv); > + ND("get_memory returned %d", error); > + if (error) > + break; > + > + /* protect access to priv from concurrent NIOCREGIF */ > + NMA_LOCK(); > + if (priv->np_ifp != NULL) { /* thread already registered */ > + error = netmap_set_ringid(priv, nmr->nr_ringid); > + NMA_UNLOCK(); > + break; > + } > + /* find the interface and a reference */ > + error = get_ifp(nmr->nr_name, &ifp); /* keep reference */ > + if (error) { > + NMA_UNLOCK(); > + break; > + } > + na = NA(ifp); /* retrieve netmap adapter */ > + > + for (i = 10; i > 0; i--) { > + na->nm_lock(ifp, NETMAP_REG_LOCK, 0); > + if (!NETMAP_DELETING(na)) > + break; > + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); > + tsleep(na, 0, "NIOCREGIF", hz/10); > + } > + if (i == 0) { > + D("too many NIOCREGIF attempts, give up"); > + error = EINVAL; > + nm_if_rele(ifp); /* return the refcount */ > + NMA_UNLOCK(); > + break; > + } > + > + /* ring configuration may have changed, fetch from the card */ > + netmap_update_config(na); > + priv->np_ifp = ifp; /* store the reference */ > + error = netmap_set_ringid(priv, nmr->nr_ringid); > + if (error) > + goto error; > + nifp = netmap_if_new(nmr->nr_name, na); > + if (nifp == NULL) { /* allocation failed */ > + error = ENOMEM; > + } else if (ifp->if_capenable & IFCAP_NETMAP) { > + /* was already set */ > + } else { > + /* Otherwise set the card in netmap mode > + * and make it use the shared buffers. > + */ > + for (i = 0 ; i < na->num_tx_rings + 1; i++) > + mtx_init(&na->tx_rings[i].q_lock, "nm_txq_lock", MTX_NETWORK_LOCK, MTX_DEF); > + for (i = 0 ; i < na->num_rx_rings + 1; i++) { > + mtx_init(&na->rx_rings[i].q_lock, "nm_rxq_lock", MTX_NETWORK_LOCK, MTX_DEF); > + } > + error = na->nm_register(ifp, 1); /* mode on */ > + if (error) { > + netmap_dtor_locked(priv); > + netmap_if_free(nifp); > + } > + } > + > + if (error) { /* reg. failed, release priv and ref */ > +error: > + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); > + nm_if_rele(ifp); /* return the refcount */ > + priv->np_ifp = NULL; > + priv->np_nifp = NULL; > + NMA_UNLOCK(); > + break; > + } > + > + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); > + > + /* the following assignment is a commitment. > + * Readers (i.e., poll and *SYNC) check for > + * np_nifp != NULL without locking > + */ > + wmb(); /* make sure previous writes are visible to all CPUs */ > + priv->np_nifp = nifp; > + NMA_UNLOCK(); > + > + /* return the offset of the netmap_if object */ > + nmr->nr_rx_rings = na->num_rx_rings; > + nmr->nr_tx_rings = na->num_tx_rings; > + nmr->nr_rx_slots = na->num_rx_desc; > + nmr->nr_tx_slots = na->num_tx_desc; > + nmr->nr_memsize = nm_mem.nm_totalsize; > + nmr->nr_offset = netmap_if_offset(nifp); > + break; > + > + case NIOCUNREGIF: > + // XXX we have no data here ? > + D("deprecated, data is %p", nmr); > + error = EINVAL; > + break; > + > + case NIOCTXSYNC: > + case NIOCRXSYNC: > + nifp = priv->np_nifp; > + > + if (nifp == NULL) { > + error = ENXIO; > + break; > + } > + rmb(); /* make sure following reads are not from cache */ > + > + > + ifp = priv->np_ifp; /* we have a reference */ > + > + if (ifp == NULL) { > + D("Internal error: nifp != NULL && ifp == NULL"); > + error = ENXIO; > + break; > + } > + > + na = NA(ifp); /* retrieve netmap adapter */ > + if (priv->np_qfirst == NETMAP_SW_RING) { /* host rings */ > + if (cmd == NIOCTXSYNC) > + netmap_sync_to_host(na); > + else > + netmap_sync_from_host(na, NULL, NULL); > + break; > + } > + /* find the last ring to scan */ > + lim = priv->np_qlast; > + if (lim == NETMAP_HW_RING) > + lim = (cmd == NIOCTXSYNC) ? > + na->num_tx_rings : na->num_rx_rings; > + > + for (i = priv->np_qfirst; i < lim; i++) { > + if (cmd == NIOCTXSYNC) { > + struct netmap_kring *kring = &na->tx_rings[i]; > + if (netmap_verbose & NM_VERB_TXSYNC) > + D("pre txsync ring %d cur %d hwcur %d", > + i, kring->ring->cur, > + kring->nr_hwcur); > + na->nm_txsync(ifp, i, 1 /* do lock */); > + if (netmap_verbose & NM_VERB_TXSYNC) > + D("post txsync ring %d cur %d hwcur %d", > + i, kring->ring->cur, > + kring->nr_hwcur); > + } else { > + na->nm_rxsync(ifp, i, 1 /* do lock */); > + microtime(&na->rx_rings[i].ring->ts); > + } > + } > + > + break; > + > +#ifdef __FreeBSD__ > + case BIOCIMMEDIATE: > + case BIOCGHDRCMPLT: > + case BIOCSHDRCMPLT: > + case BIOCSSEESENT: > + D("ignore BIOCIMMEDIATE/BIOCSHDRCMPLT/BIOCSHDRCMPLT/BIOCSSEESENT"); > + break; > + > + default: /* allow device-specific ioctls */ > + { > + struct socket so; > + bzero(&so, sizeof(so)); > + error = get_ifp(nmr->nr_name, &ifp); /* keep reference */ > + if (error) > + break; > + so.so_vnet = ifp->if_vnet; > + // so->so_proto not null. > + error = ifioctl(&so, cmd, data, td); > + nm_if_rele(ifp); > + break; > + } > + > +#else /* linux */ > + default: > + error = EOPNOTSUPP; > +#endif /* linux */ > + } > + > + CURVNET_RESTORE(); > + return (error); > +} > + > + > +/* > + * select(2) and poll(2) handlers for the "netmap" device. > + * > + * Can be called for one or more queues. > + * Return true the event mask corresponding to ready events. > + * If there are no ready events, do a selrecord on either individual > + * selfd or on the global one. > + * Device-dependent parts (locking and sync of tx/rx rings) > + * are done through callbacks. > + * > + * On linux, arguments are really pwait, the poll table, and 'td' is struct file * > + * The first one is remapped to pwait as selrecord() uses the name as an > + * hidden argument. > + */ > +static int > +netmap_poll(struct cdev *dev, int events, struct thread *td) > +{ > + struct netmap_priv_d *priv = NULL; > + struct netmap_adapter *na; > + struct ifnet *ifp; > + struct netmap_kring *kring; > + u_int core_lock, i, check_all, want_tx, want_rx, revents = 0; > + u_int lim_tx, lim_rx, host_forwarded = 0; > + struct mbq q = { NULL, NULL, 0 }; > + enum {NO_CL, NEED_CL, LOCKED_CL }; /* see below */ > + void *pwait = dev; /* linux compatibility */ > + > + (void)pwait; > + > + if (devfs_get_cdevpriv((void **)&priv) != 0 || priv == NULL) > + return POLLERR; > + > + if (priv->np_nifp == NULL) { > + D("No if registered"); > + return POLLERR; > + } > + rmb(); /* make sure following reads are not from cache */ > + > + ifp = priv->np_ifp; > + // XXX check for deleting() ? > + if ( (ifp->if_capenable & IFCAP_NETMAP) == 0) > + return POLLERR; > + > + if (netmap_verbose & 0x8000) > + D("device %s events 0x%x", ifp->if_xname, events); > + want_tx = events & (POLLOUT | POLLWRNORM); > + want_rx = events & (POLLIN | POLLRDNORM); > + > + na = NA(ifp); /* retrieve netmap adapter */ > + > + lim_tx = na->num_tx_rings; > + lim_rx = na->num_rx_rings; > + /* how many queues we are scanning */ > + if (priv->np_qfirst == NETMAP_SW_RING) { > + if (priv->np_txpoll || want_tx) { > + /* push any packets up, then we are always ready */ > + kring = &na->tx_rings[lim_tx]; > + netmap_sync_to_host(na); > + revents |= want_tx; > + } > + if (want_rx) { > + kring = &na->rx_rings[lim_rx]; > + if (kring->ring->avail == 0) > + netmap_sync_from_host(na, td, dev); > + if (kring->ring->avail > 0) { > + revents |= want_rx; > + } > + } > + return (revents); > + } > + > + /* if we are in transparent mode, check also the host rx ring */ > + kring = &na->rx_rings[lim_rx]; > + if ( (priv->np_qlast == NETMAP_HW_RING) // XXX check_all > + && want_rx > + && (netmap_fwd || kring->ring->flags & NR_FORWARD) ) { > + if (kring->ring->avail == 0) > + netmap_sync_from_host(na, td, dev); > + if (kring->ring->avail > 0) > + revents |= want_rx; > + } > + > + /* > + * check_all is set if the card has more than one queue and > + * the client is polling all of them. If true, we sleep on > + * the "global" selfd, otherwise we sleep on individual selfd > + * (we can only sleep on one of them per direction). > + * The interrupt routine in the driver should always wake on > + * the individual selfd, and also on the global one if the card > + * has more than one ring. > + * > + * If the card has only one lock, we just use that. > + * If the card has separate ring locks, we just use those > + * unless we are doing check_all, in which case the whole > + * loop is wrapped by the global lock. > + * We acquire locks only when necessary: if poll is called > + * when buffers are available, we can just return without locks. > + * > + * rxsync() is only called if we run out of buffers on a POLLIN. > + * txsync() is called if we run out of buffers on POLLOUT, or > + * there are pending packets to send. The latter can be disabled > + * passing NETMAP_NO_TX_POLL in the NIOCREG call. > + */ > + check_all = (priv->np_qlast == NETMAP_HW_RING) && (lim_tx > 1 || lim_rx > 1); > + > + /* > + * core_lock indicates what to do with the core lock. > + * The core lock is used when either the card has no individual > + * locks, or it has individual locks but we are cheking all > + * rings so we need the core lock to avoid missing wakeup events. > + * > + * It has three possible states: > + * NO_CL we don't need to use the core lock, e.g. > + * because we are protected by individual locks. > + * NEED_CL we need the core lock. In this case, when we > + * call the lock routine, move to LOCKED_CL > + * to remember to release the lock once done. > + * LOCKED_CL core lock is set, so we need to release it. > + */ > + core_lock = (check_all || !na->separate_locks) ? NEED_CL : NO_CL; > +#ifdef NM_BRIDGE > + /* the bridge uses separate locks */ > + if (na->nm_register == bdg_netmap_reg) { > + ND("not using core lock for %s", ifp->if_xname); > + core_lock = NO_CL; > + } > +#endif /* NM_BRIDGE */ > + if (priv->np_qlast != NETMAP_HW_RING) { > + lim_tx = lim_rx = priv->np_qlast; > + } > + > + /* > + * We start with a lock free round which is good if we have > + * data available. If this fails, then lock and call the sync > + * routines. > + */ > + for (i = priv->np_qfirst; want_rx && i < lim_rx; i++) { > + kring = &na->rx_rings[i]; > + if (kring->ring->avail > 0) { > + revents |= want_rx; > + want_rx = 0; /* also breaks the loop */ > + } > + } > + for (i = priv->np_qfirst; want_tx && i < lim_tx; i++) { > + kring = &na->tx_rings[i]; > + if (kring->ring->avail > 0) { > + revents |= want_tx; > + want_tx = 0; /* also breaks the loop */ > + } > + } > + > + /* > + * If we to push packets out (priv->np_txpoll) or want_tx is > + * still set, we do need to run the txsync calls (on all rings, > + * to avoid that the tx rings stall). > + */ > + if (priv->np_txpoll || want_tx) { > +flush_tx: > + for (i = priv->np_qfirst; i < lim_tx; i++) { > + kring = &na->tx_rings[i]; > + /* > + * Skip the current ring if want_tx == 0 > + * (we have already done a successful sync on > + * a previous ring) AND kring->cur == kring->hwcur > + * (there are no pending transmissions for this ring). > + */ > + if (!want_tx && kring->ring->cur == kring->nr_hwcur) > + continue; > + if (core_lock == NEED_CL) { > + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); > + core_lock = LOCKED_CL; > + } > + if (na->separate_locks) > + na->nm_lock(ifp, NETMAP_TX_LOCK, i); > + if (netmap_verbose & NM_VERB_TXSYNC) > + D("send %d on %s %d", > + kring->ring->cur, > + ifp->if_xname, i); > + if (na->nm_txsync(ifp, i, 0 /* no lock */)) > + revents |= POLLERR; > + > + /* Check avail/call selrecord only if called with POLLOUT */ > + if (want_tx) { > + if (kring->ring->avail > 0) { > + /* stop at the first ring. We don't risk > + * starvation. > + */ > + revents |= want_tx; > + want_tx = 0; > + } else if (!check_all) > + selrecord(td, &kring->si); > + } > + if (na->separate_locks) > + na->nm_lock(ifp, NETMAP_TX_UNLOCK, i); > + } > + } > + > + /* > + * now if want_rx is still set we need to lock and rxsync. > + * Do it on all rings because otherwise we starve. > + */ > + if (want_rx) { > + for (i = priv->np_qfirst; i < lim_rx; i++) { > + kring = &na->rx_rings[i]; > + if (core_lock == NEED_CL) { > + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); > + core_lock = LOCKED_CL; > + } > + if (na->separate_locks) > + na->nm_lock(ifp, NETMAP_RX_LOCK, i); > + if (netmap_fwd ||kring->ring->flags & NR_FORWARD) { > + ND(10, "forwarding some buffers up %d to %d", > + kring->nr_hwcur, kring->ring->cur); > + netmap_grab_packets(kring, &q, netmap_fwd); > + } > + > + if (na->nm_rxsync(ifp, i, 0 /* no lock */)) > + revents |= POLLERR; > + if (netmap_no_timestamp == 0 || > + kring->ring->flags & NR_TIMESTAMP) { > + microtime(&kring->ring->ts); > + } > + > + if (kring->ring->avail > 0) > + revents |= want_rx; > + else if (!check_all) > + selrecord(td, &kring->si); > + if (na->separate_locks) > + na->nm_lock(ifp, NETMAP_RX_UNLOCK, i); > + } > + } > + if (check_all && revents == 0) { /* signal on the global queue */ > + if (want_tx) > + selrecord(td, &na->tx_si); > + if (want_rx) > + selrecord(td, &na->rx_si); > + } > + > + /* forward host to the netmap ring */ > + kring = &na->rx_rings[lim_rx]; > + if (kring->nr_hwavail > 0) > + ND("host rx %d has %d packets", lim_rx, kring->nr_hwavail); > + if ( (priv->np_qlast == NETMAP_HW_RING) // XXX check_all > + && (netmap_fwd || kring->ring->flags & NR_FORWARD) > + && kring->nr_hwavail > 0 && !host_forwarded) { > + if (core_lock == NEED_CL) { > + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); > + core_lock = LOCKED_CL; > + } > + netmap_sw_to_nic(na); > + host_forwarded = 1; /* prevent another pass */ > + want_rx = 0; > + goto flush_tx; > + } > + > + if (core_lock == LOCKED_CL) > + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0); > + if (q.head) > + netmap_send_up(na->ifp, q.head); > + > + return (revents); > +} > + > +/*------- driver support routines ------*/ > + > +/* > + * default lock wrapper. > + */ > +static void > +netmap_lock_wrapper(struct ifnet *dev, int what, u_int queueid) > +{ > + struct netmap_adapter *na = NA(dev); > + > + switch (what) { > +#ifdef linux /* some system do not need lock on register */ > + case NETMAP_REG_LOCK: > + case NETMAP_REG_UNLOCK: > + break; > +#endif /* linux */ > + > + case NETMAP_CORE_LOCK: > + mtx_lock(&na->core_lock); > + break; > + > + case NETMAP_CORE_UNLOCK: > + mtx_unlock(&na->core_lock); > + break; > + > + case NETMAP_TX_LOCK: > + mtx_lock(&na->tx_rings[queueid].q_lock); > + break; > + > + case NETMAP_TX_UNLOCK: > + mtx_unlock(&na->tx_rings[queueid].q_lock); > + break; > + > + case NETMAP_RX_LOCK: > + mtx_lock(&na->rx_rings[queueid].q_lock); > + break; > + > + case NETMAP_RX_UNLOCK: > + mtx_unlock(&na->rx_rings[queueid].q_lock); > + break; > + } > +} > + > + > +/* > + * Initialize a ``netmap_adapter`` object created by driver on attach. > + * We allocate a block of memory with room for a struct netmap_adapter > + * plus two sets of N+2 struct netmap_kring (where N is the number > + * of hardware rings): > + * krings 0..N-1 are for the hardware queues. > + * kring N is for the host stack queue > + * kring N+1 is only used for the selinfo for all queues. > + * Return 0 on success, ENOMEM otherwise. > + * > + * By default the receive and transmit adapter ring counts are both initialized > + * to num_queues. na->num_tx_rings can be set for cards with different tx/rx > + * setups. > + */ > +int > +netmap_attach(struct netmap_adapter *arg, int num_queues) > +{ > + struct netmap_adapter *na = NULL; > + struct ifnet *ifp = arg ? arg->ifp : NULL; > + > + if (arg == NULL || ifp == NULL) > + goto fail; > + na = malloc(sizeof(*na), M_DEVBUF, M_NOWAIT | M_ZERO); > + if (na == NULL) > + goto fail; > + WNA(ifp) = na; > + *na = *arg; /* copy everything, trust the driver to not pass junk */ > + NETMAP_SET_CAPABLE(ifp); > + if (na->num_tx_rings == 0) > + na->num_tx_rings = num_queues; > + na->num_rx_rings = num_queues; > + na->refcount = na->na_single = na->na_multi = 0; > + /* Core lock initialized here, others after netmap_if_new. */ > + mtx_init(&na->core_lock, "netmap core lock", MTX_NETWORK_LOCK, MTX_DEF); > + if (na->nm_lock == NULL) { > + ND("using default locks for %s", ifp->if_xname); > + na->nm_lock = netmap_lock_wrapper; > + } > +#ifdef linux > + if (ifp->netdev_ops) { > + ND("netdev_ops %p", ifp->netdev_ops); > + /* prepare a clone of the netdev ops */ > + na->nm_ndo = *ifp->netdev_ops; > + } > + na->nm_ndo.ndo_start_xmit = linux_netmap_start; > +#endif > + D("success for %s", ifp->if_xname); > + return 0; > + > +fail: > + D("fail, arg %p ifp %p na %p", arg, ifp, na); > + return (na ? EINVAL : ENOMEM); > +} > + > + > +/* > + * Free the allocated memory linked to the given ``netmap_adapter`` > + * object. > + */ > +void > +netmap_detach(struct ifnet *ifp) > +{ > + struct netmap_adapter *na = NA(ifp); > + > + if (!na) > + return; > + > + mtx_destroy(&na->core_lock); > + > + if (na->tx_rings) { /* XXX should not happen */ > + D("freeing leftover tx_rings"); > + free(na->tx_rings, M_DEVBUF); > + } > + bzero(na, sizeof(*na)); > + WNA(ifp) = NULL; > + free(na, M_DEVBUF); > +} > + > + > +/* > + * Intercept packets from the network stack and pass them > + * to netmap as incoming packets on the 'software' ring. > + * We are not locked when called. > + */ > +int > +netmap_start(struct ifnet *ifp, struct mbuf *m) > +{ > + struct netmap_adapter *na = NA(ifp); > + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings]; > + u_int i, len = MBUF_LEN(m); > + u_int error = EBUSY, lim = kring->nkr_num_slots - 1; > + struct netmap_slot *slot; > + > + if (netmap_verbose & NM_VERB_HOST) > + D("%s packet %d len %d from the stack", ifp->if_xname, > + kring->nr_hwcur + kring->nr_hwavail, len); > + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); > + if (kring->nr_hwavail >= lim) { > + if (netmap_verbose) > + D("stack ring %s full\n", ifp->if_xname); > + goto done; /* no space */ > + } > + if (len > NETMAP_BUF_SIZE) { > + D("%s from_host, drop packet size %d > %d", ifp->if_xname, > + len, NETMAP_BUF_SIZE); > + goto done; /* too long for us */ > + } > + > + /* compute the insert position */ > + i = kring->nr_hwcur + kring->nr_hwavail; > + if (i > lim) > + i -= lim + 1; > + slot = &kring->ring->slot[i]; > + m_copydata(m, 0, len, NMB(slot)); > + slot->len = len; > + slot->flags = kring->nkr_slot_flags; > + kring->nr_hwavail++; > + if (netmap_verbose & NM_VERB_HOST) > + D("wake up host ring %s %d", na->ifp->if_xname, na->num_rx_rings); > + selwakeuppri(&kring->si, PI_NET); > + error = 0; > +done: > + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0); > + > + /* release the mbuf in either cases of success or failure. As an > + * alternative, put the mbuf in a free list and free the list > + * only when really necessary. > + */ > + m_freem(m); > + > + return (error); > +} > + > + > +/* > + * netmap_reset() is called by the driver routines when reinitializing > + * a ring. The driver is in charge of locking to protect the kring. > + * If netmap mode is not set just return NULL. > + */ > +struct netmap_slot * > +netmap_reset(struct netmap_adapter *na, enum txrx tx, int n, > + u_int new_cur) > +{ > + struct netmap_kring *kring; > + int new_hwofs, lim; > + > + if (na == NULL) > + return NULL; /* no netmap support here */ > + if (!(na->ifp->if_capenable & IFCAP_NETMAP)) > + return NULL; /* nothing to reinitialize */ > + > + if (tx == NR_TX) { > + if (n >= na->num_tx_rings) > + return NULL; > + kring = na->tx_rings + n; > + new_hwofs = kring->nr_hwcur - new_cur; > + } else { > + if (n >= na->num_rx_rings) > + return NULL; > + kring = na->rx_rings + n; > + new_hwofs = kring->nr_hwcur + kring->nr_hwavail - new_cur; > + } > + lim = kring->nkr_num_slots - 1; > + if (new_hwofs > lim) > + new_hwofs -= lim + 1; > + > + /* Alwayws set the new offset value and realign the ring. */ > + kring->nkr_hwofs = new_hwofs; > + if (tx == NR_TX) > + kring->nr_hwavail = kring->nkr_num_slots - 1; > + ND(10, "new hwofs %d on %s %s[%d]", > + kring->nkr_hwofs, na->ifp->if_xname, > + tx == NR_TX ? "TX" : "RX", n); > + > +#if 0 // def linux > + /* XXX check that the mappings are correct */ > + /* need ring_nr, adapter->pdev, direction */ > + buffer_info->dma = dma_map_single(&pdev->dev, addr, adapter->rx_buffer_len, DMA_FROM_DEVICE); > + if (dma_mapping_error(&adapter->pdev->dev, buffer_info->dma)) { > + D("error mapping rx netmap buffer %d", i); > + // XXX fix error handling > + } > + > +#endif /* linux */ > + /* > + * Wakeup on the individual and global lock > + * We do the wakeup here, but the ring is not yet reconfigured. > + * However, we are under lock so there are no races. > + */ > + selwakeuppri(&kring->si, PI_NET); > + selwakeuppri(tx == NR_TX ? &na->tx_si : &na->rx_si, PI_NET); > + return kring->ring->slot; > +} > + > + > +/* > + * Default functions to handle rx/tx interrupts > + * we have 4 cases: > + * 1 ring, single lock: > + * lock(core); wake(i=0); unlock(core) > + * N rings, single lock: > + * lock(core); wake(i); wake(N+1) unlock(core) > + * 1 ring, separate locks: (i=0) > + * lock(i); wake(i); unlock(i) > + * N rings, separate locks: > + * lock(i); wake(i); unlock(i); lock(core) wake(N+1) unlock(core) > + * work_done is non-null on the RX path. > + */ > +int > +netmap_rx_irq(struct ifnet *ifp, int q, int *work_done) > +{ > + struct netmap_adapter *na; > + struct netmap_kring *r; > + NM_SELINFO_T *main_wq; > + > + if (!(ifp->if_capenable & IFCAP_NETMAP)) > + return 0; > + ND(5, "received %s queue %d", work_done ? "RX" : "TX" , q); > + na = NA(ifp); > + if (na->na_flags & NAF_SKIP_INTR) { > + ND("use regular interrupt"); > + return 0; > + } > + > + if (work_done) { /* RX path */ > + if (q >= na->num_rx_rings) > + return 0; // regular queue > + r = na->rx_rings + q; > + r->nr_kflags |= NKR_PENDINTR; > + main_wq = (na->num_rx_rings > 1) ? &na->rx_si : NULL; > + } else { /* tx path */ > + if (q >= na->num_tx_rings) > + return 0; // regular queue > + r = na->tx_rings + q; > + main_wq = (na->num_tx_rings > 1) ? &na->tx_si : NULL; > + work_done = &q; /* dummy */ > + } > + if (na->separate_locks) { > + mtx_lock(&r->q_lock); > + selwakeuppri(&r->si, PI_NET); > + mtx_unlock(&r->q_lock); > + if (main_wq) { > + mtx_lock(&na->core_lock); > + selwakeuppri(main_wq, PI_NET); > + mtx_unlock(&na->core_lock); > + } > + } else { > + mtx_lock(&na->core_lock); > + selwakeuppri(&r->si, PI_NET); > + if (main_wq) > + selwakeuppri(main_wq, PI_NET); > + mtx_unlock(&na->core_lock); > + } > + *work_done = 1; /* do not fire napi again */ > + return 1; > +} > + > + > +#ifdef linux /* linux-specific routines */ > + > +/* > + * Remap linux arguments into the FreeBSD call. > + * - pwait is the poll table, passed as 'dev'; > + * If pwait == NULL someone else already woke up before. We can report > + * events but they are filtered upstream. > + * If pwait != NULL, then pwait->key contains the list of events. > + * - events is computed from pwait as above. > + * - file is passed as 'td'; > + */ > +static u_int > +linux_netmap_poll(struct file * file, struct poll_table_struct *pwait) > +{ > +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,4,0) > + int events = pwait ? pwait->key : POLLIN | POLLOUT; > +#else /* in 3.4.0 field 'key' was renamed to '_key' */ > + int events = pwait ? pwait->_key : POLLIN | POLLOUT; > +#endif > + return netmap_poll((void *)pwait, events, (void *)file); > +} > + > +static int > +linux_netmap_mmap(struct file *f, struct vm_area_struct *vma) > +{ > + int lut_skip, i, j; > + int user_skip = 0; > + struct lut_entry *l_entry; > + int error = 0; > + unsigned long off, tomap; > + /* > + * vma->vm_start: start of mapping user address space > + * vma->vm_end: end of the mapping user address space > + * vma->vm_pfoff: offset of first page in the device > + */ > + > + // XXX security checks > + > + error = netmap_get_memory(f->private_data); > + ND("get_memory returned %d", error); > + if (error) > + return -error; > + > + off = vma->vm_pgoff << PAGE_SHIFT; /* offset in bytes */ > + tomap = vma->vm_end - vma->vm_start; > + for (i = 0; i < NETMAP_POOLS_NR; i++) { /* loop through obj_pools */ > + const struct netmap_obj_pool *p = &nm_mem.pools[i]; > + /* > + * In each pool memory is allocated in clusters > + * of size _clustsize, each containing clustentries > + * entries. For each object k we already store the > + * vtophys mapping in lut[k] so we use that, scanning > + * the lut[] array in steps of clustentries, > + * and we map each cluster (not individual pages, > + * it would be overkill). > + */ > + > + /* > + * We interpret vm_pgoff as an offset into the whole > + * netmap memory, as if all clusters where contiguous. > + */ > + for (lut_skip = 0, j = 0; j < p->_numclusters; j++, lut_skip += p->clustentries) { > + unsigned long paddr, mapsize; > + if (p->_clustsize <= off) { > + off -= p->_clustsize; > + continue; > + } > + l_entry = &p->lut[lut_skip]; /* first obj in the cluster */ > + paddr = l_entry->paddr + off; > + mapsize = p->_clustsize - off; > + off = 0; > + if (mapsize > tomap) > + mapsize = tomap; > + ND("remap_pfn_range(%lx, %lx, %lx)", > + vma->vm_start + user_skip, > + paddr >> PAGE_SHIFT, mapsize); > + if (remap_pfn_range(vma, vma->vm_start + user_skip, > + paddr >> PAGE_SHIFT, mapsize, > + vma->vm_page_prot)) > + return -EAGAIN; // XXX check return value > + user_skip += mapsize; > + tomap -= mapsize; > + if (tomap == 0) > + goto done; > + } > + } > +done: > + > + return 0; > +} > + > +static netdev_tx_t > +linux_netmap_start(struct sk_buff *skb, struct net_device *dev) > +{ > + netmap_start(dev, skb); > + return (NETDEV_TX_OK); > +} > + > + > +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,37) // XXX was 38 > +#define LIN_IOCTL_NAME .ioctl > +int > +linux_netmap_ioctl(struct inode *inode, struct file *file, u_int cmd, u_long data /* arg */) > +#else > +#define LIN_IOCTL_NAME .unlocked_ioctl > +long > +linux_netmap_ioctl(struct file *file, u_int cmd, u_long data /* arg */) > +#endif > +{ > + int ret; > + struct nmreq nmr; > + bzero(&nmr, sizeof(nmr)); > + > + if (data && copy_from_user(&nmr, (void *)data, sizeof(nmr) ) != 0) > + return -EFAULT; > + ret = netmap_ioctl(NULL, cmd, (caddr_t)&nmr, 0, (void *)file); > + pr_info("netmap ioctl %u ret = %d\n", cmd, ret); > + if (data && copy_to_user((void*)data, &nmr, sizeof(nmr) ) != 0) > + return -EFAULT; > + return -ret; > +} > + > + > +static int > +netmap_release(struct inode *inode, struct file *file) > +{ > + (void)inode; /* UNUSED */ > + if (file->private_data) > + netmap_dtor(file->private_data); > + return (0); > +} > + > +static int > +linux_netmap_open(struct inode *inode, struct file *file) > +{ > + struct netmap_priv_d *priv; > + (void)inode; /* UNUSED */ > + > + priv = malloc(sizeof(struct netmap_priv_d), M_DEVBUF, > + M_NOWAIT | M_ZERO); > + if (priv == NULL) > + return -ENOMEM; > + > + file->private_data = priv; > + > + return (0); > +} > + > +static struct file_operations netmap_fops = { > + .open = linux_netmap_open, > + .mmap = linux_netmap_mmap, > + LIN_IOCTL_NAME = linux_netmap_ioctl, > + .poll = linux_netmap_poll, > + .release = netmap_release, > +}; > + > +static struct miscdevice netmap_cdevsw = { /* same name as FreeBSD */ > + MISC_DYNAMIC_MINOR, > + "netmap", > + &netmap_fops, > +}; > + > +static int netmap_init(void); > +static void netmap_fini(void); > + > +/* Errors have negative values on linux */ > +static int linux_netmap_init(void) > +{ > + return -netmap_init(); > +} > + > +module_init(linux_netmap_init); > +module_exit(netmap_fini); > +/* export certain symbols to other modules */ > +EXPORT_SYMBOL(netmap_attach); // driver attach routines > +EXPORT_SYMBOL(netmap_detach); // driver detach routines > +EXPORT_SYMBOL(netmap_ring_reinit); // ring init on error > +EXPORT_SYMBOL(netmap_buffer_lut); > +EXPORT_SYMBOL(netmap_total_buffers); // index check > +EXPORT_SYMBOL(netmap_buffer_base); > +EXPORT_SYMBOL(netmap_reset); // ring init routines > +EXPORT_SYMBOL(netmap_buf_size); > +EXPORT_SYMBOL(netmap_rx_irq); // default irq handler > +EXPORT_SYMBOL(netmap_no_pendintr); // XXX mitigation - should go away > + > + > +MODULE_AUTHOR("Matteo Landi, Luigi Rizzo"); > +MODULE_DESCRIPTION("The netmap packet I/O framework"); > +MODULE_LICENSE("Dual BSD/GPL"); /* the code here is all BSD. */ > + > +#else /* __FreeBSD__ */ > + > +static struct cdevsw netmap_cdevsw = { > + .d_version = D_VERSION, > + .d_name = "netmap", > + .d_open = netmap_open, > + .d_mmap = netmap_mmap, > + .d_mmap_single = netmap_mmap_single, > + .d_ioctl = netmap_ioctl, > + .d_poll = netmap_poll, > + .d_close = netmap_close, > +}; > +#endif /* __FreeBSD__ */ > + > +#ifdef NM_BRIDGE > +/* > + *---- support for virtual bridge ----- > + */ > + > +/* ----- FreeBSD if_bridge hash function ------- */ > + > +/* > + * The following hash function is adapted from "Hash Functions" by Bob Jenkins > + * ("Algorithm Alley", Dr. Dobbs Journal, September 1997). > + * > + * http://www.burtleburtle.net/bob/hash/spooky.html > + */ > +#define mix(a, b, c) \ > +do { \ > + a -= b; a -= c; a ^= (c >> 13); \ > + b -= c; b -= a; b ^= (a << 8); \ > + c -= a; c -= b; c ^= (b >> 13); \ > + a -= b; a -= c; a ^= (c >> 12); \ > + b -= c; b -= a; b ^= (a << 16); \ > + c -= a; c -= b; c ^= (b >> 5); \ > + a -= b; a -= c; a ^= (c >> 3); \ > + b -= c; b -= a; b ^= (a << 10); \ > + c -= a; c -= b; c ^= (b >> 15); \ > +} while (/*CONSTCOND*/0) > + > +static __inline uint32_t > +nm_bridge_rthash(const uint8_t *addr) > +{ > + uint32_t a = 0x9e3779b9, b = 0x9e3779b9, c = 0; // hask key > + > + b += addr[5] << 8; > + b += addr[4]; > + a += addr[3] << 24; > + a += addr[2] << 16; > + a += addr[1] << 8; > + a += addr[0]; > + > + mix(a, b, c); > +#define BRIDGE_RTHASH_MASK (NM_BDG_HASH-1) > + return (c & BRIDGE_RTHASH_MASK); > +} > + > +#undef mix > + > + > +static int > +bdg_netmap_reg(struct ifnet *ifp, int onoff) > +{ > + int i, err = 0; > + struct nm_bridge *b = ifp->if_bridge; > + > + BDG_LOCK(b); > + if (onoff) { > + /* the interface must be already in the list. > + * only need to mark the port as active > + */ > + ND("should attach %s to the bridge", ifp->if_xname); > + for (i=0; i < NM_BDG_MAXPORTS; i++) > + if (b->bdg_ports[i] == ifp) > + break; > + if (i == NM_BDG_MAXPORTS) { > + D("no more ports available"); > + err = EINVAL; > + goto done; > + } > + ND("setting %s in netmap mode", ifp->if_xname); > + ifp->if_capenable |= IFCAP_NETMAP; > + NA(ifp)->bdg_port = i; > + b->act_ports |= (1<<i); > + b->bdg_ports[i] = ifp; > + } else { > + /* should be in the list, too -- remove from the mask */ > + ND("removing %s from netmap mode", ifp->if_xname); > + ifp->if_capenable &= ~IFCAP_NETMAP; > + i = NA(ifp)->bdg_port; > + b->act_ports &= ~(1<<i); > + } > +done: > + BDG_UNLOCK(b); > + return err; > +} > + > + > +static int > +nm_bdg_flush(struct nm_bdg_fwd *ft, int n, struct ifnet *ifp) > +{ > + int i, ifn; > + uint64_t all_dst, dst; > + uint32_t sh, dh; > + uint64_t mysrc = 1 << NA(ifp)->bdg_port; > + uint64_t smac, dmac; > + struct netmap_slot *slot; > + struct nm_bridge *b = ifp->if_bridge; > + > + ND("prepare to send %d packets, act_ports 0x%x", n, b->act_ports); > + /* only consider valid destinations */ > + all_dst = (b->act_ports & ~mysrc); > + /* first pass: hash and find destinations */ > + for (i = 0; likely(i < n); i++) { > + uint8_t *buf = ft[i].buf; > + dmac = le64toh(*(uint64_t *)(buf)) & 0xffffffffffff; > + smac = le64toh(*(uint64_t *)(buf + 4)); > + smac >>= 16; > + if (unlikely(netmap_verbose)) { > + uint8_t *s = buf+6, *d = buf; > + D("%d len %4d %02x:%02x:%02x:%02x:%02x:%02x -> %02x:%02x:%02x:%02x:%02x:%02x", > + i, > + ft[i].len, > + s[0], s[1], s[2], s[3], s[4], s[5], > + d[0], d[1], d[2], d[3], d[4], d[5]); > + } > + /* > + * The hash is somewhat expensive, there might be some > + * worthwhile optimizations here. > + */ > + if ((buf[6] & 1) == 0) { /* valid src */ > + uint8_t *s = buf+6; > + sh = nm_bridge_rthash(buf+6); // XXX hash of source > + /* update source port forwarding entry */ > + b->ht[sh].mac = smac; /* XXX expire ? */ > + b->ht[sh].ports = mysrc; > + if (netmap_verbose) > + D("src %02x:%02x:%02x:%02x:%02x:%02x on port %d", > + s[0], s[1], s[2], s[3], s[4], s[5], NA(ifp)->bdg_port); > + } > + dst = 0; > + if ( (buf[0] & 1) == 0) { /* unicast */ > + uint8_t *d = buf; > + dh = nm_bridge_rthash(buf); // XXX hash of dst > + if (b->ht[dh].mac == dmac) { /* found dst */ > + dst = b->ht[dh].ports; > + if (netmap_verbose) > + D("dst %02x:%02x:%02x:%02x:%02x:%02x to port %x", > + d[0], d[1], d[2], d[3], d[4], d[5], (uint32_t)(dst >> 16)); > + } > + } > + if (dst == 0) > + dst = all_dst; > + dst &= all_dst; /* only consider valid ports */ > + if (unlikely(netmap_verbose)) > + D("pkt goes to ports 0x%x", (uint32_t)dst); > + ft[i].dst = dst; > + } > + > + /* second pass, scan interfaces and forward */ > + all_dst = (b->act_ports & ~mysrc); > + for (ifn = 0; all_dst; ifn++) { > + struct ifnet *dst_ifp = b->bdg_ports[ifn]; > + struct netmap_adapter *na; > + struct netmap_kring *kring; > + struct netmap_ring *ring; > + int j, lim, sent, locked; > + > + if (!dst_ifp) > + continue; > + ND("scan port %d %s", ifn, dst_ifp->if_xname); > + dst = 1 << ifn; > + if ((dst & all_dst) == 0) /* skip if not set */ > + continue; > + all_dst &= ~dst; /* clear current node */ > + na = NA(dst_ifp); > + > + ring = NULL; > + kring = NULL; > + lim = sent = locked = 0; > + /* inside, scan slots */ > + for (i = 0; likely(i < n); i++) { > + if ((ft[i].dst & dst) == 0) > + continue; /* not here */ > + if (!locked) { > + kring = &na->rx_rings[0]; > + ring = kring->ring; > + lim = kring->nkr_num_slots - 1; > + na->nm_lock(dst_ifp, NETMAP_RX_LOCK, 0); > + locked = 1; > + } > + if (unlikely(kring->nr_hwavail >= lim)) { > + if (netmap_verbose) > + D("rx ring full on %s", ifp->if_xname); > + break; > + } > + j = kring->nr_hwcur + kring->nr_hwavail; > + if (j > lim) > + j -= kring->nkr_num_slots; > + slot = &ring->slot[j]; > + ND("send %d %d bytes at %s:%d", i, ft[i].len, dst_ifp->if_xname, j); > + pkt_copy(ft[i].buf, NMB(slot), ft[i].len); > + slot->len = ft[i].len; > + kring->nr_hwavail++; > + sent++; > + } > + if (locked) { > + ND("sent %d on %s", sent, dst_ifp->if_xname); > + if (sent) > + selwakeuppri(&kring->si, PI_NET); > + na->nm_lock(dst_ifp, NETMAP_RX_UNLOCK, 0); > + } > + } > + return 0; > +} > + > +/* > + * main dispatch routine > + */ > +static int > +bdg_netmap_txsync(struct ifnet *ifp, u_int ring_nr, int do_lock) > +{ > + struct netmap_adapter *na = NA(ifp); > + struct netmap_kring *kring = &na->tx_rings[ring_nr]; > + struct netmap_ring *ring = kring->ring; > + int i, j, k, lim = kring->nkr_num_slots - 1; > + struct nm_bdg_fwd *ft = (struct nm_bdg_fwd *)(ifp + 1); > + int ft_i; /* position in the forwarding table */ > + > + k = ring->cur; > + if (k > lim) > + return netmap_ring_reinit(kring); > + if (do_lock) > + na->nm_lock(ifp, NETMAP_TX_LOCK, ring_nr); > + > + if (netmap_bridge <= 0) { /* testing only */ > + j = k; // used all > + goto done; > + } > + if (netmap_bridge > NM_BDG_BATCH) > + netmap_bridge = NM_BDG_BATCH; > + > + ft_i = 0; /* start from 0 */ > + for (j = kring->nr_hwcur; likely(j != k); j = unlikely(j == lim) ? 0 : j+1) { > + struct netmap_slot *slot = &ring->slot[j]; > + int len = ft[ft_i].len = slot->len; > + char *buf = ft[ft_i].buf = NMB(slot); > + > + prefetch(buf); > + if (unlikely(len < 14)) > + continue; > + if (unlikely(++ft_i == netmap_bridge)) > + ft_i = nm_bdg_flush(ft, ft_i, ifp); > + } > + if (ft_i) > + ft_i = nm_bdg_flush(ft, ft_i, ifp); > + /* count how many packets we sent */ > + i = k - j; > + if (i < 0) > + i += kring->nkr_num_slots; > + kring->nr_hwavail = kring->nkr_num_slots - 1 - i; > + if (j != k) > + D("early break at %d/ %d, avail %d", j, k, kring->nr_hwavail); > + > +done: > + kring->nr_hwcur = j; > + ring->avail = kring->nr_hwavail; > + if (do_lock) > + na->nm_lock(ifp, NETMAP_TX_UNLOCK, ring_nr); > + > + if (netmap_verbose) > + D("%s ring %d lock %d", ifp->if_xname, ring_nr, do_lock); > + return 0; > +} > + > +static int > +bdg_netmap_rxsync(struct ifnet *ifp, u_int ring_nr, int do_lock) > +{ > + struct netmap_adapter *na = NA(ifp); > + struct netmap_kring *kring = &na->rx_rings[ring_nr]; > + struct netmap_ring *ring = kring->ring; > + u_int j, n, lim = kring->nkr_num_slots - 1; > + u_int k = ring->cur, resvd = ring->reserved; > + > + ND("%s ring %d lock %d avail %d", > + ifp->if_xname, ring_nr, do_lock, kring->nr_hwavail); > + > + if (k > lim) > + return netmap_ring_reinit(kring); > + if (do_lock) > + na->nm_lock(ifp, NETMAP_RX_LOCK, ring_nr); > + > + /* skip past packets that userspace has released */ > + j = kring->nr_hwcur; /* netmap ring index */ > + if (resvd > 0) { > + if (resvd + ring->avail >= lim + 1) { > + D("XXX invalid reserve/avail %d %d", resvd, ring->avail); > + ring->reserved = resvd = 0; // XXX panic... > + } > + k = (k >= resvd) ? k - resvd : k + lim + 1 - resvd; > + } > + > + if (j != k) { /* userspace has released some packets. */ > + n = k - j; > + if (n < 0) > + n += kring->nkr_num_slots; > + ND("userspace releases %d packets", n); > + for (n = 0; likely(j != k); n++) { > + struct netmap_slot *slot = &ring->slot[j]; > + void *addr = NMB(slot); > + > + if (addr == netmap_buffer_base) { /* bad buf */ > + if (do_lock) > + na->nm_lock(ifp, NETMAP_RX_UNLOCK, ring_nr); > + return netmap_ring_reinit(kring); > + } > + /* decrease refcount for buffer */ > + > + slot->flags &= ~NS_BUF_CHANGED; > + j = unlikely(j == lim) ? 0 : j + 1; > + } > + kring->nr_hwavail -= n; > + kring->nr_hwcur = k; > + } > + /* tell userspace that there are new packets */ > + ring->avail = kring->nr_hwavail - resvd; > + > + if (do_lock) > + na->nm_lock(ifp, NETMAP_RX_UNLOCK, ring_nr); > + return 0; > +} > + > +static void > +bdg_netmap_attach(struct ifnet *ifp) > +{ > + struct netmap_adapter na; > + > + ND("attaching virtual bridge"); > + bzero(&na, sizeof(na)); > + > + na.ifp = ifp; > + na.separate_locks = 1; > + na.num_tx_desc = NM_BRIDGE_RINGSIZE; > + na.num_rx_desc = NM_BRIDGE_RINGSIZE; > + na.nm_txsync = bdg_netmap_txsync; > + na.nm_rxsync = bdg_netmap_rxsync; > + na.nm_register = bdg_netmap_reg; > + netmap_attach(&na, 1); > +} > + > +#endif /* NM_BRIDGE */ > + > +static struct cdev *netmap_dev; /* /dev/netmap character device. */ > + > + > +/* > + * Module loader. > + * > + * Create the /dev/netmap device and initialize all global > + * variables. > + * > + * Return 0 on success, errno on failure. > + */ > +static int > +netmap_init(void) > +{ > + int error; > + > + error = netmap_memory_init(); > + if (error != 0) { > + printf("netmap: unable to initialize the memory allocator.\n"); > + return (error); > + } > + printf("netmap: loaded module\n"); > + netmap_dev = make_dev(&netmap_cdevsw, 0, UID_ROOT, GID_WHEEL, 0660, > + "netmap"); > + > +#ifdef NM_BRIDGE > + { > + int i; > + for (i = 0; i < NM_BRIDGES; i++) > + mtx_init(&nm_bridges[i].bdg_lock, "bdg lock", "bdg_lock", MTX_DEF); > + } > +#endif > + return (error); > +} > + > + > +/* > + * Module unloader. > + * > + * Free all the memory, and destroy the ``/dev/netmap`` device. > + */ > +static void > +netmap_fini(void) > +{ > + destroy_dev(netmap_dev); > + netmap_memory_fini(); > + printf("netmap: unloaded module.\n"); > +} > + > + > +#ifdef __FreeBSD__ > +/* > + * Kernel entry point. > + * > + * Initialize/finalize the module and return. > + * > + * Return 0 on success, errno on failure. > + */ > +static int > +netmap_loader(__unused struct module *module, int event, __unused void *arg) > +{ > + int error = 0; > + > + switch (event) { > + case MOD_LOAD: > + error = netmap_init(); > + break; > + > + case MOD_UNLOAD: > + netmap_fini(); > + break; > + > + default: > + error = EOPNOTSUPP; > + break; > + } > + > + return (error); > +} > + > + > +DEV_MODULE(netmap, netmap_loader, NULL); > +#endif /* __FreeBSD__ */ > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/drivers/staging/netmap/README 2013-03-10 10:08:20.327671428 -0700 > @@ -0,0 +1,127 @@ > +# $Id: README 10832 2012-03-22 18:22:42Z luigi $ > + > +NETMAP FOR LINUX > +---------------- > + > +This directory contains a version of the "netmap" code for Linux. > + > +Netmap is a BSD-licensed framework that supports line-rate direct packet > +I/O even on 10GBit/s interfaces (14.88Mpps) with limited system load, > +and includes a libpcap emulation library to port applications. See > + > + http://info.iet.unipi.it/~luigi/netmap/ > + > +for more details. There you can also find the latest versions > +of the code and documentation as well as pre-built TinyCore > +images based on linux 3.0.3 and containing the netmap modules > +and some test applications. > + > +This is a preliminary version supporting the ixgbe and e1000/e1000e > +driver. Patches for other devices (igb, r8169, forcedeth) are > +untested and probably not working yet. > + > +Netmap relies on a kernel module (netmap_lin.ko) and slightly modified > +device drivers. Userspace programs can use the native API (documented > +in netmap.4) or a libpcap emulation library. > + > + Directory structure for this archive > + > + . documentation, patches etc. > + include/net header files for user programs > + net/netmap kernel core files, > + sample applications, manpage > + net/* patched device drivers for a 3.0.x linux version. > + > +HOW TO BUILD THE CODE > +--------------------- > + > +1. make sure you have kernel sources/headers matching your installed system > + > +2. do the following > + (cd net; make KSRC=/usr/src/linux-kernel-source-or-headers ) > + this produces net/netmap/netmap_lin.ko and other kernel modules. > + > +3. to build sample applications, run > + (cd net/netmap; make apps ) > + (you will need the pthreads and libpcap-dev packages to build them) > + > +HOW TO USE THE CODE > +------------------- > + > + REMEMBER > + THIS IS EXPERIMENTAL CODE WHICH MAY CRASH YOUR SYSTEM. > + USE IT AT YOUR OWN RISk. > + > +Whether you built your own modules, or are using the prebuilt > +TinyCore image, the following steps can be used for initial testing: > + > +1. unload any modules for the network cards you want to use, e.g. > + sudo rmmod ixgbe > + sudo rmmod e1000 > + sudo rmmod e1000e > + > +2. load netmap and device driver module > + sudo insmod net/netmap/netmap_lin.ko > + sudo insmod net/ixgbe/ixgbe.ko > + sudo insmod net/ixgbe/e1000.ko > + sudo insmod net/ixgbe/e1000e.ko > + > +3. turn the interface(s) up > + > + sudo ifconfig eth0 up # and same for others > + > +4. Run test applications -- as an example, pkt-gen is a raw packet > + sender/receiver which can do line rate on a 10G interface > + > + # send about 500 million packets of 64 bytes each. > + # wait 5s before starting, so the link can go up > + sudo pkt-gen -i eth0 -t 500111222 -l 64 -w 5 > + # you should see about 14.2 Mpps > + > + sudo pkt-gen -i eth0 # act as a receiver > + > + > +COMMON PROBLEMS > +---------------- > + > +* switching in/out of netmap mode causes the link to go down and up. > + If your card is connected to a switch with spanning tree enabled, > + the switch will likely MUTE THE LINK FOR 10 SECONDS while it is > + detecting the new topology. Either disable the spanning tree on > + the switch or use long pauses before sending data; > + > +* Not all cards can do line rate no matter how fast is your software or > + CPU. Several have hardware limitations that prevent reaching the peak > + speed, especially for small packet sizes. Examples: > + > + - ixgbe cannot receive at line rate with packet sizes that are > + not multiple of 64 (after CRC stripping). > + This is especially evident with minimum-sized frames (-l 60 ) > + > + - some of the low-end 'e1000' cards can send 1.2 - 1.3Mpps instead > + of the theoretical maximum (1.488Mpps) > + > + - the 'realtek' cards seem unable to send more than 450-500Kpps > + even though they can receive at least 1.1Mpps > + > +* if the link is not up when the packet generator starts, you will > + see frequent messages about a link reset. While we work on a fix, > + use the '-w' argument on the generator to specify a longer timeout > + > +* the ixgbe driver (and perhaps others) is severely slowed down if the > + remote party is senting flow control frames to slow down traffic. > + If that happens try to use the ethtool command to disable flow control. > + > + > +REVISION HISTORY > +----------------- > + > +20120322 - fixed the 'igb' driver, now it can send and receive correctly > + (the problem was in netmap_rx_irq() so it might have affected > + other multiqueue cards). > + Also tested the 'r8169' in transmit mode. > + Added comments on switches and spanning tree. > + > +20120217 - initial version. Only ixgbe, e1000 and e1000e are working. > + Other drivers (igb, r8169, forcedeth) are supplied only as a > + proof of concept. > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/drivers/staging/netmap/TODO 2013-03-10 16:45:40.377960319 -0700 > @@ -0,0 +1,21 @@ > + > +This driver is functional but needs some cleanup before it is ready > +for included in networking subsystem. > + > + - Use unifdef to elimiate non Linux code > + - Get rid of wrapper code including bsd_glue.h > + - Fix coding style to be Linux rather than BSD > + - Fix whitespace warnings > + - Add stubs so that ethernet drivers can use without #ifdef's > + - Autoload the module, assign a real minor number and have aliases > + - Rework documentation and put in Documentation/networking/ > + - Remove memory allocator kludge wrapper > + - Remove devfs kludge wrappers > + - Review configuration API's and management (counters) > + - Locking should be consistent (ie get rid of separate_locks option)? > + - Doesn't work with DMA remapping > + - Use __u32 rather than uint32_t in headers > + - netmap_user.h headers should use inline instead of macros > + > +Fundamentally this will break source compatiablity with FreeBSD, > +but this is not our problem. > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/drivers/staging/netmap/netmap_mem2.c 2013-03-10 10:08:20.327671428 -0700 > @@ -0,0 +1,974 @@ > +/* > + * Copyright (C) 2012 Matteo Landi, Luigi Rizzo, Giuseppe Lettieri. All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > + > +/* > + * $FreeBSD: head/sys/dev/netmap/netmap_mem2.c 234290 2012-04-14 16:44:18Z luigi $ > + * $Id: netmap_mem2.c 12010 2013-01-23 03:57:30Z luigi $ > + * > + * (New) memory allocator for netmap > + */ > + > +/* > + * This allocator creates three memory regions: > + * nm_if_pool for the struct netmap_if > + * nm_ring_pool for the struct netmap_ring > + * nm_buf_pool for the packet buffers. > + * > + * All regions need to be multiple of a page size as we export them to > + * userspace through mmap. Only the latter needs to be dma-able, > + * but for convenience use the same type of allocator for all. > + * > + * Once mapped, the three regions are exported to userspace > + * as a contiguous block, starting from nm_if_pool. Each > + * cluster (and pool) is an integral number of pages. > + * [ . . . ][ . . . . . .][ . . . . . . . . . .] > + * nm_if nm_ring nm_buf > + * > + * The userspace areas contain offsets of the objects in userspace. > + * When (at init time) we write these offsets, we find out the index > + * of the object, and from there locate the offset from the beginning > + * of the region. > + * > + * The invididual allocators manage a pool of memory for objects of > + * the same size. > + * The pool is split into smaller clusters, whose size is a > + * multiple of the page size. The cluster size is chosen > + * to minimize the waste for a given max cluster size > + * (we do it by brute force, as we have relatively few object > + * per cluster). > + * > + * Objects are aligned to the cache line (64 bytes) rounding up object > + * sizes when needed. A bitmap contains the state of each object. > + * Allocation scans the bitmap; this is done only on attach, so we are not > + * too worried about performance > + * > + * For each allocator we can define (thorugh sysctl) the size and > + * number of each object. Memory is allocated at the first use of a > + * netmap file descriptor, and can be freed when all such descriptors > + * have been released (including unmapping the memory). > + * If memory is scarce, the system tries to get as much as possible > + * and the sysctl values reflect the actual allocation. > + * Together with desired values, the sysctl export also absolute > + * min and maximum values that cannot be overridden. > + * > + * struct netmap_if: > + * variable size, max 16 bytes per ring pair plus some fixed amount. > + * 1024 bytes should be large enough in practice. > + * > + * In the worst case we have one netmap_if per ring in the system. > + * > + * struct netmap_ring > + * variable too, 8 byte per slot plus some fixed amount. > + * Rings can be large (e.g. 4k slots, or >32Kbytes). > + * We default to 36 KB (9 pages), and a few hundred rings. > + * > + * struct netmap_buffer > + * The more the better, both because fast interfaces tend to have > + * many slots, and because we may want to use buffers to store > + * packets in userspace avoiding copies. > + * Must contain a full frame (eg 1518, or more for vlans, jumbo > + * frames etc.) plus be nicely aligned, plus some NICs restrict > + * the size to multiple of 1K or so. Default to 2K > + */ > + > +#ifndef CONSERVATIVE > +#define NETMAP_BUF_MAX_NUM 20*4096*2 /* large machine */ > +#else /* CONSERVATIVE */ > +#define NETMAP_BUF_MAX_NUM 20000 /* 40MB */ > +#endif > + > +#ifdef linux > +#define NMA_LOCK_T struct semaphore > +#define NMA_LOCK_INIT() sema_init(&nm_mem.nm_mtx, 1) > +#define NMA_LOCK_DESTROY() > +#define NMA_LOCK() down(&nm_mem.nm_mtx) > +#define NMA_UNLOCK() up(&nm_mem.nm_mtx) > +#else /* !linux */ > +#define NMA_LOCK_T struct mtx > +#define NMA_LOCK_INIT() mtx_init(&nm_mem.nm_mtx, "netmap memory allocator lock", NULL, MTX_DEF) > +#define NMA_LOCK_DESTROY() mtx_destroy(&nm_mem.nm_mtx) > +#define NMA_LOCK() mtx_lock(&nm_mem.nm_mtx) > +#define NMA_UNLOCK() mtx_unlock(&nm_mem.nm_mtx) > +#endif /* linux */ > + > +enum { > + NETMAP_IF_POOL = 0, > + NETMAP_RING_POOL, > + NETMAP_BUF_POOL, > + NETMAP_POOLS_NR > +}; > + > + > +struct netmap_obj_params { > + u_int size; > + u_int num; > +}; > + > + > +struct netmap_obj_params netmap_params[NETMAP_POOLS_NR] = { > + [NETMAP_IF_POOL] = { > + .size = 1024, > + .num = 100, > + }, > + [NETMAP_RING_POOL] = { > + .size = 9*PAGE_SIZE, > + .num = 200, > + }, > + [NETMAP_BUF_POOL] = { > + .size = 2048, > + .num = NETMAP_BUF_MAX_NUM, > + }, > +}; > + > + > +struct netmap_obj_pool { > + char name[16]; /* name of the allocator */ > + u_int objtotal; /* actual total number of objects. */ > + u_int objfree; /* number of free objects. */ > + u_int clustentries; /* actual objects per cluster */ > + > + /* limits */ > + u_int objminsize; /* minimum object size */ > + u_int objmaxsize; /* maximum object size */ > + u_int nummin; /* minimum number of objects */ > + u_int nummax; /* maximum number of objects */ > + > + /* the total memory space is _numclusters*_clustsize */ > + u_int _numclusters; /* how many clusters */ > + u_int _clustsize; /* cluster size */ > + u_int _objsize; /* actual object size */ > + > + u_int _memtotal; /* _numclusters*_clustsize */ > + struct lut_entry *lut; /* virt,phys addresses, objtotal entries */ > + uint32_t *bitmap; /* one bit per buffer, 1 means free */ > + uint32_t bitmap_slots; /* number of uint32 entries in bitmap */ > +}; > + > + > +struct netmap_mem_d { > + NMA_LOCK_T nm_mtx; /* protect the allocator */ > + u_int nm_totalsize; /* shorthand */ > + > + int finalized; /* !=0 iff preallocation done */ > + int lasterr; /* last error for curr config */ > + int refcount; /* existing priv structures */ > + /* the three allocators */ > + struct netmap_obj_pool pools[NETMAP_POOLS_NR]; > +}; > + > + > +static struct netmap_mem_d nm_mem = { /* Our memory allocator. */ > + .pools = { > + [NETMAP_IF_POOL] = { > + .name = "netmap_if", > + .objminsize = sizeof(struct netmap_if), > + .objmaxsize = 4096, > + .nummin = 10, /* don't be stingy */ > + .nummax = 10000, /* XXX very large */ > + }, > + [NETMAP_RING_POOL] = { > + .name = "netmap_ring", > + .objminsize = sizeof(struct netmap_ring), > + .objmaxsize = 32*PAGE_SIZE, > + .nummin = 2, > + .nummax = 1024, > + }, > + [NETMAP_BUF_POOL] = { > + .name = "netmap_buf", > + .objminsize = 64, > + .objmaxsize = 65536, > + .nummin = 4, > + .nummax = 1000000, /* one million! */ > + }, > + }, > +}; > + > +struct lut_entry *netmap_buffer_lut; /* exported */ > + > +/* memory allocator related sysctls */ > + > +#define STRINGIFY(x) #x > + > +#define DECLARE_SYSCTLS(id, name) \ > + /* TUNABLE_INT("hw.netmap." STRINGIFY(name) "_size", &netmap_params[id].size); */ \ > + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_size, \ > + CTLFLAG_RW, &netmap_params[id].size, 0, "Requested size of netmap " STRINGIFY(name) "s"); \ > + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_curr_size, \ > + CTLFLAG_RD, &nm_mem.pools[id]._objsize, 0, "Current size of netmap " STRINGIFY(name) "s"); \ > + /* TUNABLE_INT("hw.netmap." STRINGIFY(name) "_num", &netmap_params[id].num); */ \ > + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_num, \ > + CTLFLAG_RW, &netmap_params[id].num, 0, "Requested number of netmap " STRINGIFY(name) "s"); \ > + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_curr_num, \ > + CTLFLAG_RD, &nm_mem.pools[id].objtotal, 0, "Current number of netmap " STRINGIFY(name) "s") > + > +DECLARE_SYSCTLS(NETMAP_IF_POOL, if); > +DECLARE_SYSCTLS(NETMAP_RING_POOL, ring); > +DECLARE_SYSCTLS(NETMAP_BUF_POOL, buf); > + > +/* > + * Convert a userspace offset to a phisical address. > + * XXX re-do in a simpler way. > + * > + * The idea here is to hide userspace applications the fact that pre-allocated > + * memory is not contiguous, but fragmented across different clusters and > + * smaller memory allocators. Consequently, first of all we need to find which > + * allocator is owning provided offset, then we need to find out the physical > + * address associated to target page (this is done using the look-up table. > + */ > +static inline vm_paddr_t > +netmap_ofstophys(vm_offset_t offset) > +{ > + int i; > + vm_offset_t o = offset; > + struct netmap_obj_pool *p = nm_mem.pools; > + > + for (i = 0; i < NETMAP_POOLS_NR; offset -= p[i]._memtotal, i++) { > + if (offset >= p[i]._memtotal) > + continue; > + // XXX now scan the clusters > + return p[i].lut[offset / p[i]._objsize].paddr + > + offset % p[i]._objsize; > + } > + /* this is only in case of errors */ > + D("invalid ofs 0x%x out of 0x%x 0x%x 0x%x", (u_int)o, > + p[NETMAP_IF_POOL]._memtotal, > + p[NETMAP_IF_POOL]._memtotal > + + p[NETMAP_RING_POOL]._memtotal, > + p[NETMAP_IF_POOL]._memtotal > + + p[NETMAP_RING_POOL]._memtotal > + + p[NETMAP_BUF_POOL]._memtotal); > + return 0; // XXX bad address > +} > + > +/* > + * we store objects by kernel address, need to find the offset > + * within the pool to export the value to userspace. > + * Algorithm: scan until we find the cluster, then add the > + * actual offset in the cluster > + */ > +static ssize_t > +netmap_obj_offset(struct netmap_obj_pool *p, const void *vaddr) > +{ > + int i, k = p->clustentries, n = p->objtotal; > + ssize_t ofs = 0; > + > + for (i = 0; i < n; i += k, ofs += p->_clustsize) { > + const char *base = p->lut[i].vaddr; > + ssize_t relofs = (const char *) vaddr - base; > + > + if (relofs < 0 || relofs > p->_clustsize) > + continue; > + > + ofs = ofs + relofs; > + ND("%s: return offset %d (cluster %d) for pointer %p", > + p->name, ofs, i, vaddr); > + return ofs; > + } > + D("address %p is not contained inside any cluster (%s)", > + vaddr, p->name); > + return 0; /* An error occurred */ > +} > + > +/* Helper functions which convert virtual addresses to offsets */ > +#define netmap_if_offset(v) \ > + netmap_obj_offset(&nm_mem.pools[NETMAP_IF_POOL], (v)) > + > +#define netmap_ring_offset(v) \ > + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + \ > + netmap_obj_offset(&nm_mem.pools[NETMAP_RING_POOL], (v))) > + > +#define netmap_buf_offset(v) \ > + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + \ > + nm_mem.pools[NETMAP_RING_POOL]._memtotal + \ > + netmap_obj_offset(&nm_mem.pools[NETMAP_BUF_POOL], (v))) > + > + > +/* > + * report the index, and use start position as a hint, > + * otherwise buffer allocation becomes terribly expensive. > + */ > +static void * > +netmap_obj_malloc(struct netmap_obj_pool *p, int len, uint32_t *start, uint32_t *index) > +{ > + uint32_t i = 0; /* index in the bitmap */ > + uint32_t mask, j; /* slot counter */ > + void *vaddr = NULL; > + > + if (len > p->_objsize) { > + D("%s request size %d too large", p->name, len); > + // XXX cannot reduce the size > + return NULL; > + } > + > + if (p->objfree == 0) { > + D("%s allocator: run out of memory", p->name); > + return NULL; > + } > + if (start) > + i = *start; > + > + /* termination is guaranteed by p->free, but better check bounds on i */ > + while (vaddr == NULL && i < p->bitmap_slots) { > + uint32_t cur = p->bitmap[i]; > + if (cur == 0) { /* bitmask is fully used */ > + i++; > + continue; > + } > + /* locate a slot */ > + for (j = 0, mask = 1; (cur & mask) == 0; j++, mask <<= 1) > + ; > + > + p->bitmap[i] &= ~mask; /* mark object as in use */ > + p->objfree--; > + > + vaddr = p->lut[i * 32 + j].vaddr; > + if (index) > + *index = i * 32 + j; > + } > + ND("%s allocator: allocated object @ [%d][%d]: vaddr %p", i, j, vaddr); > + > + if (start) > + *start = i; > + return vaddr; > +} > + > + > +/* > + * free by index, not by address > + */ > +static void > +netmap_obj_free(struct netmap_obj_pool *p, uint32_t j) > +{ > + if (j >= p->objtotal) { > + D("invalid index %u, max %u", j, p->objtotal); > + return; > + } > + p->bitmap[j / 32] |= (1 << (j % 32)); > + p->objfree++; > + return; > +} > + > +static void > +netmap_obj_free_va(struct netmap_obj_pool *p, void *vaddr) > +{ > + int i, j, n = p->_memtotal / p->_clustsize; > + > + for (i = 0, j = 0; i < n; i++, j += p->clustentries) { > + void *base = p->lut[i * p->clustentries].vaddr; > + ssize_t relofs = (ssize_t) vaddr - (ssize_t) base; > + > + /* Given address, is out of the scope of the current cluster.*/ > + if (vaddr < base || relofs > p->_clustsize) > + continue; > + > + j = j + relofs / p->_objsize; > + KASSERT(j != 0, ("Cannot free object 0")); > + netmap_obj_free(p, j); > + return; > + } > + D("address %p is not contained inside any cluster (%s)", > + vaddr, p->name); > +} > + > +#define netmap_if_malloc(len) netmap_obj_malloc(&nm_mem.pools[NETMAP_IF_POOL], len, NULL, NULL) > +#define netmap_if_free(v) netmap_obj_free_va(&nm_mem.pools[NETMAP_IF_POOL], (v)) > +#define netmap_ring_malloc(len) netmap_obj_malloc(&nm_mem.pools[NETMAP_RING_POOL], len, NULL, NULL) > +#define netmap_ring_free(v) netmap_obj_free_va(&nm_mem.pools[NETMAP_RING_POOL], (v)) > +#define netmap_buf_malloc(_pos, _index) \ > + netmap_obj_malloc(&nm_mem.pools[NETMAP_BUF_POOL], NETMAP_BUF_SIZE, _pos, _index) > + > + > +/* Return the index associated to the given packet buffer */ > +#define netmap_buf_index(v) \ > + (netmap_obj_offset(&nm_mem.pools[NETMAP_BUF_POOL], (v)) / nm_mem.pools[NETMAP_BUF_POOL]._objsize) > + > + > +/* Return nonzero on error */ > +static int > +netmap_new_bufs(struct netmap_if *nifp, > + struct netmap_slot *slot, u_int n) > +{ > + struct netmap_obj_pool *p = &nm_mem.pools[NETMAP_BUF_POOL]; > + int i = 0; /* slot counter */ > + uint32_t pos = 0; /* slot in p->bitmap */ > + uint32_t index = 0; /* buffer index */ > + > + (void)nifp; /* UNUSED */ > + for (i = 0; i < n; i++) { > + void *vaddr = netmap_buf_malloc(&pos, &index); > + if (vaddr == NULL) { > + D("unable to locate empty packet buffer"); > + goto cleanup; > + } > + slot[i].buf_idx = index; > + slot[i].len = p->_objsize; > + /* XXX setting flags=NS_BUF_CHANGED forces a pointer reload > + * in the NIC ring. This is a hack that hides missing > + * initializations in the drivers, and should go away. > + */ > + slot[i].flags = NS_BUF_CHANGED; > + } > + > + ND("allocated %d buffers, %d available, first at %d", n, p->objfree, pos); > + return (0); > + > +cleanup: > + while (i > 0) { > + i--; > + netmap_obj_free(p, slot[i].buf_idx); > + } > + bzero(slot, n * sizeof(slot[0])); > + return (ENOMEM); > +} > + > + > +static void > +netmap_free_buf(struct netmap_if *nifp, uint32_t i) > +{ > + struct netmap_obj_pool *p = &nm_mem.pools[NETMAP_BUF_POOL]; > + > + if (i < 2 || i >= p->objtotal) { > + D("Cannot free buf#%d: should be in [2, %d[", i, p->objtotal); > + return; > + } > + netmap_obj_free(p, i); > +} > + > +static void > +netmap_reset_obj_allocator(struct netmap_obj_pool *p) > +{ > + if (p == NULL) > + return; > + if (p->bitmap) > + free(p->bitmap, M_NETMAP); > + p->bitmap = NULL; > + if (p->lut) { > + int i; > + for (i = 0; i < p->objtotal; i += p->clustentries) { > + if (p->lut[i].vaddr) > + contigfree(p->lut[i].vaddr, p->_clustsize, M_NETMAP); > + } > + bzero(p->lut, sizeof(struct lut_entry) * p->objtotal); > +#ifdef linux > + vfree(p->lut); > +#else > + free(p->lut, M_NETMAP); > +#endif > + } > + p->lut = NULL; > +} > + > +/* > + * Free all resources related to an allocator. > + */ > +static void > +netmap_destroy_obj_allocator(struct netmap_obj_pool *p) > +{ > + if (p == NULL) > + return; > + netmap_reset_obj_allocator(p); > +} > + > +/* > + * We receive a request for objtotal objects, of size objsize each. > + * Internally we may round up both numbers, as we allocate objects > + * in small clusters multiple of the page size. > + * In the allocator we don't need to store the objsize, > + * but we do need to keep track of objtotal' and clustentries, > + * as they are needed when freeing memory. > + * > + * XXX note -- userspace needs the buffers to be contiguous, > + * so we cannot afford gaps at the end of a cluster. > + */ > + > + > +/* call with NMA_LOCK held */ > +static int > +netmap_config_obj_allocator(struct netmap_obj_pool *p, u_int objtotal, u_int objsize) > +{ > + int i, n; > + u_int clustsize; /* the cluster size, multiple of page size */ > + u_int clustentries; /* how many objects per entry */ > + > +#define MAX_CLUSTSIZE (1<<17) > +#define LINE_ROUND 64 > + if (objsize >= MAX_CLUSTSIZE) { > + /* we could do it but there is no point */ > + D("unsupported allocation for %d bytes", objsize); > + goto error; > + } > + /* make sure objsize is a multiple of LINE_ROUND */ > + i = (objsize & (LINE_ROUND - 1)); > + if (i) { > + D("XXX aligning object by %d bytes", LINE_ROUND - i); > + objsize += LINE_ROUND - i; > + } > + if (objsize < p->objminsize || objsize > p->objmaxsize) { > + D("requested objsize %d out of range [%d, %d]", > + objsize, p->objminsize, p->objmaxsize); > + goto error; > + } > + if (objtotal < p->nummin || objtotal > p->nummax) { > + D("requested objtotal %d out of range [%d, %d]", > + objtotal, p->nummin, p->nummax); > + goto error; > + } > + /* > + * Compute number of objects using a brute-force approach: > + * given a max cluster size, > + * we try to fill it with objects keeping track of the > + * wasted space to the next page boundary. > + */ > + for (clustentries = 0, i = 1;; i++) { > + u_int delta, used = i * objsize; > + if (used > MAX_CLUSTSIZE) > + break; > + delta = used % PAGE_SIZE; > + if (delta == 0) { // exact solution > + clustentries = i; > + break; > + } > + if (delta > ( (clustentries*objsize) % PAGE_SIZE) ) > + clustentries = i; > + } > + // D("XXX --- ouch, delta %d (bad for buffers)", delta); > + /* compute clustsize and round to the next page */ > + clustsize = clustentries * objsize; > + i = (clustsize & (PAGE_SIZE - 1)); > + if (i) > + clustsize += PAGE_SIZE - i; > + if (netmap_verbose) > + D("objsize %d clustsize %d objects %d", > + objsize, clustsize, clustentries); > + > + /* > + * The number of clusters is n = ceil(objtotal/clustentries) > + * objtotal' = n * clustentries > + */ > + p->clustentries = clustentries; > + p->_clustsize = clustsize; > + n = (objtotal + clustentries - 1) / clustentries; > + p->_numclusters = n; > + p->objtotal = n * clustentries; > + p->objfree = p->objtotal - 2; /* obj 0 and 1 are reserved */ > + p->_memtotal = p->_numclusters * p->_clustsize; > + p->_objsize = objsize; > + > + return 0; > + > +error: > + p->_objsize = objsize; > + p->objtotal = objtotal; > + > + return EINVAL; > +} > + > + > +/* call with NMA_LOCK held */ > +static int > +netmap_finalize_obj_allocator(struct netmap_obj_pool *p) > +{ > + int i, n; > + > + n = sizeof(struct lut_entry) * p->objtotal; > +#ifdef linux > + p->lut = vmalloc(n); > +#else > + p->lut = malloc(n, M_NETMAP, M_NOWAIT | M_ZERO); > +#endif > + if (p->lut == NULL) { > + D("Unable to create lookup table (%d bytes) for '%s'", n, p->name); > + goto clean; > + } > + > + /* Allocate the bitmap */ > + n = (p->objtotal + 31) / 32; > + p->bitmap = malloc(sizeof(uint32_t) * n, M_NETMAP, M_NOWAIT | M_ZERO); > + if (p->bitmap == NULL) { > + D("Unable to create bitmap (%d entries) for allocator '%s'", n, > + p->name); > + goto clean; > + } > + p->bitmap_slots = n; > + > + /* > + * Allocate clusters, init pointers and bitmap > + */ > + for (i = 0; i < p->objtotal;) { > + int lim = i + p->clustentries; > + char *clust; > + > + clust = contigmalloc(p->_clustsize, M_NETMAP, M_NOWAIT | M_ZERO, > + 0, -1UL, PAGE_SIZE, 0); > + if (clust == NULL) { > + /* > + * If we get here, there is a severe memory shortage, > + * so halve the allocated memory to reclaim some. > + * XXX check boundaries > + */ > + D("Unable to create cluster at %d for '%s' allocator", > + i, p->name); > + lim = i / 2; > + for (i--; i >= lim; i--) { > + p->bitmap[ (i>>5) ] &= ~( 1 << (i & 31) ); > + if (i % p->clustentries == 0 && p->lut[i].vaddr) > + contigfree(p->lut[i].vaddr, > + p->_clustsize, M_NETMAP); > + } > + p->objtotal = i; > + p->objfree = p->objtotal - 2; > + p->_numclusters = i / p->clustentries; > + p->_memtotal = p->_numclusters * p->_clustsize; > + break; > + } > + for (; i < lim; i++, clust += p->_objsize) { > + p->bitmap[ (i>>5) ] |= ( 1 << (i & 31) ); > + p->lut[i].vaddr = clust; > + p->lut[i].paddr = vtophys(clust); > + } > + } > + p->bitmap[0] = ~3; /* objs 0 and 1 is always busy */ > + if (netmap_verbose) > + D("Pre-allocated %d clusters (%d/%dKB) for '%s'", > + p->_numclusters, p->_clustsize >> 10, > + p->_memtotal >> 10, p->name); > + > + return 0; > + > +clean: > + netmap_reset_obj_allocator(p); > + return ENOMEM; > +} > + > +/* call with lock held */ > +static int > +netmap_memory_config_changed(void) > +{ > + int i; > + > + for (i = 0; i < NETMAP_POOLS_NR; i++) { > + if (nm_mem.pools[i]._objsize != netmap_params[i].size || > + nm_mem.pools[i].objtotal != netmap_params[i].num) > + return 1; > + } > + return 0; > +} > + > + > +/* call with lock held */ > +static int > +netmap_memory_config(void) > +{ > + int i; > + > + > + if (!netmap_memory_config_changed()) > + goto out; > + > + D("reconfiguring"); > + > + if (nm_mem.finalized) { > + /* reset previous allocation */ > + for (i = 0; i < NETMAP_POOLS_NR; i++) { > + netmap_reset_obj_allocator(&nm_mem.pools[i]); > + } > + nm_mem.finalized = 0; > + } > + > + for (i = 0; i < NETMAP_POOLS_NR; i++) { > + nm_mem.lasterr = netmap_config_obj_allocator(&nm_mem.pools[i], > + netmap_params[i].num, netmap_params[i].size); > + if (nm_mem.lasterr) > + goto out; > + } > + > + D("Have %d KB for interfaces, %d KB for rings and %d MB for buffers", > + nm_mem.pools[NETMAP_IF_POOL]._memtotal >> 10, > + nm_mem.pools[NETMAP_RING_POOL]._memtotal >> 10, > + nm_mem.pools[NETMAP_BUF_POOL]._memtotal >> 20); > + > +out: > + > + return nm_mem.lasterr; > +} > + > +/* call with lock held */ > +static int > +netmap_memory_finalize(void) > +{ > + int i; > + u_int totalsize = 0; > + > + nm_mem.refcount++; > + if (nm_mem.refcount > 1) { > + ND("busy (refcount %d)", nm_mem.refcount); > + goto out; > + } > + > + /* update configuration if changed */ > + if (netmap_memory_config()) > + goto out; > + > + if (nm_mem.finalized) { > + /* may happen if config is not changed */ > + ND("nothing to do"); > + goto out; > + } > + > + for (i = 0; i < NETMAP_POOLS_NR; i++) { > + nm_mem.lasterr = netmap_finalize_obj_allocator(&nm_mem.pools[i]); > + if (nm_mem.lasterr) > + goto cleanup; > + totalsize += nm_mem.pools[i]._memtotal; > + } > + nm_mem.nm_totalsize = totalsize; > + > + /* backward compatibility */ > + netmap_buf_size = nm_mem.pools[NETMAP_BUF_POOL]._objsize; > + netmap_total_buffers = nm_mem.pools[NETMAP_BUF_POOL].objtotal; > + > + netmap_buffer_lut = nm_mem.pools[NETMAP_BUF_POOL].lut; > + netmap_buffer_base = nm_mem.pools[NETMAP_BUF_POOL].lut[0].vaddr; > + > + nm_mem.finalized = 1; > + nm_mem.lasterr = 0; > + > + /* make sysctl values match actual values in the pools */ > + for (i = 0; i < NETMAP_POOLS_NR; i++) { > + netmap_params[i].size = nm_mem.pools[i]._objsize; > + netmap_params[i].num = nm_mem.pools[i].objtotal; > + } > + > +out: > + if (nm_mem.lasterr) > + nm_mem.refcount--; > + > + return nm_mem.lasterr; > + > +cleanup: > + for (i = 0; i < NETMAP_POOLS_NR; i++) { > + netmap_reset_obj_allocator(&nm_mem.pools[i]); > + } > + nm_mem.refcount--; > + > + return nm_mem.lasterr; > +} > + > +static int > +netmap_memory_init(void) > +{ > + NMA_LOCK_INIT(); > + return (0); > +} > + > +static void > +netmap_memory_fini(void) > +{ > + int i; > + > + for (i = 0; i < NETMAP_POOLS_NR; i++) { > + netmap_destroy_obj_allocator(&nm_mem.pools[i]); > + } > + NMA_LOCK_DESTROY(); > +} > + > +static void > +netmap_free_rings(struct netmap_adapter *na) > +{ > + int i; > + if (!na->tx_rings) > + return; > + for (i = 0; i < na->num_tx_rings + 1; i++) { > + netmap_ring_free(na->tx_rings[i].ring); > + na->tx_rings[i].ring = NULL; > + } > + for (i = 0; i < na->num_rx_rings + 1; i++) { > + netmap_ring_free(na->rx_rings[i].ring); > + na->rx_rings[i].ring = NULL; > + } > + free(na->tx_rings, M_DEVBUF); > + na->tx_rings = na->rx_rings = NULL; > +} > + > + > + > +/* call with NMA_LOCK held */ > +/* > + * Allocate the per-fd structure netmap_if. > + * If this is the first instance, also allocate the krings, rings etc. > + */ > +static void * > +netmap_if_new(const char *ifname, struct netmap_adapter *na) > +{ > + struct netmap_if *nifp; > + struct netmap_ring *ring; > + ssize_t base; /* handy for relative offsets between rings and nifp */ > + u_int i, len, ndesc, ntx, nrx; > + struct netmap_kring *kring; > + > + if (netmap_update_config(na)) { > + /* configuration mismatch, report and fail */ > + return NULL; > + } > + ntx = na->num_tx_rings + 1; /* shorthand, include stack ring */ > + nrx = na->num_rx_rings + 1; /* shorthand, include stack ring */ > + /* > + * the descriptor is followed inline by an array of offsets > + * to the tx and rx rings in the shared memory region. > + */ > + len = sizeof(struct netmap_if) + (nrx + ntx) * sizeof(ssize_t); > + nifp = netmap_if_malloc(len); > + if (nifp == NULL) { > + return NULL; > + } > + > + /* initialize base fields -- override const */ > + *(int *)(uintptr_t)&nifp->ni_tx_rings = na->num_tx_rings; > + *(int *)(uintptr_t)&nifp->ni_rx_rings = na->num_rx_rings; > + strncpy(nifp->ni_name, ifname, IFNAMSIZ); > + > + (na->refcount)++; /* XXX atomic ? we are under lock */ > + if (na->refcount > 1) { /* already setup, we are done */ > + goto final; > + } > + > + len = (ntx + nrx) * sizeof(struct netmap_kring); > + na->tx_rings = malloc(len, M_DEVBUF, M_NOWAIT | M_ZERO); > + if (na->tx_rings == NULL) { > + D("Cannot allocate krings for %s", ifname); > + goto cleanup; > + } > + na->rx_rings = na->tx_rings + ntx; > + > + /* > + * First instance, allocate netmap rings and buffers for this card > + * The rings are contiguous, but have variable size. > + */ > + for (i = 0; i < ntx; i++) { /* Transmit rings */ > + kring = &na->tx_rings[i]; > + ndesc = na->num_tx_desc; > + bzero(kring, sizeof(*kring)); > + len = sizeof(struct netmap_ring) + > + ndesc * sizeof(struct netmap_slot); > + ring = netmap_ring_malloc(len); > + if (ring == NULL) { > + D("Cannot allocate tx_ring[%d] for %s", i, ifname); > + goto cleanup; > + } > + ND("txring[%d] at %p ofs %d", i, ring); > + kring->na = na; > + kring->ring = ring; > + *(int *)(uintptr_t)&ring->num_slots = kring->nkr_num_slots = ndesc; > + *(ssize_t *)(uintptr_t)&ring->buf_ofs = > + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + > + nm_mem.pools[NETMAP_RING_POOL]._memtotal) - > + netmap_ring_offset(ring); > + > + /* > + * IMPORTANT: > + * Always keep one slot empty, so we can detect new > + * transmissions comparing cur and nr_hwcur (they are > + * the same only if there are no new transmissions). > + */ > + ring->avail = kring->nr_hwavail = ndesc - 1; > + ring->cur = kring->nr_hwcur = 0; > + *(int *)(uintptr_t)&ring->nr_buf_size = NETMAP_BUF_SIZE; > + ND("initializing slots for txring[%d]", i); > + if (netmap_new_bufs(nifp, ring->slot, ndesc)) { > + D("Cannot allocate buffers for tx_ring[%d] for %s", i, ifname); > + goto cleanup; > + } > + } > + > + for (i = 0; i < nrx; i++) { /* Receive rings */ > + kring = &na->rx_rings[i]; > + ndesc = na->num_rx_desc; > + bzero(kring, sizeof(*kring)); > + len = sizeof(struct netmap_ring) + > + ndesc * sizeof(struct netmap_slot); > + ring = netmap_ring_malloc(len); > + if (ring == NULL) { > + D("Cannot allocate rx_ring[%d] for %s", i, ifname); > + goto cleanup; > + } > + ND("rxring[%d] at %p ofs %d", i, ring); > + > + kring->na = na; > + kring->ring = ring; > + *(int *)(uintptr_t)&ring->num_slots = kring->nkr_num_slots = ndesc; > + *(ssize_t *)(uintptr_t)&ring->buf_ofs = > + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + > + nm_mem.pools[NETMAP_RING_POOL]._memtotal) - > + netmap_ring_offset(ring); > + > + ring->cur = kring->nr_hwcur = 0; > + ring->avail = kring->nr_hwavail = 0; /* empty */ > + *(int *)(uintptr_t)&ring->nr_buf_size = NETMAP_BUF_SIZE; > + ND("initializing slots for rxring[%d]", i); > + if (netmap_new_bufs(nifp, ring->slot, ndesc)) { > + D("Cannot allocate buffers for rx_ring[%d] for %s", i, ifname); > + goto cleanup; > + } > + } > +#ifdef linux > + // XXX initialize the selrecord structs. > + for (i = 0; i < ntx; i++) > + init_waitqueue_head(&na->tx_rings[i].si); > + for (i = 0; i < nrx; i++) > + init_waitqueue_head(&na->rx_rings[i].si); > + init_waitqueue_head(&na->tx_si); > + init_waitqueue_head(&na->rx_si); > +#endif > +final: > + /* > + * fill the slots for the rx and tx rings. They contain the offset > + * between the ring and nifp, so the information is usable in > + * userspace to reach the ring from the nifp. > + */ > + base = netmap_if_offset(nifp); > + for (i = 0; i < ntx; i++) { > + *(ssize_t *)(uintptr_t)&nifp->ring_ofs[i] = > + netmap_ring_offset(na->tx_rings[i].ring) - base; > + } > + for (i = 0; i < nrx; i++) { > + *(ssize_t *)(uintptr_t)&nifp->ring_ofs[i+ntx] = > + netmap_ring_offset(na->rx_rings[i].ring) - base; > + } > + return (nifp); > +cleanup: > + netmap_free_rings(na); > + netmap_if_free(nifp); > + (na->refcount)--; > + return NULL; > +} > + > +/* call with NMA_LOCK held */ > +static void > +netmap_memory_deref(void) > +{ > + nm_mem.refcount--; > + if (netmap_verbose) > + D("refcount = %d", nm_mem.refcount); > +} > --- a/include/uapi/Kbuild 2013-02-26 10:19:28.000000000 -0800 > +++ b/include/uapi/Kbuild 2013-03-10 10:08:20.327671428 -0700 > @@ -12,3 +12,4 @@ header-y += video/ > header-y += drm/ > header-y += xen/ > header-y += scsi/ > +header-y += netmap/ > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/include/uapi/netmap/Kbuild 2013-03-10 10:08:20.327671428 -0700 > @@ -0,0 +1,3 @@ > +# UAPI Header export list > +header-y += netmap.h > +header-y += netmap_user.h > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/include/uapi/netmap/netmap.h 2013-03-10 10:08:20.327671428 -0700 > @@ -0,0 +1,289 @@ > +/* > + * Copyright (C) 2011 Matteo Landi, Luigi Rizzo. All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions are > + * met: > + * > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the > + * distribution. > + * > + * 3. Neither the name of the authors nor the names of their contributors > + * may be used to endorse or promote products derived from this > + * software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY MATTEO LANDI AND CONTRIBUTORS "AS IS" AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR > + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL MATTEO LANDI OR CONTRIBUTORS > + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS > + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN > + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) > + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF > + * THE POSSIBILITY OF SUCH DAMAGE. > + */ > + > +/* > + * $FreeBSD: head/sys/net/netmap.h 231198 2012-02-08 11:43:29Z luigi $ > + * $Id: netmap.h 10879 2012-04-12 22:48:59Z luigi $ > + * > + * Definitions of constants and the structures used by the netmap > + * framework, for the part visible to both kernel and userspace. > + * Detailed info on netmap is available with "man netmap" or at > + * > + * http://info.iet.unipi.it/~luigi/netmap/ > + */ > + > +#ifndef _NET_NETMAP_H_ > +#define _NET_NETMAP_H_ > + > +/* > + * --- Netmap data structures --- > + * > + * The data structures used by netmap are shown below. Those in > + * capital letters are in an mmapp()ed area shared with userspace, > + * while others are private to the kernel. > + * Shared structures do not contain pointers but only memory > + * offsets, so that addressing is portable between kernel and userspace. > + > + > + softc > ++----------------+ > +| standard fields| > +| if_pspare[0] ----------+ > ++----------------+ | > + | > ++----------------+<------+ > +|(netmap_adapter)| > +| | netmap_kring > +| tx_rings *--------------------------------->+---------------+ > +| | netmap_kring | ring *---------. > +| rx_rings *--------->+---------------+ | nr_hwcur | | > ++----------------+ | ring *--------. | nr_hwavail | V > + | nr_hwcur | | | selinfo | | > + | nr_hwavail | | +---------------+ . > + | selinfo | | | ... | . > + +---------------+ | |(ntx+1 entries)| > + | .... | | | | > + |(nrx+1 entries)| | +---------------+ > + | | | > + KERNEL +---------------+ | > + | > + ==================================================================== > + | > + USERSPACE | NETMAP_RING > + +---->+-------------+ > + / | cur | > + NETMAP_IF (nifp, one per file desc.) / | avail | > + +---------------+ / | buf_ofs | > + | ni_tx_rings | / +=============+ > + | ni_rx_rings | / | buf_idx | slot[0] > + | | / | len, flags | > + | | / +-------------+ > + +===============+ / | buf_idx | slot[1] > + | txring_ofs[0] | (rel.to nifp)--' | len, flags | > + | txring_ofs[1] | +-------------+ > + (num_rings+1 entries) (nr_num_slots entries) > + | txring_ofs[n] | | buf_idx | slot[n-1] > + +---------------+ | len, flags | > + | rxring_ofs[0] | +-------------+ > + | rxring_ofs[1] | > + (num_rings+1 entries) > + | txring_ofs[n] | > + +---------------+ > + > + * The private descriptor ('softc' or 'adapter') of each interface > + * is extended with a "struct netmap_adapter" containing netmap-related > + * info (see description in dev/netmap/netmap_kernel.h. > + * Among other things, tx_rings and rx_rings point to the arrays of > + * "struct netmap_kring" which in turn reache the various > + * "struct netmap_ring", shared with userspace. > + > + * The NETMAP_RING is the userspace-visible replica of the NIC ring. > + * Each slot has the index of a buffer, its length and some flags. > + * In user space, the buffer address is computed as > + * (char *)ring + buf_ofs + index*NETMAP_BUF_SIZE > + * In the kernel, buffers do not necessarily need to be contiguous, > + * and the virtual and physical addresses are derived through > + * a lookup table. > + * To associate a different buffer to a slot, applications must > + * write the new index in buf_idx, and set NS_BUF_CHANGED flag to > + * make sure that the kernel updates the hardware ring as needed. > + * > + * Normally the driver is not requested to report the result of > + * transmissions (this can dramatically speed up operation). > + * However the user may request to report completion by setting > + * NS_REPORT. > + */ > +struct netmap_slot { > + uint32_t buf_idx; /* buffer index */ > + uint16_t len; /* packet length, to be copied to/from the hw ring */ > + uint16_t flags; /* buf changed, etc. */ > +#define NS_BUF_CHANGED 0x0001 /* must resync the map, buffer changed */ > +#define NS_REPORT 0x0002 /* ask the hardware to report results > + * e.g. by generating an interrupt > + */ > +}; > + > +/* > + * Netmap representation of a TX or RX ring (also known as "queue"). > + * This is a queue implemented as a fixed-size circular array. > + * At the software level, two fields are important: avail and cur. > + * > + * In TX rings: > + * avail indicates the number of slots available for transmission. > + * It is updated by the kernel after every netmap system call. > + * It MUST BE decremented by the application when it appends a > + * packet. > + * cur indicates the slot to use for the next packet > + * to send (i.e. the "tail" of the queue). > + * It MUST BE incremented by the application before > + * netmap system calls to reflect the number of newly > + * sent packets. > + * It is checked by the kernel on netmap system calls > + * (normally unmodified by the kernel unless invalid). > + * > + * The kernel side of netmap uses two additional fields in its own > + * private ring structure, netmap_kring: > + * nr_hwcur is a copy of nr_cur on an NIOCTXSYNC. > + * nr_hwavail is the number of slots known as available by the > + * hardware. It is updated on an INTR (inc by the > + * number of packets sent) and on a NIOCTXSYNC > + * (decrease by nr_cur - nr_hwcur) > + * A special case, nr_hwavail is -1 if the transmit > + * side is idle (no pending transmits). > + * > + * In RX rings: > + * avail is the number of packets available (possibly 0). > + * It MUST BE decremented by the application when it consumes > + * a packet, and it is updated to nr_hwavail on a NIOCRXSYNC > + * cur indicates the first slot that contains a packet not > + * processed yet (the "head" of the queue). > + * It MUST BE incremented by the software when it consumes > + * a packet. > + * reserved indicates the number of buffers before 'cur' > + * that the application has still in use. Normally 0, > + * it MUST BE incremented by the application when it > + * does not return the buffer immediately, and decremented > + * when the buffer is finally freed. > + * > + * The kernel side of netmap uses two additional fields in the kring: > + * nr_hwcur is a copy of nr_cur on an NIOCRXSYNC > + * nr_hwavail is the number of packets available. It is updated > + * on INTR (inc by the number of new packets arrived) > + * and on NIOCRXSYNC (decreased by nr_cur - nr_hwcur). > + * > + * DATA OWNERSHIP/LOCKING: > + * The netmap_ring is owned by the user program and it is only > + * accessed or modified in the upper half of the kernel during > + * a system call. > + * > + * The netmap_kring is only modified by the upper half of the kernel. > + */ > +struct netmap_ring { > + /* > + * nr_buf_base_ofs is meant to be used through macros. > + * It contains the offset of the buffer region from this > + * descriptor. > + */ > + const ssize_t buf_ofs; > + const uint32_t num_slots; /* number of slots in the ring. */ > + uint32_t avail; /* number of usable slots */ > + uint32_t cur; /* 'current' r/w position */ > + uint32_t reserved; /* not refilled before current */ > + > + const uint16_t nr_buf_size; > + uint16_t flags; > +#define NR_TIMESTAMP 0x0002 /* set timestamp on *sync() */ > + > + struct timeval ts; /* time of last *sync() */ > + > + /* the slots follow. This struct has variable size */ > + struct netmap_slot slot[0]; /* array of slots. */ > +}; > + > + > +/* > + * Netmap representation of an interface and its queue(s). > + * There is one netmap_if for each file descriptor on which we want > + * to select/poll. We assume that on each interface has the same number > + * of receive and transmit queues. > + * select/poll operates on one or all pairs depending on the value of > + * nmr_queueid passed on the ioctl. > + */ > +struct netmap_if { > + char ni_name[IFNAMSIZ]; /* name of the interface. */ > + const u_int ni_version; /* API version, currently unused */ > + const u_int ni_rx_rings; /* number of rx rings */ > + const u_int ni_tx_rings; /* if zero, same as ni_rx_rings */ > + /* > + * The following array contains the offset of each netmap ring > + * from this structure. The first ni_tx_queues+1 entries refer > + * to the tx rings, the next ni_rx_queues+1 refer to the rx rings > + * (the last entry in each block refers to the host stack rings). > + * The area is filled up by the kernel on NIOCREG, > + * and then only read by userspace code. > + */ > + const ssize_t ring_ofs[0]; > +}; > + > +#ifndef NIOCREGIF > +/* > + * ioctl names and related fields > + * > + * NIOCGINFO takes a struct ifreq, the interface name is the input, > + * the outputs are number of queues and number of descriptor > + * for each queue (useful to set number of threads etc.). > + * > + * NIOCREGIF takes an interface name within a struct ifreq, > + * and activates netmap mode on the interface (if possible). > + * > + * NIOCUNREGIF unregisters the interface associated to the fd. > + * > + * NIOCTXSYNC, NIOCRXSYNC synchronize tx or rx queues, > + * whose identity is set in NIOCREGIF through nr_ringid > + */ > + > +/* > + * struct nmreq overlays a struct ifreq > + */ > +struct nmreq { > + char nr_name[IFNAMSIZ]; > + uint32_t nr_version; /* API version */ > +#define NETMAP_API 3 /* current version */ > + uint32_t nr_offset; /* nifp offset in the shared region */ > + uint32_t nr_memsize; /* size of the shared region */ > + uint32_t nr_tx_slots; /* slots in tx rings */ > + uint32_t nr_rx_slots; /* slots in rx rings */ > + uint16_t nr_tx_rings; /* number of tx rings */ > + uint16_t nr_rx_rings; /* number of rx rings */ > + uint16_t nr_ringid; /* ring(s) we care about */ > +#define NETMAP_HW_RING 0x4000 /* low bits indicate one hw ring */ > +#define NETMAP_SW_RING 0x2000 /* process the sw ring */ > +#define NETMAP_NO_TX_POLL 0x1000 /* no automatic txsync on poll */ > +#define NETMAP_RING_MASK 0xfff /* the ring number */ > + uint16_t spare1; > + uint32_t spare2[4]; > +}; > + > +/* > + * FreeBSD uses the size value embedded in the _IOWR to determine > + * how much to copy in/out. So we need it to match the actual > + * data structure we pass. We put some spares in the structure > + * to ease compatibility with other versions > + */ > +#define NIOCGINFO _IOWR('i', 145, struct nmreq) /* return IF info */ > +#define NIOCREGIF _IOWR('i', 146, struct nmreq) /* interface register */ > +#define NIOCUNREGIF _IO('i', 147) /* interface unregister */ > +#define NIOCTXSYNC _IO('i', 148) /* sync tx queues */ > +#define NIOCRXSYNC _IO('i', 149) /* sync rx queues */ > +#endif /* !NIOCREGIF */ > + > +#endif /* _NET_NETMAP_H_ */ > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/include/uapi/netmap/netmap_user.h 2013-03-10 10:08:20.327671428 -0700 > @@ -0,0 +1,95 @@ > +/* > + * Copyright (C) 2011 Matteo Landi, Luigi Rizzo. All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions are > + * met: > + * > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the > + * distribution. > + * > + * 3. Neither the name of the authors nor the names of their contributors > + * may be used to endorse or promote products derived from this > + * software without specific prior written permission. > + * > + * THIS SOFTWARE IS PROVIDED BY MATTEO LANDI AND CONTRIBUTORS "AS IS" AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR > + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL MATTEO LANDI OR CONTRIBUTORS > + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR > + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF > + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS > + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN > + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) > + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF > + * THE POSSIBILITY OF SUCH DAMAGE. > + */ > + > +/* > + * $FreeBSD: head/sys/net/netmap_user.h 231198 2012-02-08 11:43:29Z luigi $ > + * $Id: netmap_user.h 10879 2012-04-12 22:48:59Z luigi $ > + * > + * This header contains the macros used to manipulate netmap structures > + * and packets in userspace. See netmap(4) for more information. > + * > + * The address of the struct netmap_if, say nifp, is computed from the > + * value returned from ioctl(.., NIOCREG, ...) and the mmap region: > + * ioctl(fd, NIOCREG, &req); > + * mem = mmap(0, ... ); > + * nifp = NETMAP_IF(mem, req.nr_nifp); > + * (so simple, we could just do it manually) > + * > + * From there: > + * struct netmap_ring *NETMAP_TXRING(nifp, index) > + * struct netmap_ring *NETMAP_RXRING(nifp, index) > + * we can access ring->nr_cur, ring->nr_avail, ring->nr_flags > + * > + * ring->slot[i] gives us the i-th slot (we can access > + * directly plen, flags, bufindex) > + * > + * char *buf = NETMAP_BUF(ring, index) returns a pointer to > + * the i-th buffer > + * > + * Since rings are circular, we have macros to compute the next index > + * i = NETMAP_RING_NEXT(ring, i); > + */ > + > +#ifndef _NET_NETMAP_USER_H_ > +#define _NET_NETMAP_USER_H_ > + > +#define NETMAP_IF(b, o) (struct netmap_if *)((char *)(b) + (o)) > + > +#define NETMAP_TXRING(nifp, index) \ > + ((struct netmap_ring *)((char *)(nifp) + \ > + (nifp)->ring_ofs[index] ) ) > + > +#define NETMAP_RXRING(nifp, index) \ > + ((struct netmap_ring *)((char *)(nifp) + \ > + (nifp)->ring_ofs[index + (nifp)->ni_tx_rings + 1] ) ) > + > +#define NETMAP_BUF(ring, index) \ > + ((char *)(ring) + (ring)->buf_ofs + ((index)*(ring)->nr_buf_size)) > + > +#define NETMAP_BUF_IDX(ring, buf) \ > + ( ((char *)(buf) - ((char *)(ring) + (ring)->buf_ofs) ) / \ > + (ring)->nr_buf_size ) > + > +#define NETMAP_RING_NEXT(r, i) \ > + ((i)+1 == (r)->num_slots ? 0 : (i) + 1 ) > + > +#define NETMAP_RING_FIRST_RESERVED(r) \ > + ( (r)->cur < (r)->reserved ? \ > + (r)->cur + (r)->num_slots - (r)->reserved : \ > + (r)->cur - (r)->reserved ) > + > +/* > + * Return 1 if the given tx ring is empty. > + */ > +#define NETMAP_TX_RING_EMPTY(r) ((r)->avail >= (r)->num_slots - 1) > + > +#endif /* _NET_NETMAP_USER_H_ */ > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/include/netmap/bsd_glue.h 2013-03-10 10:08:20.327671428 -0700 > @@ -0,0 +1,263 @@ > +/* > + * (C) 2012 Luigi Rizzo - Universita` di Pisa > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > + > +/* > + * glue code to build the netmap bsd code under linux. > + * Some of these tweaks are generic, some are specific for > + * character device drivers and network code/device drivers. > + */ > + > +#ifndef _BSD_GLUE_H > +#define _BSD_GLUE_H > + > +/* a set of headers used in netmap */ > +#include <linux/version.h> > +#include <linux/if.h> > +#include <linux/list.h> > +#include <linux/mutex.h> > +#include <linux/types.h> > +#include <linux/time.h> > +#include <linux/mm.h> > +#include <linux/poll.h> > +#include <linux/netdevice.h> > +#include <linux/sched.h> > +#include <linux/wait.h> > +#include <linux/miscdevice.h> > +//#include <linux/log2.h> // ilog2 > +#include <linux/etherdevice.h> // eth_type_trans > +#include <linux/module.h> > +#include <linux/moduleparam.h> > +#include <linux/virtio.h> // virt_to_phys > + > +#define printf(fmt, arg...) printk(KERN_ERR fmt, ##arg) > +#define KASSERT(a, b) BUG_ON(!(a)) > + > +/* Type redefinitions. XXX check them */ > +typedef void * bus_dma_tag_t; > +typedef void * bus_dmamap_t; > +typedef int bus_size_t; > +typedef int bus_dma_segment_t; > +typedef void * bus_addr_t; > +#define vm_paddr_t phys_addr_t > +/* XXX the 'off_t' on Linux corresponds to a 'long' */ > +#define vm_offset_t uint32_t > +struct thread; > + > +/* endianness macros/functions */ > +#define le16toh le16_to_cpu > +#define le32toh le32_to_cpu > +#define le64toh le64_to_cpu > +#define be64toh be64_to_cpu > +#define htole32 cpu_to_le32 > +#define htole64 cpu_to_le64 > + > +#include <linux/jiffies.h> > +#define time_second (jiffies_to_msecs(jiffies) / 1000U ) > + > +#define bzero(a, len) memset(a, 0, len) > +#define bcopy(_s, _d, len) memcpy(_d, _s, len) > + > + > +// XXX maybe implement it as a proper function somewhere > +// it is important to set s->len before the copy. > +#define m_devget(_buf, _len, _ofs, _dev, _fn) ( { \ > + struct sk_buff *s = netdev_alloc_skb(_dev, _len); \ > + if (s) { \ > + s->len += _len; \ > + skb_copy_to_linear_data_offset(s, _ofs, _buf, _len); \ > + s->protocol = eth_type_trans(s, _dev); \ > + } \ > + s; } ) > + > +#define mbuf sk_buff > +#define m_nextpkt next // chain of mbufs > +#define m_freem(m) dev_kfree_skb_any(m) // free a sk_buff > + > +/* > + * m_copydata() copies from mbuf to buffer following the mbuf chain. > + * XXX check which linux equivalent we should use to follow fragmented > + * skbufs. > + */ > + > +//#define m_copydata(m, o, l, b) skb_copy_bits(m, o, b, l) > +#define m_copydata(m, o, l, b) skb_copy_from_linear_data_offset(m, o, b, l) > + > +/* > + * struct ifnet is remapped into struct net_device on linux. > + * ifnet has an if_softc field pointing to the device-specific struct > + * (adapter). > + * On linux the ifnet/net_device is at the beginning of the device-specific > + * structure, so a pointer to the first field of the ifnet works. > + * We don't use this in netmap, though. > + * > + * if_xname name device name > + * if_capabilities flags // XXX not used > + * if_capenable priv_flags > + * we would use "features" but it is all taken. > + * XXX check for conflict in flags use. > + * > + * if_bridge atalk_ptr struct nm_bridge (only for VALE ports) > + * > + * In netmap we use if_pspare[0] to point to the netmap_adapter, > + * in linux we have no spares so we overload ax25_ptr, and the detection > + * for netmap-capable is some magic in the area pointed by that. > + */ > +#define WNA(_ifp) (_ifp)->ax25_ptr > + > +#define ifnet net_device /* remap */ > +#define if_xname name /* field ifnet-> net_device */ > +//#define if_capabilities flags /* IFCAP_NETMAP */ > +#define if_capenable priv_flags /* IFCAP_NETMAP */ > +#define if_bridge atalk_ptr /* remap, only for VALE ports */ > +#define ifunit_ref(_x) dev_get_by_name(&init_net, _x); > +#define if_rele(ifp) dev_put(ifp) > +#define CURVNET_SET(x) > +#define CURVNET_RESTORE(x) > + > + > +/* > + * We use spin_lock_irqsave() because we use the lock in the > + * (hard) interrupt context. > + */ > +typedef struct { > + spinlock_t sl; > + ulong flags; > +} safe_spinlock_t; > + > +static inline void mtx_lock(safe_spinlock_t *m) > +{ > + spin_lock_irqsave(&(m->sl), m->flags); > +} > + > +static inline void mtx_unlock(safe_spinlock_t *m) > +{ > + ulong flags = ACCESS_ONCE(m->flags); > + spin_unlock_irqrestore(&(m->sl), flags); > +} > + > +#define mtx_init(a, b, c, d) spin_lock_init(&((a)->sl)) > +#define mtx_destroy(a) // XXX spin_lock_destroy(a) > + > +/* use volatile to fix a probable compiler error on 2.6.25 */ > +#define malloc(_size, type, flags) \ > + ({ volatile int _v = _size; kmalloc(_v, GFP_ATOMIC | __GFP_ZERO); }) > + > +#define free(a, t) kfree(a) > + > +// XXX do we need GPF_ZERO ? > +// XXX do we need GFP_DMA for slots ? > +// http://www.mjmwired.net/kernel/Documentation/DMA-API.txt > + > +#define contigmalloc(sz, ty, flags, a, b, pgsz, c) \ > + (char *) __get_free_pages(GFP_ATOMIC | __GFP_ZERO, \ > + ilog2(roundup_pow_of_two((sz)/PAGE_SIZE))) > +#define contigfree(va, sz, ty) free_pages((unsigned long)va, \ > + ilog2(roundup_pow_of_two(sz)/PAGE_SIZE)) > + > +#define vtophys virt_to_phys > + > +/*--- selrecord and friends ---*/ > +/* wake_up() or wake_up_interruptible() ? */ > +#define selwakeuppri(sw, pri) wake_up(sw) > +#define selrecord(x, y) poll_wait((struct file *)x, y, pwait) > +#define knlist_destroy(x) // XXX todo > + > +/* we use tsleep/wakeup to sleep a bit. */ > +#define tsleep(a, b, c, t) msleep(10) // XXX > +#define wakeup(sw) // XXX double check > +#define microtime do_gettimeofday > + > + > +/* > + * The following trick is to map a struct cdev into a struct miscdevice > + */ > +#define cdev miscdevice > + > + > +/* > + * XXX to complete - the dmamap interface > + */ > +#define BUS_DMA_NOWAIT 0 > +#define bus_dmamap_load(_1, _2, _3, _4, _5, _6, _7) > +#define bus_dmamap_unload(_1, _2) > + > +typedef int (d_mmap_t)(struct file *f, struct vm_area_struct *vma); > +typedef unsigned int (d_poll_t)(struct file * file, struct poll_table_struct *pwait); > + > +/* > + * make_dev will set an error and return the first argument. > + * This relies on the availability of the 'error' local variable. > + */ > +#define make_dev(_cdev, _zero, _uid, _gid, _perm, _name) \ > + ({error = misc_register(_cdev); \ > + D("run mknod /dev/%s c %d %d # error %d", \ > + (_cdev)->name, MISC_MAJOR, (_cdev)->minor, error); \ > + _cdev; } ) > +#define destroy_dev(_cdev) misc_deregister(_cdev) > + > +/*--- sysctl API ----*/ > +/* > + * linux: sysctl are mapped into /sys/module/ipfw_mod parameters > + * windows: they are emulated via get/setsockopt > + */ > +#define CTLFLAG_RD 1 > +#define CTLFLAG_RW 2 > + > +struct sysctl_oid; > +struct sysctl_req; > + > + > +#define SYSCTL_DECL(_1) > +#define SYSCTL_OID(_1, _2, _3, _4, _5, _6, _7, _8) > +#define SYSCTL_NODE(_1, _2, _3, _4, _5, _6) > +#define _SYSCTL_BASE(_name, _var, _ty, _perm) \ > + module_param_named(_name, *(_var), _ty, \ > + ( (_perm) == CTLFLAG_RD) ? 0444: 0644 ) > + > +#define SYSCTL_PROC(_base, _oid, _name, _mode, _var, _val, _desc, _a, _b) > + > +#define SYSCTL_INT(_base, _oid, _name, _mode, _var, _val, _desc) \ > + _SYSCTL_BASE(_name, _var, int, _mode) > + > +#define SYSCTL_LONG(_base, _oid, _name, _mode, _var, _val, _desc) \ > + _SYSCTL_BASE(_name, _var, long, _mode) > + > +#define SYSCTL_ULONG(_base, _oid, _name, _mode, _var, _val, _desc) \ > + _SYSCTL_BASE(_name, _var, ulong, _mode) > + > +#define SYSCTL_UINT(_base, _oid, _name, _mode, _var, _val, _desc) \ > + _SYSCTL_BASE(_name, _var, uint, _mode) > + > +#define TUNABLE_INT(_name, _ptr) > + > +#define SYSCTL_VNET_PROC SYSCTL_PROC > +#define SYSCTL_VNET_INT SYSCTL_INT > + > +#define SYSCTL_HANDLER_ARGS \ > + struct sysctl_oid *oidp, void *arg1, int arg2, struct sysctl_req *req > +int sysctl_handle_int(SYSCTL_HANDLER_ARGS); > +int sysctl_handle_long(SYSCTL_HANDLER_ARGS); > + > +#endif /* _BSD_GLUE_H */ > --- /dev/null 1970-01-01 00:00:00.000000000 +0000 > +++ b/include/netmap/netmap_kern.h 2013-03-10 11:30:37.253528570 -0700 > @@ -0,0 +1,474 @@ > +/* > + * Copyright (C) 2011-2012 Matteo Landi, Luigi Rizzo. All rights reserved. > + * > + * Redistribution and use in source and binary forms, with or without > + * modification, are permitted provided that the following conditions > + * are met: > + * 1. Redistributions of source code must retain the above copyright > + * notice, this list of conditions and the following disclaimer. > + * 2. Redistributions in binary form must reproduce the above copyright > + * notice, this list of conditions and the following disclaimer in the > + * documentation and/or other materials provided with the distribution. > + * > + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND > + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE > + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE > + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE > + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL > + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS > + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) > + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY > + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF > + * SUCH DAMAGE. > + */ > + > +/* > + * $FreeBSD: head/sys/dev/netmap/netmap_kern.h 238985 2012-08-02 11:59:43Z luigi $ > + * $Id: netmap_kern.h 12089 2013-02-15 01:20:25Z luigi $ > + * > + * The header contains the definitions of constants and function > + * prototypes used only in kernelspace. > + */ > + > +#ifndef _NET_NETMAP_KERN_H_ > +#define _NET_NETMAP_KERN_H_ > + > +#define NETMAP_MEM2 // use the new memory allocator > + > +#if defined(__FreeBSD__) > +#define likely(x) __builtin_expect(!!(x), 1) > +#define unlikely(x) __builtin_expect(!!(x), 0) > + > +#define NM_LOCK_T struct mtx > +#define NM_SELINFO_T struct selinfo > +#define MBUF_LEN(m) ((m)->m_pkthdr.len) > +#define NM_SEND_UP(ifp, m) ((ifp)->if_input)(ifp, m) > +#elif defined (linux) > + > +#define NM_LOCK_T safe_spinlock_t // see bsd_glue.h > +#define NM_SELINFO_T wait_queue_head_t > +#define MBUF_LEN(m) ((m)->len) > +#define NM_SEND_UP(ifp, m) netif_rx(m) > + > +#ifndef DEV_NETMAP > +#define DEV_NETMAP > +#endif > + > +/* > + * IFF_NETMAP goes into net_device's priv_flags (if_capenable). > + * Map it to BSD style cap until this driver is cleaned up. > + */ > +#define IFCAP_NETMAP IFF_NETMAP > + > + > +#elif defined (__APPLE__) > +#warning apple support is incomplete. > +#define likely(x) __builtin_expect(!!(x), 1) > +#define unlikely(x) __builtin_expect(!!(x), 0) > +#define NM_LOCK_T IOLock * > +#define NM_SELINFO_T struct selinfo > +#define MBUF_LEN(m) ((m)->m_pkthdr.len) > +#define NM_SEND_UP(ifp, m) ((ifp)->if_input)(ifp, m) > + > +#else > +#error unsupported platform > +#endif > + > +#define ND(format, ...) > +#define D(format, ...) \ > + do { \ > + struct timeval __xxts; \ > + microtime(&__xxts); \ > + printf("%03d.%06d %s [%d] " format "\n", \ > + (int)__xxts.tv_sec % 1000, (int)__xxts.tv_usec, \ > + __FUNCTION__, __LINE__, ##__VA_ARGS__); \ > + } while (0) > + > +/* rate limited, lps indicates how many per second */ > +#define RD(lps, format, ...) \ > + do { \ > + static int t0, __cnt; \ > + if (t0 != time_second) { \ > + t0 = time_second; \ > + __cnt = 0; \ > + } \ > + if (__cnt++ < lps) \ > + D(format, ##__VA_ARGS__); \ > + } while (0) > + > +struct netmap_adapter; > + > +/* > + * private, kernel view of a ring. Keeps track of the status of > + * a ring across system calls. > + * > + * nr_hwcur index of the next buffer to refill. > + * It corresponds to ring->cur - ring->reserved > + * > + * nr_hwavail the number of slots "owned" by userspace. > + * nr_hwavail =:= ring->avail + ring->reserved > + * > + * The indexes in the NIC and netmap rings are offset by nkr_hwofs slots. > + * This is so that, on a reset, buffers owned by userspace are not > + * modified by the kernel. In particular: > + * RX rings: the next empty buffer (hwcur + hwavail + hwofs) coincides with > + * the next empty buffer as known by the hardware (next_to_check or so). > + * TX rings: hwcur + hwofs coincides with next_to_send > + * > + * For received packets, slot->flags is set to nkr_slot_flags > + * so we can provide a proper initial value (e.g. set NS_FORWARD > + * when operating in 'transparent' mode). > + */ > +struct netmap_kring { > + struct netmap_ring *ring; > + u_int nr_hwcur; > + int nr_hwavail; > + u_int nr_kflags; /* private driver flags */ > +#define NKR_PENDINTR 0x1 // Pending interrupt. > + u_int nkr_num_slots; > + > + uint16_t nkr_slot_flags; /* initial value for flags */ > + int nkr_hwofs; /* offset between NIC and netmap ring */ > + struct netmap_adapter *na; > + NM_SELINFO_T si; /* poll/select wait queue */ > + NM_LOCK_T q_lock; /* used if no device lock available */ > +} __attribute__((__aligned__(64))); > + > +/* > + * This struct extends the 'struct adapter' (or > + * equivalent) device descriptor. It contains all fields needed to > + * support netmap operation. > + */ > +struct netmap_adapter { > + /* > + * On linux we do not have a good way to tell if an interface > + * is netmap-capable. So we use the following trick: > + * NA(ifp) points here, and the first entry (which hopefully > + * always exists and is at least 32 bits) contains a magic > + * value which we can use to detect that the interface is good. > + */ > + uint32_t magic; > + uint32_t na_flags; /* future place for IFCAP_NETMAP */ > +#define NAF_SKIP_INTR 1 /* use the regular interrupt handler. > + * useful during initialization > + */ > + int refcount; /* number of user-space descriptors using this > + interface, which is equal to the number of > + struct netmap_if objs in the mapped region. */ > + /* > + * The selwakeup in the interrupt thread can use per-ring > + * and/or global wait queues. We track how many clients > + * of each type we have so we can optimize the drivers, > + * and especially avoid huge contention on the locks. > + */ > + int na_single; /* threads attached to a single hw queue */ > + int na_multi; /* threads attached to multiple hw queues */ > + > + int separate_locks; /* set if the interface suports different > + locks for rx, tx and core. */ > + > + u_int num_rx_rings; /* number of adapter receive rings */ > + u_int num_tx_rings; /* number of adapter transmit rings */ > + > + u_int num_tx_desc; /* number of descriptor in each queue */ > + u_int num_rx_desc; > + > + /* tx_rings and rx_rings are private but allocated > + * as a contiguous chunk of memory. Each array has > + * N+1 entries, for the adapter queues and for the host queue. > + */ > + struct netmap_kring *tx_rings; /* array of TX rings. */ > + struct netmap_kring *rx_rings; /* array of RX rings. */ > + > + NM_SELINFO_T tx_si, rx_si; /* global wait queues */ > + > + /* copy of if_qflush and if_transmit pointers, to intercept > + * packets from the network stack when netmap is active. > + */ > + int (*if_transmit)(struct ifnet *, struct mbuf *); > + > + /* references to the ifnet and device routines, used by > + * the generic netmap functions. > + */ > + struct ifnet *ifp; /* adapter is ifp->if_softc */ > + > + NM_LOCK_T core_lock; /* used if no device lock available */ > + > + int (*nm_register)(struct ifnet *, int onoff); > + void (*nm_lock)(struct ifnet *, int what, u_int ringid); > + int (*nm_txsync)(struct ifnet *, u_int ring, int lock); > + int (*nm_rxsync)(struct ifnet *, u_int ring, int lock); > + /* return configuration information */ > + int (*nm_config)(struct ifnet *, u_int *txr, u_int *txd, > + u_int *rxr, u_int *rxd); > + > + int bdg_port; > +#ifdef linux > + struct net_device_ops nm_ndo; > + int if_refcount; // XXX additions for bridge > +#endif /* linux */ > +}; > + > +/* > + * The combination of "enable" (ifp->if_capenable & IFCAP_NETMAP) > + * and refcount gives the status of the interface, namely: > + * > + * enable refcount Status > + * > + * FALSE 0 normal operation > + * FALSE != 0 -- (impossible) > + * TRUE 1 netmap mode > + * TRUE 0 being deleted. > + */ > + > +#define NETMAP_DELETING(_na) ( ((_na)->refcount == 0) && \ > + ( (_na)->ifp->if_capenable & IFCAP_NETMAP) ) > + > +/* > + * parameters for (*nm_lock)(adapter, what, index) > + */ > +enum { > + NETMAP_NO_LOCK = 0, > + NETMAP_CORE_LOCK, NETMAP_CORE_UNLOCK, > + NETMAP_TX_LOCK, NETMAP_TX_UNLOCK, > + NETMAP_RX_LOCK, NETMAP_RX_UNLOCK, > +#ifdef __FreeBSD__ > +#define NETMAP_REG_LOCK NETMAP_CORE_LOCK > +#define NETMAP_REG_UNLOCK NETMAP_CORE_UNLOCK > +#else > + NETMAP_REG_LOCK, NETMAP_REG_UNLOCK > +#endif > +}; > + > +/* > + * The following are support routines used by individual drivers to > + * support netmap operation. > + * > + * netmap_attach() initializes a struct netmap_adapter, allocating the > + * struct netmap_ring's and the struct selinfo. > + * > + * netmap_detach() frees the memory allocated by netmap_attach(). > + * > + * netmap_start() replaces the if_transmit routine of the interface, > + * and is used to intercept packets coming from the stack. > + * > + * netmap_load_map/netmap_reload_map are helper routines to set/reset > + * the dmamap for a packet buffer > + * > + * netmap_reset() is a helper routine to be called in the driver > + * when reinitializing a ring. > + */ > +int netmap_attach(struct netmap_adapter *, int); > +void netmap_detach(struct ifnet *); > +int netmap_start(struct ifnet *, struct mbuf *); > +enum txrx { NR_RX = 0, NR_TX = 1 }; > +struct netmap_slot *netmap_reset(struct netmap_adapter *na, > + enum txrx tx, int n, u_int new_cur); > +int netmap_ring_reinit(struct netmap_kring *); > + > +extern u_int netmap_buf_size; > +#define NETMAP_BUF_SIZE netmap_buf_size > +extern int netmap_mitigate; > +extern int netmap_no_pendintr; > +extern u_int netmap_total_buffers; > +extern char *netmap_buffer_base; > +extern int netmap_verbose; // XXX debugging > +enum { /* verbose flags */ > + NM_VERB_ON = 1, /* generic verbose */ > + NM_VERB_HOST = 0x2, /* verbose host stack */ > + NM_VERB_RXSYNC = 0x10, /* verbose on rxsync/txsync */ > + NM_VERB_TXSYNC = 0x20, > + NM_VERB_RXINTR = 0x100, /* verbose on rx/tx intr (driver) */ > + NM_VERB_TXINTR = 0x200, > + NM_VERB_NIC_RXSYNC = 0x1000, /* verbose on rx/tx intr (driver) */ > + NM_VERB_NIC_TXSYNC = 0x2000, > +}; > + > +/* > + * NA returns a pointer to the struct netmap adapter from the ifp, > + * WNA is used to write it. > + */ > +#ifndef WNA > +#define WNA(_ifp) (_ifp)->if_pspare[0] > +#endif > +#define NA(_ifp) ((struct netmap_adapter *)WNA(_ifp)) > + > +/* > + * Macros to determine if an interface is netmap capable or netmap enabled. > + * See the magic field in struct netmap_adapter. > + */ > +#ifdef __FreeBSD__ > +/* > + * on FreeBSD just use if_capabilities and if_capenable. > + */ > +#define NETMAP_CAPABLE(ifp) (NA(ifp) && \ > + (ifp)->if_capabilities & IFCAP_NETMAP ) > + > +#define NETMAP_SET_CAPABLE(ifp) \ > + (ifp)->if_capabilities |= IFCAP_NETMAP > + > +#else /* linux */ > + > +/* > + * on linux: > + * we check if NA(ifp) is set and its first element has a related > + * magic value. The capenable is within the struct netmap_adapter. > + */ > +#define NETMAP_MAGIC 0x52697a7a > + > +#define NETMAP_CAPABLE(ifp) (NA(ifp) && \ > + ((uint32_t)(uintptr_t)NA(ifp) ^ NA(ifp)->magic) == NETMAP_MAGIC ) > + > +#define NETMAP_SET_CAPABLE(ifp) \ > + NA(ifp)->magic = ((uint32_t)(uintptr_t)NA(ifp)) ^ NETMAP_MAGIC > + > +#endif /* linux */ > + > +#ifdef __FreeBSD__ > +/* Callback invoked by the dma machinery after a successfull dmamap_load */ > +static void netmap_dmamap_cb(__unused void *arg, > + __unused bus_dma_segment_t * segs, __unused int nseg, __unused int error) > +{ > +} > + > +/* bus_dmamap_load wrapper: call aforementioned function if map != NULL. > + * XXX can we do it without a callback ? > + */ > +static inline void > +netmap_load_map(bus_dma_tag_t tag, bus_dmamap_t map, void *buf) > +{ > + if (map) > + bus_dmamap_load(tag, map, buf, NETMAP_BUF_SIZE, > + netmap_dmamap_cb, NULL, BUS_DMA_NOWAIT); > +} > + > +/* update the map when a buffer changes. */ > +static inline void > +netmap_reload_map(bus_dma_tag_t tag, bus_dmamap_t map, void *buf) > +{ > + if (map) { > + bus_dmamap_unload(tag, map); > + bus_dmamap_load(tag, map, buf, NETMAP_BUF_SIZE, > + netmap_dmamap_cb, NULL, BUS_DMA_NOWAIT); > + } > +} > +#else /* linux */ > + > +/* > + * XXX How do we redefine these functions: > + * > + * on linux we need > + * dma_map_single(&pdev->dev, virt_addr, len, direction) > + * dma_unmap_single(&adapter->pdev->dev, phys_addr, len, direction > + * The len can be implicit (on netmap it is NETMAP_BUF_SIZE) > + * unfortunately the direction is not, so we need to change > + * something to have a cross API > + */ > +#define netmap_load_map(_t, _m, _b) > +#define netmap_reload_map(_t, _m, _b) > +#if 0 > + struct e1000_buffer *buffer_info = &tx_ring->buffer_info[l]; > + /* set time_stamp *before* dma to help avoid a possible race */ > + buffer_info->time_stamp = jiffies; > + buffer_info->mapped_as_page = false; > + buffer_info->length = len; > + //buffer_info->next_to_watch = l; > + /* reload dma map */ > + dma_unmap_single(&adapter->pdev->dev, buffer_info->dma, > + NETMAP_BUF_SIZE, DMA_TO_DEVICE); > + buffer_info->dma = dma_map_single(&adapter->pdev->dev, > + addr, NETMAP_BUF_SIZE, DMA_TO_DEVICE); > + > + if (dma_mapping_error(&adapter->pdev->dev, buffer_info->dma)) { > + D("dma mapping error"); > + /* goto dma_error; See e1000_put_txbuf() */ > + /* XXX reset */ > + } > + tx_desc->buffer_addr = htole64(buffer_info->dma); //XXX > + > +#endif > + > +/* > + * The bus_dmamap_sync() can be one of wmb() or rmb() depending on direction. > + */ > +#define bus_dmamap_sync(_a, _b, _c) > + > +#endif /* linux */ > + > +/* > + * functions to map NIC to KRING indexes (n2k) and vice versa (k2n) > + */ > +static inline int > +netmap_idx_n2k(struct netmap_kring *kr, int idx) > +{ > + int n = kr->nkr_num_slots; > + idx += kr->nkr_hwofs; > + if (idx < 0) > + return idx + n; > + else if (idx < n) > + return idx; > + else > + return idx - n; > +} > + > + > +static inline int > +netmap_idx_k2n(struct netmap_kring *kr, int idx) > +{ > + int n = kr->nkr_num_slots; > + idx -= kr->nkr_hwofs; > + if (idx < 0) > + return idx + n; > + else if (idx < n) > + return idx; > + else > + return idx - n; > +} > + > + > +#ifdef NETMAP_MEM2 > +/* Entries of the look-up table. */ > +struct lut_entry { > + void *vaddr; /* virtual address. */ > + vm_paddr_t paddr; /* phisical address. */ > +}; > + > +struct netmap_obj_pool; > +extern struct lut_entry *netmap_buffer_lut; > +#define NMB_VA(i) (netmap_buffer_lut[i].vaddr) > +#define NMB_PA(i) (netmap_buffer_lut[i].paddr) > +#else /* NETMAP_MEM1 */ > +#define NMB_VA(i) (netmap_buffer_base + (i * NETMAP_BUF_SIZE) ) > +#endif /* NETMAP_MEM2 */ > + > +/* > + * NMB return the virtual address of a buffer (buffer 0 on bad index) > + * PNMB also fills the physical address > + */ > +static inline void * > +NMB(struct netmap_slot *slot) > +{ > + uint32_t i = slot->buf_idx; > + return (unlikely(i >= netmap_total_buffers)) ? NMB_VA(0) : NMB_VA(i); > +} > + > +static inline void * > +PNMB(struct netmap_slot *slot, uint64_t *pp) > +{ > + uint32_t i = slot->buf_idx; > + void *ret = (i >= netmap_total_buffers) ? NMB_VA(0) : NMB_VA(i); > +#ifdef NETMAP_MEM2 > + *pp = (i >= netmap_total_buffers) ? NMB_PA(0) : NMB_PA(i); > +#else > + *pp = vtophys(ret); > +#endif > + return ret; > +} > + > +/* default functions to handle rx/tx interrupts */ > +int netmap_rx_irq(struct ifnet *, int, int *); > +#define netmap_tx_irq(_n, _q) netmap_rx_irq(_n, _q, NULL) > + > +extern int netmap_copy; > +#endif /* _NET_NETMAP_KERN_H_ */ > --- a/include/uapi/linux/if.h 2013-02-16 09:10:41.000000000 -0800 > +++ b/include/uapi/linux/if.h 2013-03-10 11:26:44.500548075 -0700 > @@ -83,6 +83,7 @@ > #define IFF_SUPP_NOFCS 0x80000 /* device supports sending custom FCS */ > #define IFF_LIVE_ADDR_CHANGE 0x100000 /* device supports hardware address > * change when it's running */ > +#define IFF_NETMAP 0x200000 /* device used with netmap */ > > > #define IF_GET_IFACE 0x0001 /* for querying only */ > -- > To unsubscribe from this list: send the line "unsubscribe netdev" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html

--- a/drivers/staging/Kconfig 2013-02-26 10:19:35.000000000 -0800 +++ b/drivers/staging/Kconfig 2013-03-10 10:08:20.323671490 -0700 @@ -140,4 +140,6 @@ source "drivers/staging/zcache/Kconfig" source "drivers/staging/goldfish/Kconfig" +source "drivers/staging/netmap/Kconfig" + endif # STAGING --- a/drivers/staging/Makefile 2013-02-26 10:19:35.000000000 -0800 +++ b/drivers/staging/Makefile 2013-03-10 10:08:48.555305267 -0700 @@ -62,3 +62,4 @@ obj-$(CONFIG_SB105X) += sb105x/ obj-$(CONFIG_FIREWIRE_SERIAL) += fwserial/ obj-$(CONFIG_ZCACHE) += zcache/ obj-$(CONFIG_GOLDFISH) += goldfish/ +obj-$(CONFIG_NETMAP) += netmap/ --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/drivers/staging/netmap/Kconfig 2013-03-10 10:08:20.323671490 -0700 @@ -0,0 +1,16 @@ +# +# Netmap - user mode packet processing framework +# + +config NETMAP + tristate "Netmap - user mode networking" + depends on NET && !IOMMU_API + default n + help + If you say Y here, you will get experimental support for + netmap a framework for fast packet I/O using memory mapped + buffers. + + See <http:/nfo.iet.unipi.it/~luigi/netmap/> for more information. + + If unsure, say N. --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/drivers/staging/netmap/Makefile 2013-03-10 10:08:20.323671490 -0700 @@ -0,0 +1,2 @@ +EXTRA_CFLAGS := -DDEBUG +obj-$(CONFIG_NETMAP) += netmap.o --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/drivers/staging/netmap/netmap.c 2013-03-10 16:52:34.160592316 -0700 @@ -0,0 +1,2515 @@ +/* + * Copyright (C) 2011-2012 Matteo Landi, Luigi Rizzo. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#define NM_BRIDGE + +/* + * This module supports memory mapped access to network devices, + * see netmap(4). + * + * The module uses a large, memory pool allocated by the kernel + * and accessible as mmapped memory by multiple userspace threads/processes. + * The memory pool contains packet buffers and "netmap rings", + * i.e. user-accessible copies of the interface's queues. + * + * Access to the network card works like this: + * 1. a process/thread issues one or more open() on /dev/netmap, to create + * select()able file descriptor on which events are reported. + * 2. on each descriptor, the process issues an ioctl() to identify + * the interface that should report events to the file descriptor. + * 3. on each descriptor, the process issues an mmap() request to + * map the shared memory region within the process' address space. + * The list of interesting queues is indicated by a location in + * the shared memory region. + * 4. using the functions in the netmap(4) userspace API, a process + * can look up the occupation state of a queue, access memory buffers, + * and retrieve received packets or enqueue packets to transmit. + * 5. using some ioctl()s the process can synchronize the userspace view + * of the queue with the actual status in the kernel. This includes both + * receiving the notification of new packets, and transmitting new + * packets on the output interface. + * 6. select() or poll() can be used to wait for events on individual + * transmit or receive queues (or all queues for a given interface). + */ + +#ifdef linux +#include <netmap/bsd_glue.h> +static netdev_tx_t linux_netmap_start(struct sk_buff *skb, struct net_device *dev); +#endif /* linux */ + +#ifdef __APPLE__ +#include "osx_glue.h" +#endif /* __APPLE__ */ + +#ifdef __FreeBSD__ +#include <sys/cdefs.h> /* prerequisite */ +__FBSDID("$FreeBSD: head/sys/dev/netmap/netmap.c 241723 2012-10-19 09:41:45Z glebius $"); + +#include <sys/types.h> +#include <sys/module.h> +#include <sys/errno.h> +#include <sys/param.h> /* defines used in kernel.h */ +#include <sys/jail.h> +#include <sys/kernel.h> /* types used in module initialization */ +#include <sys/conf.h> /* cdevsw struct */ +#include <sys/uio.h> /* uio struct */ +#include <sys/sockio.h> +#include <sys/socketvar.h> /* struct socket */ +#include <sys/malloc.h> +#include <sys/mman.h> /* PROT_EXEC */ +#include <sys/poll.h> +#include <sys/proc.h> +#include <vm/vm.h> /* vtophys */ +#include <vm/pmap.h> /* vtophys */ +#include <sys/socket.h> /* sockaddrs */ +#include <machine/bus.h> +#include <sys/selinfo.h> +#include <sys/sysctl.h> +#include <net/if.h> +#include <net/bpf.h> /* BIOCIMMEDIATE */ +#include <net/vnet.h> +#include <machine/bus.h> /* bus_dmamap_* */ + +MALLOC_DEFINE(M_NETMAP, "netmap", "Network memory map"); +#endif /* __FreeBSD__ */ + +#include <netmap/netmap.h> +#include <netmap/netmap_kern.h> + +u_int netmap_total_buffers; +u_int netmap_buf_size; +char *netmap_buffer_base; /* address of an invalid buffer */ + +/* user-controlled variables */ +int netmap_verbose = 1; + +static int netmap_no_timestamp; /* don't timestamp on rxsync */ + +SYSCTL_NODE(_dev, OID_AUTO, netmap, CTLFLAG_RW, 0, "Netmap args"); +SYSCTL_INT(_dev_netmap, OID_AUTO, verbose, + CTLFLAG_RW, &netmap_verbose, 0, "Verbose mode"); +SYSCTL_INT(_dev_netmap, OID_AUTO, no_timestamp, + CTLFLAG_RW, &netmap_no_timestamp, 0, "no_timestamp"); +int netmap_mitigate = 1; +SYSCTL_INT(_dev_netmap, OID_AUTO, mitigate, CTLFLAG_RW, &netmap_mitigate, 0, ""); +int netmap_no_pendintr = 1; +SYSCTL_INT(_dev_netmap, OID_AUTO, no_pendintr, + CTLFLAG_RW, &netmap_no_pendintr, 0, "Always look for new received packets."); + +int netmap_drop = 0; /* debugging */ +int netmap_flags = 0; /* debug flags */ +int netmap_fwd = 0; /* force transparent mode */ +int netmap_copy = 0; /* debugging, copy content */ + +SYSCTL_INT(_dev_netmap, OID_AUTO, drop, CTLFLAG_RW, &netmap_drop, 0 , ""); +SYSCTL_INT(_dev_netmap, OID_AUTO, flags, CTLFLAG_RW, &netmap_flags, 0 , ""); +SYSCTL_INT(_dev_netmap, OID_AUTO, fwd, CTLFLAG_RW, &netmap_fwd, 0 , ""); +SYSCTL_INT(_dev_netmap, OID_AUTO, copy, CTLFLAG_RW, &netmap_copy, 0 , ""); + +#ifdef NM_BRIDGE /* support for netmap bridge */ + +/* + * system parameters. + * + * All switched ports have prefix NM_NAME. + * The switch has a max of NM_BDG_MAXPORTS ports (often stored in a bitmap, + * so a practical upper bound is 64). + * Each tx ring is read-write, whereas rx rings are readonly (XXX not done yet). + * The virtual interfaces use per-queue lock instead of core lock. + * In the tx loop, we aggregate traffic in batches to make all operations + * faster. The batch size is NM_BDG_BATCH + */ +#define NM_NAME "vale" /* prefix for the interface */ +#define NM_BDG_MAXPORTS 16 /* up to 64 ? */ +#define NM_BRIDGE_RINGSIZE 1024 /* in the device */ +#define NM_BDG_HASH 1024 /* forwarding table entries */ +#define NM_BDG_BATCH 1024 /* entries in the forwarding buffer */ +#define NM_BRIDGES 4 /* number of bridges */ +int netmap_bridge = NM_BDG_BATCH; /* bridge batch size */ +SYSCTL_INT(_dev_netmap, OID_AUTO, bridge, CTLFLAG_RW, &netmap_bridge, 0 , ""); + +#ifdef linux +#define ADD_BDG_REF(ifp) (NA(ifp)->if_refcount++) +#define DROP_BDG_REF(ifp) (NA(ifp)->if_refcount-- <= 1) +#else /* !linux */ +#define ADD_BDG_REF(ifp) (ifp)->if_refcount++ +#define DROP_BDG_REF(ifp) refcount_release(&(ifp)->if_refcount) +#ifdef __FreeBSD__ +#include <sys/endian.h> +#include <sys/refcount.h> +#endif /* __FreeBSD__ */ +#define prefetch(x) __builtin_prefetch(x) +#endif /* !linux */ + +static void bdg_netmap_attach(struct ifnet *ifp); +static int bdg_netmap_reg(struct ifnet *ifp, int onoff); +/* per-tx-queue entry */ +struct nm_bdg_fwd { /* forwarding entry for a bridge */ + void *buf; + uint64_t dst; /* dst mask */ + uint32_t src; /* src index ? */ + uint16_t len; /* src len */ +}; + +struct nm_hash_ent { + uint64_t mac; /* the top 2 bytes are the epoch */ + uint64_t ports; +}; + +/* + * Interfaces for a bridge are all in ports[]. + * The array has fixed size, an empty entry does not terminate + * the search. + */ +struct nm_bridge { + struct ifnet *bdg_ports[NM_BDG_MAXPORTS]; + int n_ports; + uint64_t act_ports; + int freelist; /* first buffer index */ + NM_SELINFO_T si; /* poll/select wait queue */ + NM_LOCK_T bdg_lock; /* protect the selinfo ? */ + + /* the forwarding table, MAC+ports */ + struct nm_hash_ent ht[NM_BDG_HASH]; + + int namelen; /* 0 means free */ + char basename[IFNAMSIZ]; +}; + +struct nm_bridge nm_bridges[NM_BRIDGES]; + +#define BDG_LOCK(b) mtx_lock(&(b)->bdg_lock) +#define BDG_UNLOCK(b) mtx_unlock(&(b)->bdg_lock) + +/* + * NA(ifp)->bdg_port port index + */ + +// XXX only for multiples of 64 bytes, non overlapped. +static inline void +pkt_copy(void *_src, void *_dst, int l) +{ + uint64_t *src = _src; + uint64_t *dst = _dst; + if (unlikely(l >= 1024)) { + bcopy(src, dst, l); + return; + } + for (; likely(l > 0); l-=64) { + *dst++ = *src++; + *dst++ = *src++; + *dst++ = *src++; + *dst++ = *src++; + *dst++ = *src++; + *dst++ = *src++; + *dst++ = *src++; + *dst++ = *src++; + } +} + +/* + * locate a bridge among the existing ones. + * a ':' in the name terminates the bridge name. Otherwise, just NM_NAME. + * We assume that this is called with a name of at least NM_NAME chars. + */ +static struct nm_bridge * +nm_find_bridge(const char *name) +{ + int i, l, namelen, e; + struct nm_bridge *b = NULL; + + namelen = strlen(NM_NAME); /* base length */ + l = strlen(name); /* actual length */ + for (i = namelen + 1; i < l; i++) { + if (name[i] == ':') { + namelen = i; + break; + } + } + if (namelen >= IFNAMSIZ) + namelen = IFNAMSIZ; + ND("--- prefix is '%.*s' ---", namelen, name); + + /* use the first entry for locking */ + BDG_LOCK(nm_bridges); // XXX do better + for (e = -1, i = 1; i < NM_BRIDGES; i++) { + b = nm_bridges + i; + if (b->namelen == 0) + e = i; /* record empty slot */ + else if (strncmp(name, b->basename, namelen) == 0) { + ND("found '%.*s' at %d", namelen, name, i); + break; + } + } + if (i == NM_BRIDGES) { /* all full */ + if (e == -1) { /* no empty slot */ + b = NULL; + } else { + b = nm_bridges + e; + strncpy(b->basename, name, namelen); + b->namelen = namelen; + } + } + BDG_UNLOCK(nm_bridges); + return b; +} +#endif /* NM_BRIDGE */ + + +/* + * Fetch configuration from the device, to cope with dynamic + * reconfigurations after loading the module. + */ +static int +netmap_update_config(struct netmap_adapter *na) +{ + struct ifnet *ifp = na->ifp; + u_int txr, txd, rxr, rxd; + + txr = txd = rxr = rxd = 0; + if (na->nm_config) { + na->nm_config(ifp, &txr, &txd, &rxr, &rxd); + } else { + /* take whatever we had at init time */ + txr = na->num_tx_rings; + txd = na->num_tx_desc; + rxr = na->num_rx_rings; + rxd = na->num_rx_desc; + } + + if (na->num_tx_rings == txr && na->num_tx_desc == txd && + na->num_rx_rings == rxr && na->num_rx_desc == rxd) + return 0; /* nothing changed */ + if (netmap_verbose || na->refcount > 0) { + D("stored config %s: txring %d x %d, rxring %d x %d", + ifp->if_xname, + na->num_tx_rings, na->num_tx_desc, + na->num_rx_rings, na->num_rx_desc); + D("new config %s: txring %d x %d, rxring %d x %d", + ifp->if_xname, txr, txd, rxr, rxd); + } + if (na->refcount == 0) { + D("configuration changed (but fine)"); + na->num_tx_rings = txr; + na->num_tx_desc = txd; + na->num_rx_rings = rxr; + na->num_rx_desc = rxd; + return 0; + } + D("configuration changed while active, this is bad..."); + return 1; +} + +/*------------- memory allocator -----------------*/ +#ifdef NETMAP_MEM2 +#include "netmap_mem2.c" +#else /* !NETMAP_MEM2 */ +#include "netmap_mem1.c" +#endif /* !NETMAP_MEM2 */ +/*------------ end of memory allocator ----------*/ + + +/* Structure associated to each thread which registered an interface. + * + * The first 4 fields of this structure are written by NIOCREGIF and + * read by poll() and NIOC?XSYNC. + * There is low contention among writers (actually, a correct user program + * should have no contention among writers) and among writers and readers, + * so we use a single global lock to protect the structure initialization. + * Since initialization involves the allocation of memory, we reuse the memory + * allocator lock. + * Read access to the structure is lock free. Readers must check that + * np_nifp is not NULL before using the other fields. + * If np_nifp is NULL initialization has not been performed, so they should + * return an error to userlevel. + * + * The ref_done field is used to regulate access to the refcount in the + * memory allocator. The refcount must be incremented at most once for + * each open("/dev/netmap"). The increment is performed by the first + * function that calls netmap_get_memory() (currently called by + * mmap(), NIOCGINFO and NIOCREGIF). + * If the refcount is incremented, it is then decremented when the + * private structure is destroyed. + */ +struct netmap_priv_d { + struct netmap_if * volatile np_nifp; /* netmap interface descriptor. */ + + struct ifnet *np_ifp; /* device for which we hold a reference */ + int np_ringid; /* from the ioctl */ + u_int np_qfirst, np_qlast; /* range of rings to scan */ + uint16_t np_txpoll; + + unsigned long ref_done; /* use with NMA_LOCK held */ +}; + + +static int +netmap_get_memory(struct netmap_priv_d* p) +{ + int error = 0; + NMA_LOCK(); + if (!p->ref_done) { + error = netmap_memory_finalize(); + if (!error) + p->ref_done = 1; + } + NMA_UNLOCK(); + return error; +} + +/* + * File descriptor's private data destructor. + * + * Call nm_register(ifp,0) to stop netmap mode on the interface and + * revert to normal operation. We expect that np_ifp has not gone. + */ +/* call with NMA_LOCK held */ +static void +netmap_dtor_locked(void *data) +{ + struct netmap_priv_d *priv = data; + struct ifnet *ifp = priv->np_ifp; + struct netmap_adapter *na = NA(ifp); + struct netmap_if *nifp = priv->np_nifp; + + na->refcount--; + if (na->refcount <= 0) { /* last instance */ + u_int i, j, lim; + + if (netmap_verbose) + D("deleting last instance for %s", ifp->if_xname); + /* + * there is a race here with *_netmap_task() and + * netmap_poll(), which don't run under NETMAP_REG_LOCK. + * na->refcount == 0 && na->ifp->if_capenable & IFCAP_NETMAP + * (aka NETMAP_DELETING(na)) are a unique marker that the + * device is dying. + * Before destroying stuff we sleep a bit, and then complete + * the job. NIOCREG should realize the condition and + * loop until they can continue; the other routines + * should check the condition at entry and quit if + * they cannot run. + */ + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); + tsleep(na, 0, "NIOCUNREG", 4); + na->nm_lock(ifp, NETMAP_REG_LOCK, 0); + na->nm_register(ifp, 0); /* off, clear IFCAP_NETMAP */ + /* Wake up any sleeping threads. netmap_poll will + * then return POLLERR + */ + for (i = 0; i < na->num_tx_rings + 1; i++) + selwakeuppri(&na->tx_rings[i].si, PI_NET); + for (i = 0; i < na->num_rx_rings + 1; i++) + selwakeuppri(&na->rx_rings[i].si, PI_NET); + selwakeuppri(&na->tx_si, PI_NET); + selwakeuppri(&na->rx_si, PI_NET); + /* release all buffers */ + for (i = 0; i < na->num_tx_rings + 1; i++) { + struct netmap_ring *ring = na->tx_rings[i].ring; + lim = na->tx_rings[i].nkr_num_slots; + for (j = 0; j < lim; j++) + netmap_free_buf(nifp, ring->slot[j].buf_idx); + /* knlist_destroy(&na->tx_rings[i].si.si_note); */ + mtx_destroy(&na->tx_rings[i].q_lock); + } + for (i = 0; i < na->num_rx_rings + 1; i++) { + struct netmap_ring *ring = na->rx_rings[i].ring; + lim = na->rx_rings[i].nkr_num_slots; + for (j = 0; j < lim; j++) + netmap_free_buf(nifp, ring->slot[j].buf_idx); + /* knlist_destroy(&na->rx_rings[i].si.si_note); */ + mtx_destroy(&na->rx_rings[i].q_lock); + } + /* XXX kqueue(9) needed; these will mirror knlist_init. */ + /* knlist_destroy(&na->tx_si.si_note); */ + /* knlist_destroy(&na->rx_si.si_note); */ + netmap_free_rings(na); + wakeup(na); + } + netmap_if_free(nifp); +} + +static void +nm_if_rele(struct ifnet *ifp) +{ +#ifndef NM_BRIDGE + if_rele(ifp); +#else /* NM_BRIDGE */ + int i, full; + struct nm_bridge *b; + + if (strncmp(ifp->if_xname, NM_NAME, sizeof(NM_NAME) - 1)) { + if_rele(ifp); + return; + } + if (!DROP_BDG_REF(ifp)) + return; + b = ifp->if_bridge; + BDG_LOCK(nm_bridges); + BDG_LOCK(b); + ND("want to disconnect %s from the bridge", ifp->if_xname); + full = 0; + for (i = 0; i < NM_BDG_MAXPORTS; i++) { + if (b->bdg_ports[i] == ifp) { + b->bdg_ports[i] = NULL; + bzero(ifp, sizeof(*ifp)); + free(ifp, M_DEVBUF); + break; + } + else if (b->bdg_ports[i] != NULL) + full = 1; + } + BDG_UNLOCK(b); + if (full == 0) { + ND("freeing bridge %d", b - nm_bridges); + b->namelen = 0; + } + BDG_UNLOCK(nm_bridges); + if (i == NM_BDG_MAXPORTS) + D("ouch, cannot find ifp to remove"); +#endif /* NM_BRIDGE */ +} + +static void +netmap_dtor(void *data) +{ + struct netmap_priv_d *priv = data; + struct ifnet *ifp = priv->np_ifp; + struct netmap_adapter *na; + + NMA_LOCK(); + if (ifp) { + na = NA(ifp); + na->nm_lock(ifp, NETMAP_REG_LOCK, 0); + netmap_dtor_locked(data); + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); + + nm_if_rele(ifp); + } + if (priv->ref_done) { + netmap_memory_deref(); + } + NMA_UNLOCK(); + bzero(priv, sizeof(*priv)); /* XXX for safety */ + free(priv, M_DEVBUF); +} + +#ifdef __FreeBSD__ +#include <vm/vm.h> +#include <vm/vm_param.h> +#include <vm/vm_object.h> +#include <vm/vm_page.h> +#include <vm/vm_pager.h> +#include <vm/uma.h> + +static struct cdev_pager_ops saved_cdev_pager_ops; + +static int +netmap_dev_pager_ctor(void *handle, vm_ooffset_t size, vm_prot_t prot, + vm_ooffset_t foff, struct ucred *cred, u_short *color) +{ + if (netmap_verbose) + D("first mmap for %p", handle); + return saved_cdev_pager_ops.cdev_pg_ctor(handle, + size, prot, foff, cred, color); +} + +static void +netmap_dev_pager_dtor(void *handle) +{ + saved_cdev_pager_ops.cdev_pg_dtor(handle); + ND("ready to release memory for %p", handle); +} + + +static struct cdev_pager_ops netmap_cdev_pager_ops = { + .cdev_pg_ctor = netmap_dev_pager_ctor, + .cdev_pg_dtor = netmap_dev_pager_dtor, + .cdev_pg_fault = NULL, +}; + +static int +netmap_mmap_single(struct cdev *cdev, vm_ooffset_t *foff, + vm_size_t objsize, vm_object_t *objp, int prot) +{ + vm_object_t obj; + + ND("cdev %p foff %jd size %jd objp %p prot %d", cdev, + (intmax_t )*foff, (intmax_t )objsize, objp, prot); + obj = vm_pager_allocate(OBJT_DEVICE, cdev, objsize, prot, *foff, + curthread->td_ucred); + ND("returns obj %p", obj); + if (obj == NULL) + return EINVAL; + if (saved_cdev_pager_ops.cdev_pg_fault == NULL) { + ND("initialize cdev_pager_ops"); + saved_cdev_pager_ops = *(obj->un_pager.devp.ops); + netmap_cdev_pager_ops.cdev_pg_fault = + saved_cdev_pager_ops.cdev_pg_fault; + }; + obj->un_pager.devp.ops = &netmap_cdev_pager_ops; + *objp = obj; + return 0; +} +#endif /* __FreeBSD__ */ + + +/* + * mmap(2) support for the "netmap" device. + * + * Expose all the memory previously allocated by our custom memory + * allocator: this way the user has only to issue a single mmap(2), and + * can work on all the data structures flawlessly. + * + * Return 0 on success, -1 otherwise. + */ + +#ifdef __FreeBSD__ +static int +netmap_mmap(__unused struct cdev *dev, +#if __FreeBSD_version < 900000 + vm_offset_t offset, vm_paddr_t *paddr, int nprot +#else + vm_ooffset_t offset, vm_paddr_t *paddr, int nprot, + __unused vm_memattr_t *memattr +#endif + ) +{ + int error = 0; + struct netmap_priv_d *priv; + + if (nprot & PROT_EXEC) + return (-1); // XXX -1 or EINVAL ? + + error = devfs_get_cdevpriv((void **)&priv); + if (error == EBADF) { /* called on fault, memory is initialized */ + ND(5, "handling fault at ofs 0x%x", offset); + error = 0; + } else if (error == 0) /* make sure memory is set */ + error = netmap_get_memory(priv); + if (error) + return (error); + + ND("request for offset 0x%x", (uint32_t)offset); + *paddr = netmap_ofstophys(offset); + + return (*paddr ? 0 : ENOMEM); +} + +static int +netmap_close(struct cdev *dev, int fflag, int devtype, struct thread *td) +{ + if (netmap_verbose) + D("dev %p fflag 0x%x devtype %d td %p", + dev, fflag, devtype, td); + return 0; +} + +static int +netmap_open(struct cdev *dev, int oflags, int devtype, struct thread *td) +{ + struct netmap_priv_d *priv; + int error; + + priv = malloc(sizeof(struct netmap_priv_d), M_DEVBUF, + M_NOWAIT | M_ZERO); + if (priv == NULL) + return ENOMEM; + + error = devfs_set_cdevpriv(priv, netmap_dtor); + if (error) + return error; + + return 0; +} +#endif /* __FreeBSD__ */ + + +/* + * Handlers for synchronization of the queues from/to the host. + * Netmap has two operating modes: + * - in the default mode, the rings connected to the host stack are + * just another ring pair managed by userspace; + * - in transparent mode (XXX to be defined) incoming packets + * (from the host or the NIC) are marked as NS_FORWARD upon + * arrival, and the user application has a chance to reset the + * flag for packets that should be dropped. + * On the RXSYNC or poll(), packets in RX rings between + * kring->nr_kcur and ring->cur with NS_FORWARD still set are moved + * to the other side. + * The transfer NIC --> host is relatively easy, just encapsulate + * into mbufs and we are done. The host --> NIC side is slightly + * harder because there might not be room in the tx ring so it + * might take a while before releasing the buffer. + */ + +/* + * pass a chain of buffers to the host stack as coming from 'dst' + */ +static void +netmap_send_up(struct ifnet *dst, struct mbuf *head) +{ + struct mbuf *m; + + /* send packets up, outside the lock */ + while ((m = head) != NULL) { + head = head->m_nextpkt; + m->m_nextpkt = NULL; + if (netmap_verbose & NM_VERB_HOST) + D("sending up pkt %p size %d", m, MBUF_LEN(m)); + NM_SEND_UP(dst, m); + } +} + +struct mbq { + struct mbuf *head; + struct mbuf *tail; + int count; +}; + +/* + * put a copy of the buffers marked NS_FORWARD into an mbuf chain. + * Run from hwcur to cur - reserved + */ +static void +netmap_grab_packets(struct netmap_kring *kring, struct mbq *q, int force) +{ + /* Take packets from hwcur to cur-reserved and pass them up. + * In case of no buffers we give up. At the end of the loop, + * the queue is drained in all cases. + * XXX handle reserved + */ + int k = kring->ring->cur - kring->ring->reserved; + u_int n, lim = kring->nkr_num_slots - 1; + struct mbuf *m, *tail = q->tail; + + if (k < 0) + k = k + kring->nkr_num_slots; + for (n = kring->nr_hwcur; n != k;) { + struct netmap_slot *slot = &kring->ring->slot[n]; + + n = (n == lim) ? 0 : n + 1; + if ((slot->flags & NS_FORWARD) == 0 && !force) + continue; + if (slot->len < 14 || slot->len > NETMAP_BUF_SIZE) { + D("bad pkt at %d len %d", n, slot->len); + continue; + } + slot->flags &= ~NS_FORWARD; // XXX needed ? + m = m_devget(NMB(slot), slot->len, 0, kring->na->ifp, NULL); + + if (m == NULL) + break; + if (tail) + tail->m_nextpkt = m; + else + q->head = m; + tail = m; + q->count++; + m->m_nextpkt = NULL; + } + q->tail = tail; +} + +/* + * called under main lock to send packets from the host to the NIC + * The host ring has packets from nr_hwcur to (cur - reserved) + * to be sent down. We scan the tx rings, which have just been + * flushed so nr_hwcur == cur. Pushing packets down means + * increment cur and decrement avail. + * XXX to be verified + */ +static void +netmap_sw_to_nic(struct netmap_adapter *na) +{ + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings]; + struct netmap_kring *k1 = &na->tx_rings[0]; + int i, howmany, src_lim, dst_lim; + + howmany = kring->nr_hwavail; /* XXX otherwise cur - reserved - nr_hwcur */ + + src_lim = kring->nkr_num_slots; + for (i = 0; howmany > 0 && i < na->num_tx_rings; i++, k1++) { + ND("%d packets left to ring %d (space %d)", howmany, i, k1->nr_hwavail); + dst_lim = k1->nkr_num_slots; + while (howmany > 0 && k1->ring->avail > 0) { + struct netmap_slot *src, *dst, tmp; + src = &kring->ring->slot[kring->nr_hwcur]; + dst = &k1->ring->slot[k1->ring->cur]; + tmp = *src; + src->buf_idx = dst->buf_idx; + src->flags = NS_BUF_CHANGED; + + dst->buf_idx = tmp.buf_idx; + dst->len = tmp.len; + dst->flags = NS_BUF_CHANGED; + ND("out len %d buf %d from %d to %d", + dst->len, dst->buf_idx, + kring->nr_hwcur, k1->ring->cur); + + if (++kring->nr_hwcur >= src_lim) + kring->nr_hwcur = 0; + howmany--; + kring->nr_hwavail--; + if (++k1->ring->cur >= dst_lim) + k1->ring->cur = 0; + k1->ring->avail--; + } + kring->ring->cur = kring->nr_hwcur; // XXX + k1++; + } +} + +/* + * netmap_sync_to_host() passes packets up. We are called from a + * system call in user process context, and the only contention + * can be among multiple user threads erroneously calling + * this routine concurrently. + */ +static void +netmap_sync_to_host(struct netmap_adapter *na) +{ + struct netmap_kring *kring = &na->tx_rings[na->num_tx_rings]; + struct netmap_ring *ring = kring->ring; + u_int k, lim = kring->nkr_num_slots - 1; + struct mbq q = { NULL, NULL }; + + k = ring->cur; + if (k > lim) { + netmap_ring_reinit(kring); + return; + } + // na->nm_lock(na->ifp, NETMAP_CORE_LOCK, 0); + + /* Take packets from hwcur to cur and pass them up. + * In case of no buffers we give up. At the end of the loop, + * the queue is drained in all cases. + */ + netmap_grab_packets(kring, &q, 1); + kring->nr_hwcur = k; + kring->nr_hwavail = ring->avail = lim; + // na->nm_lock(na->ifp, NETMAP_CORE_UNLOCK, 0); + + netmap_send_up(na->ifp, q.head); +} + +/* + * rxsync backend for packets coming from the host stack. + * They have been put in the queue by netmap_start() so we + * need to protect access to the kring using a lock. + * + * This routine also does the selrecord if called from the poll handler + * (we know because td != NULL). + * + * NOTE: on linux, selrecord() is defined as a macro and uses pwait + * as an additional hidden argument. + */ +static void +netmap_sync_from_host(struct netmap_adapter *na, struct thread *td, void *pwait) +{ + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings]; + struct netmap_ring *ring = kring->ring; + u_int j, n, lim = kring->nkr_num_slots; + u_int k = ring->cur, resvd = ring->reserved; + + (void)pwait; /* disable unused warnings */ + na->nm_lock(na->ifp, NETMAP_CORE_LOCK, 0); + if (k >= lim) { + netmap_ring_reinit(kring); + return; + } + /* new packets are already set in nr_hwavail */ + /* skip past packets that userspace has released */ + j = kring->nr_hwcur; + if (resvd > 0) { + if (resvd + ring->avail >= lim + 1) { + D("XXX invalid reserve/avail %d %d", resvd, ring->avail); + ring->reserved = resvd = 0; // XXX panic... + } + k = (k >= resvd) ? k - resvd : k + lim - resvd; + } + if (j != k) { + n = k >= j ? k - j : k + lim - j; + kring->nr_hwavail -= n; + kring->nr_hwcur = k; + } + k = ring->avail = kring->nr_hwavail - resvd; + if (k == 0 && td) + selrecord(td, &kring->si); + if (k && (netmap_verbose & NM_VERB_HOST)) + D("%d pkts from stack", k); + na->nm_lock(na->ifp, NETMAP_CORE_UNLOCK, 0); +} + + +/* + * get a refcounted reference to an interface. + * Return ENXIO if the interface does not exist, EINVAL if netmap + * is not supported by the interface. + * If successful, hold a reference. + */ +static int +get_ifp(const char *name, struct ifnet **ifp) +{ +#ifdef NM_BRIDGE + struct ifnet *iter = NULL; + + do { + struct nm_bridge *b; + int i, l, cand = -1; + + if (strncmp(name, NM_NAME, sizeof(NM_NAME) - 1)) + break; + b = nm_find_bridge(name); + if (b == NULL) { + D("no bridges available for '%s'", name); + return (ENXIO); + } + /* XXX locking */ + BDG_LOCK(b); + /* lookup in the local list of ports */ + for (i = 0; i < NM_BDG_MAXPORTS; i++) { + iter = b->bdg_ports[i]; + if (iter == NULL) { + if (cand == -1) + cand = i; /* potential insert point */ + continue; + } + if (!strcmp(iter->if_xname, name)) { + ADD_BDG_REF(iter); + ND("found existing interface"); + BDG_UNLOCK(b); + break; + } + } + if (i < NM_BDG_MAXPORTS) /* already unlocked */ + break; + if (cand == -1) { + D("bridge full, cannot create new port"); +no_port: + BDG_UNLOCK(b); + *ifp = NULL; + return EINVAL; + } + ND("create new bridge port %s", name); + /* space for forwarding list after the ifnet */ + l = sizeof(*iter) + + sizeof(struct nm_bdg_fwd)*NM_BDG_BATCH ; + iter = malloc(l, M_DEVBUF, M_NOWAIT | M_ZERO); + if (!iter) + goto no_port; + strcpy(iter->if_xname, name); + bdg_netmap_attach(iter); + b->bdg_ports[cand] = iter; + iter->if_bridge = b; + ADD_BDG_REF(iter); + BDG_UNLOCK(b); + ND("attaching virtual bridge %p", b); + } while (0); + *ifp = iter; + if (! *ifp) +#endif /* NM_BRIDGE */ + *ifp = ifunit_ref(name); + if (*ifp == NULL) + return (ENXIO); + /* can do this if the capability exists and if_pspare[0] + * points to the netmap descriptor. + */ + if (NETMAP_CAPABLE(*ifp)) + return 0; /* valid pointer, we hold the refcount */ + nm_if_rele(*ifp); + return EINVAL; // not NETMAP capable +} + + +/* + * Error routine called when txsync/rxsync detects an error. + * Can't do much more than resetting cur = hwcur, avail = hwavail. + * Return 1 on reinit. + * + * This routine is only called by the upper half of the kernel. + * It only reads hwcur (which is changed only by the upper half, too) + * and hwavail (which may be changed by the lower half, but only on + * a tx ring and only to increase it, so any error will be recovered + * on the next call). For the above, we don't strictly need to call + * it under lock. + */ +int +netmap_ring_reinit(struct netmap_kring *kring) +{ + struct netmap_ring *ring = kring->ring; + u_int i, lim = kring->nkr_num_slots - 1; + int errors = 0; + + RD(10, "called for %s", kring->na->ifp->if_xname); + if (ring->cur > lim) + errors++; + for (i = 0; i <= lim; i++) { + u_int idx = ring->slot[i].buf_idx; + u_int len = ring->slot[i].len; + if (idx < 2 || idx >= netmap_total_buffers) { + if (!errors++) + D("bad buffer at slot %d idx %d len %d ", i, idx, len); + ring->slot[i].buf_idx = 0; + ring->slot[i].len = 0; + } else if (len > NETMAP_BUF_SIZE) { + ring->slot[i].len = 0; + if (!errors++) + D("bad len %d at slot %d idx %d", + len, i, idx); + } + } + if (errors) { + int pos = kring - kring->na->tx_rings; + int n = kring->na->num_tx_rings + 1; + + RD(10, "total %d errors", errors); + errors++; + RD(10, "%s %s[%d] reinit, cur %d -> %d avail %d -> %d", + kring->na->ifp->if_xname, + pos < n ? "TX" : "RX", pos < n ? pos : pos - n, + ring->cur, kring->nr_hwcur, + ring->avail, kring->nr_hwavail); + ring->cur = kring->nr_hwcur; + ring->avail = kring->nr_hwavail; + } + return (errors ? 1 : 0); +} + + +/* + * Set the ring ID. For devices with a single queue, a request + * for all rings is the same as a single ring. + */ +static int +netmap_set_ringid(struct netmap_priv_d *priv, u_int ringid) +{ + struct ifnet *ifp = priv->np_ifp; + struct netmap_adapter *na = NA(ifp); + u_int i = ringid & NETMAP_RING_MASK; + /* initially (np_qfirst == np_qlast) we don't want to lock */ + int need_lock = (priv->np_qfirst != priv->np_qlast); + int lim = na->num_rx_rings; + + if (na->num_tx_rings > lim) + lim = na->num_tx_rings; + if ( (ringid & NETMAP_HW_RING) && i >= lim) { + D("invalid ring id %d", i); + return (EINVAL); + } + if (need_lock) + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); + priv->np_ringid = ringid; + if (ringid & NETMAP_SW_RING) { + priv->np_qfirst = NETMAP_SW_RING; + priv->np_qlast = 0; + } else if (ringid & NETMAP_HW_RING) { + priv->np_qfirst = i; + priv->np_qlast = i + 1; + } else { + priv->np_qfirst = 0; + priv->np_qlast = NETMAP_HW_RING ; + } + priv->np_txpoll = (ringid & NETMAP_NO_TX_POLL) ? 0 : 1; + if (need_lock) + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0); + if (netmap_verbose) { + if (ringid & NETMAP_SW_RING) + D("ringid %s set to SW RING", ifp->if_xname); + else if (ringid & NETMAP_HW_RING) + D("ringid %s set to HW RING %d", ifp->if_xname, + priv->np_qfirst); + else + D("ringid %s set to all %d HW RINGS", ifp->if_xname, lim); + } + return 0; +} + +/* + * ioctl(2) support for the "netmap" device. + * + * Following a list of accepted commands: + * - NIOCGINFO + * - SIOCGIFADDR just for convenience + * - NIOCREGIF + * - NIOCUNREGIF + * - NIOCTXSYNC + * - NIOCRXSYNC + * + * Return 0 on success, errno otherwise. + */ +static int +netmap_ioctl(struct cdev *dev, u_long cmd, caddr_t data, + int fflag, struct thread *td) +{ + struct netmap_priv_d *priv = NULL; + struct ifnet *ifp; + struct nmreq *nmr = (struct nmreq *) data; + struct netmap_adapter *na; + int error; + u_int i, lim; + struct netmap_if *nifp; + + (void)dev; /* UNUSED */ + (void)fflag; /* UNUSED */ +#ifdef linux +#define devfs_get_cdevpriv(pp) \ + ({ *(struct netmap_priv_d **)pp = ((struct file *)td)->private_data; \ + (*pp ? 0 : ENOENT); }) + +/* devfs_set_cdevpriv cannot fail on linux */ +#define devfs_set_cdevpriv(p, fn) \ + ({ ((struct file *)td)->private_data = p; (p ? 0 : EINVAL); }) + + +#define devfs_clear_cdevpriv() do { \ + netmap_dtor(priv); ((struct file *)td)->private_data = 0; \ + } while (0) +#endif /* linux */ + + CURVNET_SET(TD_TO_VNET(td)); + + error = devfs_get_cdevpriv((void **)&priv); + if (error) { + CURVNET_RESTORE(); + /* XXX ENOENT should be impossible, since the priv + * is now created in the open */ + return (error == ENOENT ? ENXIO : error); + } + + nmr->nr_name[sizeof(nmr->nr_name) - 1] = '\0'; /* truncate name */ + switch (cmd) { + case NIOCGINFO: /* return capabilities etc */ + if (nmr->nr_version != NETMAP_API) { + D("API mismatch got %d have %d", + nmr->nr_version, NETMAP_API); + nmr->nr_version = NETMAP_API; + error = EINVAL; + break; + } + /* update configuration */ + error = netmap_get_memory(priv); + ND("get_memory returned %d", error); + if (error) + break; + /* memsize is always valid */ + nmr->nr_memsize = nm_mem.nm_totalsize; + nmr->nr_offset = 0; + nmr->nr_rx_rings = nmr->nr_tx_rings = 0; + nmr->nr_rx_slots = nmr->nr_tx_slots = 0; + if (nmr->nr_name[0] == '\0') /* just get memory info */ + break; + error = get_ifp(nmr->nr_name, &ifp); /* get a refcount */ + if (error) + break; + na = NA(ifp); /* retrieve netmap_adapter */ + netmap_update_config(na); + nmr->nr_rx_rings = na->num_rx_rings; + nmr->nr_tx_rings = na->num_tx_rings; + nmr->nr_rx_slots = na->num_rx_desc; + nmr->nr_tx_slots = na->num_tx_desc; + nm_if_rele(ifp); /* return the refcount */ + break; + + case NIOCREGIF: + if (nmr->nr_version != NETMAP_API) { + nmr->nr_version = NETMAP_API; + error = EINVAL; + break; + } + /* ensure allocators are ready */ + error = netmap_get_memory(priv); + ND("get_memory returned %d", error); + if (error) + break; + + /* protect access to priv from concurrent NIOCREGIF */ + NMA_LOCK(); + if (priv->np_ifp != NULL) { /* thread already registered */ + error = netmap_set_ringid(priv, nmr->nr_ringid); + NMA_UNLOCK(); + break; + } + /* find the interface and a reference */ + error = get_ifp(nmr->nr_name, &ifp); /* keep reference */ + if (error) { + NMA_UNLOCK(); + break; + } + na = NA(ifp); /* retrieve netmap adapter */ + + for (i = 10; i > 0; i--) { + na->nm_lock(ifp, NETMAP_REG_LOCK, 0); + if (!NETMAP_DELETING(na)) + break; + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); + tsleep(na, 0, "NIOCREGIF", hz/10); + } + if (i == 0) { + D("too many NIOCREGIF attempts, give up"); + error = EINVAL; + nm_if_rele(ifp); /* return the refcount */ + NMA_UNLOCK(); + break; + } + + /* ring configuration may have changed, fetch from the card */ + netmap_update_config(na); + priv->np_ifp = ifp; /* store the reference */ + error = netmap_set_ringid(priv, nmr->nr_ringid); + if (error) + goto error; + nifp = netmap_if_new(nmr->nr_name, na); + if (nifp == NULL) { /* allocation failed */ + error = ENOMEM; + } else if (ifp->if_capenable & IFCAP_NETMAP) { + /* was already set */ + } else { + /* Otherwise set the card in netmap mode + * and make it use the shared buffers. + */ + for (i = 0 ; i < na->num_tx_rings + 1; i++) + mtx_init(&na->tx_rings[i].q_lock, "nm_txq_lock", MTX_NETWORK_LOCK, MTX_DEF); + for (i = 0 ; i < na->num_rx_rings + 1; i++) { + mtx_init(&na->rx_rings[i].q_lock, "nm_rxq_lock", MTX_NETWORK_LOCK, MTX_DEF); + } + error = na->nm_register(ifp, 1); /* mode on */ + if (error) { + netmap_dtor_locked(priv); + netmap_if_free(nifp); + } + } + + if (error) { /* reg. failed, release priv and ref */ +error: + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); + nm_if_rele(ifp); /* return the refcount */ + priv->np_ifp = NULL; + priv->np_nifp = NULL; + NMA_UNLOCK(); + break; + } + + na->nm_lock(ifp, NETMAP_REG_UNLOCK, 0); + + /* the following assignment is a commitment. + * Readers (i.e., poll and *SYNC) check for + * np_nifp != NULL without locking + */ + wmb(); /* make sure previous writes are visible to all CPUs */ + priv->np_nifp = nifp; + NMA_UNLOCK(); + + /* return the offset of the netmap_if object */ + nmr->nr_rx_rings = na->num_rx_rings; + nmr->nr_tx_rings = na->num_tx_rings; + nmr->nr_rx_slots = na->num_rx_desc; + nmr->nr_tx_slots = na->num_tx_desc; + nmr->nr_memsize = nm_mem.nm_totalsize; + nmr->nr_offset = netmap_if_offset(nifp); + break; + + case NIOCUNREGIF: + // XXX we have no data here ? + D("deprecated, data is %p", nmr); + error = EINVAL; + break; + + case NIOCTXSYNC: + case NIOCRXSYNC: + nifp = priv->np_nifp; + + if (nifp == NULL) { + error = ENXIO; + break; + } + rmb(); /* make sure following reads are not from cache */ + + + ifp = priv->np_ifp; /* we have a reference */ + + if (ifp == NULL) { + D("Internal error: nifp != NULL && ifp == NULL"); + error = ENXIO; + break; + } + + na = NA(ifp); /* retrieve netmap adapter */ + if (priv->np_qfirst == NETMAP_SW_RING) { /* host rings */ + if (cmd == NIOCTXSYNC) + netmap_sync_to_host(na); + else + netmap_sync_from_host(na, NULL, NULL); + break; + } + /* find the last ring to scan */ + lim = priv->np_qlast; + if (lim == NETMAP_HW_RING) + lim = (cmd == NIOCTXSYNC) ? + na->num_tx_rings : na->num_rx_rings; + + for (i = priv->np_qfirst; i < lim; i++) { + if (cmd == NIOCTXSYNC) { + struct netmap_kring *kring = &na->tx_rings[i]; + if (netmap_verbose & NM_VERB_TXSYNC) + D("pre txsync ring %d cur %d hwcur %d", + i, kring->ring->cur, + kring->nr_hwcur); + na->nm_txsync(ifp, i, 1 /* do lock */); + if (netmap_verbose & NM_VERB_TXSYNC) + D("post txsync ring %d cur %d hwcur %d", + i, kring->ring->cur, + kring->nr_hwcur); + } else { + na->nm_rxsync(ifp, i, 1 /* do lock */); + microtime(&na->rx_rings[i].ring->ts); + } + } + + break; + +#ifdef __FreeBSD__ + case BIOCIMMEDIATE: + case BIOCGHDRCMPLT: + case BIOCSHDRCMPLT: + case BIOCSSEESENT: + D("ignore BIOCIMMEDIATE/BIOCSHDRCMPLT/BIOCSHDRCMPLT/BIOCSSEESENT"); + break; + + default: /* allow device-specific ioctls */ + { + struct socket so; + bzero(&so, sizeof(so)); + error = get_ifp(nmr->nr_name, &ifp); /* keep reference */ + if (error) + break; + so.so_vnet = ifp->if_vnet; + // so->so_proto not null. + error = ifioctl(&so, cmd, data, td); + nm_if_rele(ifp); + break; + } + +#else /* linux */ + default: + error = EOPNOTSUPP; +#endif /* linux */ + } + + CURVNET_RESTORE(); + return (error); +} + + +/* + * select(2) and poll(2) handlers for the "netmap" device. + * + * Can be called for one or more queues. + * Return true the event mask corresponding to ready events. + * If there are no ready events, do a selrecord on either individual + * selfd or on the global one. + * Device-dependent parts (locking and sync of tx/rx rings) + * are done through callbacks. + * + * On linux, arguments are really pwait, the poll table, and 'td' is struct file * + * The first one is remapped to pwait as selrecord() uses the name as an + * hidden argument. + */ +static int +netmap_poll(struct cdev *dev, int events, struct thread *td) +{ + struct netmap_priv_d *priv = NULL; + struct netmap_adapter *na; + struct ifnet *ifp; + struct netmap_kring *kring; + u_int core_lock, i, check_all, want_tx, want_rx, revents = 0; + u_int lim_tx, lim_rx, host_forwarded = 0; + struct mbq q = { NULL, NULL, 0 }; + enum {NO_CL, NEED_CL, LOCKED_CL }; /* see below */ + void *pwait = dev; /* linux compatibility */ + + (void)pwait; + + if (devfs_get_cdevpriv((void **)&priv) != 0 || priv == NULL) + return POLLERR; + + if (priv->np_nifp == NULL) { + D("No if registered"); + return POLLERR; + } + rmb(); /* make sure following reads are not from cache */ + + ifp = priv->np_ifp; + // XXX check for deleting() ? + if ( (ifp->if_capenable & IFCAP_NETMAP) == 0) + return POLLERR; + + if (netmap_verbose & 0x8000) + D("device %s events 0x%x", ifp->if_xname, events); + want_tx = events & (POLLOUT | POLLWRNORM); + want_rx = events & (POLLIN | POLLRDNORM); + + na = NA(ifp); /* retrieve netmap adapter */ + + lim_tx = na->num_tx_rings; + lim_rx = na->num_rx_rings; + /* how many queues we are scanning */ + if (priv->np_qfirst == NETMAP_SW_RING) { + if (priv->np_txpoll || want_tx) { + /* push any packets up, then we are always ready */ + kring = &na->tx_rings[lim_tx]; + netmap_sync_to_host(na); + revents |= want_tx; + } + if (want_rx) { + kring = &na->rx_rings[lim_rx]; + if (kring->ring->avail == 0) + netmap_sync_from_host(na, td, dev); + if (kring->ring->avail > 0) { + revents |= want_rx; + } + } + return (revents); + } + + /* if we are in transparent mode, check also the host rx ring */ + kring = &na->rx_rings[lim_rx]; + if ( (priv->np_qlast == NETMAP_HW_RING) // XXX check_all + && want_rx + && (netmap_fwd || kring->ring->flags & NR_FORWARD) ) { + if (kring->ring->avail == 0) + netmap_sync_from_host(na, td, dev); + if (kring->ring->avail > 0) + revents |= want_rx; + } + + /* + * check_all is set if the card has more than one queue and + * the client is polling all of them. If true, we sleep on + * the "global" selfd, otherwise we sleep on individual selfd + * (we can only sleep on one of them per direction). + * The interrupt routine in the driver should always wake on + * the individual selfd, and also on the global one if the card + * has more than one ring. + * + * If the card has only one lock, we just use that. + * If the card has separate ring locks, we just use those + * unless we are doing check_all, in which case the whole + * loop is wrapped by the global lock. + * We acquire locks only when necessary: if poll is called + * when buffers are available, we can just return without locks. + * + * rxsync() is only called if we run out of buffers on a POLLIN. + * txsync() is called if we run out of buffers on POLLOUT, or + * there are pending packets to send. The latter can be disabled + * passing NETMAP_NO_TX_POLL in the NIOCREG call. + */ + check_all = (priv->np_qlast == NETMAP_HW_RING) && (lim_tx > 1 || lim_rx > 1); + + /* + * core_lock indicates what to do with the core lock. + * The core lock is used when either the card has no individual + * locks, or it has individual locks but we are cheking all + * rings so we need the core lock to avoid missing wakeup events. + * + * It has three possible states: + * NO_CL we don't need to use the core lock, e.g. + * because we are protected by individual locks. + * NEED_CL we need the core lock. In this case, when we + * call the lock routine, move to LOCKED_CL + * to remember to release the lock once done. + * LOCKED_CL core lock is set, so we need to release it. + */ + core_lock = (check_all || !na->separate_locks) ? NEED_CL : NO_CL; +#ifdef NM_BRIDGE + /* the bridge uses separate locks */ + if (na->nm_register == bdg_netmap_reg) { + ND("not using core lock for %s", ifp->if_xname); + core_lock = NO_CL; + } +#endif /* NM_BRIDGE */ + if (priv->np_qlast != NETMAP_HW_RING) { + lim_tx = lim_rx = priv->np_qlast; + } + + /* + * We start with a lock free round which is good if we have + * data available. If this fails, then lock and call the sync + * routines. + */ + for (i = priv->np_qfirst; want_rx && i < lim_rx; i++) { + kring = &na->rx_rings[i]; + if (kring->ring->avail > 0) { + revents |= want_rx; + want_rx = 0; /* also breaks the loop */ + } + } + for (i = priv->np_qfirst; want_tx && i < lim_tx; i++) { + kring = &na->tx_rings[i]; + if (kring->ring->avail > 0) { + revents |= want_tx; + want_tx = 0; /* also breaks the loop */ + } + } + + /* + * If we to push packets out (priv->np_txpoll) or want_tx is + * still set, we do need to run the txsync calls (on all rings, + * to avoid that the tx rings stall). + */ + if (priv->np_txpoll || want_tx) { +flush_tx: + for (i = priv->np_qfirst; i < lim_tx; i++) { + kring = &na->tx_rings[i]; + /* + * Skip the current ring if want_tx == 0 + * (we have already done a successful sync on + * a previous ring) AND kring->cur == kring->hwcur + * (there are no pending transmissions for this ring). + */ + if (!want_tx && kring->ring->cur == kring->nr_hwcur) + continue; + if (core_lock == NEED_CL) { + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); + core_lock = LOCKED_CL; + } + if (na->separate_locks) + na->nm_lock(ifp, NETMAP_TX_LOCK, i); + if (netmap_verbose & NM_VERB_TXSYNC) + D("send %d on %s %d", + kring->ring->cur, + ifp->if_xname, i); + if (na->nm_txsync(ifp, i, 0 /* no lock */)) + revents |= POLLERR; + + /* Check avail/call selrecord only if called with POLLOUT */ + if (want_tx) { + if (kring->ring->avail > 0) { + /* stop at the first ring. We don't risk + * starvation. + */ + revents |= want_tx; + want_tx = 0; + } else if (!check_all) + selrecord(td, &kring->si); + } + if (na->separate_locks) + na->nm_lock(ifp, NETMAP_TX_UNLOCK, i); + } + } + + /* + * now if want_rx is still set we need to lock and rxsync. + * Do it on all rings because otherwise we starve. + */ + if (want_rx) { + for (i = priv->np_qfirst; i < lim_rx; i++) { + kring = &na->rx_rings[i]; + if (core_lock == NEED_CL) { + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); + core_lock = LOCKED_CL; + } + if (na->separate_locks) + na->nm_lock(ifp, NETMAP_RX_LOCK, i); + if (netmap_fwd ||kring->ring->flags & NR_FORWARD) { + ND(10, "forwarding some buffers up %d to %d", + kring->nr_hwcur, kring->ring->cur); + netmap_grab_packets(kring, &q, netmap_fwd); + } + + if (na->nm_rxsync(ifp, i, 0 /* no lock */)) + revents |= POLLERR; + if (netmap_no_timestamp == 0 || + kring->ring->flags & NR_TIMESTAMP) { + microtime(&kring->ring->ts); + } + + if (kring->ring->avail > 0) + revents |= want_rx; + else if (!check_all) + selrecord(td, &kring->si); + if (na->separate_locks) + na->nm_lock(ifp, NETMAP_RX_UNLOCK, i); + } + } + if (check_all && revents == 0) { /* signal on the global queue */ + if (want_tx) + selrecord(td, &na->tx_si); + if (want_rx) + selrecord(td, &na->rx_si); + } + + /* forward host to the netmap ring */ + kring = &na->rx_rings[lim_rx]; + if (kring->nr_hwavail > 0) + ND("host rx %d has %d packets", lim_rx, kring->nr_hwavail); + if ( (priv->np_qlast == NETMAP_HW_RING) // XXX check_all + && (netmap_fwd || kring->ring->flags & NR_FORWARD) + && kring->nr_hwavail > 0 && !host_forwarded) { + if (core_lock == NEED_CL) { + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); + core_lock = LOCKED_CL; + } + netmap_sw_to_nic(na); + host_forwarded = 1; /* prevent another pass */ + want_rx = 0; + goto flush_tx; + } + + if (core_lock == LOCKED_CL) + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0); + if (q.head) + netmap_send_up(na->ifp, q.head); + + return (revents); +} + +/*------- driver support routines ------*/ + +/* + * default lock wrapper. + */ +static void +netmap_lock_wrapper(struct ifnet *dev, int what, u_int queueid) +{ + struct netmap_adapter *na = NA(dev); + + switch (what) { +#ifdef linux /* some system do not need lock on register */ + case NETMAP_REG_LOCK: + case NETMAP_REG_UNLOCK: + break; +#endif /* linux */ + + case NETMAP_CORE_LOCK: + mtx_lock(&na->core_lock); + break; + + case NETMAP_CORE_UNLOCK: + mtx_unlock(&na->core_lock); + break; + + case NETMAP_TX_LOCK: + mtx_lock(&na->tx_rings[queueid].q_lock); + break; + + case NETMAP_TX_UNLOCK: + mtx_unlock(&na->tx_rings[queueid].q_lock); + break; + + case NETMAP_RX_LOCK: + mtx_lock(&na->rx_rings[queueid].q_lock); + break; + + case NETMAP_RX_UNLOCK: + mtx_unlock(&na->rx_rings[queueid].q_lock); + break; + } +} + + +/* + * Initialize a ``netmap_adapter`` object created by driver on attach. + * We allocate a block of memory with room for a struct netmap_adapter + * plus two sets of N+2 struct netmap_kring (where N is the number + * of hardware rings): + * krings 0..N-1 are for the hardware queues. + * kring N is for the host stack queue + * kring N+1 is only used for the selinfo for all queues. + * Return 0 on success, ENOMEM otherwise. + * + * By default the receive and transmit adapter ring counts are both initialized + * to num_queues. na->num_tx_rings can be set for cards with different tx/rx + * setups. + */ +int +netmap_attach(struct netmap_adapter *arg, int num_queues) +{ + struct netmap_adapter *na = NULL; + struct ifnet *ifp = arg ? arg->ifp : NULL; + + if (arg == NULL || ifp == NULL) + goto fail; + na = malloc(sizeof(*na), M_DEVBUF, M_NOWAIT | M_ZERO); + if (na == NULL) + goto fail; + WNA(ifp) = na; + *na = *arg; /* copy everything, trust the driver to not pass junk */ + NETMAP_SET_CAPABLE(ifp); + if (na->num_tx_rings == 0) + na->num_tx_rings = num_queues; + na->num_rx_rings = num_queues; + na->refcount = na->na_single = na->na_multi = 0; + /* Core lock initialized here, others after netmap_if_new. */ + mtx_init(&na->core_lock, "netmap core lock", MTX_NETWORK_LOCK, MTX_DEF); + if (na->nm_lock == NULL) { + ND("using default locks for %s", ifp->if_xname); + na->nm_lock = netmap_lock_wrapper; + } +#ifdef linux + if (ifp->netdev_ops) { + ND("netdev_ops %p", ifp->netdev_ops); + /* prepare a clone of the netdev ops */ + na->nm_ndo = *ifp->netdev_ops; + } + na->nm_ndo.ndo_start_xmit = linux_netmap_start; +#endif + D("success for %s", ifp->if_xname); + return 0; + +fail: + D("fail, arg %p ifp %p na %p", arg, ifp, na); + return (na ? EINVAL : ENOMEM); +} + + +/* + * Free the allocated memory linked to the given ``netmap_adapter`` + * object. + */ +void +netmap_detach(struct ifnet *ifp) +{ + struct netmap_adapter *na = NA(ifp); + + if (!na) + return; + + mtx_destroy(&na->core_lock); + + if (na->tx_rings) { /* XXX should not happen */ + D("freeing leftover tx_rings"); + free(na->tx_rings, M_DEVBUF); + } + bzero(na, sizeof(*na)); + WNA(ifp) = NULL; + free(na, M_DEVBUF); +} + + +/* + * Intercept packets from the network stack and pass them + * to netmap as incoming packets on the 'software' ring. + * We are not locked when called. + */ +int +netmap_start(struct ifnet *ifp, struct mbuf *m) +{ + struct netmap_adapter *na = NA(ifp); + struct netmap_kring *kring = &na->rx_rings[na->num_rx_rings]; + u_int i, len = MBUF_LEN(m); + u_int error = EBUSY, lim = kring->nkr_num_slots - 1; + struct netmap_slot *slot; + + if (netmap_verbose & NM_VERB_HOST) + D("%s packet %d len %d from the stack", ifp->if_xname, + kring->nr_hwcur + kring->nr_hwavail, len); + na->nm_lock(ifp, NETMAP_CORE_LOCK, 0); + if (kring->nr_hwavail >= lim) { + if (netmap_verbose) + D("stack ring %s full\n", ifp->if_xname); + goto done; /* no space */ + } + if (len > NETMAP_BUF_SIZE) { + D("%s from_host, drop packet size %d > %d", ifp->if_xname, + len, NETMAP_BUF_SIZE); + goto done; /* too long for us */ + } + + /* compute the insert position */ + i = kring->nr_hwcur + kring->nr_hwavail; + if (i > lim) + i -= lim + 1; + slot = &kring->ring->slot[i]; + m_copydata(m, 0, len, NMB(slot)); + slot->len = len; + slot->flags = kring->nkr_slot_flags; + kring->nr_hwavail++; + if (netmap_verbose & NM_VERB_HOST) + D("wake up host ring %s %d", na->ifp->if_xname, na->num_rx_rings); + selwakeuppri(&kring->si, PI_NET); + error = 0; +done: + na->nm_lock(ifp, NETMAP_CORE_UNLOCK, 0); + + /* release the mbuf in either cases of success or failure. As an + * alternative, put the mbuf in a free list and free the list + * only when really necessary. + */ + m_freem(m); + + return (error); +} + + +/* + * netmap_reset() is called by the driver routines when reinitializing + * a ring. The driver is in charge of locking to protect the kring. + * If netmap mode is not set just return NULL. + */ +struct netmap_slot * +netmap_reset(struct netmap_adapter *na, enum txrx tx, int n, + u_int new_cur) +{ + struct netmap_kring *kring; + int new_hwofs, lim; + + if (na == NULL) + return NULL; /* no netmap support here */ + if (!(na->ifp->if_capenable & IFCAP_NETMAP)) + return NULL; /* nothing to reinitialize */ + + if (tx == NR_TX) { + if (n >= na->num_tx_rings) + return NULL; + kring = na->tx_rings + n; + new_hwofs = kring->nr_hwcur - new_cur; + } else { + if (n >= na->num_rx_rings) + return NULL; + kring = na->rx_rings + n; + new_hwofs = kring->nr_hwcur + kring->nr_hwavail - new_cur; + } + lim = kring->nkr_num_slots - 1; + if (new_hwofs > lim) + new_hwofs -= lim + 1; + + /* Alwayws set the new offset value and realign the ring. */ + kring->nkr_hwofs = new_hwofs; + if (tx == NR_TX) + kring->nr_hwavail = kring->nkr_num_slots - 1; + ND(10, "new hwofs %d on %s %s[%d]", + kring->nkr_hwofs, na->ifp->if_xname, + tx == NR_TX ? "TX" : "RX", n); + +#if 0 // def linux + /* XXX check that the mappings are correct */ + /* need ring_nr, adapter->pdev, direction */ + buffer_info->dma = dma_map_single(&pdev->dev, addr, adapter->rx_buffer_len, DMA_FROM_DEVICE); + if (dma_mapping_error(&adapter->pdev->dev, buffer_info->dma)) { + D("error mapping rx netmap buffer %d", i); + // XXX fix error handling + } + +#endif /* linux */ + /* + * Wakeup on the individual and global lock + * We do the wakeup here, but the ring is not yet reconfigured. + * However, we are under lock so there are no races. + */ + selwakeuppri(&kring->si, PI_NET); + selwakeuppri(tx == NR_TX ? &na->tx_si : &na->rx_si, PI_NET); + return kring->ring->slot; +} + + +/* + * Default functions to handle rx/tx interrupts + * we have 4 cases: + * 1 ring, single lock: + * lock(core); wake(i=0); unlock(core) + * N rings, single lock: + * lock(core); wake(i); wake(N+1) unlock(core) + * 1 ring, separate locks: (i=0) + * lock(i); wake(i); unlock(i) + * N rings, separate locks: + * lock(i); wake(i); unlock(i); lock(core) wake(N+1) unlock(core) + * work_done is non-null on the RX path. + */ +int +netmap_rx_irq(struct ifnet *ifp, int q, int *work_done) +{ + struct netmap_adapter *na; + struct netmap_kring *r; + NM_SELINFO_T *main_wq; + + if (!(ifp->if_capenable & IFCAP_NETMAP)) + return 0; + ND(5, "received %s queue %d", work_done ? "RX" : "TX" , q); + na = NA(ifp); + if (na->na_flags & NAF_SKIP_INTR) { + ND("use regular interrupt"); + return 0; + } + + if (work_done) { /* RX path */ + if (q >= na->num_rx_rings) + return 0; // regular queue + r = na->rx_rings + q; + r->nr_kflags |= NKR_PENDINTR; + main_wq = (na->num_rx_rings > 1) ? &na->rx_si : NULL; + } else { /* tx path */ + if (q >= na->num_tx_rings) + return 0; // regular queue + r = na->tx_rings + q; + main_wq = (na->num_tx_rings > 1) ? &na->tx_si : NULL; + work_done = &q; /* dummy */ + } + if (na->separate_locks) { + mtx_lock(&r->q_lock); + selwakeuppri(&r->si, PI_NET); + mtx_unlock(&r->q_lock); + if (main_wq) { + mtx_lock(&na->core_lock); + selwakeuppri(main_wq, PI_NET); + mtx_unlock(&na->core_lock); + } + } else { + mtx_lock(&na->core_lock); + selwakeuppri(&r->si, PI_NET); + if (main_wq) + selwakeuppri(main_wq, PI_NET); + mtx_unlock(&na->core_lock); + } + *work_done = 1; /* do not fire napi again */ + return 1; +} + + +#ifdef linux /* linux-specific routines */ + +/* + * Remap linux arguments into the FreeBSD call. + * - pwait is the poll table, passed as 'dev'; + * If pwait == NULL someone else already woke up before. We can report + * events but they are filtered upstream. + * If pwait != NULL, then pwait->key contains the list of events. + * - events is computed from pwait as above. + * - file is passed as 'td'; + */ +static u_int +linux_netmap_poll(struct file * file, struct poll_table_struct *pwait) +{ +#if LINUX_VERSION_CODE < KERNEL_VERSION(3,4,0) + int events = pwait ? pwait->key : POLLIN | POLLOUT; +#else /* in 3.4.0 field 'key' was renamed to '_key' */ + int events = pwait ? pwait->_key : POLLIN | POLLOUT; +#endif + return netmap_poll((void *)pwait, events, (void *)file); +} + +static int +linux_netmap_mmap(struct file *f, struct vm_area_struct *vma) +{ + int lut_skip, i, j; + int user_skip = 0; + struct lut_entry *l_entry; + int error = 0; + unsigned long off, tomap; + /* + * vma->vm_start: start of mapping user address space + * vma->vm_end: end of the mapping user address space + * vma->vm_pfoff: offset of first page in the device + */ + + // XXX security checks + + error = netmap_get_memory(f->private_data); + ND("get_memory returned %d", error); + if (error) + return -error; + + off = vma->vm_pgoff << PAGE_SHIFT; /* offset in bytes */ + tomap = vma->vm_end - vma->vm_start; + for (i = 0; i < NETMAP_POOLS_NR; i++) { /* loop through obj_pools */ + const struct netmap_obj_pool *p = &nm_mem.pools[i]; + /* + * In each pool memory is allocated in clusters + * of size _clustsize, each containing clustentries + * entries. For each object k we already store the + * vtophys mapping in lut[k] so we use that, scanning + * the lut[] array in steps of clustentries, + * and we map each cluster (not individual pages, + * it would be overkill). + */ + + /* + * We interpret vm_pgoff as an offset into the whole + * netmap memory, as if all clusters where contiguous. + */ + for (lut_skip = 0, j = 0; j < p->_numclusters; j++, lut_skip += p->clustentries) { + unsigned long paddr, mapsize; + if (p->_clustsize <= off) { + off -= p->_clustsize; + continue; + } + l_entry = &p->lut[lut_skip]; /* first obj in the cluster */ + paddr = l_entry->paddr + off; + mapsize = p->_clustsize - off; + off = 0; + if (mapsize > tomap) + mapsize = tomap; + ND("remap_pfn_range(%lx, %lx, %lx)", + vma->vm_start + user_skip, + paddr >> PAGE_SHIFT, mapsize); + if (remap_pfn_range(vma, vma->vm_start + user_skip, + paddr >> PAGE_SHIFT, mapsize, + vma->vm_page_prot)) + return -EAGAIN; // XXX check return value + user_skip += mapsize; + tomap -= mapsize; + if (tomap == 0) + goto done; + } + } +done: + + return 0; +} + +static netdev_tx_t +linux_netmap_start(struct sk_buff *skb, struct net_device *dev) +{ + netmap_start(dev, skb); + return (NETDEV_TX_OK); +} + + +#if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,37) // XXX was 38 +#define LIN_IOCTL_NAME .ioctl +int +linux_netmap_ioctl(struct inode *inode, struct file *file, u_int cmd, u_long data /* arg */) +#else +#define LIN_IOCTL_NAME .unlocked_ioctl +long +linux_netmap_ioctl(struct file *file, u_int cmd, u_long data /* arg */) +#endif +{ + int ret; + struct nmreq nmr; + bzero(&nmr, sizeof(nmr)); + + if (data && copy_from_user(&nmr, (void *)data, sizeof(nmr) ) != 0) + return -EFAULT; + ret = netmap_ioctl(NULL, cmd, (caddr_t)&nmr, 0, (void *)file); + pr_info("netmap ioctl %u ret = %d\n", cmd, ret); + if (data && copy_to_user((void*)data, &nmr, sizeof(nmr) ) != 0) + return -EFAULT; + return -ret; +} + + +static int +netmap_release(struct inode *inode, struct file *file) +{ + (void)inode; /* UNUSED */ + if (file->private_data) + netmap_dtor(file->private_data); + return (0); +} + +static int +linux_netmap_open(struct inode *inode, struct file *file) +{ + struct netmap_priv_d *priv; + (void)inode; /* UNUSED */ + + priv = malloc(sizeof(struct netmap_priv_d), M_DEVBUF, + M_NOWAIT | M_ZERO); + if (priv == NULL) + return -ENOMEM; + + file->private_data = priv; + + return (0); +} + +static struct file_operations netmap_fops = { + .open = linux_netmap_open, + .mmap = linux_netmap_mmap, + LIN_IOCTL_NAME = linux_netmap_ioctl, + .poll = linux_netmap_poll, + .release = netmap_release, +}; + +static struct miscdevice netmap_cdevsw = { /* same name as FreeBSD */ + MISC_DYNAMIC_MINOR, + "netmap", + &netmap_fops, +}; + +static int netmap_init(void); +static void netmap_fini(void); + +/* Errors have negative values on linux */ +static int linux_netmap_init(void) +{ + return -netmap_init(); +} + +module_init(linux_netmap_init); +module_exit(netmap_fini); +/* export certain symbols to other modules */ +EXPORT_SYMBOL(netmap_attach); // driver attach routines +EXPORT_SYMBOL(netmap_detach); // driver detach routines +EXPORT_SYMBOL(netmap_ring_reinit); // ring init on error +EXPORT_SYMBOL(netmap_buffer_lut); +EXPORT_SYMBOL(netmap_total_buffers); // index check +EXPORT_SYMBOL(netmap_buffer_base); +EXPORT_SYMBOL(netmap_reset); // ring init routines +EXPORT_SYMBOL(netmap_buf_size); +EXPORT_SYMBOL(netmap_rx_irq); // default irq handler +EXPORT_SYMBOL(netmap_no_pendintr); // XXX mitigation - should go away + + +MODULE_AUTHOR("Matteo Landi, Luigi Rizzo"); +MODULE_DESCRIPTION("The netmap packet I/O framework"); +MODULE_LICENSE("Dual BSD/GPL"); /* the code here is all BSD. */ + +#else /* __FreeBSD__ */ + +static struct cdevsw netmap_cdevsw = { + .d_version = D_VERSION, + .d_name = "netmap", + .d_open = netmap_open, + .d_mmap = netmap_mmap, + .d_mmap_single = netmap_mmap_single, + .d_ioctl = netmap_ioctl, + .d_poll = netmap_poll, + .d_close = netmap_close, +}; +#endif /* __FreeBSD__ */ + +#ifdef NM_BRIDGE +/* + *---- support for virtual bridge ----- + */ + +/* ----- FreeBSD if_bridge hash function ------- */ + +/* + * The following hash function is adapted from "Hash Functions" by Bob Jenkins + * ("Algorithm Alley", Dr. Dobbs Journal, September 1997). + * + * http://www.burtleburtle.net/bob/hash/spooky.html + */ +#define mix(a, b, c) \ +do { \ + a -= b; a -= c; a ^= (c >> 13); \ + b -= c; b -= a; b ^= (a << 8); \ + c -= a; c -= b; c ^= (b >> 13); \ + a -= b; a -= c; a ^= (c >> 12); \ + b -= c; b -= a; b ^= (a << 16); \ + c -= a; c -= b; c ^= (b >> 5); \ + a -= b; a -= c; a ^= (c >> 3); \ + b -= c; b -= a; b ^= (a << 10); \ + c -= a; c -= b; c ^= (b >> 15); \ +} while (/*CONSTCOND*/0) + +static __inline uint32_t +nm_bridge_rthash(const uint8_t *addr) +{ + uint32_t a = 0x9e3779b9, b = 0x9e3779b9, c = 0; // hask key + + b += addr[5] << 8; + b += addr[4]; + a += addr[3] << 24; + a += addr[2] << 16; + a += addr[1] << 8; + a += addr[0]; + + mix(a, b, c); +#define BRIDGE_RTHASH_MASK (NM_BDG_HASH-1) + return (c & BRIDGE_RTHASH_MASK); +} + +#undef mix + + +static int +bdg_netmap_reg(struct ifnet *ifp, int onoff) +{ + int i, err = 0; + struct nm_bridge *b = ifp->if_bridge; + + BDG_LOCK(b); + if (onoff) { + /* the interface must be already in the list. + * only need to mark the port as active + */ + ND("should attach %s to the bridge", ifp->if_xname); + for (i=0; i < NM_BDG_MAXPORTS; i++) + if (b->bdg_ports[i] == ifp) + break; + if (i == NM_BDG_MAXPORTS) { + D("no more ports available"); + err = EINVAL; + goto done; + } + ND("setting %s in netmap mode", ifp->if_xname); + ifp->if_capenable |= IFCAP_NETMAP; + NA(ifp)->bdg_port = i; + b->act_ports |= (1<<i); + b->bdg_ports[i] = ifp; + } else { + /* should be in the list, too -- remove from the mask */ + ND("removing %s from netmap mode", ifp->if_xname); + ifp->if_capenable &= ~IFCAP_NETMAP; + i = NA(ifp)->bdg_port; + b->act_ports &= ~(1<<i); + } +done: + BDG_UNLOCK(b); + return err; +} + + +static int +nm_bdg_flush(struct nm_bdg_fwd *ft, int n, struct ifnet *ifp) +{ + int i, ifn; + uint64_t all_dst, dst; + uint32_t sh, dh; + uint64_t mysrc = 1 << NA(ifp)->bdg_port; + uint64_t smac, dmac; + struct netmap_slot *slot; + struct nm_bridge *b = ifp->if_bridge; + + ND("prepare to send %d packets, act_ports 0x%x", n, b->act_ports); + /* only consider valid destinations */ + all_dst = (b->act_ports & ~mysrc); + /* first pass: hash and find destinations */ + for (i = 0; likely(i < n); i++) { + uint8_t *buf = ft[i].buf; + dmac = le64toh(*(uint64_t *)(buf)) & 0xffffffffffff; + smac = le64toh(*(uint64_t *)(buf + 4)); + smac >>= 16; + if (unlikely(netmap_verbose)) { + uint8_t *s = buf+6, *d = buf; + D("%d len %4d %02x:%02x:%02x:%02x:%02x:%02x -> %02x:%02x:%02x:%02x:%02x:%02x", + i, + ft[i].len, + s[0], s[1], s[2], s[3], s[4], s[5], + d[0], d[1], d[2], d[3], d[4], d[5]); + } + /* + * The hash is somewhat expensive, there might be some + * worthwhile optimizations here. + */ + if ((buf[6] & 1) == 0) { /* valid src */ + uint8_t *s = buf+6; + sh = nm_bridge_rthash(buf+6); // XXX hash of source + /* update source port forwarding entry */ + b->ht[sh].mac = smac; /* XXX expire ? */ + b->ht[sh].ports = mysrc; + if (netmap_verbose) + D("src %02x:%02x:%02x:%02x:%02x:%02x on port %d", + s[0], s[1], s[2], s[3], s[4], s[5], NA(ifp)->bdg_port); + } + dst = 0; + if ( (buf[0] & 1) == 0) { /* unicast */ + uint8_t *d = buf; + dh = nm_bridge_rthash(buf); // XXX hash of dst + if (b->ht[dh].mac == dmac) { /* found dst */ + dst = b->ht[dh].ports; + if (netmap_verbose) + D("dst %02x:%02x:%02x:%02x:%02x:%02x to port %x", + d[0], d[1], d[2], d[3], d[4], d[5], (uint32_t)(dst >> 16)); + } + } + if (dst == 0) + dst = all_dst; + dst &= all_dst; /* only consider valid ports */ + if (unlikely(netmap_verbose)) + D("pkt goes to ports 0x%x", (uint32_t)dst); + ft[i].dst = dst; + } + + /* second pass, scan interfaces and forward */ + all_dst = (b->act_ports & ~mysrc); + for (ifn = 0; all_dst; ifn++) { + struct ifnet *dst_ifp = b->bdg_ports[ifn]; + struct netmap_adapter *na; + struct netmap_kring *kring; + struct netmap_ring *ring; + int j, lim, sent, locked; + + if (!dst_ifp) + continue; + ND("scan port %d %s", ifn, dst_ifp->if_xname); + dst = 1 << ifn; + if ((dst & all_dst) == 0) /* skip if not set */ + continue; + all_dst &= ~dst; /* clear current node */ + na = NA(dst_ifp); + + ring = NULL; + kring = NULL; + lim = sent = locked = 0; + /* inside, scan slots */ + for (i = 0; likely(i < n); i++) { + if ((ft[i].dst & dst) == 0) + continue; /* not here */ + if (!locked) { + kring = &na->rx_rings[0]; + ring = kring->ring; + lim = kring->nkr_num_slots - 1; + na->nm_lock(dst_ifp, NETMAP_RX_LOCK, 0); + locked = 1; + } + if (unlikely(kring->nr_hwavail >= lim)) { + if (netmap_verbose) + D("rx ring full on %s", ifp->if_xname); + break; + } + j = kring->nr_hwcur + kring->nr_hwavail; + if (j > lim) + j -= kring->nkr_num_slots; + slot = &ring->slot[j]; + ND("send %d %d bytes at %s:%d", i, ft[i].len, dst_ifp->if_xname, j); + pkt_copy(ft[i].buf, NMB(slot), ft[i].len); + slot->len = ft[i].len; + kring->nr_hwavail++; + sent++; + } + if (locked) { + ND("sent %d on %s", sent, dst_ifp->if_xname); + if (sent) + selwakeuppri(&kring->si, PI_NET); + na->nm_lock(dst_ifp, NETMAP_RX_UNLOCK, 0); + } + } + return 0; +} + +/* + * main dispatch routine + */ +static int +bdg_netmap_txsync(struct ifnet *ifp, u_int ring_nr, int do_lock) +{ + struct netmap_adapter *na = NA(ifp); + struct netmap_kring *kring = &na->tx_rings[ring_nr]; + struct netmap_ring *ring = kring->ring; + int i, j, k, lim = kring->nkr_num_slots - 1; + struct nm_bdg_fwd *ft = (struct nm_bdg_fwd *)(ifp + 1); + int ft_i; /* position in the forwarding table */ + + k = ring->cur; + if (k > lim) + return netmap_ring_reinit(kring); + if (do_lock) + na->nm_lock(ifp, NETMAP_TX_LOCK, ring_nr); + + if (netmap_bridge <= 0) { /* testing only */ + j = k; // used all + goto done; + } + if (netmap_bridge > NM_BDG_BATCH) + netmap_bridge = NM_BDG_BATCH; + + ft_i = 0; /* start from 0 */ + for (j = kring->nr_hwcur; likely(j != k); j = unlikely(j == lim) ? 0 : j+1) { + struct netmap_slot *slot = &ring->slot[j]; + int len = ft[ft_i].len = slot->len; + char *buf = ft[ft_i].buf = NMB(slot); + + prefetch(buf); + if (unlikely(len < 14)) + continue; + if (unlikely(++ft_i == netmap_bridge)) + ft_i = nm_bdg_flush(ft, ft_i, ifp); + } + if (ft_i) + ft_i = nm_bdg_flush(ft, ft_i, ifp); + /* count how many packets we sent */ + i = k - j; + if (i < 0) + i += kring->nkr_num_slots; + kring->nr_hwavail = kring->nkr_num_slots - 1 - i; + if (j != k) + D("early break at %d/ %d, avail %d", j, k, kring->nr_hwavail); + +done: + kring->nr_hwcur = j; + ring->avail = kring->nr_hwavail; + if (do_lock) + na->nm_lock(ifp, NETMAP_TX_UNLOCK, ring_nr); + + if (netmap_verbose) + D("%s ring %d lock %d", ifp->if_xname, ring_nr, do_lock); + return 0; +} + +static int +bdg_netmap_rxsync(struct ifnet *ifp, u_int ring_nr, int do_lock) +{ + struct netmap_adapter *na = NA(ifp); + struct netmap_kring *kring = &na->rx_rings[ring_nr]; + struct netmap_ring *ring = kring->ring; + u_int j, n, lim = kring->nkr_num_slots - 1; + u_int k = ring->cur, resvd = ring->reserved; + + ND("%s ring %d lock %d avail %d", + ifp->if_xname, ring_nr, do_lock, kring->nr_hwavail); + + if (k > lim) + return netmap_ring_reinit(kring); + if (do_lock) + na->nm_lock(ifp, NETMAP_RX_LOCK, ring_nr); + + /* skip past packets that userspace has released */ + j = kring->nr_hwcur; /* netmap ring index */ + if (resvd > 0) { + if (resvd + ring->avail >= lim + 1) { + D("XXX invalid reserve/avail %d %d", resvd, ring->avail); + ring->reserved = resvd = 0; // XXX panic... + } + k = (k >= resvd) ? k - resvd : k + lim + 1 - resvd; + } + + if (j != k) { /* userspace has released some packets. */ + n = k - j; + if (n < 0) + n += kring->nkr_num_slots; + ND("userspace releases %d packets", n); + for (n = 0; likely(j != k); n++) { + struct netmap_slot *slot = &ring->slot[j]; + void *addr = NMB(slot); + + if (addr == netmap_buffer_base) { /* bad buf */ + if (do_lock) + na->nm_lock(ifp, NETMAP_RX_UNLOCK, ring_nr); + return netmap_ring_reinit(kring); + } + /* decrease refcount for buffer */ + + slot->flags &= ~NS_BUF_CHANGED; + j = unlikely(j == lim) ? 0 : j + 1; + } + kring->nr_hwavail -= n; + kring->nr_hwcur = k; + } + /* tell userspace that there are new packets */ + ring->avail = kring->nr_hwavail - resvd; + + if (do_lock) + na->nm_lock(ifp, NETMAP_RX_UNLOCK, ring_nr); + return 0; +} + +static void +bdg_netmap_attach(struct ifnet *ifp) +{ + struct netmap_adapter na; + + ND("attaching virtual bridge"); + bzero(&na, sizeof(na)); + + na.ifp = ifp; + na.separate_locks = 1; + na.num_tx_desc = NM_BRIDGE_RINGSIZE; + na.num_rx_desc = NM_BRIDGE_RINGSIZE; + na.nm_txsync = bdg_netmap_txsync; + na.nm_rxsync = bdg_netmap_rxsync; + na.nm_register = bdg_netmap_reg; + netmap_attach(&na, 1); +} + +#endif /* NM_BRIDGE */ + +static struct cdev *netmap_dev; /* /dev/netmap character device. */ + + +/* + * Module loader. + * + * Create the /dev/netmap device and initialize all global + * variables. + * + * Return 0 on success, errno on failure. + */ +static int +netmap_init(void) +{ + int error; + + error = netmap_memory_init(); + if (error != 0) { + printf("netmap: unable to initialize the memory allocator.\n"); + return (error); + } + printf("netmap: loaded module\n"); + netmap_dev = make_dev(&netmap_cdevsw, 0, UID_ROOT, GID_WHEEL, 0660, + "netmap"); + +#ifdef NM_BRIDGE + { + int i; + for (i = 0; i < NM_BRIDGES; i++) + mtx_init(&nm_bridges[i].bdg_lock, "bdg lock", "bdg_lock", MTX_DEF); + } +#endif + return (error); +} + + +/* + * Module unloader. + * + * Free all the memory, and destroy the ``/dev/netmap`` device. + */ +static void +netmap_fini(void) +{ + destroy_dev(netmap_dev); + netmap_memory_fini(); + printf("netmap: unloaded module.\n"); +} + + +#ifdef __FreeBSD__ +/* + * Kernel entry point. + * + * Initialize/finalize the module and return. + * + * Return 0 on success, errno on failure. + */ +static int +netmap_loader(__unused struct module *module, int event, __unused void *arg) +{ + int error = 0; + + switch (event) { + case MOD_LOAD: + error = netmap_init(); + break; + + case MOD_UNLOAD: + netmap_fini(); + break; + + default: + error = EOPNOTSUPP; + break; + } + + return (error); +} + + +DEV_MODULE(netmap, netmap_loader, NULL); +#endif /* __FreeBSD__ */ --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/drivers/staging/netmap/README 2013-03-10 10:08:20.327671428 -0700 @@ -0,0 +1,127 @@ +# $Id: README 10832 2012-03-22 18:22:42Z luigi $ + +NETMAP FOR LINUX +---------------- + +This directory contains a version of the "netmap" code for Linux. + +Netmap is a BSD-licensed framework that supports line-rate direct packet +I/O even on 10GBit/s interfaces (14.88Mpps) with limited system load, +and includes a libpcap emulation library to port applications. See + + http://info.iet.unipi.it/~luigi/netmap/ + +for more details. There you can also find the latest versions +of the code and documentation as well as pre-built TinyCore +images based on linux 3.0.3 and containing the netmap modules +and some test applications. + +This is a preliminary version supporting the ixgbe and e1000/e1000e +driver. Patches for other devices (igb, r8169, forcedeth) are +untested and probably not working yet. + +Netmap relies on a kernel module (netmap_lin.ko) and slightly modified +device drivers. Userspace programs can use the native API (documented +in netmap.4) or a libpcap emulation library. + + Directory structure for this archive + + . documentation, patches etc. + include/net header files for user programs + net/netmap kernel core files, + sample applications, manpage + net/* patched device drivers for a 3.0.x linux version. + +HOW TO BUILD THE CODE +--------------------- + +1. make sure you have kernel sources/headers matching your installed system + +2. do the following + (cd net; make KSRC=/usr/src/linux-kernel-source-or-headers ) + this produces net/netmap/netmap_lin.ko and other kernel modules. + +3. to build sample applications, run + (cd net/netmap; make apps ) + (you will need the pthreads and libpcap-dev packages to build them) + +HOW TO USE THE CODE +------------------- + + REMEMBER + THIS IS EXPERIMENTAL CODE WHICH MAY CRASH YOUR SYSTEM. + USE IT AT YOUR OWN RISk. + +Whether you built your own modules, or are using the prebuilt +TinyCore image, the following steps can be used for initial testing: + +1. unload any modules for the network cards you want to use, e.g. + sudo rmmod ixgbe + sudo rmmod e1000 + sudo rmmod e1000e + +2. load netmap and device driver module + sudo insmod net/netmap/netmap_lin.ko + sudo insmod net/ixgbe/ixgbe.ko + sudo insmod net/ixgbe/e1000.ko + sudo insmod net/ixgbe/e1000e.ko + +3. turn the interface(s) up + + sudo ifconfig eth0 up # and same for others + +4. Run test applications -- as an example, pkt-gen is a raw packet + sender/receiver which can do line rate on a 10G interface + + # send about 500 million packets of 64 bytes each. + # wait 5s before starting, so the link can go up + sudo pkt-gen -i eth0 -t 500111222 -l 64 -w 5 + # you should see about 14.2 Mpps + + sudo pkt-gen -i eth0 # act as a receiver + + +COMMON PROBLEMS +---------------- + +* switching in/out of netmap mode causes the link to go down and up. + If your card is connected to a switch with spanning tree enabled, + the switch will likely MUTE THE LINK FOR 10 SECONDS while it is + detecting the new topology. Either disable the spanning tree on + the switch or use long pauses before sending data; + +* Not all cards can do line rate no matter how fast is your software or + CPU. Several have hardware limitations that prevent reaching the peak + speed, especially for small packet sizes. Examples: + + - ixgbe cannot receive at line rate with packet sizes that are + not multiple of 64 (after CRC stripping). + This is especially evident with minimum-sized frames (-l 60 ) + + - some of the low-end 'e1000' cards can send 1.2 - 1.3Mpps instead + of the theoretical maximum (1.488Mpps) + + - the 'realtek' cards seem unable to send more than 450-500Kpps + even though they can receive at least 1.1Mpps + +* if the link is not up when the packet generator starts, you will + see frequent messages about a link reset. While we work on a fix, + use the '-w' argument on the generator to specify a longer timeout + +* the ixgbe driver (and perhaps others) is severely slowed down if the + remote party is senting flow control frames to slow down traffic. + If that happens try to use the ethtool command to disable flow control. + + +REVISION HISTORY +----------------- + +20120322 - fixed the 'igb' driver, now it can send and receive correctly + (the problem was in netmap_rx_irq() so it might have affected + other multiqueue cards). + Also tested the 'r8169' in transmit mode. + Added comments on switches and spanning tree. + +20120217 - initial version. Only ixgbe, e1000 and e1000e are working. + Other drivers (igb, r8169, forcedeth) are supplied only as a + proof of concept. --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/drivers/staging/netmap/TODO 2013-03-10 16:45:40.377960319 -0700 @@ -0,0 +1,21 @@ + +This driver is functional but needs some cleanup before it is ready +for included in networking subsystem. + + - Use unifdef to elimiate non Linux code + - Get rid of wrapper code including bsd_glue.h + - Fix coding style to be Linux rather than BSD + - Fix whitespace warnings + - Add stubs so that ethernet drivers can use without #ifdef's + - Autoload the module, assign a real minor number and have aliases + - Rework documentation and put in Documentation/networking/ + - Remove memory allocator kludge wrapper + - Remove devfs kludge wrappers + - Review configuration API's and management (counters) + - Locking should be consistent (ie get rid of separate_locks option)? + - Doesn't work with DMA remapping + - Use __u32 rather than uint32_t in headers + - netmap_user.h headers should use inline instead of macros + +Fundamentally this will break source compatiablity with FreeBSD, +but this is not our problem. --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/drivers/staging/netmap/netmap_mem2.c 2013-03-10 10:08:20.327671428 -0700 @@ -0,0 +1,974 @@ +/* + * Copyright (C) 2012 Matteo Landi, Luigi Rizzo, Giuseppe Lettieri. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* + * $FreeBSD: head/sys/dev/netmap/netmap_mem2.c 234290 2012-04-14 16:44:18Z luigi $ + * $Id: netmap_mem2.c 12010 2013-01-23 03:57:30Z luigi $ + * + * (New) memory allocator for netmap + */ + +/* + * This allocator creates three memory regions: + * nm_if_pool for the struct netmap_if + * nm_ring_pool for the struct netmap_ring + * nm_buf_pool for the packet buffers. + * + * All regions need to be multiple of a page size as we export them to + * userspace through mmap. Only the latter needs to be dma-able, + * but for convenience use the same type of allocator for all. + * + * Once mapped, the three regions are exported to userspace + * as a contiguous block, starting from nm_if_pool. Each + * cluster (and pool) is an integral number of pages. + * [ . . . ][ . . . . . .][ . . . . . . . . . .] + * nm_if nm_ring nm_buf + * + * The userspace areas contain offsets of the objects in userspace. + * When (at init time) we write these offsets, we find out the index + * of the object, and from there locate the offset from the beginning + * of the region. + * + * The invididual allocators manage a pool of memory for objects of + * the same size. + * The pool is split into smaller clusters, whose size is a + * multiple of the page size. The cluster size is chosen + * to minimize the waste for a given max cluster size + * (we do it by brute force, as we have relatively few object + * per cluster). + * + * Objects are aligned to the cache line (64 bytes) rounding up object + * sizes when needed. A bitmap contains the state of each object. + * Allocation scans the bitmap; this is done only on attach, so we are not + * too worried about performance + * + * For each allocator we can define (thorugh sysctl) the size and + * number of each object. Memory is allocated at the first use of a + * netmap file descriptor, and can be freed when all such descriptors + * have been released (including unmapping the memory). + * If memory is scarce, the system tries to get as much as possible + * and the sysctl values reflect the actual allocation. + * Together with desired values, the sysctl export also absolute + * min and maximum values that cannot be overridden. + * + * struct netmap_if: + * variable size, max 16 bytes per ring pair plus some fixed amount. + * 1024 bytes should be large enough in practice. + * + * In the worst case we have one netmap_if per ring in the system. + * + * struct netmap_ring + * variable too, 8 byte per slot plus some fixed amount. + * Rings can be large (e.g. 4k slots, or >32Kbytes). + * We default to 36 KB (9 pages), and a few hundred rings. + * + * struct netmap_buffer + * The more the better, both because fast interfaces tend to have + * many slots, and because we may want to use buffers to store + * packets in userspace avoiding copies. + * Must contain a full frame (eg 1518, or more for vlans, jumbo + * frames etc.) plus be nicely aligned, plus some NICs restrict + * the size to multiple of 1K or so. Default to 2K + */ + +#ifndef CONSERVATIVE +#define NETMAP_BUF_MAX_NUM 20*4096*2 /* large machine */ +#else /* CONSERVATIVE */ +#define NETMAP_BUF_MAX_NUM 20000 /* 40MB */ +#endif + +#ifdef linux +#define NMA_LOCK_T struct semaphore +#define NMA_LOCK_INIT() sema_init(&nm_mem.nm_mtx, 1) +#define NMA_LOCK_DESTROY() +#define NMA_LOCK() down(&nm_mem.nm_mtx) +#define NMA_UNLOCK() up(&nm_mem.nm_mtx) +#else /* !linux */ +#define NMA_LOCK_T struct mtx +#define NMA_LOCK_INIT() mtx_init(&nm_mem.nm_mtx, "netmap memory allocator lock", NULL, MTX_DEF) +#define NMA_LOCK_DESTROY() mtx_destroy(&nm_mem.nm_mtx) +#define NMA_LOCK() mtx_lock(&nm_mem.nm_mtx) +#define NMA_UNLOCK() mtx_unlock(&nm_mem.nm_mtx) +#endif /* linux */ + +enum { + NETMAP_IF_POOL = 0, + NETMAP_RING_POOL, + NETMAP_BUF_POOL, + NETMAP_POOLS_NR +}; + + +struct netmap_obj_params { + u_int size; + u_int num; +}; + + +struct netmap_obj_params netmap_params[NETMAP_POOLS_NR] = { + [NETMAP_IF_POOL] = { + .size = 1024, + .num = 100, + }, + [NETMAP_RING_POOL] = { + .size = 9*PAGE_SIZE, + .num = 200, + }, + [NETMAP_BUF_POOL] = { + .size = 2048, + .num = NETMAP_BUF_MAX_NUM, + }, +}; + + +struct netmap_obj_pool { + char name[16]; /* name of the allocator */ + u_int objtotal; /* actual total number of objects. */ + u_int objfree; /* number of free objects. */ + u_int clustentries; /* actual objects per cluster */ + + /* limits */ + u_int objminsize; /* minimum object size */ + u_int objmaxsize; /* maximum object size */ + u_int nummin; /* minimum number of objects */ + u_int nummax; /* maximum number of objects */ + + /* the total memory space is _numclusters*_clustsize */ + u_int _numclusters; /* how many clusters */ + u_int _clustsize; /* cluster size */ + u_int _objsize; /* actual object size */ + + u_int _memtotal; /* _numclusters*_clustsize */ + struct lut_entry *lut; /* virt,phys addresses, objtotal entries */ + uint32_t *bitmap; /* one bit per buffer, 1 means free */ + uint32_t bitmap_slots; /* number of uint32 entries in bitmap */ +}; + + +struct netmap_mem_d { + NMA_LOCK_T nm_mtx; /* protect the allocator */ + u_int nm_totalsize; /* shorthand */ + + int finalized; /* !=0 iff preallocation done */ + int lasterr; /* last error for curr config */ + int refcount; /* existing priv structures */ + /* the three allocators */ + struct netmap_obj_pool pools[NETMAP_POOLS_NR]; +}; + + +static struct netmap_mem_d nm_mem = { /* Our memory allocator. */ + .pools = { + [NETMAP_IF_POOL] = { + .name = "netmap_if", + .objminsize = sizeof(struct netmap_if), + .objmaxsize = 4096, + .nummin = 10, /* don't be stingy */ + .nummax = 10000, /* XXX very large */ + }, + [NETMAP_RING_POOL] = { + .name = "netmap_ring", + .objminsize = sizeof(struct netmap_ring), + .objmaxsize = 32*PAGE_SIZE, + .nummin = 2, + .nummax = 1024, + }, + [NETMAP_BUF_POOL] = { + .name = "netmap_buf", + .objminsize = 64, + .objmaxsize = 65536, + .nummin = 4, + .nummax = 1000000, /* one million! */ + }, + }, +}; + +struct lut_entry *netmap_buffer_lut; /* exported */ + +/* memory allocator related sysctls */ + +#define STRINGIFY(x) #x + +#define DECLARE_SYSCTLS(id, name) \ + /* TUNABLE_INT("hw.netmap." STRINGIFY(name) "_size", &netmap_params[id].size); */ \ + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_size, \ + CTLFLAG_RW, &netmap_params[id].size, 0, "Requested size of netmap " STRINGIFY(name) "s"); \ + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_curr_size, \ + CTLFLAG_RD, &nm_mem.pools[id]._objsize, 0, "Current size of netmap " STRINGIFY(name) "s"); \ + /* TUNABLE_INT("hw.netmap." STRINGIFY(name) "_num", &netmap_params[id].num); */ \ + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_num, \ + CTLFLAG_RW, &netmap_params[id].num, 0, "Requested number of netmap " STRINGIFY(name) "s"); \ + SYSCTL_INT(_dev_netmap, OID_AUTO, name##_curr_num, \ + CTLFLAG_RD, &nm_mem.pools[id].objtotal, 0, "Current number of netmap " STRINGIFY(name) "s") + +DECLARE_SYSCTLS(NETMAP_IF_POOL, if); +DECLARE_SYSCTLS(NETMAP_RING_POOL, ring); +DECLARE_SYSCTLS(NETMAP_BUF_POOL, buf); + +/* + * Convert a userspace offset to a phisical address. + * XXX re-do in a simpler way. + * + * The idea here is to hide userspace applications the fact that pre-allocated + * memory is not contiguous, but fragmented across different clusters and + * smaller memory allocators. Consequently, first of all we need to find which + * allocator is owning provided offset, then we need to find out the physical + * address associated to target page (this is done using the look-up table. + */ +static inline vm_paddr_t +netmap_ofstophys(vm_offset_t offset) +{ + int i; + vm_offset_t o = offset; + struct netmap_obj_pool *p = nm_mem.pools; + + for (i = 0; i < NETMAP_POOLS_NR; offset -= p[i]._memtotal, i++) { + if (offset >= p[i]._memtotal) + continue; + // XXX now scan the clusters + return p[i].lut[offset / p[i]._objsize].paddr + + offset % p[i]._objsize; + } + /* this is only in case of errors */ + D("invalid ofs 0x%x out of 0x%x 0x%x 0x%x", (u_int)o, + p[NETMAP_IF_POOL]._memtotal, + p[NETMAP_IF_POOL]._memtotal + + p[NETMAP_RING_POOL]._memtotal, + p[NETMAP_IF_POOL]._memtotal + + p[NETMAP_RING_POOL]._memtotal + + p[NETMAP_BUF_POOL]._memtotal); + return 0; // XXX bad address +} + +/* + * we store objects by kernel address, need to find the offset + * within the pool to export the value to userspace. + * Algorithm: scan until we find the cluster, then add the + * actual offset in the cluster + */ +static ssize_t +netmap_obj_offset(struct netmap_obj_pool *p, const void *vaddr) +{ + int i, k = p->clustentries, n = p->objtotal; + ssize_t ofs = 0; + + for (i = 0; i < n; i += k, ofs += p->_clustsize) { + const char *base = p->lut[i].vaddr; + ssize_t relofs = (const char *) vaddr - base; + + if (relofs < 0 || relofs > p->_clustsize) + continue; + + ofs = ofs + relofs; + ND("%s: return offset %d (cluster %d) for pointer %p", + p->name, ofs, i, vaddr); + return ofs; + } + D("address %p is not contained inside any cluster (%s)", + vaddr, p->name); + return 0; /* An error occurred */ +} + +/* Helper functions which convert virtual addresses to offsets */ +#define netmap_if_offset(v) \ + netmap_obj_offset(&nm_mem.pools[NETMAP_IF_POOL], (v)) + +#define netmap_ring_offset(v) \ + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + \ + netmap_obj_offset(&nm_mem.pools[NETMAP_RING_POOL], (v))) + +#define netmap_buf_offset(v) \ + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + \ + nm_mem.pools[NETMAP_RING_POOL]._memtotal + \ + netmap_obj_offset(&nm_mem.pools[NETMAP_BUF_POOL], (v))) + + +/* + * report the index, and use start position as a hint, + * otherwise buffer allocation becomes terribly expensive. + */ +static void * +netmap_obj_malloc(struct netmap_obj_pool *p, int len, uint32_t *start, uint32_t *index) +{ + uint32_t i = 0; /* index in the bitmap */ + uint32_t mask, j; /* slot counter */ + void *vaddr = NULL; + + if (len > p->_objsize) { + D("%s request size %d too large", p->name, len); + // XXX cannot reduce the size + return NULL; + } + + if (p->objfree == 0) { + D("%s allocator: run out of memory", p->name); + return NULL; + } + if (start) + i = *start; + + /* termination is guaranteed by p->free, but better check bounds on i */ + while (vaddr == NULL && i < p->bitmap_slots) { + uint32_t cur = p->bitmap[i]; + if (cur == 0) { /* bitmask is fully used */ + i++; + continue; + } + /* locate a slot */ + for (j = 0, mask = 1; (cur & mask) == 0; j++, mask <<= 1) + ; + + p->bitmap[i] &= ~mask; /* mark object as in use */ + p->objfree--; + + vaddr = p->lut[i * 32 + j].vaddr; + if (index) + *index = i * 32 + j; + } + ND("%s allocator: allocated object @ [%d][%d]: vaddr %p", i, j, vaddr); + + if (start) + *start = i; + return vaddr; +} + + +/* + * free by index, not by address + */ +static void +netmap_obj_free(struct netmap_obj_pool *p, uint32_t j) +{ + if (j >= p->objtotal) { + D("invalid index %u, max %u", j, p->objtotal); + return; + } + p->bitmap[j / 32] |= (1 << (j % 32)); + p->objfree++; + return; +} + +static void +netmap_obj_free_va(struct netmap_obj_pool *p, void *vaddr) +{ + int i, j, n = p->_memtotal / p->_clustsize; + + for (i = 0, j = 0; i < n; i++, j += p->clustentries) { + void *base = p->lut[i * p->clustentries].vaddr; + ssize_t relofs = (ssize_t) vaddr - (ssize_t) base; + + /* Given address, is out of the scope of the current cluster.*/ + if (vaddr < base || relofs > p->_clustsize) + continue; + + j = j + relofs / p->_objsize; + KASSERT(j != 0, ("Cannot free object 0")); + netmap_obj_free(p, j); + return; + } + D("address %p is not contained inside any cluster (%s)", + vaddr, p->name); +} + +#define netmap_if_malloc(len) netmap_obj_malloc(&nm_mem.pools[NETMAP_IF_POOL], len, NULL, NULL) +#define netmap_if_free(v) netmap_obj_free_va(&nm_mem.pools[NETMAP_IF_POOL], (v)) +#define netmap_ring_malloc(len) netmap_obj_malloc(&nm_mem.pools[NETMAP_RING_POOL], len, NULL, NULL) +#define netmap_ring_free(v) netmap_obj_free_va(&nm_mem.pools[NETMAP_RING_POOL], (v)) +#define netmap_buf_malloc(_pos, _index) \ + netmap_obj_malloc(&nm_mem.pools[NETMAP_BUF_POOL], NETMAP_BUF_SIZE, _pos, _index) + + +/* Return the index associated to the given packet buffer */ +#define netmap_buf_index(v) \ + (netmap_obj_offset(&nm_mem.pools[NETMAP_BUF_POOL], (v)) / nm_mem.pools[NETMAP_BUF_POOL]._objsize) + + +/* Return nonzero on error */ +static int +netmap_new_bufs(struct netmap_if *nifp, + struct netmap_slot *slot, u_int n) +{ + struct netmap_obj_pool *p = &nm_mem.pools[NETMAP_BUF_POOL]; + int i = 0; /* slot counter */ + uint32_t pos = 0; /* slot in p->bitmap */ + uint32_t index = 0; /* buffer index */ + + (void)nifp; /* UNUSED */ + for (i = 0; i < n; i++) { + void *vaddr = netmap_buf_malloc(&pos, &index); + if (vaddr == NULL) { + D("unable to locate empty packet buffer"); + goto cleanup; + } + slot[i].buf_idx = index; + slot[i].len = p->_objsize; + /* XXX setting flags=NS_BUF_CHANGED forces a pointer reload + * in the NIC ring. This is a hack that hides missing + * initializations in the drivers, and should go away. + */ + slot[i].flags = NS_BUF_CHANGED; + } + + ND("allocated %d buffers, %d available, first at %d", n, p->objfree, pos); + return (0); + +cleanup: + while (i > 0) { + i--; + netmap_obj_free(p, slot[i].buf_idx); + } + bzero(slot, n * sizeof(slot[0])); + return (ENOMEM); +} + + +static void +netmap_free_buf(struct netmap_if *nifp, uint32_t i) +{ + struct netmap_obj_pool *p = &nm_mem.pools[NETMAP_BUF_POOL]; + + if (i < 2 || i >= p->objtotal) { + D("Cannot free buf#%d: should be in [2, %d[", i, p->objtotal); + return; + } + netmap_obj_free(p, i); +} + +static void +netmap_reset_obj_allocator(struct netmap_obj_pool *p) +{ + if (p == NULL) + return; + if (p->bitmap) + free(p->bitmap, M_NETMAP); + p->bitmap = NULL; + if (p->lut) { + int i; + for (i = 0; i < p->objtotal; i += p->clustentries) { + if (p->lut[i].vaddr) + contigfree(p->lut[i].vaddr, p->_clustsize, M_NETMAP); + } + bzero(p->lut, sizeof(struct lut_entry) * p->objtotal); +#ifdef linux + vfree(p->lut); +#else + free(p->lut, M_NETMAP); +#endif + } + p->lut = NULL; +} + +/* + * Free all resources related to an allocator. + */ +static void +netmap_destroy_obj_allocator(struct netmap_obj_pool *p) +{ + if (p == NULL) + return; + netmap_reset_obj_allocator(p); +} + +/* + * We receive a request for objtotal objects, of size objsize each. + * Internally we may round up both numbers, as we allocate objects + * in small clusters multiple of the page size. + * In the allocator we don't need to store the objsize, + * but we do need to keep track of objtotal' and clustentries, + * as they are needed when freeing memory. + * + * XXX note -- userspace needs the buffers to be contiguous, + * so we cannot afford gaps at the end of a cluster. + */ + + +/* call with NMA_LOCK held */ +static int +netmap_config_obj_allocator(struct netmap_obj_pool *p, u_int objtotal, u_int objsize) +{ + int i, n; + u_int clustsize; /* the cluster size, multiple of page size */ + u_int clustentries; /* how many objects per entry */ + +#define MAX_CLUSTSIZE (1<<17) +#define LINE_ROUND 64 + if (objsize >= MAX_CLUSTSIZE) { + /* we could do it but there is no point */ + D("unsupported allocation for %d bytes", objsize); + goto error; + } + /* make sure objsize is a multiple of LINE_ROUND */ + i = (objsize & (LINE_ROUND - 1)); + if (i) { + D("XXX aligning object by %d bytes", LINE_ROUND - i); + objsize += LINE_ROUND - i; + } + if (objsize < p->objminsize || objsize > p->objmaxsize) { + D("requested objsize %d out of range [%d, %d]", + objsize, p->objminsize, p->objmaxsize); + goto error; + } + if (objtotal < p->nummin || objtotal > p->nummax) { + D("requested objtotal %d out of range [%d, %d]", + objtotal, p->nummin, p->nummax); + goto error; + } + /* + * Compute number of objects using a brute-force approach: + * given a max cluster size, + * we try to fill it with objects keeping track of the + * wasted space to the next page boundary. + */ + for (clustentries = 0, i = 1;; i++) { + u_int delta, used = i * objsize; + if (used > MAX_CLUSTSIZE) + break; + delta = used % PAGE_SIZE; + if (delta == 0) { // exact solution + clustentries = i; + break; + } + if (delta > ( (clustentries*objsize) % PAGE_SIZE) ) + clustentries = i; + } + // D("XXX --- ouch, delta %d (bad for buffers)", delta); + /* compute clustsize and round to the next page */ + clustsize = clustentries * objsize; + i = (clustsize & (PAGE_SIZE - 1)); + if (i) + clustsize += PAGE_SIZE - i; + if (netmap_verbose) + D("objsize %d clustsize %d objects %d", + objsize, clustsize, clustentries); + + /* + * The number of clusters is n = ceil(objtotal/clustentries) + * objtotal' = n * clustentries + */ + p->clustentries = clustentries; + p->_clustsize = clustsize; + n = (objtotal + clustentries - 1) / clustentries; + p->_numclusters = n; + p->objtotal = n * clustentries; + p->objfree = p->objtotal - 2; /* obj 0 and 1 are reserved */ + p->_memtotal = p->_numclusters * p->_clustsize; + p->_objsize = objsize; + + return 0; + +error: + p->_objsize = objsize; + p->objtotal = objtotal; + + return EINVAL; +} + + +/* call with NMA_LOCK held */ +static int +netmap_finalize_obj_allocator(struct netmap_obj_pool *p) +{ + int i, n; + + n = sizeof(struct lut_entry) * p->objtotal; +#ifdef linux + p->lut = vmalloc(n); +#else + p->lut = malloc(n, M_NETMAP, M_NOWAIT | M_ZERO); +#endif + if (p->lut == NULL) { + D("Unable to create lookup table (%d bytes) for '%s'", n, p->name); + goto clean; + } + + /* Allocate the bitmap */ + n = (p->objtotal + 31) / 32; + p->bitmap = malloc(sizeof(uint32_t) * n, M_NETMAP, M_NOWAIT | M_ZERO); + if (p->bitmap == NULL) { + D("Unable to create bitmap (%d entries) for allocator '%s'", n, + p->name); + goto clean; + } + p->bitmap_slots = n; + + /* + * Allocate clusters, init pointers and bitmap + */ + for (i = 0; i < p->objtotal;) { + int lim = i + p->clustentries; + char *clust; + + clust = contigmalloc(p->_clustsize, M_NETMAP, M_NOWAIT | M_ZERO, + 0, -1UL, PAGE_SIZE, 0); + if (clust == NULL) { + /* + * If we get here, there is a severe memory shortage, + * so halve the allocated memory to reclaim some. + * XXX check boundaries + */ + D("Unable to create cluster at %d for '%s' allocator", + i, p->name); + lim = i / 2; + for (i--; i >= lim; i--) { + p->bitmap[ (i>>5) ] &= ~( 1 << (i & 31) ); + if (i % p->clustentries == 0 && p->lut[i].vaddr) + contigfree(p->lut[i].vaddr, + p->_clustsize, M_NETMAP); + } + p->objtotal = i; + p->objfree = p->objtotal - 2; + p->_numclusters = i / p->clustentries; + p->_memtotal = p->_numclusters * p->_clustsize; + break; + } + for (; i < lim; i++, clust += p->_objsize) { + p->bitmap[ (i>>5) ] |= ( 1 << (i & 31) ); + p->lut[i].vaddr = clust; + p->lut[i].paddr = vtophys(clust); + } + } + p->bitmap[0] = ~3; /* objs 0 and 1 is always busy */ + if (netmap_verbose) + D("Pre-allocated %d clusters (%d/%dKB) for '%s'", + p->_numclusters, p->_clustsize >> 10, + p->_memtotal >> 10, p->name); + + return 0; + +clean: + netmap_reset_obj_allocator(p); + return ENOMEM; +} + +/* call with lock held */ +static int +netmap_memory_config_changed(void) +{ + int i; + + for (i = 0; i < NETMAP_POOLS_NR; i++) { + if (nm_mem.pools[i]._objsize != netmap_params[i].size || + nm_mem.pools[i].objtotal != netmap_params[i].num) + return 1; + } + return 0; +} + + +/* call with lock held */ +static int +netmap_memory_config(void) +{ + int i; + + + if (!netmap_memory_config_changed()) + goto out; + + D("reconfiguring"); + + if (nm_mem.finalized) { + /* reset previous allocation */ + for (i = 0; i < NETMAP_POOLS_NR; i++) { + netmap_reset_obj_allocator(&nm_mem.pools[i]); + } + nm_mem.finalized = 0; + } + + for (i = 0; i < NETMAP_POOLS_NR; i++) { + nm_mem.lasterr = netmap_config_obj_allocator(&nm_mem.pools[i], + netmap_params[i].num, netmap_params[i].size); + if (nm_mem.lasterr) + goto out; + } + + D("Have %d KB for interfaces, %d KB for rings and %d MB for buffers", + nm_mem.pools[NETMAP_IF_POOL]._memtotal >> 10, + nm_mem.pools[NETMAP_RING_POOL]._memtotal >> 10, + nm_mem.pools[NETMAP_BUF_POOL]._memtotal >> 20); + +out: + + return nm_mem.lasterr; +} + +/* call with lock held */ +static int +netmap_memory_finalize(void) +{ + int i; + u_int totalsize = 0; + + nm_mem.refcount++; + if (nm_mem.refcount > 1) { + ND("busy (refcount %d)", nm_mem.refcount); + goto out; + } + + /* update configuration if changed */ + if (netmap_memory_config()) + goto out; + + if (nm_mem.finalized) { + /* may happen if config is not changed */ + ND("nothing to do"); + goto out; + } + + for (i = 0; i < NETMAP_POOLS_NR; i++) { + nm_mem.lasterr = netmap_finalize_obj_allocator(&nm_mem.pools[i]); + if (nm_mem.lasterr) + goto cleanup; + totalsize += nm_mem.pools[i]._memtotal; + } + nm_mem.nm_totalsize = totalsize; + + /* backward compatibility */ + netmap_buf_size = nm_mem.pools[NETMAP_BUF_POOL]._objsize; + netmap_total_buffers = nm_mem.pools[NETMAP_BUF_POOL].objtotal; + + netmap_buffer_lut = nm_mem.pools[NETMAP_BUF_POOL].lut; + netmap_buffer_base = nm_mem.pools[NETMAP_BUF_POOL].lut[0].vaddr; + + nm_mem.finalized = 1; + nm_mem.lasterr = 0; + + /* make sysctl values match actual values in the pools */ + for (i = 0; i < NETMAP_POOLS_NR; i++) { + netmap_params[i].size = nm_mem.pools[i]._objsize; + netmap_params[i].num = nm_mem.pools[i].objtotal; + } + +out: + if (nm_mem.lasterr) + nm_mem.refcount--; + + return nm_mem.lasterr; + +cleanup: + for (i = 0; i < NETMAP_POOLS_NR; i++) { + netmap_reset_obj_allocator(&nm_mem.pools[i]); + } + nm_mem.refcount--; + + return nm_mem.lasterr; +} + +static int +netmap_memory_init(void) +{ + NMA_LOCK_INIT(); + return (0); +} + +static void +netmap_memory_fini(void) +{ + int i; + + for (i = 0; i < NETMAP_POOLS_NR; i++) { + netmap_destroy_obj_allocator(&nm_mem.pools[i]); + } + NMA_LOCK_DESTROY(); +} + +static void +netmap_free_rings(struct netmap_adapter *na) +{ + int i; + if (!na->tx_rings) + return; + for (i = 0; i < na->num_tx_rings + 1; i++) { + netmap_ring_free(na->tx_rings[i].ring); + na->tx_rings[i].ring = NULL; + } + for (i = 0; i < na->num_rx_rings + 1; i++) { + netmap_ring_free(na->rx_rings[i].ring); + na->rx_rings[i].ring = NULL; + } + free(na->tx_rings, M_DEVBUF); + na->tx_rings = na->rx_rings = NULL; +} + + + +/* call with NMA_LOCK held */ +/* + * Allocate the per-fd structure netmap_if. + * If this is the first instance, also allocate the krings, rings etc. + */ +static void * +netmap_if_new(const char *ifname, struct netmap_adapter *na) +{ + struct netmap_if *nifp; + struct netmap_ring *ring; + ssize_t base; /* handy for relative offsets between rings and nifp */ + u_int i, len, ndesc, ntx, nrx; + struct netmap_kring *kring; + + if (netmap_update_config(na)) { + /* configuration mismatch, report and fail */ + return NULL; + } + ntx = na->num_tx_rings + 1; /* shorthand, include stack ring */ + nrx = na->num_rx_rings + 1; /* shorthand, include stack ring */ + /* + * the descriptor is followed inline by an array of offsets + * to the tx and rx rings in the shared memory region. + */ + len = sizeof(struct netmap_if) + (nrx + ntx) * sizeof(ssize_t); + nifp = netmap_if_malloc(len); + if (nifp == NULL) { + return NULL; + } + + /* initialize base fields -- override const */ + *(int *)(uintptr_t)&nifp->ni_tx_rings = na->num_tx_rings; + *(int *)(uintptr_t)&nifp->ni_rx_rings = na->num_rx_rings; + strncpy(nifp->ni_name, ifname, IFNAMSIZ); + + (na->refcount)++; /* XXX atomic ? we are under lock */ + if (na->refcount > 1) { /* already setup, we are done */ + goto final; + } + + len = (ntx + nrx) * sizeof(struct netmap_kring); + na->tx_rings = malloc(len, M_DEVBUF, M_NOWAIT | M_ZERO); + if (na->tx_rings == NULL) { + D("Cannot allocate krings for %s", ifname); + goto cleanup; + } + na->rx_rings = na->tx_rings + ntx; + + /* + * First instance, allocate netmap rings and buffers for this card + * The rings are contiguous, but have variable size. + */ + for (i = 0; i < ntx; i++) { /* Transmit rings */ + kring = &na->tx_rings[i]; + ndesc = na->num_tx_desc; + bzero(kring, sizeof(*kring)); + len = sizeof(struct netmap_ring) + + ndesc * sizeof(struct netmap_slot); + ring = netmap_ring_malloc(len); + if (ring == NULL) { + D("Cannot allocate tx_ring[%d] for %s", i, ifname); + goto cleanup; + } + ND("txring[%d] at %p ofs %d", i, ring); + kring->na = na; + kring->ring = ring; + *(int *)(uintptr_t)&ring->num_slots = kring->nkr_num_slots = ndesc; + *(ssize_t *)(uintptr_t)&ring->buf_ofs = + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + + nm_mem.pools[NETMAP_RING_POOL]._memtotal) - + netmap_ring_offset(ring); + + /* + * IMPORTANT: + * Always keep one slot empty, so we can detect new + * transmissions comparing cur and nr_hwcur (they are + * the same only if there are no new transmissions). + */ + ring->avail = kring->nr_hwavail = ndesc - 1; + ring->cur = kring->nr_hwcur = 0; + *(int *)(uintptr_t)&ring->nr_buf_size = NETMAP_BUF_SIZE; + ND("initializing slots for txring[%d]", i); + if (netmap_new_bufs(nifp, ring->slot, ndesc)) { + D("Cannot allocate buffers for tx_ring[%d] for %s", i, ifname); + goto cleanup; + } + } + + for (i = 0; i < nrx; i++) { /* Receive rings */ + kring = &na->rx_rings[i]; + ndesc = na->num_rx_desc; + bzero(kring, sizeof(*kring)); + len = sizeof(struct netmap_ring) + + ndesc * sizeof(struct netmap_slot); + ring = netmap_ring_malloc(len); + if (ring == NULL) { + D("Cannot allocate rx_ring[%d] for %s", i, ifname); + goto cleanup; + } + ND("rxring[%d] at %p ofs %d", i, ring); + + kring->na = na; + kring->ring = ring; + *(int *)(uintptr_t)&ring->num_slots = kring->nkr_num_slots = ndesc; + *(ssize_t *)(uintptr_t)&ring->buf_ofs = + (nm_mem.pools[NETMAP_IF_POOL]._memtotal + + nm_mem.pools[NETMAP_RING_POOL]._memtotal) - + netmap_ring_offset(ring); + + ring->cur = kring->nr_hwcur = 0; + ring->avail = kring->nr_hwavail = 0; /* empty */ + *(int *)(uintptr_t)&ring->nr_buf_size = NETMAP_BUF_SIZE; + ND("initializing slots for rxring[%d]", i); + if (netmap_new_bufs(nifp, ring->slot, ndesc)) { + D("Cannot allocate buffers for rx_ring[%d] for %s", i, ifname); + goto cleanup; + } + } +#ifdef linux + // XXX initialize the selrecord structs. + for (i = 0; i < ntx; i++) + init_waitqueue_head(&na->tx_rings[i].si); + for (i = 0; i < nrx; i++) + init_waitqueue_head(&na->rx_rings[i].si); + init_waitqueue_head(&na->tx_si); + init_waitqueue_head(&na->rx_si); +#endif +final: + /* + * fill the slots for the rx and tx rings. They contain the offset + * between the ring and nifp, so the information is usable in + * userspace to reach the ring from the nifp. + */ + base = netmap_if_offset(nifp); + for (i = 0; i < ntx; i++) { + *(ssize_t *)(uintptr_t)&nifp->ring_ofs[i] = + netmap_ring_offset(na->tx_rings[i].ring) - base; + } + for (i = 0; i < nrx; i++) { + *(ssize_t *)(uintptr_t)&nifp->ring_ofs[i+ntx] = + netmap_ring_offset(na->rx_rings[i].ring) - base; + } + return (nifp); +cleanup: + netmap_free_rings(na); + netmap_if_free(nifp); + (na->refcount)--; + return NULL; +} + +/* call with NMA_LOCK held */ +static void +netmap_memory_deref(void) +{ + nm_mem.refcount--; + if (netmap_verbose) + D("refcount = %d", nm_mem.refcount); +} --- a/include/uapi/Kbuild 2013-02-26 10:19:28.000000000 -0800 +++ b/include/uapi/Kbuild 2013-03-10 10:08:20.327671428 -0700 @@ -12,3 +12,4 @@ header-y += video/ header-y += drm/ header-y += xen/ header-y += scsi/ +header-y += netmap/ --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/include/uapi/netmap/Kbuild 2013-03-10 10:08:20.327671428 -0700 @@ -0,0 +1,3 @@ +# UAPI Header export list +header-y += netmap.h +header-y += netmap_user.h --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/include/uapi/netmap/netmap.h 2013-03-10 10:08:20.327671428 -0700 @@ -0,0 +1,289 @@ +/* + * Copyright (C) 2011 Matteo Landi, Luigi Rizzo. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the + * distribution. + * + * 3. Neither the name of the authors nor the names of their contributors + * may be used to endorse or promote products derived from this + * software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY MATTEO LANDI AND CONTRIBUTORS "AS IS" AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL MATTEO LANDI OR CONTRIBUTORS + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF + * THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* + * $FreeBSD: head/sys/net/netmap.h 231198 2012-02-08 11:43:29Z luigi $ + * $Id: netmap.h 10879 2012-04-12 22:48:59Z luigi $ + * + * Definitions of constants and the structures used by the netmap + * framework, for the part visible to both kernel and userspace. + * Detailed info on netmap is available with "man netmap" or at + * + * http://info.iet.unipi.it/~luigi/netmap/ + */ + +#ifndef _NET_NETMAP_H_ +#define _NET_NETMAP_H_ + +/* + * --- Netmap data structures --- + * + * The data structures used by netmap are shown below. Those in + * capital letters are in an mmapp()ed area shared with userspace, + * while others are private to the kernel. + * Shared structures do not contain pointers but only memory + * offsets, so that addressing is portable between kernel and userspace. + + + softc ++----------------+ +| standard fields| +| if_pspare[0] ----------+ ++----------------+ | + | ++----------------+<------+ +|(netmap_adapter)| +| | netmap_kring +| tx_rings *--------------------------------->+---------------+ +| | netmap_kring | ring *---------. +| rx_rings *--------->+---------------+ | nr_hwcur | | ++----------------+ | ring *--------. | nr_hwavail | V + | nr_hwcur | | | selinfo | | + | nr_hwavail | | +---------------+ . + | selinfo | | | ... | . + +---------------+ | |(ntx+1 entries)| + | .... | | | | + |(nrx+1 entries)| | +---------------+ + | | | + KERNEL +---------------+ | + | + ==================================================================== + | + USERSPACE | NETMAP_RING + +---->+-------------+ + / | cur | + NETMAP_IF (nifp, one per file desc.) / | avail | + +---------------+ / | buf_ofs | + | ni_tx_rings | / +=============+ + | ni_rx_rings | / | buf_idx | slot[0] + | | / | len, flags | + | | / +-------------+ + +===============+ / | buf_idx | slot[1] + | txring_ofs[0] | (rel.to nifp)--' | len, flags | + | txring_ofs[1] | +-------------+ + (num_rings+1 entries) (nr_num_slots entries) + | txring_ofs[n] | | buf_idx | slot[n-1] + +---------------+ | len, flags | + | rxring_ofs[0] | +-------------+ + | rxring_ofs[1] | + (num_rings+1 entries) + | txring_ofs[n] | + +---------------+ + + * The private descriptor ('softc' or 'adapter') of each interface + * is extended with a "struct netmap_adapter" containing netmap-related + * info (see description in dev/netmap/netmap_kernel.h. + * Among other things, tx_rings and rx_rings point to the arrays of + * "struct netmap_kring" which in turn reache the various + * "struct netmap_ring", shared with userspace. + + * The NETMAP_RING is the userspace-visible replica of the NIC ring. + * Each slot has the index of a buffer, its length and some flags. + * In user space, the buffer address is computed as + * (char *)ring + buf_ofs + index*NETMAP_BUF_SIZE + * In the kernel, buffers do not necessarily need to be contiguous, + * and the virtual and physical addresses are derived through + * a lookup table. + * To associate a different buffer to a slot, applications must + * write the new index in buf_idx, and set NS_BUF_CHANGED flag to + * make sure that the kernel updates the hardware ring as needed. + * + * Normally the driver is not requested to report the result of + * transmissions (this can dramatically speed up operation). + * However the user may request to report completion by setting + * NS_REPORT. + */ +struct netmap_slot { + uint32_t buf_idx; /* buffer index */ + uint16_t len; /* packet length, to be copied to/from the hw ring */ + uint16_t flags; /* buf changed, etc. */ +#define NS_BUF_CHANGED 0x0001 /* must resync the map, buffer changed */ +#define NS_REPORT 0x0002 /* ask the hardware to report results + * e.g. by generating an interrupt + */ +}; + +/* + * Netmap representation of a TX or RX ring (also known as "queue"). + * This is a queue implemented as a fixed-size circular array. + * At the software level, two fields are important: avail and cur. + * + * In TX rings: + * avail indicates the number of slots available for transmission. + * It is updated by the kernel after every netmap system call. + * It MUST BE decremented by the application when it appends a + * packet. + * cur indicates the slot to use for the next packet + * to send (i.e. the "tail" of the queue). + * It MUST BE incremented by the application before + * netmap system calls to reflect the number of newly + * sent packets. + * It is checked by the kernel on netmap system calls + * (normally unmodified by the kernel unless invalid). + * + * The kernel side of netmap uses two additional fields in its own + * private ring structure, netmap_kring: + * nr_hwcur is a copy of nr_cur on an NIOCTXSYNC. + * nr_hwavail is the number of slots known as available by the + * hardware. It is updated on an INTR (inc by the + * number of packets sent) and on a NIOCTXSYNC + * (decrease by nr_cur - nr_hwcur) + * A special case, nr_hwavail is -1 if the transmit + * side is idle (no pending transmits). + * + * In RX rings: + * avail is the number of packets available (possibly 0). + * It MUST BE decremented by the application when it consumes + * a packet, and it is updated to nr_hwavail on a NIOCRXSYNC + * cur indicates the first slot that contains a packet not + * processed yet (the "head" of the queue). + * It MUST BE incremented by the software when it consumes + * a packet. + * reserved indicates the number of buffers before 'cur' + * that the application has still in use. Normally 0, + * it MUST BE incremented by the application when it + * does not return the buffer immediately, and decremented + * when the buffer is finally freed. + * + * The kernel side of netmap uses two additional fields in the kring: + * nr_hwcur is a copy of nr_cur on an NIOCRXSYNC + * nr_hwavail is the number of packets available. It is updated + * on INTR (inc by the number of new packets arrived) + * and on NIOCRXSYNC (decreased by nr_cur - nr_hwcur). + * + * DATA OWNERSHIP/LOCKING: + * The netmap_ring is owned by the user program and it is only + * accessed or modified in the upper half of the kernel during + * a system call. + * + * The netmap_kring is only modified by the upper half of the kernel. + */ +struct netmap_ring { + /* + * nr_buf_base_ofs is meant to be used through macros. + * It contains the offset of the buffer region from this + * descriptor. + */ + const ssize_t buf_ofs; + const uint32_t num_slots; /* number of slots in the ring. */ + uint32_t avail; /* number of usable slots */ + uint32_t cur; /* 'current' r/w position */ + uint32_t reserved; /* not refilled before current */ + + const uint16_t nr_buf_size; + uint16_t flags; +#define NR_TIMESTAMP 0x0002 /* set timestamp on *sync() */ + + struct timeval ts; /* time of last *sync() */ + + /* the slots follow. This struct has variable size */ + struct netmap_slot slot[0]; /* array of slots. */ +}; + + +/* + * Netmap representation of an interface and its queue(s). + * There is one netmap_if for each file descriptor on which we want + * to select/poll. We assume that on each interface has the same number + * of receive and transmit queues. + * select/poll operates on one or all pairs depending on the value of + * nmr_queueid passed on the ioctl. + */ +struct netmap_if { + char ni_name[IFNAMSIZ]; /* name of the interface. */ + const u_int ni_version; /* API version, currently unused */ + const u_int ni_rx_rings; /* number of rx rings */ + const u_int ni_tx_rings; /* if zero, same as ni_rx_rings */ + /* + * The following array contains the offset of each netmap ring + * from this structure. The first ni_tx_queues+1 entries refer + * to the tx rings, the next ni_rx_queues+1 refer to the rx rings + * (the last entry in each block refers to the host stack rings). + * The area is filled up by the kernel on NIOCREG, + * and then only read by userspace code. + */ + const ssize_t ring_ofs[0]; +}; + +#ifndef NIOCREGIF +/* + * ioctl names and related fields + * + * NIOCGINFO takes a struct ifreq, the interface name is the input, + * the outputs are number of queues and number of descriptor + * for each queue (useful to set number of threads etc.). + * + * NIOCREGIF takes an interface name within a struct ifreq, + * and activates netmap mode on the interface (if possible). + * + * NIOCUNREGIF unregisters the interface associated to the fd. + * + * NIOCTXSYNC, NIOCRXSYNC synchronize tx or rx queues, + * whose identity is set in NIOCREGIF through nr_ringid + */ + +/* + * struct nmreq overlays a struct ifreq + */ +struct nmreq { + char nr_name[IFNAMSIZ]; + uint32_t nr_version; /* API version */ +#define NETMAP_API 3 /* current version */ + uint32_t nr_offset; /* nifp offset in the shared region */ + uint32_t nr_memsize; /* size of the shared region */ + uint32_t nr_tx_slots; /* slots in tx rings */ + uint32_t nr_rx_slots; /* slots in rx rings */ + uint16_t nr_tx_rings; /* number of tx rings */ + uint16_t nr_rx_rings; /* number of rx rings */ + uint16_t nr_ringid; /* ring(s) we care about */ +#define NETMAP_HW_RING 0x4000 /* low bits indicate one hw ring */ +#define NETMAP_SW_RING 0x2000 /* process the sw ring */ +#define NETMAP_NO_TX_POLL 0x1000 /* no automatic txsync on poll */ +#define NETMAP_RING_MASK 0xfff /* the ring number */ + uint16_t spare1; + uint32_t spare2[4]; +}; + +/* + * FreeBSD uses the size value embedded in the _IOWR to determine + * how much to copy in/out. So we need it to match the actual + * data structure we pass. We put some spares in the structure + * to ease compatibility with other versions + */ +#define NIOCGINFO _IOWR('i', 145, struct nmreq) /* return IF info */ +#define NIOCREGIF _IOWR('i', 146, struct nmreq) /* interface register */ +#define NIOCUNREGIF _IO('i', 147) /* interface unregister */ +#define NIOCTXSYNC _IO('i', 148) /* sync tx queues */ +#define NIOCRXSYNC _IO('i', 149) /* sync rx queues */ +#endif /* !NIOCREGIF */ + +#endif /* _NET_NETMAP_H_ */ --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/include/uapi/netmap/netmap_user.h 2013-03-10 10:08:20.327671428 -0700 @@ -0,0 +1,95 @@ +/* + * Copyright (C) 2011 Matteo Landi, Luigi Rizzo. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are + * met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the + * distribution. + * + * 3. Neither the name of the authors nor the names of their contributors + * may be used to endorse or promote products derived from this + * software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY MATTEO LANDI AND CONTRIBUTORS "AS IS" AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL MATTEO LANDI OR CONTRIBUTORS + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF + * THE POSSIBILITY OF SUCH DAMAGE. + */ + +/* + * $FreeBSD: head/sys/net/netmap_user.h 231198 2012-02-08 11:43:29Z luigi $ + * $Id: netmap_user.h 10879 2012-04-12 22:48:59Z luigi $ + * + * This header contains the macros used to manipulate netmap structures + * and packets in userspace. See netmap(4) for more information. + * + * The address of the struct netmap_if, say nifp, is computed from the + * value returned from ioctl(.., NIOCREG, ...) and the mmap region: + * ioctl(fd, NIOCREG, &req); + * mem = mmap(0, ... ); + * nifp = NETMAP_IF(mem, req.nr_nifp); + * (so simple, we could just do it manually) + * + * From there: + * struct netmap_ring *NETMAP_TXRING(nifp, index) + * struct netmap_ring *NETMAP_RXRING(nifp, index) + * we can access ring->nr_cur, ring->nr_avail, ring->nr_flags + * + * ring->slot[i] gives us the i-th slot (we can access + * directly plen, flags, bufindex) + * + * char *buf = NETMAP_BUF(ring, index) returns a pointer to + * the i-th buffer + * + * Since rings are circular, we have macros to compute the next index + * i = NETMAP_RING_NEXT(ring, i); + */ + +#ifndef _NET_NETMAP_USER_H_ +#define _NET_NETMAP_USER_H_ + +#define NETMAP_IF(b, o) (struct netmap_if *)((char *)(b) + (o)) + +#define NETMAP_TXRING(nifp, index) \ + ((struct netmap_ring *)((char *)(nifp) + \ + (nifp)->ring_ofs[index] ) ) + +#define NETMAP_RXRING(nifp, index) \ + ((struct netmap_ring *)((char *)(nifp) + \ + (nifp)->ring_ofs[index + (nifp)->ni_tx_rings + 1] ) ) + +#define NETMAP_BUF(ring, index) \ + ((char *)(ring) + (ring)->buf_ofs + ((index)*(ring)->nr_buf_size)) + +#define NETMAP_BUF_IDX(ring, buf) \ + ( ((char *)(buf) - ((char *)(ring) + (ring)->buf_ofs) ) / \ + (ring)->nr_buf_size ) + +#define NETMAP_RING_NEXT(r, i) \ + ((i)+1 == (r)->num_slots ? 0 : (i) + 1 ) + +#define NETMAP_RING_FIRST_RESERVED(r) \ + ( (r)->cur < (r)->reserved ? \ + (r)->cur + (r)->num_slots - (r)->reserved : \ + (r)->cur - (r)->reserved ) + +/* + * Return 1 if the given tx ring is empty. + */ +#define NETMAP_TX_RING_EMPTY(r) ((r)->avail >= (r)->num_slots - 1) + +#endif /* _NET_NETMAP_USER_H_ */ --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/include/netmap/bsd_glue.h 2013-03-10 10:08:20.327671428 -0700 @@ -0,0 +1,263 @@ +/* + * (C) 2012 Luigi Rizzo - Universita` di Pisa + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* + * glue code to build the netmap bsd code under linux. + * Some of these tweaks are generic, some are specific for + * character device drivers and network code/device drivers. + */ + +#ifndef _BSD_GLUE_H +#define _BSD_GLUE_H + +/* a set of headers used in netmap */ +#include <linux/version.h> +#include <linux/if.h> +#include <linux/list.h> +#include <linux/mutex.h> +#include <linux/types.h> +#include <linux/time.h> +#include <linux/mm.h> +#include <linux/poll.h> +#include <linux/netdevice.h> +#include <linux/sched.h> +#include <linux/wait.h> +#include <linux/miscdevice.h> +//#include <linux/log2.h> // ilog2 +#include <linux/etherdevice.h> // eth_type_trans +#include <linux/module.h> +#include <linux/moduleparam.h> +#include <linux/virtio.h> // virt_to_phys + +#define printf(fmt, arg...) printk(KERN_ERR fmt, ##arg) +#define KASSERT(a, b) BUG_ON(!(a)) + +/* Type redefinitions. XXX check them */ +typedef void * bus_dma_tag_t; +typedef void * bus_dmamap_t; +typedef int bus_size_t; +typedef int bus_dma_segment_t; +typedef void * bus_addr_t; +#define vm_paddr_t phys_addr_t +/* XXX the 'off_t' on Linux corresponds to a 'long' */ +#define vm_offset_t uint32_t +struct thread; + +/* endianness macros/functions */ +#define le16toh le16_to_cpu +#define le32toh le32_to_cpu +#define le64toh le64_to_cpu +#define be64toh be64_to_cpu +#define htole32 cpu_to_le32 +#define htole64 cpu_to_le64 + +#include <linux/jiffies.h> +#define time_second (jiffies_to_msecs(jiffies) / 1000U ) + +#define bzero(a, len) memset(a, 0, len) +#define bcopy(_s, _d, len) memcpy(_d, _s, len) + + +// XXX maybe implement it as a proper function somewhere +// it is important to set s->len before the copy. +#define m_devget(_buf, _len, _ofs, _dev, _fn) ( { \ + struct sk_buff *s = netdev_alloc_skb(_dev, _len); \ + if (s) { \ + s->len += _len; \ + skb_copy_to_linear_data_offset(s, _ofs, _buf, _len); \ + s->protocol = eth_type_trans(s, _dev); \ + } \ + s; } ) + +#define mbuf sk_buff +#define m_nextpkt next // chain of mbufs +#define m_freem(m) dev_kfree_skb_any(m) // free a sk_buff + +/* + * m_copydata() copies from mbuf to buffer following the mbuf chain. + * XXX check which linux equivalent we should use to follow fragmented + * skbufs. + */ + +//#define m_copydata(m, o, l, b) skb_copy_bits(m, o, b, l) +#define m_copydata(m, o, l, b) skb_copy_from_linear_data_offset(m, o, b, l) + +/* + * struct ifnet is remapped into struct net_device on linux. + * ifnet has an if_softc field pointing to the device-specific struct + * (adapter). + * On linux the ifnet/net_device is at the beginning of the device-specific + * structure, so a pointer to the first field of the ifnet works. + * We don't use this in netmap, though. + * + * if_xname name device name + * if_capabilities flags // XXX not used + * if_capenable priv_flags + * we would use "features" but it is all taken. + * XXX check for conflict in flags use. + * + * if_bridge atalk_ptr struct nm_bridge (only for VALE ports) + * + * In netmap we use if_pspare[0] to point to the netmap_adapter, + * in linux we have no spares so we overload ax25_ptr, and the detection + * for netmap-capable is some magic in the area pointed by that. + */ +#define WNA(_ifp) (_ifp)->ax25_ptr + +#define ifnet net_device /* remap */ +#define if_xname name /* field ifnet-> net_device */ +//#define if_capabilities flags /* IFCAP_NETMAP */ +#define if_capenable priv_flags /* IFCAP_NETMAP */ +#define if_bridge atalk_ptr /* remap, only for VALE ports */ +#define ifunit_ref(_x) dev_get_by_name(&init_net, _x); +#define if_rele(ifp) dev_put(ifp) +#define CURVNET_SET(x) +#define CURVNET_RESTORE(x) + + +/* + * We use spin_lock_irqsave() because we use the lock in the + * (hard) interrupt context. + */ +typedef struct { + spinlock_t sl; + ulong flags; +} safe_spinlock_t; + +static inline void mtx_lock(safe_spinlock_t *m) +{ + spin_lock_irqsave(&(m->sl), m->flags); +} + +static inline void mtx_unlock(safe_spinlock_t *m) +{ + ulong flags = ACCESS_ONCE(m->flags); + spin_unlock_irqrestore(&(m->sl), flags); +} + +#define mtx_init(a, b, c, d) spin_lock_init(&((a)->sl)) +#define mtx_destroy(a) // XXX spin_lock_destroy(a) + +/* use volatile to fix a probable compiler error on 2.6.25 */ +#define malloc(_size, type, flags) \ + ({ volatile int _v = _size; kmalloc(_v, GFP_ATOMIC | __GFP_ZERO); }) + +#define free(a, t) kfree(a) + +// XXX do we need GPF_ZERO ? +// XXX do we need GFP_DMA for slots ? +// http://www.mjmwired.net/kernel/Documentation/DMA-API.txt + +#define contigmalloc(sz, ty, flags, a, b, pgsz, c) \ + (char *) __get_free_pages(GFP_ATOMIC | __GFP_ZERO, \ + ilog2(roundup_pow_of_two((sz)/PAGE_SIZE))) +#define contigfree(va, sz, ty) free_pages((unsigned long)va, \ + ilog2(roundup_pow_of_two(sz)/PAGE_SIZE)) + +#define vtophys virt_to_phys + +/*--- selrecord and friends ---*/ +/* wake_up() or wake_up_interruptible() ? */ +#define selwakeuppri(sw, pri) wake_up(sw) +#define selrecord(x, y) poll_wait((struct file *)x, y, pwait) +#define knlist_destroy(x) // XXX todo + +/* we use tsleep/wakeup to sleep a bit. */ +#define tsleep(a, b, c, t) msleep(10) // XXX +#define wakeup(sw) // XXX double check +#define microtime do_gettimeofday + + +/* + * The following trick is to map a struct cdev into a struct miscdevice + */ +#define cdev miscdevice + + +/* + * XXX to complete - the dmamap interface + */ +#define BUS_DMA_NOWAIT 0 +#define bus_dmamap_load(_1, _2, _3, _4, _5, _6, _7) +#define bus_dmamap_unload(_1, _2) + +typedef int (d_mmap_t)(struct file *f, struct vm_area_struct *vma); +typedef unsigned int (d_poll_t)(struct file * file, struct poll_table_struct *pwait); + +/* + * make_dev will set an error and return the first argument. + * This relies on the availability of the 'error' local variable. + */ +#define make_dev(_cdev, _zero, _uid, _gid, _perm, _name) \ + ({error = misc_register(_cdev); \ + D("run mknod /dev/%s c %d %d # error %d", \ + (_cdev)->name, MISC_MAJOR, (_cdev)->minor, error); \ + _cdev; } ) +#define destroy_dev(_cdev) misc_deregister(_cdev) + +/*--- sysctl API ----*/ +/* + * linux: sysctl are mapped into /sys/module/ipfw_mod parameters + * windows: they are emulated via get/setsockopt + */ +#define CTLFLAG_RD 1 +#define CTLFLAG_RW 2 + +struct sysctl_oid; +struct sysctl_req; + + +#define SYSCTL_DECL(_1) +#define SYSCTL_OID(_1, _2, _3, _4, _5, _6, _7, _8) +#define SYSCTL_NODE(_1, _2, _3, _4, _5, _6) +#define _SYSCTL_BASE(_name, _var, _ty, _perm) \ + module_param_named(_name, *(_var), _ty, \ + ( (_perm) == CTLFLAG_RD) ? 0444: 0644 ) + +#define SYSCTL_PROC(_base, _oid, _name, _mode, _var, _val, _desc, _a, _b) + +#define SYSCTL_INT(_base, _oid, _name, _mode, _var, _val, _desc) \ + _SYSCTL_BASE(_name, _var, int, _mode) + +#define SYSCTL_LONG(_base, _oid, _name, _mode, _var, _val, _desc) \ + _SYSCTL_BASE(_name, _var, long, _mode) + +#define SYSCTL_ULONG(_base, _oid, _name, _mode, _var, _val, _desc) \ + _SYSCTL_BASE(_name, _var, ulong, _mode) + +#define SYSCTL_UINT(_base, _oid, _name, _mode, _var, _val, _desc) \ + _SYSCTL_BASE(_name, _var, uint, _mode) + +#define TUNABLE_INT(_name, _ptr) + +#define SYSCTL_VNET_PROC SYSCTL_PROC +#define SYSCTL_VNET_INT SYSCTL_INT + +#define SYSCTL_HANDLER_ARGS \ + struct sysctl_oid *oidp, void *arg1, int arg2, struct sysctl_req *req +int sysctl_handle_int(SYSCTL_HANDLER_ARGS); +int sysctl_handle_long(SYSCTL_HANDLER_ARGS); + +#endif /* _BSD_GLUE_H */ --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ b/include/netmap/netmap_kern.h 2013-03-10 11:30:37.253528570 -0700 @@ -0,0 +1,474 @@ +/* + * Copyright (C) 2011-2012 Matteo Landi, Luigi Rizzo. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* + * $FreeBSD: head/sys/dev/netmap/netmap_kern.h 238985 2012-08-02 11:59:43Z luigi $ + * $Id: netmap_kern.h 12089 2013-02-15 01:20:25Z luigi $ + * + * The header contains the definitions of constants and function + * prototypes used only in kernelspace. + */ + +#ifndef _NET_NETMAP_KERN_H_ +#define _NET_NETMAP_KERN_H_ + +#define NETMAP_MEM2 // use the new memory allocator + +#if defined(__FreeBSD__) +#define likely(x) __builtin_expect(!!(x), 1) +#define unlikely(x) __builtin_expect(!!(x), 0) + +#define NM_LOCK_T struct mtx +#define NM_SELINFO_T struct selinfo +#define MBUF_LEN(m) ((m)->m_pkthdr.len) +#define NM_SEND_UP(ifp, m) ((ifp)->if_input)(ifp, m) +#elif defined (linux) + +#define NM_LOCK_T safe_spinlock_t // see bsd_glue.h +#define NM_SELINFO_T wait_queue_head_t +#define MBUF_LEN(m) ((m)->len) +#define NM_SEND_UP(ifp, m) netif_rx(m) + +#ifndef DEV_NETMAP +#define DEV_NETMAP +#endif + +/* + * IFF_NETMAP goes into net_device's priv_flags (if_capenable). + * Map it to BSD style cap until this driver is cleaned up. + */ +#define IFCAP_NETMAP IFF_NETMAP + + +#elif defined (__APPLE__) +#warning apple support is incomplete. +#define likely(x) __builtin_expect(!!(x), 1) +#define unlikely(x) __builtin_expect(!!(x), 0) +#define NM_LOCK_T IOLock * +#define NM_SELINFO_T struct selinfo +#define MBUF_LEN(m) ((m)->m_pkthdr.len) +#define NM_SEND_UP(ifp, m) ((ifp)->if_input)(ifp, m) + +#else +#error unsupported platform +#endif + +#define ND(format, ...) +#define D(format, ...) \ + do { \ + struct timeval __xxts; \ + microtime(&__xxts); \ + printf("%03d.%06d %s [%d] " format "\n", \ + (int)__xxts.tv_sec % 1000, (int)__xxts.tv_usec, \ + __FUNCTION__, __LINE__, ##__VA_ARGS__); \ + } while (0) + +/* rate limited, lps indicates how many per second */ +#define RD(lps, format, ...) \ + do { \ + static int t0, __cnt; \ + if (t0 != time_second) { \ + t0 = time_second; \ + __cnt = 0; \ + } \ + if (__cnt++ < lps) \ + D(format, ##__VA_ARGS__); \ + } while (0) + +struct netmap_adapter; + +/* + * private, kernel view of a ring. Keeps track of the status of + * a ring across system calls. + * + * nr_hwcur index of the next buffer to refill. + * It corresponds to ring->cur - ring->reserved + * + * nr_hwavail the number of slots "owned" by userspace. + * nr_hwavail =:= ring->avail + ring->reserved + * + * The indexes in the NIC and netmap rings are offset by nkr_hwofs slots. + * This is so that, on a reset, buffers owned by userspace are not + * modified by the kernel. In particular: + * RX rings: the next empty buffer (hwcur + hwavail + hwofs) coincides with + * the next empty buffer as known by the hardware (next_to_check or so). + * TX rings: hwcur + hwofs coincides with next_to_send + * + * For received packets, slot->flags is set to nkr_slot_flags + * so we can provide a proper initial value (e.g. set NS_FORWARD + * when operating in 'transparent' mode). + */ +struct netmap_kring { + struct netmap_ring *ring; + u_int nr_hwcur; + int nr_hwavail; + u_int nr_kflags; /* private driver flags */ +#define NKR_PENDINTR 0x1 // Pending interrupt. + u_int nkr_num_slots; + + uint16_t nkr_slot_flags; /* initial value for flags */ + int nkr_hwofs; /* offset between NIC and netmap ring */ + struct netmap_adapter *na; + NM_SELINFO_T si; /* poll/select wait queue */ + NM_LOCK_T q_lock; /* used if no device lock available */ +} __attribute__((__aligned__(64))); + +/* + * This struct extends the 'struct adapter' (or + * equivalent) device descriptor. It contains all fields needed to + * support netmap operation. + */ +struct netmap_adapter { + /* + * On linux we do not have a good way to tell if an interface + * is netmap-capable. So we use the following trick: + * NA(ifp) points here, and the first entry (which hopefully + * always exists and is at least 32 bits) contains a magic + * value which we can use to detect that the interface is good. + */ + uint32_t magic; + uint32_t na_flags; /* future place for IFCAP_NETMAP */ +#define NAF_SKIP_INTR 1 /* use the regular interrupt handler. + * useful during initialization + */ + int refcount; /* number of user-space descriptors using this + interface, which is equal to the number of + struct netmap_if objs in the mapped region. */ + /* + * The selwakeup in the interrupt thread can use per-ring + * and/or global wait queues. We track how many clients + * of each type we have so we can optimize the drivers, + * and especially avoid huge contention on the locks. + */ + int na_single; /* threads attached to a single hw queue */ + int na_multi; /* threads attached to multiple hw queues */ + + int separate_locks; /* set if the interface suports different + locks for rx, tx and core. */ + + u_int num_rx_rings; /* number of adapter receive rings */ + u_int num_tx_rings; /* number of adapter transmit rings */ + + u_int num_tx_desc; /* number of descriptor in each queue */ + u_int num_rx_desc; + + /* tx_rings and rx_rings are private but allocated + * as a contiguous chunk of memory. Each array has + * N+1 entries, for the adapter queues and for the host queue. + */ + struct netmap_kring *tx_rings; /* array of TX rings. */ + struct netmap_kring *rx_rings; /* array of RX rings. */ + + NM_SELINFO_T tx_si, rx_si; /* global wait queues */ + + /* copy of if_qflush and if_transmit pointers, to intercept + * packets from the network stack when netmap is active. + */ + int (*if_transmit)(struct ifnet *, struct mbuf *); + + /* references to the ifnet and device routines, used by + * the generic netmap functions. + */ + struct ifnet *ifp; /* adapter is ifp->if_softc */ + + NM_LOCK_T core_lock; /* used if no device lock available */ + + int (*nm_register)(struct ifnet *, int onoff); + void (*nm_lock)(struct ifnet *, int what, u_int ringid); + int (*nm_txsync)(struct ifnet *, u_int ring, int lock); + int (*nm_rxsync)(struct ifnet *, u_int ring, int lock); + /* return configuration information */ + int (*nm_config)(struct ifnet *, u_int *txr, u_int *txd, + u_int *rxr, u_int *rxd); + + int bdg_port; +#ifdef linux + struct net_device_ops nm_ndo; + int if_refcount; // XXX additions for bridge +#endif /* linux */ +}; + +/* + * The combination of "enable" (ifp->if_capenable & IFCAP_NETMAP) + * and refcount gives the status of the interface, namely: + * + * enable refcount Status + * + * FALSE 0 normal operation + * FALSE != 0 -- (impossible) + * TRUE 1 netmap mode + * TRUE 0 being deleted. + */ + +#define NETMAP_DELETING(_na) ( ((_na)->refcount == 0) && \ + ( (_na)->ifp->if_capenable & IFCAP_NETMAP) ) + +/* + * parameters for (*nm_lock)(adapter, what, index) + */ +enum { + NETMAP_NO_LOCK = 0, + NETMAP_CORE_LOCK, NETMAP_CORE_UNLOCK, + NETMAP_TX_LOCK, NETMAP_TX_UNLOCK, + NETMAP_RX_LOCK, NETMAP_RX_UNLOCK, +#ifdef __FreeBSD__ +#define NETMAP_REG_LOCK NETMAP_CORE_LOCK +#define NETMAP_REG_UNLOCK NETMAP_CORE_UNLOCK +#else + NETMAP_REG_LOCK, NETMAP_REG_UNLOCK +#endif +}; + +/* + * The following are support routines used by individual drivers to + * support netmap operation. + * + * netmap_attach() initializes a struct netmap_adapter, allocating the + * struct netmap_ring's and the struct selinfo. + * + * netmap_detach() frees the memory allocated by netmap_attach(). + * + * netmap_start() replaces the if_transmit routine of the interface, + * and is used to intercept packets coming from the stack. + * + * netmap_load_map/netmap_reload_map are helper routines to set/reset + * the dmamap for a packet buffer + * + * netmap_reset() is a helper routine to be called in the driver + * when reinitializing a ring. + */ +int netmap_attach(struct netmap_adapter *, int); +void netmap_detach(struct ifnet *); +int netmap_start(struct ifnet *, struct mbuf *); +enum txrx { NR_RX = 0, NR_TX = 1 }; +struct netmap_slot *netmap_reset(struct netmap_adapter *na, + enum txrx tx, int n, u_int new_cur); +int netmap_ring_reinit(struct netmap_kring *); + +extern u_int netmap_buf_size; +#define NETMAP_BUF_SIZE netmap_buf_size +extern int netmap_mitigate; +extern int netmap_no_pendintr; +extern u_int netmap_total_buffers; +extern char *netmap_buffer_base; +extern int netmap_verbose; // XXX debugging +enum { /* verbose flags */ + NM_VERB_ON = 1, /* generic verbose */ + NM_VERB_HOST = 0x2, /* verbose host stack */ + NM_VERB_RXSYNC = 0x10, /* verbose on rxsync/txsync */ + NM_VERB_TXSYNC = 0x20, + NM_VERB_RXINTR = 0x100, /* verbose on rx/tx intr (driver) */ + NM_VERB_TXINTR = 0x200, + NM_VERB_NIC_RXSYNC = 0x1000, /* verbose on rx/tx intr (driver) */ + NM_VERB_NIC_TXSYNC = 0x2000, +}; + +/* + * NA returns a pointer to the struct netmap adapter from the ifp, + * WNA is used to write it. + */ +#ifndef WNA +#define WNA(_ifp) (_ifp)->if_pspare[0] +#endif +#define NA(_ifp) ((struct netmap_adapter *)WNA(_ifp)) + +/* + * Macros to determine if an interface is netmap capable or netmap enabled. + * See the magic field in struct netmap_adapter. + */ +#ifdef __FreeBSD__ +/* + * on FreeBSD just use if_capabilities and if_capenable. + */ +#define NETMAP_CAPABLE(ifp) (NA(ifp) && \ + (ifp)->if_capabilities & IFCAP_NETMAP ) + +#define NETMAP_SET_CAPABLE(ifp) \ + (ifp)->if_capabilities |= IFCAP_NETMAP + +#else /* linux */ + +/* + * on linux: + * we check if NA(ifp) is set and its first element has a related + * magic value. The capenable is within the struct netmap_adapter. + */ +#define NETMAP_MAGIC 0x52697a7a + +#define NETMAP_CAPABLE(ifp) (NA(ifp) && \ + ((uint32_t)(uintptr_t)NA(ifp) ^ NA(ifp)->magic) == NETMAP_MAGIC ) + +#define NETMAP_SET_CAPABLE(ifp) \ + NA(ifp)->magic = ((uint32_t)(uintptr_t)NA(ifp)) ^ NETMAP_MAGIC + +#endif /* linux */ + +#ifdef __FreeBSD__ +/* Callback invoked by the dma machinery after a successfull dmamap_load */ +static void netmap_dmamap_cb(__unused void *arg, + __unused bus_dma_segment_t * segs, __unused int nseg, __unused int error) +{ +} + +/* bus_dmamap_load wrapper: call aforementioned function if map != NULL. + * XXX can we do it without a callback ? + */ +static inline void +netmap_load_map(bus_dma_tag_t tag, bus_dmamap_t map, void *buf) +{ + if (map) + bus_dmamap_load(tag, map, buf, NETMAP_BUF_SIZE, + netmap_dmamap_cb, NULL, BUS_DMA_NOWAIT); +} + +/* update the map when a buffer changes. */ +static inline void +netmap_reload_map(bus_dma_tag_t tag, bus_dmamap_t map, void *buf) +{ + if (map) { + bus_dmamap_unload(tag, map); + bus_dmamap_load(tag, map, buf, NETMAP_BUF_SIZE, + netmap_dmamap_cb, NULL, BUS_DMA_NOWAIT); + } +} +#else /* linux */ + +/* + * XXX How do we redefine these functions: + * + * on linux we need + * dma_map_single(&pdev->dev, virt_addr, len, direction) + * dma_unmap_single(&adapter->pdev->dev, phys_addr, len, direction + * The len can be implicit (on netmap it is NETMAP_BUF_SIZE) + * unfortunately the direction is not, so we need to change + * something to have a cross API + */ +#define netmap_load_map(_t, _m, _b) +#define netmap_reload_map(_t, _m, _b) +#if 0 + struct e1000_buffer *buffer_info = &tx_ring->buffer_info[l]; + /* set time_stamp *before* dma to help avoid a possible race */ + buffer_info->time_stamp = jiffies; + buffer_info->mapped_as_page = false; + buffer_info->length = len; + //buffer_info->next_to_watch = l; + /* reload dma map */ + dma_unmap_single(&adapter->pdev->dev, buffer_info->dma, + NETMAP_BUF_SIZE, DMA_TO_DEVICE); + buffer_info->dma = dma_map_single(&adapter->pdev->dev, + addr, NETMAP_BUF_SIZE, DMA_TO_DEVICE); + + if (dma_mapping_error(&adapter->pdev->dev, buffer_info->dma)) { + D("dma mapping error"); + /* goto dma_error; See e1000_put_txbuf() */ + /* XXX reset */ + } + tx_desc->buffer_addr = htole64(buffer_info->dma); //XXX + +#endif + +/* + * The bus_dmamap_sync() can be one of wmb() or rmb() depending on direction. + */ +#define bus_dmamap_sync(_a, _b, _c) + +#endif /* linux */ + +/* + * functions to map NIC to KRING indexes (n2k) and vice versa (k2n) + */ +static inline int +netmap_idx_n2k(struct netmap_kring *kr, int idx) +{ + int n = kr->nkr_num_slots; + idx += kr->nkr_hwofs; + if (idx < 0) + return idx + n; + else if (idx < n) + return idx; + else + return idx - n; +} + + +static inline int +netmap_idx_k2n(struct netmap_kring *kr, int idx) +{ + int n = kr->nkr_num_slots; + idx -= kr->nkr_hwofs; + if (idx < 0) + return idx + n; + else if (idx < n) + return idx; + else + return idx - n; +} + + +#ifdef NETMAP_MEM2 +/* Entries of the look-up table. */ +struct lut_entry { + void *vaddr; /* virtual address. */ + vm_paddr_t paddr; /* phisical address. */ +}; + +struct netmap_obj_pool; +extern struct lut_entry *netmap_buffer_lut; +#define NMB_VA(i) (netmap_buffer_lut[i].vaddr) +#define NMB_PA(i) (netmap_buffer_lut[i].paddr) +#else /* NETMAP_MEM1 */ +#define NMB_VA(i) (netmap_buffer_base + (i * NETMAP_BUF_SIZE) ) +#endif /* NETMAP_MEM2 */ + +/* + * NMB return the virtual address of a buffer (buffer 0 on bad index) + * PNMB also fills the physical address + */ +static inline void * +NMB(struct netmap_slot *slot) +{ + uint32_t i = slot->buf_idx; + return (unlikely(i >= netmap_total_buffers)) ? NMB_VA(0) : NMB_VA(i); +} + +static inline void * +PNMB(struct netmap_slot *slot, uint64_t *pp) +{ + uint32_t i = slot->buf_idx; + void *ret = (i >= netmap_total_buffers) ? NMB_VA(0) : NMB_VA(i); +#ifdef NETMAP_MEM2 + *pp = (i >= netmap_total_buffers) ? NMB_PA(0) : NMB_PA(i); +#else + *pp = vtophys(ret); +#endif + return ret; +} + +/* default functions to handle rx/tx interrupts */ +int netmap_rx_irq(struct ifnet *, int, int *); +#define netmap_tx_irq(_n, _q) netmap_rx_irq(_n, _q, NULL) + +extern int netmap_copy; +#endif /* _NET_NETMAP_KERN_H_ */ --- a/include/uapi/linux/if.h 2013-02-16 09:10:41.000000000 -0800 +++ b/include/uapi/linux/if.h 2013-03-10 11:26:44.500548075 -0700 @@ -83,6 +83,7 @@ #define IFF_SUPP_NOFCS 0x80000 /* device supports sending custom FCS */ #define IFF_LIVE_ADDR_CHANGE 0x100000 /* device supports hardware address * change when it's running */ +#define IFF_NETMAP 0x200000 /* device used with netmap */ #define IF_GET_IFACE 0x0001 /* for querying only */

[RFC,1/2] netmap: infrastructure (in staging)

Commit Message

Comments

Patch