From patchwork Wed Aug 1 22:46:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yi-Hung Wei X-Patchwork-Id: 952458 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="F3VtzpOv"; dkim-atps=neutral Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41gpT5107Jz9s3Z for ; Thu, 2 Aug 2018 08:51:37 +1000 (AEST) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id C2A7ADA2; Wed, 1 Aug 2018 22:50:37 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 806CFD9B for ; Wed, 1 Aug 2018 22:50:36 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.7.6 Received: from mail-pl0-f65.google.com (mail-pl0-f65.google.com [209.85.160.65]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id D90E76C5 for ; Wed, 1 Aug 2018 22:50:34 +0000 (UTC) Received: by mail-pl0-f65.google.com with SMTP id x6-v6so92861plv.10 for ; Wed, 01 Aug 2018 15:50:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=HBCL7QLLULMt3R7Uvg3jOmyy5fBjUr+EGbGcE9O9cxA=; b=F3VtzpOvKfUEnbKZunV24oN0U/kvFgxLggzN4nc9Ps/9wo6mm2IE169EPgXrt+GX4N zbHxExghDxupUrBFdY2gJEQBcV7hLA0dEc4SYjElpUw27unViqPYYu0SckLkY4b9/ISo 2/UJTKhaUVRQb6Qmpd6+vOdVDdKaFUok9hB2mkiloQhDzN5wtJY50vKZ71A6sXIIencq nOGqjjZw+e+4Zl5iwkE99tGxA5h6ngCAFdm0HSgZNzDIr+3xfsSs5vtLsIXvKhI/rcjW SLwnnFlVyfNUMOc1vap823seaPGZHrI12J1tUMqKZM4HDuji9zpraui6cwzP7wUooDlf qvuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=HBCL7QLLULMt3R7Uvg3jOmyy5fBjUr+EGbGcE9O9cxA=; b=p26hdJZWw2/eHJ5yOikkXt7Bx+UGef12jtEFs5oW8lGOq83BPT25CpMeSSNvXbBbfv 6HANHNs1gzbUPVYLuZTKPULNR2losELSLKT6muLtH6FJlCoN9sEeuCQgl0Vd8NqnJ9lv bYRz0vB/Vlg3sZT/Tq/uF6NOsLsJS408LgA4weiJ2D3tHibYks0/EXv9uwSTWfKpi5j9 a3Q3VQeHhUdzgluzQkoDjg/sb5iMqBkNRqfYEQCz0YpyFOltCSyEIZUDhbojvP133aBG BcwWgayVV81xvOOV7/lTVmGVWsq8IramCryd7vu39TdFYzxlzTS5/PmmkpHX75LTTVT3 AEFg== X-Gm-Message-State: AOUpUlFB+UiQIbUGJCF4skyxyqVpwa6VkV1WvDx37zTRXEMfZfQYKqTd RaSpZAA9z9eUY2ellh7igMh79vfX X-Google-Smtp-Source: AAOMgpcjomqa4I59GSFJwljbarpABL1Llru6bhAAv2zfDVhVSPfNj6h0P0iqI9F05pIgEzZ+yjZMMQ== X-Received: by 2002:a17:902:32a4:: with SMTP id z33-v6mr223388plb.226.1533163833831; Wed, 01 Aug 2018 15:50:33 -0700 (PDT) Received: from Husky.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id l85-v6sm196899pfk.34.2018.08.01.15.50.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 01 Aug 2018 15:50:32 -0700 (PDT) From: Yi-Hung Wei To: dev@openvswitch.org Date: Wed, 1 Aug 2018 15:46:11 -0700 Message-Id: <1533163580-27989-3-git-send-email-yihung.wei@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1533163580-27989-1-git-send-email-yihung.wei@gmail.com> References: <1533163580-27989-1-git-send-email-yihung.wei@gmail.com> X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 02/11] datapath: compat: Backports nf_conncount X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This patch backports the nf_conncount backend that counts the number of connections matching an arbitrary key. The following patch will use the feature to support connection tracking zone limit in ovs kernel datapath. This backport is based on an upstream net-next commit 5c789e131cbb ("netfilter: nf_conncount: Add list lock and gc worker, and RCU for init tree search") that applies a couple of techniques to optimize nf_conncount performance. The upstream nf_conncount has a couple of export functions while this patch only export the ones that ovs kernel module needs. Signed-off-by: Yi-Hung Wei --- acinclude.m4 | 2 + datapath/linux/Modules.mk | 2 + .../include/net/netfilter/nf_conntrack_count.h | 61 ++ datapath/linux/compat/nf_conncount.c | 637 +++++++++++++++++++++ 4 files changed, 702 insertions(+) create mode 100644 datapath/linux/compat/include/net/netfilter/nf_conntrack_count.h create mode 100644 datapath/linux/compat/nf_conncount.c diff --git a/acinclude.m4 b/acinclude.m4 index 731bc07be8fa..96fcad55a7cb 100644 --- a/acinclude.m4 +++ b/acinclude.m4 @@ -625,6 +625,8 @@ AC_DEFUN([OVS_CHECK_LINUX_COMPAT], [ OVS_GREP_IFELSE([$KSRC/include/net/netfilter/nf_nat.h], [nf_nat_alloc_null_binding]) OVS_GREP_IFELSE([$KSRC/include/net/netfilter/nf_nat.h], [nf_nat_range2]) OVS_GREP_IFELSE([$KSRC/include/net/netfilter/nf_conntrack_seqadj.h], [nf_ct_seq_adjust]) + OVS_GREP_IFELSE([$KSRC/include/net/netfilter/nf_conntrack_count.h], [nf_conncount_gc_list], + [OVS_DEFINE([HAVE_UPSTREAM_NF_CONNCOUNT])]) OVS_GREP_IFELSE([$KSRC/include/linux/random.h], [prandom_u32]) OVS_GREP_IFELSE([$KSRC/include/linux/random.h], [prandom_u32_max]) diff --git a/datapath/linux/Modules.mk b/datapath/linux/Modules.mk index 04ea5b756b6c..f5c3b6580ad7 100644 --- a/datapath/linux/Modules.mk +++ b/datapath/linux/Modules.mk @@ -17,6 +17,7 @@ openvswitch_sources += \ linux/compat/ip6_tunnel.c \ linux/compat/lisp.c \ linux/compat/netdevice.c \ + linux/compat/nf_conncount.c \ linux/compat/nf_conntrack_core.c \ linux/compat/nf_conntrack_proto.c \ linux/compat/nf_conntrack_reasm.c \ @@ -100,6 +101,7 @@ openvswitch_headers += \ linux/compat/include/net/vxlan.h \ linux/compat/include/net/netfilter/nf_conntrack.h \ linux/compat/include/net/netfilter/nf_conntrack_core.h \ + linux/compat/include/net/netfilter/nf_conntrack_count.h \ linux/compat/include/net/netfilter/nf_conntrack_expect.h \ linux/compat/include/net/netfilter/nf_conntrack_helper.h \ linux/compat/include/net/netfilter/nf_conntrack_labels.h \ diff --git a/datapath/linux/compat/include/net/netfilter/nf_conntrack_count.h b/datapath/linux/compat/include/net/netfilter/nf_conntrack_count.h new file mode 100644 index 000000000000..fd536f3e1854 --- /dev/null +++ b/datapath/linux/compat/include/net/netfilter/nf_conntrack_count.h @@ -0,0 +1,61 @@ +#ifndef _NF_CONNTRACK_COUNT_WRAPPER_H +#define _NF_CONNTRACK_COUNT_WRAPPER_H + +#include + +#ifdef HAVE_UPSTREAM_NF_CONNCOUNT +#include_next + +static inline int rpl_nf_conncount_modinit(void) +{ + return 0; +} + +static inline void rpl_nf_conncount_modexit(void) +{ +} + +#else +#include +#include +#define CONFIG_NETFILTER_CONNCOUNT 1 +struct nf_conncount_data; + +enum nf_conncount_list_add { + NF_CONNCOUNT_ADDED, /* list add was ok */ + NF_CONNCOUNT_ERR, /* -ENOMEM, must drop skb */ + NF_CONNCOUNT_SKIP, /* list is already reclaimed by gc */ +}; + +struct nf_conncount_list { + spinlock_t list_lock; + struct list_head head; /* connections with the same filtering key */ + unsigned int count; /* length of list */ + bool dead; +}; + +struct nf_conncount_data +*rpl_nf_conncount_init(struct net *net, unsigned int family, + unsigned int keylen); + +void rpl_nf_conncount_destroy(struct net *net, unsigned int family, + struct nf_conncount_data *data); + +unsigned int rpl_nf_conncount_count(struct net *net, + struct nf_conncount_data *data, + const u32 *key, + const struct nf_conntrack_tuple *tuple, + const struct nf_conntrack_zone *zone); + +#define nf_conncount_init rpl_nf_conncount_init +#define nf_conncount_destroy rpl_nf_conncount_destroy +#define nf_conncount_count rpl_nf_conncount_count + +int rpl_nf_conncount_modinit(void); +void rpl_nf_conncount_modexit(void); +#endif /* HAVE_UPSTREAM_NF_CONNCOUNT */ + +#define nf_conncount_mod_init rpl_nf_conncount_modinit +#define nf_conncount_modexit rpl_nf_conncount_modexit + +#endif /* _NF_CONNTRACK_COUNT_WRAPPER_H */ diff --git a/datapath/linux/compat/nf_conncount.c b/datapath/linux/compat/nf_conncount.c new file mode 100644 index 000000000000..0bee96274b00 --- /dev/null +++ b/datapath/linux/compat/nf_conncount.c @@ -0,0 +1,637 @@ +/* + * Backported from upstream commit 5c789e131cbb ("netfilter: + * nf_conncount: Add list lock and gc worker, and RCU for init tree search") + * + * count the number of connections matching an arbitrary key. + * + * (C) 2017 Red Hat GmbH + * Author: Florian Westphal + * + * split from xt_connlimit.c: + * (c) 2000 Gerd Knorr + * Nov 2002: Martin Bene : + * only ignore TIME_WAIT or gone connections + * (C) CC Computer Consultants GmbH, 2007 + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define CONNCOUNT_SLOTS 256U + +#ifdef CONFIG_LOCKDEP +#define CONNCOUNT_LOCK_SLOTS 8U +#else +#define CONNCOUNT_LOCK_SLOTS 256U +#endif + +#define CONNCOUNT_GC_MAX_NODES 8 +#define MAX_KEYLEN 5 + +/* we will save the tuples of all connections we care about */ +struct nf_conncount_tuple { + struct list_head node; + struct nf_conntrack_tuple tuple; + struct nf_conntrack_zone zone; + int cpu; + u32 jiffies32; + struct rcu_head rcu_head; +}; + +struct nf_conncount_rb { + struct rb_node node; + struct nf_conncount_list list; + u32 key[MAX_KEYLEN]; + struct rcu_head rcu_head; +}; + +static spinlock_t nf_conncount_locks[CONNCOUNT_LOCK_SLOTS] __cacheline_aligned_in_smp; + +struct nf_conncount_data { + unsigned int keylen; + struct rb_root root[CONNCOUNT_SLOTS]; + struct net *net; + struct work_struct gc_work; + unsigned long pending_trees[BITS_TO_LONGS(CONNCOUNT_SLOTS)]; + unsigned int gc_tree; +}; + +static u_int32_t conncount_rnd __read_mostly; +static struct kmem_cache *conncount_rb_cachep __read_mostly; +static struct kmem_cache *conncount_conn_cachep __read_mostly; + +static inline bool already_closed(const struct nf_conn *conn) +{ + if (nf_ct_protonum(conn) == IPPROTO_TCP) + return conn->proto.tcp.state == TCP_CONNTRACK_TIME_WAIT || + conn->proto.tcp.state == TCP_CONNTRACK_CLOSE; + else + return false; +} + +static int key_diff(const u32 *a, const u32 *b, unsigned int klen) +{ + return memcmp(a, b, klen * sizeof(u32)); +} + +static enum nf_conncount_list_add +nf_conncount_add(struct nf_conncount_list *list, + const struct nf_conntrack_tuple *tuple, + const struct nf_conntrack_zone *zone) +{ + struct nf_conncount_tuple *conn; + + if (WARN_ON_ONCE(list->count > INT_MAX)) + return NF_CONNCOUNT_ERR; + + conn = kmem_cache_alloc(conncount_conn_cachep, GFP_ATOMIC); + if (conn == NULL) + return NF_CONNCOUNT_ERR; + + conn->tuple = *tuple; + conn->zone = *zone; + conn->cpu = raw_smp_processor_id(); + conn->jiffies32 = (u32)jiffies; + spin_lock(&list->list_lock); + if (list->dead == true) { + kmem_cache_free(conncount_conn_cachep, conn); + spin_unlock(&list->list_lock); + return NF_CONNCOUNT_SKIP; + } + list_add_tail(&conn->node, &list->head); + list->count++; + spin_unlock(&list->list_lock); + return NF_CONNCOUNT_ADDED; +} + +static void __conn_free(struct rcu_head *h) +{ + struct nf_conncount_tuple *conn; + + conn = container_of(h, struct nf_conncount_tuple, rcu_head); + kmem_cache_free(conncount_conn_cachep, conn); +} + +static bool conn_free(struct nf_conncount_list *list, + struct nf_conncount_tuple *conn) +{ + bool free_entry = false; + + spin_lock(&list->list_lock); + + if (list->count == 0) { + spin_unlock(&list->list_lock); + return free_entry; + } + + list->count--; + list_del_rcu(&conn->node); + if (list->count == 0) + free_entry = true; + + spin_unlock(&list->list_lock); + call_rcu(&conn->rcu_head, __conn_free); + return free_entry; +} + +static const struct nf_conntrack_tuple_hash * +find_or_evict(struct net *net, struct nf_conncount_list *list, + struct nf_conncount_tuple *conn, bool *free_entry) +{ + const struct nf_conntrack_tuple_hash *found; + unsigned long a, b; + int cpu = raw_smp_processor_id(); + __s32 age; + + found = nf_conntrack_find_get(net, &conn->zone, &conn->tuple); + if (found) + return found; + b = conn->jiffies32; + a = (u32)jiffies; + + /* conn might have been added just before by another cpu and + * might still be unconfirmed. In this case, nf_conntrack_find() + * returns no result. Thus only evict if this cpu added the + * stale entry or if the entry is older than two jiffies. + */ + age = a - b; + if (conn->cpu == cpu || age >= 2) { + *free_entry = conn_free(list, conn); + return ERR_PTR(-ENOENT); + } + + return ERR_PTR(-EAGAIN); +} + +static void nf_conncount_lookup(struct net *net, + struct nf_conncount_list *list, + const struct nf_conntrack_tuple *tuple, + const struct nf_conntrack_zone *zone, + bool *addit) +{ + const struct nf_conntrack_tuple_hash *found; + struct nf_conncount_tuple *conn, *conn_n; + struct nf_conn *found_ct; + unsigned int collect = 0; + bool free_entry = false; + + /* best effort only */ + *addit = tuple ? true : false; + + /* check the saved connections */ + list_for_each_entry_safe(conn, conn_n, &list->head, node) { + if (collect > CONNCOUNT_GC_MAX_NODES) + break; + + found = find_or_evict(net, list, conn, &free_entry); + if (IS_ERR(found)) { + /* Not found, but might be about to be confirmed */ + if (PTR_ERR(found) == -EAGAIN) { + if (!tuple) + continue; + + if (nf_ct_tuple_equal(&conn->tuple, tuple) && + nf_ct_zone_id(&conn->zone, conn->zone.dir) == + nf_ct_zone_id(zone, zone->dir)) + *addit = false; + } else if (PTR_ERR(found) == -ENOENT) + collect++; + continue; + } + + found_ct = nf_ct_tuplehash_to_ctrack(found); + + if (tuple && nf_ct_tuple_equal(&conn->tuple, tuple) && + nf_ct_zone_equal(found_ct, zone, zone->dir)) { + /* + * We should not see tuples twice unless someone hooks + * this into a table without "-p tcp --syn". + * + * Attempt to avoid a re-add in this case. + */ + *addit = false; + } else if (already_closed(found_ct)) { + /* + * we do not care about connections which are + * closed already -> ditch it + */ + nf_ct_put(found_ct); + conn_free(list, conn); + collect++; + continue; + } + + nf_ct_put(found_ct); + } +} + +static void nf_conncount_list_init(struct nf_conncount_list *list) +{ + spin_lock_init(&list->list_lock); + INIT_LIST_HEAD(&list->head); + list->count = 1; + list->dead = false; +} + +/* Return true if the list is empty */ +static bool nf_conncount_gc_list(struct net *net, + struct nf_conncount_list *list) +{ + const struct nf_conntrack_tuple_hash *found; + struct nf_conncount_tuple *conn, *conn_n; + struct nf_conn *found_ct; + unsigned int collected = 0; + bool free_entry = false; + + list_for_each_entry_safe(conn, conn_n, &list->head, node) { + found = find_or_evict(net, list, conn, &free_entry); + if (IS_ERR(found)) { + if (PTR_ERR(found) == -ENOENT) { + if (free_entry) + return true; + collected++; + } + continue; + } + + found_ct = nf_ct_tuplehash_to_ctrack(found); + if (already_closed(found_ct)) { + /* + * we do not care about connections which are + * closed already -> ditch it + */ + nf_ct_put(found_ct); + if (conn_free(list, conn)) + return true; + collected++; + continue; + } + + nf_ct_put(found_ct); + if (collected > CONNCOUNT_GC_MAX_NODES) + return false; + } + return false; +} + +static void __tree_nodes_free(struct rcu_head *h) +{ + struct nf_conncount_rb *rbconn; + + rbconn = container_of(h, struct nf_conncount_rb, rcu_head); + kmem_cache_free(conncount_rb_cachep, rbconn); +} + +static void tree_nodes_free(struct rb_root *root, + struct nf_conncount_rb *gc_nodes[], + unsigned int gc_count) +{ + struct nf_conncount_rb *rbconn; + + while (gc_count) { + rbconn = gc_nodes[--gc_count]; + spin_lock(&rbconn->list.list_lock); + if (rbconn->list.count == 0 && rbconn->list.dead == false) { + rbconn->list.dead = true; + rb_erase(&rbconn->node, root); + call_rcu(&rbconn->rcu_head, __tree_nodes_free); + } + spin_unlock(&rbconn->list.list_lock); + } +} + +static void schedule_gc_worker(struct nf_conncount_data *data, int tree) +{ + set_bit(tree, data->pending_trees); + schedule_work(&data->gc_work); +} + +static unsigned int +insert_tree(struct net *net, + struct nf_conncount_data *data, + struct rb_root *root, + unsigned int hash, + const u32 *key, + u8 keylen, + const struct nf_conntrack_tuple *tuple, + const struct nf_conntrack_zone *zone) +{ + enum nf_conncount_list_add ret; + struct nf_conncount_rb *gc_nodes[CONNCOUNT_GC_MAX_NODES]; + struct rb_node **rbnode, *parent; + struct nf_conncount_rb *rbconn; + struct nf_conncount_tuple *conn; + unsigned int count = 0, gc_count = 0; + bool node_found = false; + + spin_lock_bh(&nf_conncount_locks[hash % CONNCOUNT_LOCK_SLOTS]); + + parent = NULL; + rbnode = &(root->rb_node); + while (*rbnode) { + int diff; + rbconn = rb_entry(*rbnode, struct nf_conncount_rb, node); + + parent = *rbnode; + diff = key_diff(key, rbconn->key, keylen); + if (diff < 0) { + rbnode = &((*rbnode)->rb_left); + } else if (diff > 0) { + rbnode = &((*rbnode)->rb_right); + } else { + /* unlikely: other cpu added node already */ + node_found = true; + ret = nf_conncount_add(&rbconn->list, tuple, zone); + if (ret == NF_CONNCOUNT_ERR) { + count = 0; /* hotdrop */ + } else if (ret == NF_CONNCOUNT_ADDED) { + count = rbconn->list.count; + } else { + /* NF_CONNCOUNT_SKIP, rbconn is already + * reclaimed by gc, insert a new tree node + */ + node_found = false; + } + break; + } + + if (gc_count >= ARRAY_SIZE(gc_nodes)) + continue; + + if (nf_conncount_gc_list(net, &rbconn->list)) + gc_nodes[gc_count++] = rbconn; + } + + if (gc_count) { + tree_nodes_free(root, gc_nodes, gc_count); + /* tree_node_free before new allocation permits + * allocator to re-use newly free'd object. + * + * This is a rare event; in most cases we will find + * existing node to re-use. (or gc_count is 0). + */ + + if (gc_count >= ARRAY_SIZE(gc_nodes)) + schedule_gc_worker(data, hash); + } + + if (node_found) + goto out_unlock; + + /* expected case: match, insert new node */ + rbconn = kmem_cache_alloc(conncount_rb_cachep, GFP_ATOMIC); + if (rbconn == NULL) + goto out_unlock; + + conn = kmem_cache_alloc(conncount_conn_cachep, GFP_ATOMIC); + if (conn == NULL) { + kmem_cache_free(conncount_rb_cachep, rbconn); + goto out_unlock; + } + + conn->tuple = *tuple; + conn->zone = *zone; + memcpy(rbconn->key, key, sizeof(u32) * keylen); + + nf_conncount_list_init(&rbconn->list); + list_add(&conn->node, &rbconn->list.head); + count = 1; + + rb_link_node(&rbconn->node, parent, rbnode); + rb_insert_color(&rbconn->node, root); +out_unlock: + spin_unlock_bh(&nf_conncount_locks[hash % CONNCOUNT_LOCK_SLOTS]); + return count; +} + +static unsigned int +count_tree(struct net *net, + struct nf_conncount_data *data, + const u32 *key, + const struct nf_conntrack_tuple *tuple, + const struct nf_conntrack_zone *zone) +{ + enum nf_conncount_list_add ret; + struct rb_root *root; + struct rb_node *parent; + struct nf_conncount_rb *rbconn; + unsigned int hash; + u8 keylen = data->keylen; + + hash = jhash2(key, data->keylen, conncount_rnd) % CONNCOUNT_SLOTS; + root = &data->root[hash]; + + parent = rcu_dereference_raw(root->rb_node); + while (parent) { + int diff; + bool addit; + + rbconn = rb_entry(parent, struct nf_conncount_rb, node); + + diff = key_diff(key, rbconn->key, keylen); + if (diff < 0) { + parent = rcu_dereference_raw(parent->rb_left); + } else if (diff > 0) { + parent = rcu_dereference_raw(parent->rb_right); + } else { + /* same source network -> be counted! */ + nf_conncount_lookup(net, &rbconn->list, tuple, zone, + &addit); + + if (!addit) + return rbconn->list.count; + + ret = nf_conncount_add(&rbconn->list, tuple, zone); + if (ret == NF_CONNCOUNT_ERR) { + return 0; /* hotdrop */ + } else if (ret == NF_CONNCOUNT_ADDED) { + return rbconn->list.count; + } else { + /* NF_CONNCOUNT_SKIP, rbconn is already + * reclaimed by gc, insert a new tree node + */ + break; + } + } + } + + if (!tuple) + return 0; + + return insert_tree(net, data, root, hash, key, keylen, tuple, zone); +} + +static void tree_gc_worker(struct work_struct *work) +{ + struct nf_conncount_data *data = container_of(work, struct nf_conncount_data, gc_work); + struct nf_conncount_rb *gc_nodes[CONNCOUNT_GC_MAX_NODES], *rbconn; + struct rb_root *root; + struct rb_node *node; + unsigned int tree, next_tree, gc_count = 0; + + tree = data->gc_tree % CONNCOUNT_LOCK_SLOTS; + root = &data->root[tree]; + + rcu_read_lock(); + for (node = rb_first(root); node != NULL; node = rb_next(node)) { + rbconn = rb_entry(node, struct nf_conncount_rb, node); + if (nf_conncount_gc_list(data->net, &rbconn->list)) + gc_nodes[gc_count++] = rbconn; + } + rcu_read_unlock(); + + spin_lock_bh(&nf_conncount_locks[tree]); + + if (gc_count) { + tree_nodes_free(root, gc_nodes, gc_count); + } + + clear_bit(tree, data->pending_trees); + + next_tree = (tree + 1) % CONNCOUNT_SLOTS; + next_tree = find_next_bit(data->pending_trees, next_tree, CONNCOUNT_SLOTS); + + if (next_tree < CONNCOUNT_SLOTS) { + data->gc_tree = next_tree; + schedule_work(work); + } + + spin_unlock_bh(&nf_conncount_locks[tree]); +} + +/* Count and return number of conntrack entries in 'net' with particular 'key'. + * If 'tuple' is not null, insert it into the accounting data structure. + * Call with RCU read lock. + */ +unsigned int rpl_nf_conncount_count(struct net *net, + struct nf_conncount_data *data, + const u32 *key, + const struct nf_conntrack_tuple *tuple, + const struct nf_conntrack_zone *zone) +{ + return count_tree(net, data, key, tuple, zone); +} +EXPORT_SYMBOL_GPL(rpl_nf_conncount_count); + +struct nf_conncount_data *rpl_nf_conncount_init(struct net *net, unsigned int family, + unsigned int keylen) +{ + struct nf_conncount_data *data; + int ret, i; + + if (keylen % sizeof(u32) || + keylen / sizeof(u32) > MAX_KEYLEN || + keylen == 0) + return ERR_PTR(-EINVAL); + + net_get_random_once(&conncount_rnd, sizeof(conncount_rnd)); + + data = kmalloc(sizeof(*data), GFP_KERNEL); + if (!data) + return ERR_PTR(-ENOMEM); + + ret = nf_ct_netns_get(net, family); + if (ret < 0) { + kfree(data); + return ERR_PTR(ret); + } + + for (i = 0; i < ARRAY_SIZE(data->root); ++i) + data->root[i] = RB_ROOT; + + data->keylen = keylen / sizeof(u32); + data->net = net; + INIT_WORK(&data->gc_work, tree_gc_worker); + + return data; +} +EXPORT_SYMBOL_GPL(rpl_nf_conncount_init); + +static void nf_conncount_cache_free(struct nf_conncount_list *list) +{ + struct nf_conncount_tuple *conn, *conn_n; + + list_for_each_entry_safe(conn, conn_n, &list->head, node) + kmem_cache_free(conncount_conn_cachep, conn); +} + +static void destroy_tree(struct rb_root *r) +{ + struct nf_conncount_rb *rbconn; + struct rb_node *node; + + while ((node = rb_first(r)) != NULL) { + rbconn = rb_entry(node, struct nf_conncount_rb, node); + + rb_erase(node, r); + + nf_conncount_cache_free(&rbconn->list); + + kmem_cache_free(conncount_rb_cachep, rbconn); + } +} + +void rpl_nf_conncount_destroy(struct net *net, unsigned int family, + struct nf_conncount_data *data) +{ + unsigned int i; + + cancel_work_sync(&data->gc_work); + nf_ct_netns_put(net, family); + + for (i = 0; i < ARRAY_SIZE(data->root); ++i) + destroy_tree(&data->root[i]); + + kfree(data); +} +EXPORT_SYMBOL_GPL(rpl_nf_conncount_destroy); + +int rpl_nf_conncount_modinit(void) +{ + int i; + + BUILD_BUG_ON(CONNCOUNT_LOCK_SLOTS > CONNCOUNT_SLOTS); + BUILD_BUG_ON((CONNCOUNT_SLOTS % CONNCOUNT_LOCK_SLOTS) != 0); + + for (i = 0; i < CONNCOUNT_LOCK_SLOTS; ++i) + spin_lock_init(&nf_conncount_locks[i]); + + conncount_conn_cachep = kmem_cache_create("nf_conncount_tuple", + sizeof(struct nf_conncount_tuple), + 0, 0, NULL); + if (!conncount_conn_cachep) + return -ENOMEM; + + conncount_rb_cachep = kmem_cache_create("nf_conncount_rb", + sizeof(struct nf_conncount_rb), + 0, 0, NULL); + if (!conncount_rb_cachep) { + kmem_cache_destroy(conncount_conn_cachep); + return -ENOMEM; + } + + return 0; +} + +void rpl_nf_conncount_modexit(void) +{ + kmem_cache_destroy(conncount_conn_cachep); + kmem_cache_destroy(conncount_rb_cachep); +}