[ovs-dev] OVN: initial patch of datalog engine

Message ID	SN1PR0501MB175957AD08783495E6F0FBC4D2560@SN1PR0501MB1759.namprd05.prod.outlook.com
State	Changes Requested
Headers	show Return-Path: <dev-bounces@openvswitch.org> Received-SPF: error (mx1-pf1.cudamail.com: error in processing during lookup of vmware.com: DNS problem) From: Yusheng Wang <yshwang@vmware.com> To: "dev@openvswitch.org" <dev@openvswitch.org> Thread-Topic: [PATCH] OVN: initial patch of datalog engine OVN: initial patch of datalog engine Thread-Index: AQHRx7QX9IK9iAocRUSNBFerUtK3ZQ== Date: Thu, 16 Jun 2016 09:56:55 +0000 Message-ID: <SN1PR0501MB175957AD08783495E6F0FBC4D2560@SN1PR0501MB1759.namprd05.prod.outlook.com> Accept-Language: en-US Content-Language: en-US Received-SPF: None (EX13-EDG-OU-001.vmware.com: yshwang@vmware.com does not designate permitted sender hosts) received-spf: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Jun 2016 09:56:55.1218 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted Subject: [ovs-dev] [PATCH] OVN: initial patch of datalog engine Precedence: list Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: dev-bounces@openvswitch.org Sender: "dev" <dev-bounces@openvswitch.org>

Hi, Yusheng, I am very intersted in testing your nice nlog implementation. Meanwhile, I appreciate if you could look at some questions I have about this nlog. - Reading the early proposal of nlog [1], I understand that nlog will be serving as a new computation model for ovn-northd. Furthermore, I think the plan is to replace the existing join_XXX operations (e.g., [3][4]) in populating tables in southbound database. Is my understanding correct? - How much performance improvement will you expect from using nlog? The NVP paper [2] mentioned in section 8.2 that even with nlog, the scalability issue still exists because Openflow in hypervisors requires O(N^2). Thanks in advance. - Hui [1] http://openvswitch.org/pipermail/dev/2016-February/065419.html [2] http://benpfaff.org/papers/net-virt.pdf [3] https://github.com/huikang/ovs/blob/master/ovn/northd/ovn-northd.c#L557 [4] https://github.com/huikang/ovs/blob/master/ovn/northd/ovn-northd.c#L327 "dev" <dev-bounces@openvswitch.org> wrote on 06/16/2016 05:56:55 AM: > From: Yusheng Wang <yshwang@vmware.com> > To: "dev@openvswitch.org" <dev@openvswitch.org> > Date: 06/16/2016 05:57 AM > Subject: [ovs-dev] [PATCH] OVN: initial patch of datalog engine > Sent by: "dev" <dev-bounces@openvswitch.org> > > From 62dc90f8f0a61181754e944f2101951afbd055c1 Mon Sep 17 00:00:00 2001 > From: Yusheng Wang <yshwang@vmware.com> > Date: Thu, 16 Jun 2016 15:04:26 +0800 > Subject: [PATCH] OVN: initial patch of datalog engine > > Signed-off-by: Yusheng Wang <yshwang@vmware.com> > --- > ovn/lib/automake.mk | 5 +- > ovn/lib/datalog-private.h | 808 +++++++++++ > ovn/lib/datalog.c | 3392 ++++++++++++++++++++++++++++++++++ > +++++++++++ > ovn/lib/datalog.h | 49 + > tests/automake.mk | 6 +- > tests/datalog.at | 11 + > tests/test-datalog.c | 1662 ++++++++++++++++++++++ > tests/testsuite.at | 1 + > 8 files changed, 5931 insertions(+), 3 deletions(-) > create mode 100644 ovn/lib/datalog-private.h > create mode 100644 ovn/lib/datalog.c > create mode 100644 ovn/lib/datalog.h > create mode 100644 tests/datalog.at > create mode 100644 tests/test-datalog.c > > diff --git a/ovn/lib/automake.mk b/ovn/lib/automake.mk > index 4870505..5e5b04f 100644 > --- a/ovn/lib/automake.mk > +++ b/ovn/lib/automake.mk > @@ -10,7 +10,10 @@ ovn_lib_libovn_la_SOURCES = \ > ovn/lib/expr.h \ > ovn/lib/lex.c \ > ovn/lib/lex.h \ > - ovn/lib/logical-fields.h > + ovn/lib/logical-fields.h \ > + ovn/lib/datalog.h \ > + ovn/lib/datalog-private.h \ > + ovn/lib/datalog.c > nodist_ovn_lib_libovn_la_SOURCES = \ > ovn/lib/ovn-nb-idl.c \ > ovn/lib/ovn-nb-idl.h \ > diff --git a/ovn/lib/datalog-private.h b/ovn/lib/datalog-private.h > new file mode 100644 > index 0000000..4a1acfb > --- /dev/null > +++ b/ovn/lib/datalog-private.h > @@ -0,0 +1,808 @@ > +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. > + * > + * Licensed under the Apache License, Version 2.0 (the "License"); > + * you may not use this file except in compliance with the License. > + * You may obtain a copy of the License at: > + * > + * http://www.apache.org/licenses/LICENSE-2.0 > + * > + * Unless required by applicable law or agreed to in writing, software > + * distributed under the License is distributed on an "AS IS" BASIS, > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > + * See the License for the specific language governing permissions and > + * limitations under the License. > + */ > + > +#ifndef OVN_DATALOG_PRIV_H > +#define OVN_DATALOG_PRIV_H 1 > + > +#include <string.h> > +#include <stdbool.h> > +#include <stdlib.h> > +#include <stdio.h> > +#include <ctype.h> > +#include <stdint.h> > +#include <inttypes.h> > +#include <time.h> > + > +#include "util.h" > + > +/* -------------------------------------------------------------------------- > + * CONSTANTS > + * -------------------------------------------------------------------------- > + */ > + > +#define LOG_COMP 0 /* 0 for not enable logging */ > + > +#define SZ_INIT_BITSET 16 /* initial capacity of bitset */ > +#define SZ_INIT_ARRAY 16 /* initial capacity of array */ > +#define SZ_INIT_HASH 11 /* initial capacity of hash map */ > + > +#define LOG_TOKEN_SZ 512 /* max length of token or string literal */ > + > +/* ENTITY TYPES */ > + > +#define ENT_UNKNOWN 0 > +#define ENT_INT32 1 > +#define ENT_TST_INT32 2 /* for testing only */ > +#define ENT_STR 3 /* null terminated string */ > +#define ENT_VALUE 4 /* log_value_t* */ > +#define ENT_INT_TUPLE 5 /* log_int_tuple_t* */ > +#define ENT_TUPLE 6 /* log_tuple_t* */ > +#define ENT_TABLE 7 /* log_table_t* */ > +#define ENT_RULE 8 /* log_rule_t* */ > +#define ENT_RULE_SET 9 /* log_rule_set_t* */ > +#define ENT_LOG_ENG 0x0a/* log_engine_t* */ > +#define ENT_JOIN_PARAM 0x0b/* log_join_param_t* */ > + > +#define ENT_ARRAY 0x0f/* array */ > +#define ENT_MAP 0x0e/* map */ > +#define ENT_SET 0x0d/* set */ > +#define ENT_INDEX 0x0c/* index of tuple */ > + /* the collection type must be */ > + /* (ENT_UNKNOWN, ENT_UNKNOWN, ENT_INDEX) */ > +#define ENT_BITSET 0x10 > + > +#define KEY_TYPE(t) ((t) & 0xff) > +#define KEY_ID(id) ((id)) > +#define VALUE_TYPE(t) (((t) >> 8) & 0xff) > +#define VALUE_ID(id) ((id) << 8) > +#define COLL_TYPE(t) (((t) >> 16) & 0xff) /* collection type */ > +#define COLL_ID(id) ((id) << 16) > + > +#define MAP_INT32_VALUE (KEY_ID(ENT_INT32) | VALUE_ID(ENT_VALUE)) > + > +/* conversion between pointer and int32_t. type cast will produce warning */ > +#define ptr2i(v) ((int32_t)(((union { intptr_t i; void* p; })(v)).i)) > +#define i2ptr(v) (((union { intptr_t i; void* p; })(intptr_t)(v)).p) > + > +#define TYPE(t, def) t /* showing type of object in array, set */ > +#define TYPE2(t, key, val) t /* showing type of object in map */ > + > +/* -------------------------------------------------------------------------- > + * BITSET, ARRAY, HASH MAP, AND HASH SET > + * -------------------------------------------------------------------------- > + */ > + > +/* when type is known for a collection, it could be freed using [coll]_free, > + * and all nested structure will be freed. items never cross reference by > + * default, except for log_values. when there is cross reference, the 'free' > + * function must be defined. 'global value' could be regarded as global > + * context for the object. > + * > + * naming convention for collection function: collection_method: > + * collection could be: bitset, array, set, map > + * method could be: (init, free), (add, del), (ins, rmv), (get, set). > + */ > + > +struct hash_s; > + > +typedef struct meta_s { > + int32_t type; /* must be the first field */ > + bool alloc; /* indicates if the structure is from malloc */ > + struct hash_s* glb_values; > +} meta_t; > + > +typedef struct bitset_s { > + meta_t m; > + int32_t size; /* size of items in bytes */ > + int32_t len; /* length of allocated items */ > + uint32_t* items; > +} bitset_t; > + > +typedef struct array_s { > + meta_t m; > + void** item; > + int32_t size; /* size of the array */ > + int32_t len; /* length of allocated space */ > +} array_t; > + > +typedef struct map_node_s { > + struct map_node_s* next; > + void* key; > + void* value; > +} map_node_t; > + > +/* set_node_s, index_node_s must have the same structure as map_node_s > + * except the last field. assume they have the same alignment, and > + * map_node_s is used to access both. > + */ > +typedef struct set_node_s { > + struct set_node_s* next; > + void* key; > +} set_node_t; > + > +/* for index map, the aux points to log_int_tuple which defines the > + * index. the key of each entry points to one tuple of the tuple set > + * whose elements share the same tuple key. the set is represented by > + * double link presented in the tuple. > + */ > +typedef struct index_node_s { > + struct index_node_s* next; > + void* key; /* points to one tuple of the set */ > + int32_t hash_code; /* hash_code of the tuple key */ > +} index_node_t; > + > +typedef struct hash_s { > + meta_t m; > + map_node_t** bucket; > + int32_t size; /* size of items */ > + int32_t len; /* length of bucket array */ > + void* aux; /* extra data */ > +} hash_t; > + > +typedef hash_t map_t; > +typedef hash_t set_t; > + > +/* -------------------------------------------------------------------------- > + * DATA STRUCTURE OF LOG ENGINE > + * -------------------------------------------------------------------------- > + */ > + > +typedef struct log_config_s { > + char sep1; /* field separator */ > + char sep2; /* record separator */ > +} log_config_t; > + > +typedef struct log_value_s {/* variable size structure */ > + int32_t hash_code; /* must be first field */ > + int32_t ref_no; /* number of references */ > + /* the actual size is abs(size) and it does not count terminating 0 */ > + /* 0 is always padded no matter .a or .p is used for printf */ > + int32_t size; /* size < 0 indicates using value.p */ > + /* size >= 0 indicates using value.a */ > + union { > + char a[0]; /* byte array, need not terminate with null */ > + char* p; /* only used in populating the value set */ > + } value; > +} log_value_t; > + > +/* > + * log_tuple_s.indexes is a log_tuple_t pointer array, which represents > + * an array of double linked list. for index i, indexes[i * 2] points to > + * previous node and index[i * 2 + 1] points to success node. > + */ > +typedef struct log_tuple_s {/* variable size structure */ > + /* do not have size due to memory consideration */ > + int32_t hash_code; /* must be first field */ > + int32_t n_values; /* duplicated, see table.num_fields */ > + int64_t count; /* valid only in table, when tuple is outside */ > + /* table, it has different meaning */ > + struct log_tuple_s** indexes; /* valid only in table */ > + log_value_t* values[0]; /* field array of the tuple */ > +} log_tuple_t; > + > +typedef struct log_int_tuple_s {/* variable size structure */ > + int32_t hash_code; /* must be first field */ > + int32_t n_items; /* number of integers */ > + int32_t values[0]; > +} log_int_tuple_t; > + > +struct log_engine_s; > + > +/* index_map, index_def and tuple.indexes must have the same size, > + * and aligns to each other, i.e. assume there is N indexes defined, > + * tuple.indexes has N * 2 items, and index_map and index_def > + * has N items. For index j: > + * (1) index_def[key_def]->j, defines the index, e.g., '0:3:4'->1. > + * (2) index_map[j] defines hash map from key tuples to tuple set. > + * The aux of this map points to corresponding index_def's key. > + * (3) tuple.indexes[j * 2] and tuple.indexes[j * 2 + 1] defines tuple set. > + */ > +typedef struct log_table_s { > + meta_t m; > + int32_t table_index; /* id of the table */ > + int32_t num_fields; /* also present in each tuple */ > + bool is_remove; /* valid if this is delta */ > + > + TYPE2(map_t, log_int_tuple_t*, int32_t) index_def; > + TYPE(array_t, hash_t*) index_map; > + TYPE(set_t, log_tuple_t*) tuples; > +} log_table_t; > + > +typedef struct log_rule_s { > + meta_t m; > + > + bool is_union; > + /* > + * table index start from 0. item 0 is left side > + * example X1 : X2, X3 -> 7, 3, 6 > + * rule and param has the same sequence. > + */ > + TYPE(array_t, int32_t) rule; > + > + /* > + * item 0 is left side. param index starts from 0. > + * example: X1(a,b) : X2(a, -), X3('c', b) -> ((0, 1), (0, -1), (-2, 1)) > + * -1 indicates 'ignore', -2 is the index for first constant > + */ > + TYPE(array_t, array_t*) param; /* array of array of integer */ > + /* example: -2 -> 'c', map integer to value */ > + TYPE2(map_t, int32_t, log_value_t*) const_param; > + > + /* example: 0->'a', 1->'b' */ > + TYPE2(map_t, int32_t, log_value_t*) param_name_map; > +} log_rule_t; > + > +typedef struct log_rule_set_s { > + meta_t m; > + > + /* example: 7-> 'X1' */ > + TYPE2(map_t, int32_t, log_value_t*) rule_name_map; > + TYPE2(map_t, log_value_t*, int32_t) rule_index_map; > + > + /* table index -> rule. table index starts from 0 */ > + TYPE2(map_t, int32_t, log_rule_s*) rules; > + /* table index -> table index set */ > + TYPE2(map_t, int32_t, array_t*) table_rule_map; /* array of integer */ > + /* example: 3->(7), 6->(7), list is ordered */ > + > + TYPE(set_t, int32_t) input_tables; > + TYPE(set_t, int32_t) output_tables; > + TYPE2(map_t, int32_t, int32_t) param_size; > +} log_rule_set_t; > + > +typedef struct log_io_s { > + TYPE(array_t*, log_table_t*) inp_remove; > + TYPE(array_t*, log_table_t*) inp_insert; > + TYPE(array_t*, log_table_t*) res; > + > + log_table_t* cur_tbl; > + log_tuple_t* cur_tuple; > + int32_t t_idx; /* table index */ > + int32_t f_idx; /* field index */ > + > + map_node_t* hash_node; /* current node of iterator */ > + int32_t hash_b; /* current bucket of iterator */ > +} log_io_t; > + > +typedef struct log_engine_s { > + meta_t m; > + > + log_rule_set_t rule_set; > + /* map table index to log_table_t */ > + TYPE2(map_t, int32_t, log_table_t*) tables; > + > + bool (*ext_func)( /* to reset state if last 3 param are NULL */ > + struct log_engine_s* eng, log_table_t*, log_table_t*, log_table_t*); > + > + log_io_t io; /* temporary data for calling (put|get)_(table|field) */ > +} log_engine_t; > + > +typedef struct log_join_param_s { > + meta_t m; > + > + log_int_tuple_t* index2; > + TYPE(array_t, log_value_t*) select1; > + TYPE(array_t, int32_t) select1i; /* match select1 in size */ > + TYPE(array_t, int32_t) rem1; > + TYPE(array_t, int32_t) rem2; > + TYPE(array_t, int32_t) out_param; > + bool full_join; > +} log_join_param_t; > + > +/* -------------------------------------------------------------------------- > + * PROTOTYPES > + * -------------------------------------------------------------------------- > + */ > + > +void log_topo_sort( > + TYPE2(map_t*, log_value_t*, set_t*) g, /* set of value */ > + TYPE(array_t*, log_value_t*) order, > + TYPE(set_t*, log_value_t*) in_nodes, > + TYPE(set_t*, log_value_t*) out_nodes); > +void log_sort_array( > + int start, array_t* list, array_t* sem1, array_t* sem2); > +int log_insert_item( > + int val, array_t* list, void* obj1, > + void* obj2, array_t* sem1, array_t* sem2); > + > +void log_set_sep(char sep1, char sep2); > +void log_set_global_value(set_t* s); > + > +void log_sync_init(const char* log, set_t* gv); > +void log_sync_parse(map_t* sem); > +void log_sem_process(log_rule_set_t* rule_set, map_t* sem); > +void log_eng_set_ext_func(log_engine_t* eng, void* func); > +log_engine_t* log_eng_parse(const char* rules, set_t* gv); > + > +int32_t log_hash_code(hash_t* m, const void* v); > +int32_t log_hash_code_byte(const void* v, int32_t* size); > +bool log_key_equal(hash_t* m, const void* k1, const void* k2); > +void log_coll_free(void* coll, int32_t type, set_t* gv); > + > +map_t* log_hash_init(map_t* m, int type, int sz_init, hash_t*); > +void log_hash_free(hash_t*); > +void log_hash_add(map_t* m, void* k, void* v); > +void* log_hash_del(map_t* m, void* k); > +void log_hash_rehash(map_t* m); > +bool log_hash_next(map_t* m, int32_t* b, map_node_t** cur); > +void* log_hash_get_one(map_t* m); > + > +bitset_t* log_bitset_init(bitset_t* set); > +void log_bitset_free(bitset_t*); > + > +array_t* log_array_init( > + array_t* a, int32_t type, int32_t i_size, set_t* gv); > +void log_array_free(array_t*); > +array_t* log_array_clone(array_t* a); > +int32_t log_array_look_for(array_t* a, void* v); > + > +log_value_t* log_value_init(const char* v, int32_t size, set_t* gv); > +log_int_tuple_t* log_int_tuple_init(TYPE(array_t*, int32_t) a); > +void log_int_tuple_free(log_int_tuple_t*); > + > +log_tuple_t* log_tuple_init(int32_t n_values); > +void log_tuple_free(log_tuple_t*, set_t*, bool); > +log_tuple_t* log_tuple_init_val(log_value_t** val, int32_t n_values); > +log_tuple_t* log_tuple_init_str(const char* t, char sep, set_t* gv); > +log_tuple_t* log_tuple_init_str_null( > + const char* t, char sep, set_t* gv); > + > +log_table_t* log_table_init( > + log_table_t* tbl, int32_t n, int32_t f, > + int32_t size, set_t* gv); > +void log_table_free(log_table_t* tbl); > +void log_table_add(log_table_t* tbl, log_tuple_t* t); > +void log_table_remove(log_table_t* tbl, log_tuple_t* t); > + > +int32_t log_table_add_index( > + log_table_t* tbl, log_int_tuple_t* index_key); > +log_tuple_t* log_index_get_index( > + log_table_t* tbl, log_tuple_t* t, int32_t i_idx); > + > +void log_rule_free(log_rule_t* rule); > +void log_rule_set_init(log_rule_set_t* rs, set_t* gv); > +void log_rule_set_free(log_rule_set_t*); > + > +log_engine_t* log_engine_init(log_engine_t* log, set_t* gv); > +void log_engine_free(log_engine_t*); > + > +log_join_param_t* log_join_param_init( > + log_join_param_t* jp, log_int_tuple_t* i2, > set_t* gv); > +void log_join_param_free(log_join_param_t*); > + > +log_table_t* log_tblopr_join( > + log_table_t* t1, log_table_t* t2, > + log_join_param_t* joinp); > +void log_eng_do_join( > + log_engine_t* eng, > + log_table_t* input, log_table_t* output); > +void log_eng_do_union(log_engine_t* eng, > + log_table_t* input, log_table_t* output); > + > +TYPE(array_t*, log_table_t*) log_eng_delta( > + log_engine_t* eng, > + TYPE(array_t*, log_table_t*) inp_remove, > + TYPE(array_t*, log_table_t*) inp_insert); > +TYPE(array_t*, log_table_t*) log_eng_query( > + log_engine_t* eng, > + TYPE(array_t*, log_table_t*) input); > + > +log_table_t* log_get_table(log_engine_t* eng, const char* name); > +log_table_t* log_get_org_table(log_engine_t* eng, log_table_t* t); > +log_tuple_t* log_query_on0(log_engine_t* eng, > + int32_t tid, log_value_t* value); > + > +log_table_t* log_io_table_name_reset( > + log_engine_t* eng, const char* name); > +bool log_io_parse_line( > + const char* line, log_engine_t* eng, bool use_null, > + TYPE(array_t*, log_table_t*) inp_remove, > + TYPE(array_t*, log_table_t*) inp_insert); > +int32_t log_io_gen_line(char* text, int pos, log_engine_t* eng, > + TYPE(array_t*, log_table_t*) all); > + > +int32_t log_coll_print( > + char* buf, int32_t pos, void* item, int32_t type, bool verbose); > +int32_t log_array_print(char* buf, int32_t pos, array_t* a, bool verbose); > +int32_t log_hash_print(char* buf, int32_t pos, hash_t* m, bool verbose); > +int32_t log_table_print(char* buf, int32_t pos, log_table_t* t, > bool verbose); > +int32_t log_index_print(char* buf, int32_t pos, log_table_t* t); > +int32_t log_tuple_print(char* buf, int32_t pos, log_tuple_t* t); > +int32_t log_rule_set_print(char* buf, int32_t pos, log_rule_set_t* rs); > + > +/* -------------------------------------------------------------------------- > + * MACROS AND INLINE IMPLEMENTATIONS > + * -------------------------------------------------------------------------- > + */ > + > +#define map_size(m) ((m)->size) > +#define map_has(m, k) (log_hash_get(m, k) != NULL) > +#define map_free(m) log_hash_free(m) > +#define map_del(m, k) log_hash_del(m, k) > + > +#define set_size(m) ((m)->size) > +#define set_has(m, k) (log_hash_get(m, k) != NULL) > +#define set_free(m) log_hash_free(m) > +#define set_del(m, k) log_hash_del(m, k) > + > +#define array_size(a) ((a)->size) > +#define table_size(t) (set_size(&(t)->tuples)) > + > +#define map_get_int(m, k) ptr2i(map_get(m, k)) > +#define array_get_int(a, i) ptr2i(array_get(a, i)) > + > +#define log_index_i_pre(tuple, i) ((tuple)->indexes[(i) * 2]) > +#define log_index_i_suc(tuple, i) ((tuple)->indexes[(i) * 2 + 1]) > + > +/* -------------------------------------------------------------------------- > + * META DATA > + * -------------------------------------------------------------------------- > + */ > + > +static inline void* > +c_realloc(void* ptr, size_t old_sz, size_t new_sz) > +{ > + /* ptr could be NULL and expanded area will be zeroed */ > + void* n = realloc(ptr, new_sz); > + memset(((char*)n) + old_sz, 0, new_sz - old_sz); > + return n; > +} > + > +static inline void > +hash_code_array_init(int32_t* c) > +{ > + /* sequence of 0 will have different hash code depending on itslength */ > + *c = 1; > +} > + > +static inline void > +hash_code_array_add(int32_t* c, int32_t i) > +{ > + *c = (*c) * 31 + i; > +} > + > +static inline void > +hash_code_array_final(int32_t* c) > +{ > + if (*c < 0) *c = -(*c); > +} > + > +static inline void > +coll_alloc(void* ptr, int32_t size, int32_t type, void* global_values) > +{ > + meta_t** m = (meta_t**)ptr; > + if (*m == NULL) { > + *m = malloc(size); > + (*m)->alloc = true; > + } > + else { > + (*m)->alloc = false; > + } > + > + (*m)->type = COLL_ID(type); > + (*m)->glb_values = global_values; > +} > + > +static inline void > +coll_free_ptr(void* ptr) > +{ > + meta_t* m = ptr; > + if (m->alloc) free(m); > +} > + > +static inline void > +log_value_ref(log_value_t* v) > +{ > + v->ref_no++; > +} > + > +static inline void > +log_value_free(log_value_t* v, set_t* gv) > +{ > + if (v->ref_no > 1) --v->ref_no; > + else { > + int32_t sv_type = gv->m.type; > + gv->m.type = 0; > + log_hash_del(gv, v); > + gv->m.type = sv_type; > + free(v); > + } > +} > + > +static inline void > +check_value_ref(void* v, int32_t type, set_t* gv, bool add) > +{ > + if (v == NULL) return; > + if (!add) log_coll_free(v, type, gv); > + else if (type == ENT_VALUE) log_value_ref((log_value_t*)v); > +} > + > +/* -------------------------------------------------------------------------- > + * BITSET > + * -------------------------------------------------------------------------- > + */ > + > +static inline void > +bitset_resize(bitset_t* set, int32_t len) > +{ > + /* since there is no reset operation, it always expands */ > + set->items = c_realloc(set->items, > + set->len * sizeof(int32_t), len * sizeof(int32_t)); > + /* assume align to 4 bytes for int32_t */ > + set->len = len; > +} > + > +static inline void > +bitset_set(bitset_t* set, int32_t b) > +{ > + int32_t p = b >> 5; > + if (p >= set->size) { > + set->size = p + 1; > + if (set->size > set->len) bitset_resize(set, set->size); > + } > + > + set->items[p] |= 1 << (b % 32); > +} > + > +static inline bool > +bitset_get(bitset_t* set, int b) > +{ > + int p = b >> 5; > + if (p >= set->size) return false; > + return (set->items[p] & (1 << (b % 32))) != 0; > +} > + > +static inline bool > +bitset_empty(bitset_t* set) > +{ > + int i; > + for (i = 0;i < set->size;i++) > + if (set->items[i] != 0) return false; > + return true; > +} > + > +static inline void > +bitset_and(bitset_t* dest, bitset_t* src) > +{ > + > + int32_t m_size = dest->size > src->size ? src->size : dest->size; > + > + int i; > + for (i = 0;i < m_size;i++) dest->items[i] &= src->items[i]; > + for (i = m_size;i < dest->size;i++) dest->items[i] = 0; > +} > + > +/* -------------------------------------------------------------------------- > + * ARRAY > + * -------------------------------------------------------------------------- > + */ > + > +static inline void > +array_add(array_t* a, void* i) > +{ > + ovs_assert(a->size <= a->len); > + > + if (a->size == a->len) { > + a->item = c_realloc(a->item, > + a->len * sizeof(void*), a->len * 2 * sizeof(void*)); > + a->len = a->len * 2; > + } > + a->item[a->size++] = i; > + check_value_ref(i, KEY_TYPE(a->m.type), a->m.glb_values, true); > +} > + > +static inline void > +array_ins(array_t* a, int32_t pos, void* i) > +{ > + array_add(a, NULL); /* make room for new item */ > + memmove(&a->item[pos + 1], &a->item[pos], > + (a->size - pos - 1) * sizeof(void*)); /* size has increased by 1 */ > + > + a->item[pos] = i; > + check_value_ref(i, KEY_TYPE(a->m.type), a->m.glb_values, true); > +} > + > +static inline void* > +array_rmv(array_t* a, int32_t pos) > +{ > + /* no change to ref */ > + void* org = a->item[pos]; > + memmove(&a->item[pos], &a->item[pos + 1], > + (a->size - pos - 1) * sizeof(void*)); > + a->size--; > + return org; > +} > + > +static inline void* > +array_get(array_t* a, int32_t i) > +{ > + ovs_assert(i >= 0 && i < a->size); > + return a->item[i]; > +} > + > +static inline void > +array_set(array_t* a, int i, void* v) > +{ > + ovs_assert(i >= 0 && i < a->size); > + check_value_ref(a->item[i], KEY_TYPE(a->m.type), a->m.glb_values, false); > + > + a->item[i] = v; > + check_value_ref(v, KEY_TYPE(a->m.type), a->m.glb_values, true); > +} > + > +/* -------------------------------------------------------------------------- > + * HASH TABLE > + * -------------------------------------------------------------------------- > + */ > + > +static inline map_node_t* > +log_hash_get(map_t* m, void* k) > +{ > + /* no change to reference count */ > + > + int32_t code = log_hash_code(m, k); > + int32_t slot = code % m->len; > + map_node_t* head = m->bucket[slot]; > + > + while (head != NULL) { > + if (log_key_equal(m, head->key, k)) return head; > + head = head->next; > + } > + return NULL; > +} > + > +/* -------------------------------------------------------------------------- > + * MAP > + * -------------------------------------------------------------------------- > + */ > + > +static inline map_t* > +map_init(map_t* m, int type, int sz_init, set_t* gv) > +{ > + return log_hash_init(m, type | COLL_ID(ENT_MAP), sz_init, gv); > +} > + > +static inline void* > +map_get(map_t* m, void* k) > +{ > + map_node_t* node = log_hash_get(m, k); > + if (node == NULL) return NULL; > + else return node->value; > +} > + > +static inline void > +map_add(map_t* m, void* k, void* v) > +{ > + map_node_t* node = log_hash_get(m, k); > + > + if (node != NULL) { > + void* old = node->value; > + check_value_ref(old, VALUE_ID(m->m.type), m->m.glb_values, false); > + > + node->value = v; > + check_value_ref(v, VALUE_ID(m->m.type), m->m.glb_values, true); > + > + } else log_hash_add(m, k, v); > +} > + > +/* -------------------------------------------------------------------------- > + * SET > + * -------------------------------------------------------------------------- > + */ > + > +static inline set_t* > +set_init(set_t* m, int type, int sz_init, set_t* gv) > +{ > + return log_hash_init(m, type | COLL_ID(ENT_SET), sz_init, gv); > +} > + > +static inline void* > +set_get(set_t* m, void* k) > +{ > + map_node_t* node = log_hash_get(m, k); > + if (node == NULL) return NULL; > + else return node->key; > +} > + > +static inline void > +set_add(set_t* m, void* k) > +{ > + map_node_t* node = log_hash_get(m, k); > + if (node != NULL) return; > + log_hash_add(m, k, NULL); > +} > + > +/* -------------------------------------------------------------------------- > + * ITERATIONS > + * -------------------------------------------------------------------------- > + */ > + > +/* nested usage is supported. > + * 'continue' does not work with set. > + * 'if (c) [TYPE]_EXIT; else {}' does not work with map and set. > + */ > + > +#define ARRAY_ALL_0(array, node, type, typec) \ > + { int __array_i = 0; \ > + for (;__array_i < (array)->size;__array_i++) { \ > + type node = typec((array)->item[__array_i]); { > + > +#define ARRAY_EXIT break > +#define ARRAY_END }}} > + > +#define MAP_ALL(map, node) \ > + { bool __map_exit = false; \ > + map_t* __map = (map_t*)(map); \ > + int __map_i = 0; \ > + for (;__map_i < __map->len;__map_i++) { \ > + if (__map_exit) break; \ > + struct map_node_s* __map_head = __map->bucket[__map_i]; \ > + while (__map_head != NULL) { \ > + struct map_node_s* node = __map_head; \ > + __map_head = __map_head->next; { > + > +#define MAP_EXIT {__map_exit = true; break; } > +#define MAP_END }}}} > + > +#define SET_ALL_0(set, node, type, typec) \ > + { bool __set_exit = false, __set_rmv; \ > + struct set_node_s* __set_save = NULL, *__set_pre; \ > + set_t* __set = (set_t*)(set); \ > + int32_t __set_i = 0; \ > + for (;__set_i < __set->len;__set_i++) { \ > + if (__set_exit) break; \ > + struct set_node_s* __set_this = (struct set_node_s*) \ > + (__set->bucket[__set_i]); \ > + __set_pre = NULL; \ > + while (__set_this != NULL) { \ > + type node = typec(__set_this->key); \ > + __set_rmv = false; { > + > +#define SET_REMOVE_ITEM /* will not rehash */ \ > + (__set_rmv = true, __set->size--, __set_save = __set_this, \ > + __set_pre == NULL ? (__set->bucket[__set_i] = \ > + (map_node_t*)__set_this->next) : \ > + (map_node_t*)(__set_pre->next = __set_this->next), \ > + __set_this = __set_this->next, free(__set_save)) > + > +#define SET_EXIT {__set_exit = true; break; } > + > +#define SET_END } \ > + if (!__set_rmv) { __set_pre = __set_this; \ > + __set_this = __set_this->next;} }} \ > + __set_save++; __set_pre++; } /* make no warning */ > + > +#define INDEX_ALL(tuple, i_idx, node) \ > + { bool __index_first = true; \ > + log_tuple_t* __index_head = tuple; \ > + log_tuple_t* node; \ > + if (__index_head != NULL) { for (;;) { \ > + if (__index_first) { \ > + node = __index_head; __index_first = false; } \ > + else { \ > + (node) = (node)->indexes[i_idx * 2 + 1]; \ > + if (__index_head == node) break; \ > + } > + > +#define INDEX_EXIT break > +#define INDEX_END }}} > + > +#define ARRAY_ALL(array, node, type) ARRAY_ALL_0(array, node, type, (type)) > +#define ARRAY_ALL_INT(array, node) ARRAY_ALL_0(array, node, int32_t, ptr2i) > + > +#define SET_ALL(set, node, type) SET_ALL_0(set, node, type, (type)) > +#define SET_ALL_INT(set, node) SET_ALL_0(set, node, int32_t, ptr2i) > + > +#endif /* datalog-private.h */ > diff --git a/ovn/lib/datalog.c b/ovn/lib/datalog.c > new file mode 100644 > index 0000000..3354512 > --- /dev/null > +++ b/ovn/lib/datalog.c > @@ -0,0 +1,3392 @@ > +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. > + * > + * Licensed under the Apache License, Version 2.0 (the "License"); > + * you may not use this file except in compliance with the License. > + * You may obtain a copy of the License at: > + * > + * http://www.apache.org/licenses/LICENSE-2.0 > + * > + * Unless required by applicable law or agreed to in writing, software > + * distributed under the License is distributed on an "AS IS" BASIS, > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > + * See the License for the specific language governing permissions and > + * limitations under the License. > + */ > + > +#include <config.h> > +#include "datalog.h" > +#include "datalog-private.h" > + > +/* ========================================================================== > + * BASIC COLLECTIONS > + * ========================================================================== > + */ > + > +void > +log_coll_free(void* coll, int32_t type, set_t* gv) > +{ > + switch (type) { > + case ENT_BITSET: log_bitset_free(coll); break; > + case ENT_ARRAY: log_array_free(coll); break; > + > + case ENT_MAP: > + case ENT_SET: > + case ENT_INDEX: log_hash_free(coll); break; > + > + case ENT_TUPLE: log_tuple_free(coll, gv, true); break; > + case ENT_VALUE: log_value_free(coll, gv); break; > + case ENT_TABLE: log_table_free(coll); break; > + case ENT_INT_TUPLE: log_int_tuple_free(coll); break; > + case ENT_RULE: log_rule_free(coll); break; > + case ENT_RULE_SET: log_rule_set_free(coll); break; > + case ENT_LOG_ENG: log_engine_free(coll); break; > + case ENT_JOIN_PARAM:log_join_param_free(coll); break; > + } > +} > + > +/* -------------------------------------------------------------------------- > + * BITSET > + * -------------------------------------------------------------------------- > + */ > + > +bitset_t* > +log_bitset_init(bitset_t* set) > +{ > + coll_alloc(&set, sizeof(bitset_t), ENT_BITSET, NULL); > + set->len = SZ_INIT_BITSET; > + set->items = calloc(set->len, sizeof(void*)); > + set->size = 0; > + return set; > +} > + > +void > +log_bitset_free(bitset_t* dest) > +{ > + if (dest->items != NULL) free(dest->items); > + coll_free_ptr(dest); > +} > + > +/* -------------------------------------------------------------------------- > + * ARRAY > + * -------------------------------------------------------------------------- > + */ > + > +void > +log_set_global_value(set_t* s) > +{ > + s->m.type = KEY_ID(ENT_VALUE) | COLL_ID(ENT_SET); > + s->m.glb_values = s; > +} > + > +array_t* > +log_array_init(array_t* a, int32_t type, int32_t i_size, set_t* gv) > +{ > + coll_alloc(&a, sizeof(array_t), ENT_ARRAY, gv); > + > + a->len = i_size; > + a->m.type |= type; > + if (a->len == 0) a->len = SZ_INIT_ARRAY; > + a->item = calloc(a->len, sizeof(void*)); > + a->size = 0; > + return a; > +} > + > +array_t* > +log_array_clone(array_t* a) > +{ > + int i; > + array_t* na = log_array_init(NULL, a->m.type, a->size, a-> m.glb_values); > + memcpy(na->item, a->item, a->size * sizeof(void*)); > + na->size = a->size; > + > + for (i = 0;i < a->size;i++) > + check_value_ref(na->item[i], KEY_TYPE(na->m.type), > + na->m.glb_values, true); > + return na; > +} > + > +void > +log_array_free(array_t* a) > +{ > + int i; > + for (i = 0;i < a->size;i++) > + check_value_ref(a->item[i], KEY_TYPE(a->m.type), > + a->m.glb_values, false); > + free(a->item); > + coll_free_ptr(a); > +} > + > +int32_t > +log_array_look_for(array_t* a, void* v) > +{ > + int i = 0; > + bool found = false; > + > + ARRAY_ALL(a, item, const void*) > + if (item == v) { > + found = true; > + ARRAY_EXIT; > + } > + i++; > + ARRAY_END > + > + if (found) return i; > + else return -1; > +} > + > +int32_t > +log_array_print(char* buf, int32_t pos, array_t* a, _Bool verbose) { > + int i = 0; > + buf[pos++] = '['; > + int32_t ktype = KEY_TYPE(a->m.type); > + for (i = 0; i < a->size; i++) { > + if (i > 0) > + buf[pos++] = ','; > + > + pos = log_coll_print(buf, pos, a->item[i], ktype, verbose); > + } > + buf[pos++] = ']'; > + return pos; > +} > + > +/* -------------------------------------------------------------------------- > + * HASH CODE > + * -------------------------------------------------------------------------- > + */ > + > +int32_t > +log_hash_code_byte(const void* v, int32_t* size) > +{ > + /* > + * size of 0 indicates null terminated string, and actual size not > + * including null is returned in size. > + */ > + > + const unsigned char* p = v; > + uint32_t hash = 0; > + int32_t i = 0; > + > + /* Jenkins's Hash */ > + if (*size == 0) { > + unsigned char c; > + while ((c = *p++) != 0) { > + i++; > + hash += c; > + hash += (hash << 10); > + hash ^= (hash >> 6); > + } > + *size = i; > + } > + else { > + int32_t s = *size; > + for (;i < s;i++) { > + hash += *p++; > + hash += (hash << 10); > + hash ^= (hash >> 6); > + } > + } > + > + hash += (hash << 3); > + hash ^= (hash >> 11); > + hash += (hash << 15); > + return (int32_t) hash < 0 ? -hash : hash; > +} > + > +bool > +log_key_equal(hash_t* m, const void* k1, const void* k2) > +{ > + /* if only one key is from hash_table, it must be k1 */ > + int type = KEY_TYPE(m->m.type); > + > + if (type == ENT_VALUE) { > + return k1 == k2; > + } > + else if (type == ENT_TUPLE) { > + const log_tuple_t* t1 = k1; > + const log_tuple_t* t2 = k2; > + > + if (t1->hash_code != t2->hash_code) return false; > + int32_t sz_tuples = > + (((log_table_t*)(m->aux))->num_fields) * sizeof(void*); > + return memcmp(&t1->values, &t2->values, sz_tuples) == 0; > + } > + > + else if (COLL_TYPE(m->m.type) == ENT_INDEX) { > + const log_tuple_t* t1 = k1; > + const log_tuple_t* t2 = k2; > + > + int i; > + log_int_tuple_t* index_def = m->aux; > + > + if (t1 == t2) return true; > + if (t2->count == 0) { /* compact form */ > + for (i = 0;i < index_def->n_items;i++) > + if (t1->values[index_def->values[i]] != t2->values[i]) > + return false; > + } > + else { > + for (i = 0;i < index_def->n_items;i++) > + if (t1->values[index_def->values[i]] != > + t2->values[index_def->values[i]]) { > + return false; > + } > + } > + return true; > + } > + > + else if (m == m->m.glb_values) { > + const log_value_t* v1 = k1; > + const log_value_t* v2 = k2; > + > + if (v1->size >= 0 && v2->size >= 0) return k1 == k2; > + if (v1->hash_code != v2->hash_code) return false; > + if (v1->size == 0 && v2->size == 0) return true; > + > + const void* p1 = v1->size > 0 ? v1->value.a : v1->value.p; > + const void* p2 = v2->size > 0 ? v2->value.a : v2->value.p; > + > + int32_t s1 = v1->size > 0 ? v1->size : (-v1->size); > + int32_t s2 = v2->size > 0 ? v2->size : (-v2->size); > + if (s1 != s2) return false; > + return memcmp(p1, p2, s1) == 0; > + } > + > + else if (type == ENT_INT_TUPLE) { > + const log_int_tuple_t* t1 = k1; > + const log_int_tuple_t* t2 = k2; > + > + if (t1->hash_code != t2->hash_code) return false; > + if (t1->n_items != t2->n_items) return false; > + > + int s = t1->n_items * sizeof(int32_t); > + return memcmp(&t1->values, &t2->values, s) == 0; > + } > + > + > + else if (type == ENT_INT32 || type == ENT_TST_INT32) > + return ptr2i((void*)k1) == ptr2i((void*)k2); > + else if (type == ENT_STR) return strcmp(k1, k2) == 0; > + > + ovs_assert(false); > + return false; > +} > + > +int32_t > +log_hash_code(hash_t* m, const void* v) > +{ > + > + int32_t type = KEY_TYPE(m->m.type); > + if (type == ENT_STR) { > + int32_t size0 = 0; > + return log_hash_code_byte(v, &size0); > + } > + > + else if (COLL_TYPE(m->m.type) == ENT_INDEX) { > + int32_t i, code; > + log_tuple_t* t = (log_tuple_t*)v; > + > + log_int_tuple_t* index_def = m->aux; > + hash_code_array_init(&code); > + > + if (t->count == 0) { /* compact form */ > + for (i = 0; i < index_def->n_items; i++) { > + log_value_t* v = t->values[i]; > + hash_code_array_add(&code, v->hash_code); > + } > + } else { > + for (i = 0; i < index_def->n_items; i++) { > + log_value_t* v = t->values[index_def->values[i]]; > + hash_code_array_add(&code, v->hash_code); > + } > + } > + > + hash_code_array_final(&code); > + return code; > + } > + > + else if (type == ENT_INT32) { > + int32_t n = ptr2i((void*)v); > + if (n < 0) > + n = -n; > + return n; > + } else if (type == ENT_TST_INT32) > + return ptr2i((void*)v) % 100; > + else > + return *(const int32_t*) v; > +} > + > +/* -------------------------------------------------------------------------- > + * HASH TABLE > + * -------------------------------------------------------------------------- > + */ > + > +map_t* > +log_hash_init(map_t* m, int type, int sz_init, struct hash_s* values) > +{ > + coll_alloc(&m, sizeof(map_t), 0, values); > + m->m.type = type; > + m->size = 0; > + m->aux = NULL; > + m->len = sz_init == 0 ? SZ_INIT_HASH : sz_init; > + m->bucket = calloc(m->len, sizeof(void*)); > + return m; > +} > + > +void > +log_hash_free(map_t* m) > +{ > + int i; > + int32_t k_type = KEY_TYPE(m->m.type); > + int32_t v_type = VALUE_TYPE(m->m.type); > + int32_t c_type = COLL_TYPE(m->m.type); > + > + for (i = 0;i < m->len;i++) { > + map_node_t* head = m->bucket[i]; > + while (head != NULL) { > + if (m == m->m.glb_values) free((void*)head->key); > + else { > + void* key = c_type != ENT_INDEX ? head->key : NULL; > + void* value = c_type == ENT_MAP ? head->value : NULL; > + check_value_ref(key, k_type, m->m.glb_values, false); > + check_value_ref(value, v_type, m->m.glb_values, false); > + } > + > + map_node_t* next = head->next; > + free(head); > + head = next; > + } > + } > + free(m->bucket); > + coll_free_ptr(m); > +} > + > +void > +log_hash_rehash(map_t* m) > +{ > + int nlen; > + if (m->size > m->len * 2 / 3) nlen = m->len * 2; > + else if (m->size < m->len / 5 && m->size > 50) nlen = m->len / 2; > + else return; /* 50 is arbitrary number */ > + > + int i; > + map_node_t** nb = calloc(nlen, sizeof(void*)); > + > + for (i = 0;i < m->len;i++) { > + map_node_t* head = m->bucket[i]; > + while (head != NULL) { > + map_node_t* next = head->next; > + > + int b; > + if (COLL_TYPE(m->m.type) == ENT_INDEX) > + b = ptr2i(head->value); > + else b = log_hash_code(m, head->key); > + > + /* the list is reversed in some sense */ > + map_node_t* nhead = nb[b % nlen]; > + if (nhead == NULL) head->next = NULL; > + else head->next = nhead; > + > + nb[b % nlen] = head; > + head = next; > + } > + } > + > + free(m->bucket); > + m->bucket = nb; > + m->len = nlen; > +} > + > +bool > +log_hash_next(map_t* m, int32_t* b, map_node_t** cur) > +{ > + /* return false if there is no next */ > + int32_t i = *b; > + for (;;) { > + if (*cur == NULL) { > + while (i < m->len && m->bucket[i] == NULL) > + i++; > + if (i >= m->len) > + return false; > + > + *b = i; > + *cur = m->bucket[i]; > + return true; > + } > + *cur = (*cur)->next; > + if (*cur != NULL) > + return true; > + i++; > + } > + return false; /* never reach here */ > +} > + > +void > +log_hash_add(map_t* m, void* k, void* v) > +{ > + /* check key existence before calling this */ > + > + int32_t type = COLL_TYPE(m->m.type); > + int32_t code = log_hash_code(m, k); > + int32_t slot = code % m->len; > + > + map_node_t* head = m->bucket[slot]; > + map_node_t* item; > + > + if (type == ENT_MAP) { > + item = malloc(sizeof (map_node_t)); > + item->value = v; > + } > + else if (type == ENT_SET) { > + item = malloc(sizeof (set_node_t)); > + } > + else if (type == ENT_INDEX) { > + item = malloc(sizeof (index_node_t)); > + item->value = i2ptr(code); > + } > + else ovs_assert(false); > + > + item->key = k; > + item->next = head; > + m->bucket[slot] = item; > + m->size++; > + > + check_value_ref(k, KEY_TYPE(m->m.type), m->m.glb_values, true); > + check_value_ref(v, VALUE_TYPE(m->m.type), m->m.glb_values, true); > + log_hash_rehash(m); > +} > + > +void* > +log_hash_del(map_t* m, void* k) > +{ > + /* reference counter of value will NOT be changed, key is freed. */ > + int32_t slot = log_hash_code(m, k) % m->len; > + > + map_node_t* head = m->bucket[slot]; > + map_node_t* pre = NULL; > + > + while (head != NULL) { > + if (log_key_equal(m, head->key, k)) { > + if (pre == NULL) m->bucket[slot] = head->next; > + else pre->next = head->next; > + > + const void* value = COLL_TYPE(m->m.type) == ENT_MAP ? > + head->value : NULL; > + free(head); > + > + check_value_ref(k, KEY_TYPE(m->m.type), m->m.glb_values, false); > + /* value ref not changed, so that caller may still use > + * value, even only for free. > + */ > + m->size--; > + log_hash_rehash(m); > + return (void*)value; > + } > + pre = head; > + head = head->next; > + } > + return NULL; > +} > + > +void* > +log_hash_get_one(map_t* m) > +{ > + if (m->size == 0) return NULL; > + MAP_ALL(m, node) > + return node->key; > + MAP_END > + return NULL; /* not reachable */ > +} > + > +int32_t > +log_hash_print(char* buf, int32_t pos, hash_t* m, bool verbose) > +{ > + int i, j = 0; > + int32_t ktype = KEY_TYPE(m->m.type); > + int32_t vtype = VALUE_TYPE(m->m.type); > + int32_t htype = COLL_TYPE(m->m.type); > + > + if (verbose) > + pos += sprintf(buf + pos, " hash %s, size=%d, len=%d\n", > + htype == ENT_SET ? "set" : "map", m->size, m-> len); > + else buf[pos++] = '{'; > + > + for (i = 0;i < m->len;i++) { > + map_node_t* head = m->bucket[i]; > + if (head == NULL) continue; > + if (verbose) pos += sprintf(buf + pos, " [%d] ", i); > + > + while (head != NULL) { > + if (j > 0 && !verbose) buf[pos++] = ','; > + if (verbose) > + pos += sprintf(buf + pos, " (%x)", > + log_hash_code(m, head->key)); > + > + pos = log_coll_print(buf, pos, head->key, ktype, verbose); > + if (htype == ENT_MAP) { > + pos += sprintf(buf + pos, "->"); > + pos = log_coll_print(buf, pos, head->value, vtype, verbose); > + } > + > + head = head->next; > + if (++j > 200) { > + pos += sprintf(buf + pos, " ..."); > + if (verbose) pos += sprintf(buf + pos, "\n"); > + return pos; > + } > + } > + if (verbose) pos += sprintf(buf + pos, "\n"); > + } > + > + if (!verbose) buf[pos++] = '}'; > + return pos; > +} > + > +/* ========================================================================== > + * LOG VALUES, INDEXES, AND TABLES > + * ========================================================================== > + */ > + > +/* > + * for tuple used as key passed to hash_* function, tuple->count: > + * == 0 indicates compact form, i.e., tuple only contains key fields, and > + * hash_code has not been provisioned. > + * != 0 indicates tuple from table and hash_code is not the hash code > + * for the key fields (is the hash code for tuple) > + */ > + > +log_int_tuple_t* > +log_int_tuple_init(TYPE(array_t*, int32_t) a) > +{ > + log_int_tuple_t* it = calloc(1, > + sizeof(log_int_tuple_t) + sizeof(int32_t) * array_size(a)); > + it->n_items = array_size(a); > + > + int32_t code; > + hash_code_array_init(&code); > + int i = 0; > + > + ARRAY_ALL_INT(a, v) > + it->values[i++] = v; > + hash_code_array_add(&code, v); > + ARRAY_END > + > + hash_code_array_final(&code); > + it->hash_code = code; > + return it; > +} > + > +static log_int_tuple_t* > +log_int_tuple_clone(log_int_tuple_t* i) > +{ > + int32_t sz = sizeof(log_int_tuple_t) + sizeof(int32_t) * i->n_items; > + log_int_tuple_t* it = calloc(1, sz); > + memcpy(it, i, sz); > + return it; > +} > + > +void > +log_int_tuple_free(log_int_tuple_t* t) > +{ > + free(t); > +} > + > +static int32_t > +log_int_tuple_print(char* buf, int32_t pos, log_int_tuple_t* t) > +{ > + int i; > + buf[pos++] = '['; > + for (i = 0;i < t->n_items;i++) { > + if (i > 0) buf[pos++] = ','; > + pos += sprintf(buf + pos, "%d", t->values[i]); > + } > + buf[pos++] = ']'; > + return pos; > +} > + > +log_value_t* > +log_value_init(const char* v, int32_t size, set_t* gv) > +{ > + /* two types of values: null terminating string (size == 0) > + * and byte array (size indicates the size of byte array) > + * data will be copied and saved in global value set. > + * the ref_no is increased after calling this. > + */ > + > + log_value_t inp; > + inp.value.p = (char*)v; > + inp.size = size; > + inp.hash_code = log_hash_code_byte(v, &inp.size); > + inp.size = -inp.size; /* using value.p */ > + > + int32_t sv_type = gv->m.type; > + gv->m.type = 0; > + > + log_value_t* setv = set_get(gv, &inp); > + gv->m.type = sv_type; > + > + if (setv != NULL) { > + setv->ref_no++; > + return setv; > + } > + > + inp.size = -inp.size; > + int32_t offset = inp.value.a - ((char*)&inp); > + setv = calloc(1, offset + inp.size + 1); > + memcpy(setv, &inp, offset); > + > + memcpy(setv->value.a, v, inp.size); > + /* make it null terminate for printing when debugging */ > + *((char*)setv->value.a + inp.size) = 0; > + > + setv->ref_no = 0; > + log_hash_add(gv, setv, NULL); > + return setv; > +} > + > +static int32_t > +log_value_print(char* buf, int32_t pos, log_value_t* value, bool verbose) > +{ > + if (value == NULL) return pos + sprintf(buf + pos, "<null>"); > + else if (verbose) > + return pos + sprintf(buf + pos, "%s<r%d,s%d>", > + value->value.a, value->ref_no, value->size); > + else return pos + sprintf(buf + pos, "%s", value->value.a); > +} > + > +static log_rule_t* > +log_rule_init(log_rule_t* rule, set_t* gv) > +{ > + coll_alloc(&rule, sizeof(log_rule_t), ENT_RULE, gv); > + rule->is_union = false; > + log_array_init(&rule->rule, ENT_INT32, 0, gv); > + log_array_init(&rule->param, ENT_ARRAY, 0, gv); > + map_init(&rule->const_param, MAP_INT32_VALUE, 0, gv); > + map_init(&rule->param_name_map, MAP_INT32_VALUE, 0, gv); > + return rule; > +} > + > +void > +log_rule_free(log_rule_t* rule) > +{ > + log_array_free(&rule->rule); > + log_array_free(&rule->param); > + map_free(&rule->const_param); > + map_free(&rule->param_name_map); > + coll_free_ptr(rule); > +} > + > +void > +log_rule_set_init(log_rule_set_t* rs, set_t* gv) > +{ > + coll_alloc(&rs, sizeof(log_rule_set_t), ENT_RULE_SET, gv); > + map_init(&rs->rule_name_map, MAP_INT32_VALUE, 0, gv); > + map_init(&rs->rule_index_map, > + KEY_ID(ENT_VALUE) | VALUE_ID(ENT_INT32), 0, gv); > + map_init(&rs->rules, KEY_ID(ENT_INT32) | VALUE_ID(ENT_RULE), 0, gv); > + map_init(&rs->table_rule_map, > + KEY_ID(ENT_INT32) | VALUE_ID(ENT_ARRAY), 0, gv); > + set_init(&rs->input_tables, KEY_ID(ENT_INT32), 0, gv); > + set_init(&rs->output_tables, KEY_ID(ENT_INT32), 0, gv); > + map_init(&rs->param_size, KEY_ID(ENT_INT32) | > VALUE_ID(ENT_INT32), 0, gv); > +} > + > +void > +log_rule_set_free(log_rule_set_t* rs) > +{ > + map_free(&rs->rule_name_map); > + map_free(&rs->rule_index_map); > + map_free(&rs->rules); > + map_free(&rs->table_rule_map); > + set_free(&rs->input_tables); > + set_free(&rs->output_tables); > + map_free(&rs->param_size); > + coll_free_ptr(rs); > +} > + > +log_engine_t* > +log_engine_init(log_engine_t* log, set_t* gv) > +{ > + coll_alloc(&log, sizeof(log_engine_t), ENT_LOG_ENG, gv); > + map_init(&log->tables, KEY_ID(ENT_INT32) | VALUE_ID(ENT_TABLE), 0, gv); > + log_rule_set_init(&log->rule_set, gv); > + log->ext_func = NULL; > + return log; > +} > + > +void > +log_engine_free(log_engine_t* log) > +{ > + map_free(&log->tables); > + log_rule_set_free(&log->rule_set); > + coll_free_ptr(log); > +} > + > +/* -------------------------------------------------------------------------- > + * TUPLES > + * -------------------------------------------------------------------------- > + */ > + > +static log_config_t log_config; > + > +void log_set_sep(char sep1, char sep2) > +{ > + log_config.sep1 = sep1; > + log_config.sep2 = sep2; > +} > + > +/* indexes will never be manipulated directly. use log_table_add|remove */ > + > +void > +log_tuple_free(log_tuple_t* t, set_t* values, bool free_val) > +{ > + /* free indicates if value is freed. */ > + > + if (free_val) { > + int i; > + for (i = 0;i < t->n_values;i++) > + if (t->values[i] != NULL) > + log_value_free(t->values[i], values); > + } > + > + if (t->indexes != NULL) free(t->indexes); > + free(t); > +} > + > +static void > +log_tuple_set_hash_code(log_tuple_t* t, int32_t n_values) > +{ > + int32_t i, code; > + hash_code_array_init(&code); > + > + for (i = 0;i < n_values;i++) { > + /* NULL only for specifying query condition */ > + if (t->values[i] != NULL) > + hash_code_array_add(&code, t->values[i]->hash_code); > + } > + > + hash_code_array_final(&code); > + t->hash_code = code; > +} > + > +log_tuple_t* > +log_tuple_init(int32_t n_values) > +{ > + log_tuple_t* tuple = > + calloc(1, sizeof(log_tuple_t) + sizeof(void*) * n_values); > + tuple->n_values = n_values; > + return tuple; > +} > + > +log_tuple_t* > +log_tuple_init_val(log_value_t** val, int32_t n_values) > +{ > + /* ref will not be updated */ > + log_tuple_t* tuple = log_tuple_init(n_values); > + if (val != NULL) { > + memcpy(tuple->values, val, sizeof(void*) * n_values); > + log_tuple_set_hash_code(tuple, n_values); > + } > + return tuple; > +} > + > +static log_tuple_t* > +log_tuple_clone(log_tuple_t* tuple) > +{ > + int i; > + log_tuple_t* nt = log_tuple_init_val(tuple->values, tuple-> n_values); > + for (i = 0;i < tuple->n_values;i++) log_value_ref(tuple->values[i]); > + nt->count = tuple->count; > + return nt; > +} > + > +static log_tuple_t* > +log_tuple_init_str0(const char* t, char sep, bool use_null, set_t* gv) > +{ > + /* the value's reference will be added */ > + /* the input is in the form of n:f0: ... : fn */ > + int64_t count = atoll(t); > + if (t[0] < '0' || t[0] > '9') return NULL; > + > + TYPE(array_t, char*) pos; > + log_array_init(&pos, 0, 0, gv); > + > + char* p = (char*)t; > + for (;*p != 0;p++) > + if (*p == sep) array_add(&pos, p); > + > + int32_t size = array_size(&pos); > + array_add(&pos, p); > + > + if (size == 0) { > + log_array_free(&pos); > + return NULL; > + } > + > + log_tuple_t* tuple = log_tuple_init(size); > + tuple->count = count; > + > + int i; > + for (i = 0;i < size;i++) { > + if (use_null) { > + char c = ((char*)array_get(&pos, i))[1]; > + if (c == 0 || c == sep) { > + tuple->values[i] = NULL; > + continue; > + } > + } > + > + log_value_t* val = log_value_init((char*)array_get(&pos, i) + 1, > + (char*)array_get(&pos, i + 1) - > + ((char*)array_get(&pos, i) + 1), gv); > + tuple->values[i] = val; > + } > + > + log_array_free(&pos); > + log_tuple_set_hash_code(tuple, size); > + return tuple; > +} > + > +log_tuple_t* > +log_tuple_init_str_null(const char* t, char sep, set_t* gv) > +{ > + return log_tuple_init_str0(t, sep, true, gv); > +} > + > +log_tuple_t* > +log_tuple_init_str(const char* t, char sep, set_t* gv) > +{ > + return log_tuple_init_str0(t, sep, false, gv); > +} > + > +int32_t > +log_tuple_print(char* buf, int32_t pos, log_tuple_t* t) > +{ > + pos += sprintf(buf + pos, "%" PRId64, t->count); > + buf[pos++] = log_config.sep1; > + > + int32_t i; > + for (i = 0;i < t->n_values;i++) { > + if (i > 0) buf[pos++] = log_config.sep1; > + pos = log_value_print(buf, pos, t->values[i], false); > + } > + return pos; > +} > + > +/* -------------------------------------------------------------------------- > + * TABLES > + * -------------------------------------------------------------------------- > + */ > + > +log_table_t* > +log_table_init(log_table_t* tbl, int32_t n, int32_t f, > + int32_t size, set_t* gv) > +{ > + coll_alloc(&tbl, sizeof(log_table_t), ENT_TABLE, gv); > + > + tbl->table_index = n; > + tbl->num_fields = f; > + tbl->is_remove = false; > + > + map_init(&tbl->index_def, > + KEY_ID(ENT_INT_TUPLE) | VALUE_ID(ENT_INT32), 0, gv); > + log_array_init(&tbl->index_map, KEY_ID(ENT_INDEX), 0, gv); > + set_init(&tbl->tuples, KEY_ID(ENT_TUPLE), size, gv); > + > + tbl->tuples.aux = tbl; > + return tbl; > +} > + > +void > +log_table_free(log_table_t* tbl) > +{ > + map_free(&tbl->index_def); > + log_array_free(&tbl->index_map); > + set_free(&tbl->tuples); > + coll_free_ptr(tbl); > +} > + > +static void > +log_index_add_node0(log_tuple_t* t, int32_t i_idx) > +{ > + /* add first tuple for the key of index */ > + log_index_i_pre(t, i_idx) = log_index_i_suc(t, i_idx) = t; > +} > + > +static void > +log_index_add_node1(log_tuple_t* t, log_tuple_t* head, int32_t i_idx) > +{ > + log_index_i_suc(t, i_idx) = head; > + log_index_i_pre(t, i_idx) = log_index_i_pre(head, i_idx); > + log_index_i_suc(log_index_i_pre(head, i_idx), i_idx) = t; > + log_index_i_pre(head, i_idx) = t; > +} > + > +static bool > +log_index_del_node(log_tuple_t* t, int32_t i_idx) > +{ > + /* returns true if this is the last node in link */ > + if (log_index_i_pre(t, i_idx) == t) return true; > + > + log_index_i_suc(log_index_i_pre(t, i_idx), i_idx) = > + log_index_i_suc(t, i_idx); > + > + log_index_i_pre(log_index_i_suc(t, i_idx), i_idx) = > + log_index_i_pre(t, i_idx); > + return false; > +} > + > +static void > +log_index_add_tuple(hash_t* index, int32_t i, log_tuple_t* t) > +{ > + map_node_t* node = log_hash_get(index, t); > + if (node == NULL) { > + log_index_add_node0(t, i); > + log_hash_add(index, t, NULL); > + } > + else { > + log_index_add_node1(t, (log_tuple_t*)node->key, i); > + node->key = t; > + } > +} > + > +log_tuple_t* > +log_index_get_index(log_table_t* tbl, log_tuple_t* t, int32_t i_idx) > +{ > + /* t is key tuple */ > + hash_t* index = array_get(&tbl->index_map, i_idx); > + map_node_t* node = log_hash_get(index, t); > + > + if (node == NULL) return NULL; > + return (log_tuple_t*)log_hash_get(index, t)->key; > +} > + > +static void > +log_table_add0(log_table_t* tbl, log_tuple_t* t) > +{ > + /* assume tuple is not present in table, need not free t afterwards */ > + /* check where tuple count is set to 1 */ > + log_hash_add(&tbl->tuples, t, NULL); > + int n_idx = array_size(&tbl->index_def); > + if (n_idx > 0) t->indexes = calloc(n_idx * 2, sizeof(void*)); > + > + /* update index */ > + int i; > + for (i = 0;i < n_idx;i++) { > + hash_t* index = array_get(&tbl->index_map, i); > + log_index_add_tuple(index, i, t); > + } > +} > + > +static void > +log_table_remove0(log_table_t* tbl, log_tuple_t* t) > +{ > + /* assume tuple is present in table, t has been freed afterwards */ > + int i; > + /* update index */ > + int n_idx = array_size(&tbl->index_def); > + > + for (i = 0;i < n_idx;i++) { > + bool last = log_index_del_node(t, i); > + hash_t* index = array_get(&tbl->index_map, i); > + map_node_t* node = log_hash_get(index, t); > + > + if (last) log_hash_del(index, t); > + else if (node->key == t) node->key = log_index_i_suc(t, i); > + } > + > + log_hash_del(&tbl->tuples, t); > +} > + > +static void > +log_table_add_extra(log_table_t* tbl, log_tuple_t* t) > +{ > + /* add or merge count, will be referred (add) or freed (merge count) */ > + log_tuple_t* et = set_get(&tbl->tuples, t); > + if (et == NULL) log_table_add0(tbl, t); > + else { > + et->count += t->count; > + log_tuple_free(t, tbl->m.glb_values, true); > + } > +} > + > +void > +log_table_add(log_table_t* tbl, log_tuple_t* t) > +{ > + /* validation, t not in table */ > + ovs_assert(!set_has(&tbl->tuples, t) && tbl->num_fields == t-> n_values); > + log_table_add0(tbl, t); > +} > + > +void > +log_table_remove(log_table_t* tbl, log_tuple_t* t) > +{ > + /* validation, must have this and t is from table */ > + ovs_assert(set_get(&tbl->tuples, t) == t); > + log_table_remove0(tbl, t); > +} > + > +int32_t > +log_table_add_index(log_table_t* tbl, log_int_tuple_t* index_key) > +{ > + /* add new index for table or returning existing one */ > + /* index_key will be cloned */ > + > + map_node_t* index = log_hash_get(&tbl->index_def, index_key); > + if (index != NULL) return ptr2i(index->value); > + > + log_int_tuple_t* ikey = log_int_tuple_clone(index_key); > + int32_t index_id = map_size(&tbl->index_def); > + map_add(&tbl->index_def, ikey, i2ptr(index_id)); > + > + map_t* new_i = log_hash_init(NULL, COLL_ID(ENT_INDEX), > + SZ_INIT_HASH, tbl->m.glb_values); > + array_add(&tbl->index_map, new_i); > + new_i->aux = ikey; > + > + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) > + tuple->indexes = realloc( > + tuple->indexes, (index_id + 1) * 2 * sizeof(void*)); > + log_index_add_tuple(new_i, index_id, tuple); > + SET_END > + > + return index_id; > +} > + > +int32_t > +log_index_print(char* buf, int32_t pos, log_table_t* t) > +{ > + int i = 0, j = 0; > + ARRAY_ALL(&t->index_map, index, hash_t*) > + log_int_tuple_t* def = index->aux; > + pos += sprintf(buf + pos, " index="); > + pos = log_int_tuple_print(buf, pos, def); > + buf[pos++] = '\n'; > + > + SET_ALL(index, key, log_tuple_t*) > + pos += sprintf(buf + pos, " key set: "); > + INDEX_ALL(key, i, t1) > + pos = log_tuple_print(buf, pos, t1); > + buf[pos++] = ' '; > + > + if (++j > 200) { > + pos += sprintf(buf + pos, " ...\n"); > + return pos; > + } > + INDEX_END > + buf[pos++] = '\n'; > + SET_END > + i++; > + ARRAY_END > + return pos; > +} > + > +int32_t > +log_table_print(char* buf, int32_t pos, log_table_t* t, bool verbose) > +{ > + if (verbose) { > + pos += sprintf(buf + pos, " tbl id=%d fd=%d sz=%d\n", > + t->table_index, t->num_fields, table_size(t)); > + } > + pos = log_hash_print(buf, pos, &t->tuples, verbose); > + if (verbose) pos = log_index_print(buf, pos, t); > + return pos; > +} > + > +int32_t > +log_rule_set_print(char* buf, int32_t pos, log_rule_set_t* rs) > +{ > + pos += sprintf(buf + pos, "rule_name_map\n"); > + pos = log_hash_print(buf, pos, &rs->rule_name_map, false); > + > + pos += sprintf(buf + pos, "\nrule_index_map\n"); > + pos = log_hash_print(buf, pos, &rs->rule_index_map, false); > + > + pos += sprintf(buf + pos, "\ntable_rule_map\n"); > + pos = log_hash_print(buf, pos, &rs->table_rule_map, false); > + > + pos += sprintf(buf + pos, "\nparam_size\n"); > + pos = log_hash_print(buf, pos, &rs->param_size, false); > + > + pos += sprintf(buf + pos, "\ninput_tables\n"); > + pos = log_hash_print(buf, pos, &rs->input_tables, false); > + > + pos += sprintf(buf + pos, "\noutput_tables\n"); > + pos = log_hash_print(buf, pos, &rs->output_tables, false); > + > + pos += sprintf(buf + pos, "\nrules\n"); > + pos = log_hash_print(buf, pos, &rs->rules, false); > + return pos; > +} > + > +static int32_t > +log_rule_print(char* buf, int32_t pos, log_rule_t* r) > +{ > + pos += sprintf(buf + pos, r->is_union ? "(union," : "(join,"); > + pos += sprintf(buf + pos, "rule="); > + pos = log_array_print(buf, pos, &r->rule, false); > + pos += sprintf(buf + pos, ",param="); > + pos = log_array_print(buf, pos, &r->param, false); > + pos += sprintf(buf + pos, ",name="); > + pos = log_hash_print(buf, pos, &r->param_name_map, false); > + pos += sprintf(buf + pos, ",const="); > + pos = log_hash_print(buf, pos, &r->const_param, false); > + pos += sprintf(buf + pos, ")\n"); > + return pos; > +} > + > +int32_t > +log_coll_print(char* buf, int pos, void* item, int32_t type, bool verbose) > +{ > + /* do not check the buf limit */ > + if (item == NULL && type != ENT_INT32 && type != ENT_TST_INT32) > + return pos; > + > + else if (type == ENT_INT32 || type == ENT_TST_INT32) > + pos += sprintf(buf + pos, "%d", ptr2i(item)); > + else if (type == ENT_STR) > + pos += sprintf(buf + pos, "%s", (const char*)item); > + else if (type == ENT_VALUE) > + pos = log_value_print(buf, pos, item, verbose); > + else if (type == ENT_INT_TUPLE) > + pos = log_int_tuple_print(buf, pos, item); > + else if (type == ENT_TUPLE) > + pos = log_tuple_print(buf, pos, item); > + else if (type == ENT_ARRAY) > + pos = log_array_print(buf, pos, item, verbose); > + else if (type == ENT_SET || type == ENT_MAP) > + pos = log_hash_print(buf, pos, item, verbose); > + else if (type == ENT_TABLE) > + pos = log_table_print(buf, pos, item, verbose); > + else if (type == ENT_RULE) > + pos = log_rule_print(buf, pos, item); > + else if (type == ENT_RULE_SET) > + pos = log_rule_set_print(buf, pos, item); > + return pos; > +} > + > +/* ========================================================================== > + * SYNTAX PROCESSING > + * ========================================================================== > + */ > + > +/* > + * token of id: [a-zA-Z_][a-zA-Z0-9_]* > + * all upper case: output table; all lower case: input table; others: > + * intermediate. > + * <table_name> ( <param_name>, ... ) ':'|'>' <table_name> ( > + * <param_name> | 'value' | - ), ... ; > + * ':' is for join. '>' is for union. order is always important > + * special table: not used now, e.g., could be used to specify language > + * parameters. join is preferred with external function as there is more > + * flexibility in param. comments start with # document started with ## > + */ > + > +struct log_sync_s { > + const char* text; > + int len; > + > + int curpos; > + char curchar; > + char curtoken; > + > + char token[LOG_TOKEN_SZ]; > + int token_pos; > + set_t* gv; > +}; > + > +static struct log_sync_s sync; > + > +void > +log_sync_init(const char* log, set_t* gv) > +{ > + sync.len = strlen(log); > + sync.text = log; > + sync.curpos = 0; > + sync.token_pos = 0; > + sync.gv = gv; > +} > + > +static void > +sync_getc(void) > +{ > + sync.curchar = sync.text[sync.curpos++]; > +} > + > +static bool > +sync_eof(void) > +{ > + return sync.curpos >= sync.len; > +} > + > +static void > +sync_error(const char* s0, const log_value_t* s1) > +{ > + printf("syntax error: %s %s \n%s\n", s0, > + s1 == NULL ? "" : s1->value.a, > + sync.text + sync.curpos); > + exit(1); > +} > + > +static void > +sync_init_token(void) > +{ > + sync.token_pos = 0; > +} > + > +static log_value_t* > +sync_get_value(void) > +{ > + log_value_t* token = log_value_init(sync.token, sync.token_pos,sync.gv); > + return token; > +} > + > +static log_value_t* > +sync_get_const(const char* c) > +{ > + log_value_t* token = log_value_init(c, 0, sync.gv); > + return token; > +} > + > +static void > +sync_append_c(void) > +{ > + sync.token[sync.token_pos++] = sync.curchar; > + sync_getc(); > + if (sync.token_pos >= LOG_TOKEN_SZ) sync_error("token too long", 0); > +} > + > +static void > +sync_gett(void) > +{ > + bool in_comment = false; > + bool in_literal = false; > + bool in_ident = false; > + > + while (!sync_eof()) { > + if (in_comment) { > + if (sync.curchar == '\n') { > + in_comment = false; > + } > + sync_append_c(); > + } > + else if (in_literal) { > + if (sync.curchar == '\'') { > + sync_getc(); > + sync.curtoken = 's'; > + return; > + } > + else { > + sync_append_c(); > + } > + } > + else if (in_ident) { > + if (sync.curchar != '_' && !isalpha(sync.curchar) > + && !isdigit(sync.curchar)) { > + sync.curtoken = 't'; > + return; > + } > + sync_append_c(); > + } > + else { > + if (sync.curchar == '#') { > + in_comment = true; > + sync_getc(); > + } > + else if (sync.curchar == '\'') { > + in_literal = true; > + sync_init_token(); > + sync_getc(); > + } > + else if (sync.curchar == '_' || isalpha(sync.curchar)) { > + sync_init_token(); > + sync_append_c(); > + in_ident = true; > + } > + else if (strchr(":>().,-;", sync.curchar) != NULL) { > + sync.curtoken = sync.curchar; > + sync_getc(); > + return; > + } > + else if (isspace(sync.curchar)) sync_getc(); > + else if (sync.curchar == '\n') sync_getc(); > + else sync_error("unknown char near", 0); > + } > + } > + > + /* for last period without following chars */ > + sync.curtoken = sync.curchar; > +} > + > +static void > +sync_nt_params(array_t* list) > +{ > + /* ( ( param | 'literal' | - )* , ) */ > + if (sync.curtoken != '(') sync_error("expecting (", 0); > + sync_gett(); > + > + for (;;) { > + if (sync.curtoken == 't') { > + array_add(list, sync_get_const("t")); > + array_add(list, sync_get_value()); > + } > + else if (sync.curtoken == '-') { > + array_add(list, sync_get_const("-")); > + array_add(list, NULL); > + } > + else if (sync.curtoken == 's') { > + array_add(list, sync_get_const("s")); > + array_add(list, sync_get_value()); > + } > + else sync_error("expecting param, literal, or -", NULL); > + > + sync_gett(); > + if (sync.curtoken == ',') { > + sync_gett(); > + continue; > + } > + else if (sync.curtoken == ')') { > + sync_gett(); > + return; > + } > + else sync_error("expecting , or )", NULL); > + } > +} > + > +static void > +sync_nt_table(array_t* list) > +{ > + if (sync.curtoken != 't') sync_error("table name expected", NULL); > + log_value_t* table_name = sync_get_value(); > + > + sync_gett(); > + array_add(list, NULL); /* will be overrided later */ > + array_add(list, table_name); > + sync_nt_params(list); > +} > + > +void > +log_sync_parse(map_t* sem) > +{ > + /* > + * tableName -> (left side, right side table 0, right side table 1, ...) > + * each table contains (table name, param0, param1, ...) > + */ > + > + map_init(sem, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_ARRAY), 0, sync.gv); > + sync.gv = sem->m.glb_values; > + > + sync_getc(); > + sync_gett(); > + > + for (;;) { > + array_t* tables = log_array_init(NULL, KEY_ID(ENT_ARRAY), > 0, sync.gv); > + array_t* tbl = log_array_init(NULL, KEY_ID(ENT_VALUE), 0, sync.gv); > + > + sync_nt_table(tbl); > + array_add(tables, tbl); > + log_value_t* table_name = array_get(tbl, 1); > + > + if (sync.curtoken != ':' && sync.curtoken != '>') { > + sync_error("expecting : or >", NULL); > + } > + > + array_set(tbl, 0, sync_get_const(sync.curtoken == '>' ? "u" : "j")); > + sync_gett(); > + > + for (;;) { > + tbl = log_array_init(NULL, KEY_ID(ENT_VALUE), 0, sync.gv); > + sync_nt_table(tbl); > + array_add(tables, tbl); > + if (sync.curtoken == ';' || sync.curtoken == '.') break; > + } > + > + if (map_get(sem, table_name) != NULL) { > + sync_error("definition existed: ", table_name); > + } > + > + map_add(sem, table_name, tables); > + if (sync.curtoken == ';') sync_gett(); > + else if (sync.curtoken == '.') return; > + } > +} > + > +/* ========================================================================== > + * SYNTAX PROCESSING > + * ========================================================================== > + */ > + > +void > +log_sort_array(int start, array_t* list, array_t* sem1, array_t* sem2) > +{ > + /* this must be stable sort. see the sort for table size */ > + > + int i, j; > + for (i = start;i < list->size;i++) { > + int index = i; > + > + for (j = i + 1;j < list->size;j++) > + if (ptr2i(list->item[j]) < ptr2i(list->item[index])) > + index = j; > + > + void* newi = (void*)list->item[index]; > + void* newv1 = sem1 == NULL ? NULL : (void*)sem1->item[index]; > + void* newv2 = sem2 == NULL ? NULL : (void*)sem2->item[index]; > + > + memmove(&list->item[i + 1], &list->item[i], > + (index - i) * sizeof(void*)); > + > + if (sem1 != NULL) > + memmove(&sem1->item[i + 1], &sem1->item[i], > + (index - i) * sizeof(void*)); > + if (sem2 != NULL) > + memmove(&sem2->item[i + 1], &sem2->item[i], > + (index - i) * sizeof(void*)); > + > + list->item[i] = newi; > + if (sem1 != NULL) sem1->item[i] = newv1; > + if (sem2 != NULL) sem2->item[i] = newv2; > + } > +} > + > +int > +log_insert_item(int val, array_t* list, void* obj1, > + void* obj2, array_t* sem1, array_t* sem2) > +{ > + /* > + * returns -1 if there is no equal value in the list, or the position to > + * insert. If there is tie, the position is after all items of equal > + * value. will insert only if there is no equal value. > + */ > + > + int count; > + for (count = 0;count < list->size;count++) { > + if (val < ptr2i(list->item[count])) break; > + } > + > + array_ins(list, count, i2ptr(val)); > + if (sem1 != NULL) array_ins(sem1, count, obj1); > + if (sem2 != NULL) array_ins(sem2, count, obj2); > + return count; > +} > + > +void > +log_topo_sort(TYPE2(map_t*, log_value_t*, set_t*) g, /* set of value */ > + TYPE(array_t* order, log_value_t*), > + TYPE(set_t*, log_value_t*) in_nodes, > + TYPE(set_t*, log_value_t*) out_nodes) > +{ > + /* input g will be destroyed after sort */ > + > + set_t* gv = g->m.glb_values; > + log_array_init(order, KEY_ID(ENT_VALUE), 0, gv); > + set_init(in_nodes, KEY_ID(ENT_VALUE), 0, gv); > + set_init(out_nodes, KEY_ID(ENT_VALUE), 0, gv); > + > + set_t right_nodes, all_nodes, to_remove; > + set_init(&right_nodes, KEY_ID(ENT_VALUE), 0, gv); > + set_init(&all_nodes, KEY_ID(ENT_VALUE), 0, gv); > + > + MAP_ALL(g, node) > + set_add(&all_nodes, node->key); > + > + SET_ALL(node->value, node1, log_value_t*) > + set_add(&all_nodes, node1); > + set_add(&right_nodes, node1); > + SET_END /* end of iteration */ > + MAP_END /* end of iteration */ > + > + SET_ALL(&all_nodes, key, log_value_t*) > + if (!map_has(g, key)) set_add(in_nodes, key); > + else if (!set_has(&right_nodes, key)) > + set_add(out_nodes, key); > + SET_END /* end of iteration */ > + > + while (set_size(&all_nodes) > 0) { > + log_value_t* next = NULL; > + > + SET_ALL(&all_nodes, key, log_value_t*) > + if (!map_has(g, key)) { > + next = key; > + SET_EXIT; > + } > + SET_END > + > + if (next == NULL) > + sync_error("circular graph, check around ", > + log_hash_get_one(&all_nodes)); > + > + array_add(order, next); > + set_del(&all_nodes, next); > + set_init(&to_remove, KEY_ID(ENT_VALUE), 0, gv); > + > + MAP_ALL(g, node) > + set_del(node->value, next); > + if (set_size((set_t*)node->value) == 0) > + set_add(&to_remove, node->key); > + MAP_END > + > + SET_ALL(&to_remove, key, log_value_t*) > + set_free(map_del(g, key)); > + SET_END > + set_free(&to_remove); > + } > + > + set_free(&right_nodes); > + set_free(&all_nodes); > + map_free(g); > +} > + > +static int > +check_name(log_value_t* t) > +{ > + /* check if string are in all lower case (> 0), all upper case (<0) > + * or mixed (== 0) > + */ > + const char* s = t->value.a; > + bool lower = false; > + bool upper = false; > + char c; > + > + while ((c = *s++) != 0) { > + if (islower(c)) lower = true; > + else if (isupper(c)) upper = true; > + } > + > + if (lower && upper) return 0; > + else if (lower) return 1; > + else if (upper) return -1; > + else return 0; > +} > + > +/* ========================================================================== > + * SEMANTICS DEFINITION > + * ========================================================================== > + */ > + > +void > +log_sem_process(log_rule_set_t* rule_set, map_t* sem) > +{ > + /* string -> array of array of string */ > + set_t* gv = sem->m.glb_values; > + /* STEP 1: check table dependency and assign digit table / rule index */ > + map_t rules; /* map string to (set of value) */ > + map_init(&rules, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_SET), 0, gv); > + > + log_value_t* rname; > + MAP_ALL(sem, node) > + rname = NULL; > + set_t* dep = set_init(NULL, KEY_ID(ENT_VALUE), 0, gv); > + > + ARRAY_ALL((array_t*)node->value, t, array_t*) > + log_value_t* nm = array_get(t, 1); > + if (rname == NULL) rname = nm; > + else set_add(dep, nm); > + ARRAY_END > + map_add(&rules, rname, dep); > + MAP_END > + > + array_t topo_order; > + set_t topo_in, topo_out; > + log_topo_sort(&rules, &topo_order, &topo_in, &topo_out); > + > + int32_t rule_index = 0; > + ARRAY_ALL(&topo_order, name, log_value_t*) > + map_add(&rule_set->rule_name_map, i2ptr(rule_index), name); > + map_add(&rule_set->rule_index_map, name, i2ptr(rule_index++)); > + ARRAY_END > + > + /* STEP 2: check upper case, lower case and mixed */ > + SET_ALL(&topo_in, name, log_value_t*) > + if (check_name(name) <= 0) > + sync_error("input must be all lower case: ", name); > + > + set_add(&rule_set->input_tables, > + map_get(&rule_set->rule_index_map, name)); > + SET_END > + > + SET_ALL(&topo_out, name, log_value_t*) > + if (check_name(name) >= 0) > + sync_error("output must be all upper case: ", name); > + > + set_add(&rule_set->output_tables, > + map_get(&rule_set->rule_index_map, name)); > + SET_END > + > + set_t intr; > + set_init(&intr, KEY_ID(ENT_VALUE), 0, gv); > + > + MAP_ALL(sem, name) > + if (map_has(&topo_out, i2ptr(name->key))) continue; > + if (check_name(i2ptr(name->key)) != 0) > + sync_error("intermediate must be mixed: ", name->key); > + MAP_END > + > + /* > + * STEP 3: check table size consistency and assign digit index. > + * and if union / join on itself, it could only perform twice on > + * each param, i.e., X : A, A, A is not supported. > + */ > + map_t table_size; > + map_init(&table_size, KEY_ID(ENT_VALUE), 0, gv); /* map string to int */ > + > + MAP_ALL(sem, name) > + array_t* val = name->value; /* array of array of value */ > + log_rule_t* rule = log_rule_init(NULL, gv); > + > + /* rule_id is integer */ > + void* rule_id = map_get(&rule_set->rule_index_map, name->key); > + map_add(&rule_set->rules, rule_id, rule); > + > + log_value_t* rule_t = array_get(array_get(val, 0), 0); > + rule->is_union = strcmp("u", rule_t->value.a) == 0; > + > + int const_value = -2; /* -1 is for ignore */ > + > + /* reorder table sequence */ > + ARRAY_ALL(val, t, array_t*) > + log_value_t* table_name = array_get(t, 1); > + void* rule_id = map_get(&rule_set->rule_index_map, table_name); > + array_add(&rule->rule, rule_id); > + ARRAY_END > + > + int i; > + log_sort_array(1, &rule->rule, val, NULL); > + > + for (i = 2;i < rule->rule.size - 1;i++) { > + if (array_get(&rule->rule, i - 1) == array_get(&rule->rule, i) && > + array_get(&rule->rule, i + 1) == array_get(&rule->rule, i)) > + sync_error( > + "cannot join / union on itself for more > than twice: ", > + name->key); > + } > + > + int param_size = 0; > + bool is_left = true; > + bool left_has_param = false; > + bool left_has_value = false; > + > + map_t param_map; /* map string to integer */ > + map_init(&param_map, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_INT32), 0, gv); > + > + ARRAY_ALL(val, t, array_t*) > + array_t* param_list = log_array_init(NULL, KEY_ID (ENT_INT32), > + 0, gv); > + array_add(&rule->param, param_list); > + log_value_t* table_name = array_get(t, 1); > + > + int32_t size = ((array_t*)t)->size / 2 - 1; > + int32_t table_id = map_get_int( > + &rule_set->rule_index_map, table_name); > + map_add(&rule_set->param_size, i2ptr(table_id), i2ptr (size)); > + > + /* check table size */ > + if (rule->is_union) { > + if (is_left) param_size = size; > + else if (size != param_size) > + sync_error("table param size mismatch in union rule: ", > + name->key); > + } > + else { > + struct map_node_s* ksize = log_hash_get( > + &table_size, table_name); > + if (ksize == NULL) > + map_add(&table_size, table_name, i2ptr(size)); > + > + else if (ptr2i(ksize->value) != size) > + sync_error("table param size mismatch in join rule: " , > + name->key); > + } /* if check table size */ > + > + /* assign index to each param */ > + for (i = 0; i < size;i++) { > + log_value_t* param_type = array_get(t, i * 2 + 2); > + log_value_t* param_value = array_get(t, i * 2 + 3); > + > + if (is_left && strcmp(param_type->value.a, "-") == 0) > + sync_error("left cannot have 'ignore': ", name-> key); > + > + if (strcmp(param_type->value.a, "-") == 0) { > + array_add(param_list, i2ptr(-1)); > + } > + > + else if (strcmp(param_type->value.a, "t") == 0) { > + if (is_left) left_has_param = true; > + map_node_t* param_no = log_hash_get( > + &param_map, param_value); > + > + void* no; /* type is int */ > + if (param_no == NULL) { > + no = i2ptr(param_map.size); > + map_add(&param_map, param_value, no); > + map_add(&rule->param_name_map, no, param_value); > + } else no = param_no->value; > + array_add(param_list, no); > + } > + > + else if (strcmp(param_type->value.a, "s") == 0) { > + if (is_left) left_has_value = true; > + void* c_value = param_value; > + map_add(&rule->const_param, i2ptr (const_value),c_value); > + array_add(param_list, i2ptr(const_value--)); > + } > + } /* for each table param */ > + is_left = false; > + ARRAY_END /* for each table */ > + > + map_free(&param_map); > + if (!left_has_param) > + sync_error("left must have param: ", name->key); > + if (rule->is_union && left_has_value) > + sync_error("left cannot have const: ", name->key); > + MAP_END /* for each rule */ > + > + /* STEP 4: check param reference. */ > + MAP_ALL(&rule_set->rules, rule_no) > + array_t* left_param = NULL; /* array of int, for checking union */ > + > + log_rule_t* rule = map_get(&rule_set->rules, rule_no->key); > + log_value_t* rule_name = map_get( > + &rule_set->rule_name_map, rule_no->key); > + > + /* add table rule map */ > + int32_t i; > + for (i = 1;i < rule->rule.size;i++) { > + array_t* set = map_get(&rule_set->table_rule_map, > + array_get(&rule->rule, i)); > + > + if (set == NULL) { > + set = log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + map_add(&rule_set->table_rule_map, > + array_get(&rule->rule, i), set); > + } > + > + int32_t found = log_array_look_for(set, rule_no->key); > + if (found < 0) array_add(set, rule_no->key); > + } > + > + ARRAY_ALL(&rule->param, param, array_t*) > + /* param is array of int */ > + if (left_param == NULL) left_param = param; > + > + if (rule->is_union) { > + if (left_param != NULL) { > + bool not_found = false; > + > + ARRAY_ALL(left_param, item, void*) > + if (log_array_look_for(param, item) < 0) { > + not_found = true; > + ARRAY_EXIT; > + } > + ARRAY_END > + > + if (not_found) > + sync_error("union param not found in ", rule_name); > + } > + continue; > + } > + > + /* > + * for right side param, it must also appear either in left > + * side or right side or being 'ignored'; for left side param, > + * it must appear in right side > + */ > + ARRAY_ALL((array_t*)param, p0no, void*) > + int32_t p0 = ptr2i(p0no); > + if (p0 < 0) continue; > + > + bool found = false; > + ARRAY_ALL(&rule->param, param1, array_t*) > + if (param == param1) continue; > + > + if (log_array_look_for(param1, p0no) >= 0) { > + found = true; > + ARRAY_EXIT; > + } > + ARRAY_END > + > + if (!found) > + sync_error("not used / undefined param ", rule_name); > + ARRAY_END > + ARRAY_END /* for each table */ > + MAP_END /* for each rule */ > + > + /* STEP 5: sort table_rule_map */ > + MAP_ALL(&rule_set->table_rule_map, node) > + array_t* list = node->value; > + log_sort_array(0, list, NULL, NULL); > + MAP_END > + > + set_free(&intr); > + set_free(&topo_in); > + set_free(&topo_out); > + map_free(&table_size); > + log_array_free(&topo_order); > +} > + > +/* ========================================================================== > + * TABLE OPERATION > + * ========================================================================== > + */ > + > +static log_tuple_t* > +tblopr_reorder_tuple(log_tuple_t* t, TYPE(array_t*, int32_t) order, > + TYPE2(map_t*, int32_t, log_value_t*) const_map) > +{ > + /* input and output table could be different. order is sequence > + * instead of param index. > + * (a, b, c, d, e) + (2, -2, 1, 4) => (c, C[-2], b, e) > + */ > + log_tuple_t* newt = log_tuple_init(array_size(order)); > + int i; > + for (i = 0;i < array_size(order);i++) { > + log_value_t* value; > + int32_t order_i = array_get_int(order, i); > + > + if (order_i >= 0) value = t->values[order_i]; > + else if (order_i < -1) value = map_get(const_map, i2ptr (order_i)); > + else ovs_assert(false); /* reorder sees constants */ > + > + log_value_ref(value); > + newt->values[i] = value; > + } > + > + newt->count = t->count; > + log_tuple_set_hash_code(newt, array_size(order)); > + return newt; > +} > + > +static void > +tblopr_reorder_table(log_table_t* input, TYPE(array_t*, int32_t) order, > + TYPE2(map_t*, int32_t, log_value_t*) const_map, > + log_table_t* output) > +{ > + /* input and output table could be different */ > + SET_ALL(&input->tuples, t, log_tuple_t*) > + log_table_add(output, tblopr_reorder_tuple(t, order, const_map)); > + SET_END > +} > + > +static bool > +tblopr_match_const(log_tuple_t* t, TYPE(array_t*, int32_t) cpos, > + log_value_t** cval) > +{ > + int i; > + for (i = 0;i < array_size(cpos);i++) { > + int32_t ci = array_get_int(cpos, i); > + if (t->values[ci] != cval[i]) return false; > + } > + return true; > +} > + > +static log_table_t* > +tblopr_query_table(log_table_t* input, TYPE(array_t*, int32_t) param, > + log_value_t** val) > +{ > + /* input only used for get table param */ > + > + log_table_t* output = > + log_table_init(NULL, input->table_index, input->num_fields, > + 0, input->m.glb_values); > + > + if (array_size(param) == 0) { > + SET_ALL(&input->tuples, t, log_tuple_t*) > + log_tuple_t* nt = log_tuple_clone(t); > + log_table_add(output, nt); > + SET_END > + return output; > + } > + > + log_int_tuple_t* index = log_int_tuple_init(param); > + int32_t idx = log_table_add_index(input, index); > + log_int_tuple_free(index); > + > + log_tuple_t* key = log_tuple_init_val(val, array_size(param)); > + log_tuple_t* set = log_index_get_index(input, key, idx); > + > + /* check set is null? */ > + INDEX_ALL(set, idx, t) > + log_tuple_t* nt = log_tuple_clone(t); > + log_table_add(output, nt); > + INDEX_END > + > + log_tuple_free(key, input->m.glb_values, false); > + return output; > +} > + > +static void > +tblopr_merge_tuple(log_table_t* tbl, log_tuple_t* tup, > + bool negative, bool free_t) > +{ > + /* tuple untouched */ > + log_tuple_t* org_tuple = set_get(&tbl->tuples, tup); > + > + if (org_tuple == NULL) { > + log_tuple_t* nt = free_t ? tup : log_tuple_clone(tup); > + log_table_add0(tbl, nt); > + } > + else { > + if (negative) org_tuple->count -= tup->count; > + else org_tuple->count += tup->count; > + if (free_t) log_tuple_free(tup, tbl->m.glb_values, true); > + } > +} > + > +static void > +tblopr_merge_table(log_table_t* src, log_table_t* dst, bool negative) > +{ > + SET_ALL(&src->tuples, t, log_tuple_t*) > + tblopr_merge_tuple(dst, t, negative, false); > + SET_END > +} > + > +static void > +tblopr_final_delta(log_table_t* source, log_table_t* dest) > +{ > + if (source->is_remove) { > + SET_ALL(&source->tuples, t, log_tuple_t*) > + log_tuple_t* ot = set_get(&dest->tuples, t); > + log_table_remove0(dest, ot); > + SET_END > + } > + else { > + SET_ALL(&source->tuples, t, log_tuple_t*) > + ovs_assert(set_get(&dest->tuples, t) == NULL); > + log_tuple_t* nt = log_tuple_clone(t); > + log_table_add0(dest, nt); > + SET_END > + } > +} > + > +static void > +tblopr_match_reorder_and_merge(log_table_t* input, log_table_t* output, > + TYPE(array_t*, int32_t) parami, > + TYPE(array_t*, int32_t) paramo, > + TYPE2(map_t*, int32_t, log_value_t*)const_map) > +{ > + /* example: output(0, 1, 2) input(2, 1, -, 'const', 0) > + * input count is ignored. parami/o is param number, not > + * sequence number. > + */ > + > + set_t* gv = input->m.glb_values; > + TYPE(array_t*, int32_t) cpos = log_array_init( > + NULL, KEY_ID(ENT_INT32), 0, gv); > + TYPE(array_t*, int32_t) order = log_array_init( > + NULL, KEY_ID(ENT_INT32), 0, gv); > + int pos = 0, i; > + > + for (i = 0;i < array_size(parami);i++) { > + int32_t c = array_get_int(parami, i); > + if (c < -1) array_add(cpos, i2ptr(i)); > + else if (c == -1) continue; > + else { > + int32_t op = log_array_look_for(parami, > array_get(paramo, pos++)); > + if (op >= 0) array_add(order, i2ptr(op)); > + } > + } > + > + log_value_t** cval = calloc(array_size(cpos), sizeof(void*)); > + for (i = 0;i < array_size(cpos);i++) > + cval[i] = (log_value_t*) > + map_get(const_map, array_get(parami, > + array_get_int(cpos, i))); > + > + SET_ALL(&input->tuples, t, log_tuple_t*) > + if (tblopr_match_const(t, cpos, cval)) { > + log_tuple_t* nt = tblopr_reorder_tuple(t, order, NULL); > + log_table_add_extra(output, nt); > + } > + SET_END > + > + free(cval); > + log_array_free(cpos); > + log_array_free(order); > +} > + > +static void > +tblopr_combine_tuple(log_table_t* res, int32_t tuple1_count, > + log_join_param_t* joinp, int r1_fnum, > + log_tuple_t* tuple2, log_value_t** tuple_values) > +{ > + int i; > + for (i = 0;i < array_size(&joinp->rem2);i++) > + tuple_values[i + r1_fnum] = tuple2->values[ > + ptr2i(array_get(&joinp->rem2, i))]; > + > + int32_t res_num_fields = res->num_fields; > + log_tuple_t* nt = log_tuple_init_val(tuple_values, res->num_fields); > + nt->count = tuple1_count; > + > + for (i = 0;i < res_num_fields;i++) log_value_ref(nt->values[i]); > + tblopr_merge_tuple(res, nt, false, true); > +} > + > +static log_table_t* > +tblopr_cond_join(log_table_t* t1, log_table_t* t2, log_join_param_t* joinp) > +{ > + /* > + * t1 is intermediate table during join; t2 is original table. > + * select values joinp.select1 from t1 and const, and match that with > + * params indicated by joinp.index2. > + */ > + set_t* gv = t1->m.glb_values; > + > + int32_t r1_fnum = array_size(&joinp->rem1); > + int32_t r2_fnum = array_size(&joinp->rem2); > + > + log_table_t* res = log_table_init(NULL, -1, r1_fnum + r2_fnum, 0, gv); > + int32_t t2_index = map_get_int(&t2->index_def, joinp->index2); > + > + int t1val_sz = array_size(&joinp->select1); > + TYPE(array_t*, int32_t) t1param = log_array_init( > + NULL, KEY_ID(ENT_INT32), t1val_sz, gv); > + log_tuple_t* key_tuple = log_tuple_init_val(NULL, t1val_sz); > + > + int i; > + for (i = 0;i < t1val_sz;i++) { > + log_value_t* obj = array_get(&joinp->select1, i); > + if (obj != NULL) { > + array_add(t1param, i2ptr(-1)); /* mark as not set */ > + key_tuple->values[i] = obj; > + } > + else array_add(t1param, array_get(&joinp->select1i, i)); > + } > + > + /* loop over t1 and join */ > + log_value_t** tuple_values = calloc(sizeof(void*), res->num_fields); > + SET_ALL(&t1->tuples, tuple1, log_tuple_t*) > + > + for (i = 0;i < array_size(t1param);i++) { > + int32_t t1p = ptr2i(array_get(t1param, i)); > + if (t1p < 0) continue; > + key_tuple->values[i] = tuple1->values[t1p]; > + } > + > + log_tuple_set_hash_code(key_tuple, t1val_sz); > + log_tuple_t* match_tuples = > + log_index_get_index(t2, key_tuple, t2_index); > + > + if (match_tuples != NULL) { /* do not use continue */ > + for (i = 0;i < r1_fnum;i++) > + tuple_values[i] = > + tuple1->values[ptr2i(array_get(&joinp->rem1, i))]; > + > + /* join the value */ > + INDEX_ALL(match_tuples, t2_index, tuple2) > + tblopr_combine_tuple(res, tuple1->count, > + joinp, r1_fnum, tuple2, tuple_values); > + INDEX_END > + } > + SET_END > + > + log_tuple_free(key_tuple, t1->m.glb_values, false); > + log_array_free(t1param); > + free(tuple_values); > + return res; > +} > + > +static void > +tblopr_gen_delta(log_table_t* source, log_table_t* dest) > +{ > + bool is_remove = source->is_remove; > + set_t* gv = dest->m.glb_values; > + > + SET_ALL(&source->tuples, st, log_tuple_t*) > + ovs_assert(st->indexes == NULL); > + > + int64_t st_count = st->count; > + log_tuple_t* dt = set_get(&dest->tuples, st); > + > + if (is_remove) { > + ovs_assert(dt != NULL && dt->count >= st_count); > + if (dt->count > st_count) { > + dt->count -= st_count; > + SET_REMOVE_ITEM; > + log_tuple_free(st, gv, true); > + } > + } > + else { > + if (dt != NULL) { > + dt->count += st_count; > + SET_REMOVE_ITEM; > + log_tuple_free(st, gv, true); > + } > + } > + SET_END > +} > + > +static void > +tblopr_merge_delta(log_table_t* source, log_table_t* dest, > + log_table_t* dest_ivt) > +{ > + > + /* dest is of the same operation as source, while destIvt is opposite. */ > + ovs_assert(source->table_index == dest->table_index); > + set_t* gv = dest->m.glb_values; > + > + SET_ALL(&source->tuples, st, log_tuple_t*) > + ovs_assert(st->indexes == NULL); > + log_tuple_t* dt = set_get(&dest->tuples, st); > + > + if (dt != NULL) { > + SET_REMOVE_ITEM; > + dt->count += st->count; /* count will not change hash */ > + log_tuple_free(st, gv, true); > + /* will not match opposite */ > + } > + else { > + log_tuple_t* dt_ivt = set_get(&dest_ivt->tuples, st); > + /* will be added later for dt_ivt == NULL */ > + if (dt_ivt != NULL) { > + /* cross merge */ > + long st_count = st->count; > + dt_ivt->count -= st_count; > + > + if (dt_ivt->count >= 0) { > + SET_REMOVE_ITEM; > + log_tuple_free(st, gv, true); > + } > + > + if (dt_ivt->count == 0) log_table_remove0(dest_ivt, dt_ivt); > + else if (dt_ivt->count < 0) { /* move to opposite table */ > + dt_ivt->count = -dt_ivt->count; > + log_tuple_t* nt = log_tuple_clone(dt_ivt); > + log_table_add(dest, nt); > + log_table_remove0(dest_ivt, dt_ivt); > + } > + } > + } > + SET_END /* for source tuple */ > + > + /* add remaining */ > + SET_ALL(&source->tuples, t, log_tuple_t*) > + log_tuple_t* nt = log_tuple_clone(t); > + log_table_add0(dest, nt); > + SET_END > +} > + > +static log_table_t* > +tblopr_full_join(log_table_t* t1, log_table_t* t2, log_join_param_t* joinp) > +{ > + /* select1 contains only constants */ > + > + int32_t r1_fnum = array_size(&joinp->rem1); > + log_table_t* res = log_table_init( > + NULL, -1, r1_fnum + array_size(&joinp->rem2), 0, t1-> m.glb_values); > + > + int32_t i; > + log_value_t** tuple_values = calloc(sizeof(void*), res->num_fields); > + > + if (joinp->index2 == NULL) { > + SET_ALL(&t1->tuples, tuple1, log_tuple_t*) > + for (i = 0;i < r1_fnum;i++) > + tuple_values[i] = tuple1->values[ > + ptr2i(array_get(&joinp->rem1, i))]; > + > + SET_ALL(&t2->tuples, tuple2, log_tuple_t*) > + tblopr_combine_tuple(res, tuple1->count, > + joinp, r1_fnum, tuple2, tuple_values); > + SET_END > + SET_END > + } > + > + else { > + int32_t t2_index = map_get_int(&t2->index_def, joinp->index2); > + int t1val_sz = array_size(&joinp->select1); > + log_value_t** t1val = calloc(sizeof(void*), t1val_sz); > + > + for (i = 0;i < t1val_sz;i++) > + t1val[i] = array_get(&joinp->select1, i); > + > + log_tuple_t* key_tuple = log_tuple_init_val(t1val, t1val_sz); > + log_tuple_t* match_tuples = log_index_get_index( > + t2, key_tuple, t2_index); > + > + free(t1val); > + log_tuple_free(key_tuple, t1->m.glb_values, false); > + > + SET_ALL(&t1->tuples, tuple1, log_tuple_t*) > + for (i = 0;i < r1_fnum;i++) > + tuple_values[i] = tuple1->values[ > + ptr2i(array_get(&joinp->rem1, i))]; > + > + /* join the value */ > + INDEX_ALL(match_tuples, t2_index, tuple2) > + tblopr_combine_tuple(res, tuple1->count, > + joinp, r1_fnum, tuple2, tuple_values); > + INDEX_END > + SET_END > + } > + > + free(tuple_values); > + return res; /* isEmpty? */ > +} > + > +log_table_t* > +log_tblopr_join(log_table_t* t1, log_table_t* t2, log_join_param_t* joinp) > +{ > + if (joinp->index2 != NULL && > + !map_has(&t2->index_def, joinp->index2)) > + log_table_add_index(t2, joinp->index2); > + > + if (joinp->full_join) return tblopr_full_join(t1, t2, joinp); > + else return tblopr_cond_join(t1, t2, joinp); > +} > + > +/* ========================================================================== > + * LOG ENGINE > + * ========================================================================== > + */ > + > +log_join_param_t* > +log_join_param_init(log_join_param_t* jp, log_int_tuple_t* i2, set_t* gv) > +{ > + coll_alloc(&jp, sizeof(log_join_param_t), ENT_JOIN_PARAM, gv); > + > + jp->full_join = false; > + jp->index2 = i2; > + log_array_init(&jp->select1, KEY_ID(ENT_VALUE), 0, gv); > + log_array_init(&jp->select1i, KEY_ID(ENT_INT32), 0, gv); > + log_array_init(&jp->rem1, KEY_ID(ENT_INT32), 0, gv); > + log_array_init(&jp->rem2, KEY_ID(ENT_INT32), 0, gv); > + log_array_init(&jp->out_param, KEY_ID(ENT_INT32), 0, gv); > + return jp; > +} > + > +void > +log_join_param_free(log_join_param_t* jp) > +{ > + log_array_free(&jp->select1); > + log_array_free(&jp->select1i); > + log_array_free(&jp->rem1); > + log_array_free(&jp->rem2); > + log_array_free(&jp->out_param); > + if (jp->index2 != NULL) log_int_tuple_free(jp->index2); > + coll_free_ptr(jp); > +} > + > +static array_t* > +eng_get_cond_param(array_t* param) > +{ > + /* input and output are array of integer */ > + > + int i; > + array_t* res = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, param->m.glb_values); > + for (i = 0;i < array_size(param);i++) { > + int32_t p = array_get_int(param, i); > + if (p >= 0) array_add(res, i2ptr(p)); > + } > + return res; > +} > + > +static bool > +eng_check_will_use(int32_t p, TYPE(array_t*, int32_t) not_used, > + TYPE(array_t*, int32_t) reorder_list, log_rule_t* rule) > +{ > + int i; > + for (i = 0;i < array_size(not_used);i++) { > + if (ptr2i(array_get(not_used, i)) < 0) continue; > + > + array_t* a = (array_t*)array_get(&rule->param, > + array_get_int(reorder_list, i)); > + if (log_array_look_for(a, i2ptr(p)) >= 0) return true; > + } > + return false; > +} > + > +static bitset_t* > +eng_gen_bitset(TYPE(array_t*, int32_t) inp) > +{ > + int i; > + bitset_t* b = log_bitset_init(NULL); > + > + for (i = 0;i < array_size(inp);i++) { > + int32_t idx = array_get_int(inp, i); > + if (idx >= 0) bitset_set(b, idx); > + } > + return b; > +} > + > +static int32_t > +eng_get_joinable(log_rule_t* rule, TYPE(array_t*, int32_t) cur_param, > + TYPE(array_t*, int32_t) table_sz, > + TYPE(array_t*, int32_t) reorder) > +{ > + /* returns the seq id of the table to be joined. */ > + > + int32_t i, tb_index; > + bitset_t* bitset1 = eng_gen_bitset(cur_param); > + > + for (i = 1;i < array_size(table_sz);i++) { > + > + if (ptr2i(array_get(table_sz, i)) < 0) continue; > + /* had joined */ > + > + tb_index = ptr2i(array_get(reorder, i)); > + > + bitset_t* bitset2 = eng_gen_bitset(array_get(&rule->param, > tb_index)); > + bitset_and(bitset2, bitset1); > + > + bool empty = bitset_empty(bitset2); > + log_bitset_free(bitset2); > + > + if (!empty) { > + log_bitset_free(bitset1); > + return i; /* cond join */ > + } > + } > + > + log_bitset_free(bitset1); > + for (i = 1;i < array_size(table_sz);i++) > + if (ptr2i(array_get(table_sz, i)) >= 0) return i; > + /* full join */ > + return -1; > +} > + > +static log_join_param_t* > +eng_gen_join_param(TYPE(array_t*, int32_t) param1, > + TYPE(array_t*, int32_t) param2, > + TYPE(array_t*, int32_t) not_used, > + TYPE(array_t*, int32_t) reorder, > + log_rule_t* rule) > +{ > + /* > + * full join is false: > + * p1(7, 3, 2) p2(2, 3, -1, -3, 6) => outParam(7, 2, 6) > + * for join: select1(2, 1, v[-3]), index2(0, 1, 3) > + * for keep: rem1(0, 2), rem2(4) > + * > + * full join is true: > + * p1(7, 2, 6) p2(-4, 9, -1) => outParam(7, 2, 6, 9) > + * for join: select1(v[-4]), index2(0) > + * for keep: rem1(0, 1), rem2(1) > + */ > + > + log_join_param_t* joinp = log_join_param_init( > + NULL, /*idx2*/NULL, rule->m.glb_values); > + > + TYPE(array_t*, int32_t) index2 = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, param1-> m.glb_values); > + > + bitset_t* join_set = eng_gen_bitset(param1); > + bitset_t* param2_set = eng_gen_bitset(param2); > + > + int32_t i; > + bitset_and(join_set, param2_set); > + joinp->full_join = true; > + > + /* set select1 and index2 */ > + for (i = 0;i < array_size(param2);i++) { > + int32_t p2 = array_get_int(param2, i); > + > + if (p2 < -1) { > + array_add(index2, i2ptr(i)); > + log_value_t* v = (log_value_t*) > + map_get(&rule->const_param, i2ptr(p2)); > + array_add(&joinp->select1, v); > + array_add(&joinp->select1i, 0); > + } > + else if (p2 >= 0 && bitset_get(join_set, p2)) { > + int32_t pos1 = log_array_look_for(param1, i2ptr(p2)); > + array_add(index2, i2ptr(i)); > + > + array_add(&joinp->select1, NULL); > + array_add(&joinp->select1i, i2ptr(pos1)); > + joinp->full_join = false; > + } > + } > + > + /* set rem1 */ > + for (i = 0;i < array_size(param1);i++) { > + // if in left or right excluding processed tables, keep. > + int32_t p1 = array_get_int(param1, i); > + if (!eng_check_will_use(p1, not_used, reorder, rule)) continue; > + array_add(&joinp->rem1, i2ptr(i)); > + array_add(&joinp->out_param, i2ptr(p1)); > + } > + > + /* set rem2 */ > + for (i = 0;i < array_size(param2);i++) { > + /* > + * excluding all join set (because it is included in param1) > + * if in left or right excluding processed tables, keep. > + */ > + int32_t p2 = array_get_int(param2, i); > + if (p2 <= -1) continue; > + if (bitset_get(join_set, p2)) continue; > + if (!eng_check_will_use(p2, not_used, reorder, rule)) continue; > + > + array_add(&joinp->rem2, i2ptr(i)); > + array_add(&joinp->out_param, i2ptr(p2)); > + } > + > + /* void* ->int array not correct */ > + int32_t sz2 = array_size(index2); > + joinp->index2 = sz2 > 0 ? log_int_tuple_init(index2) : NULL; > + > + log_array_free(index2); > + log_bitset_free(join_set); > + log_bitset_free(param2_set); > + return joinp; > +} > + > +log_engine_t* > +log_eng_parse(const char* rules, set_t* gv) > +{ > + log_engine_t* eng = log_engine_init(NULL, gv); > + TYPE(map_t, array_t*) sem; > + > + log_sync_init(rules, gv); > + log_sync_parse(&sem); > + log_sem_process(&eng->rule_set, &sem); > + map_free(&sem); > + > + /* create tables */ > + MAP_ALL(&eng->rule_set.param_size, rule) > + int32_t tsize = ptr2i(rule->value); > + if (map_has(&eng->rule_set.input_tables, rule->key) || > + !map_has(&eng->rule_set.output_tables, rule->key)) > + > + map_add(&eng->tables, rule->key, > + log_table_init(NULL, ptr2i(rule->key), tsize, 0, gv)); > + MAP_END > + return eng; > +} > + > +static void > +eng_check_tuples(log_engine_t* eng, log_table_t* table) > +{ > + /* check if tuple already present for add; or does not exist for remove. > + * the check is not good if there are multiple same items in the input. > + */ > + > + bool is_remove = table->is_remove; > + log_table_t* org_table = map_get(&eng->tables, > i2ptr(table->table_index)); > + TYPE(array_t*, log_tuple_t*) to_be_removed = /* no ID to prevent free */ > + log_array_init(NULL, 0, 0, table->m.glb_values); > + > + SET_ALL(&table->tuples, t, log_tuple_t*) > + log_tuple_t* ot = set_get(&org_table->tuples, t); > + if ((is_remove && ot == NULL) || (!is_remove && ot != NULL)) > + array_add(to_be_removed, t); > + /*array_add(to_be_removed, ot); */ > + SET_END > + > + ARRAY_ALL(to_be_removed, t, log_tuple_t*) > + /* log_table_remove(org_table, t); ?? */ > + log_table_remove(table, t); > + ARRAY_END > + log_array_free(to_be_removed); > +} > + > +static void > +eng_align_tables(log_engine_t* eng, > + TYPE(array_t*, log_table_t*) inp_remove, > + TYPE(array_t*, log_table_t*) inp_insert) > +{ > + TYPE(array_t*, int32_t) del_ids = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); > + TYPE(array_t*, int32_t) add_ids = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); > + > + ARRAY_ALL(inp_remove, tbl, log_table_t*) > + eng_check_tuples(eng, tbl); > + array_add(del_ids, i2ptr(tbl->table_index)); > + ARRAY_END > + > + ARRAY_ALL(inp_insert, tbl, log_table_t*) > + eng_check_tuples(eng, tbl); > + array_add(add_ids, i2ptr(tbl->table_index)); > + ARRAY_END > + > + log_sort_array(0, del_ids, inp_remove, NULL); > + log_sort_array(0, add_ids, inp_insert, NULL); > + > + int del_i, add_i; > + for (del_i = 0, add_i = 0; > + del_i < array_size(del_ids) || add_i < array_size(add_ids);) { > + > + bool align_add = false; > + bool align_del = false; > + > + if (del_i < array_size(del_ids) && add_i < array_size(add_ids)) { > + if (array_get(del_ids, del_i) == > + array_get(add_ids, add_i)) { > + add_i++; del_i++; continue; > + } > + else if (array_get(del_ids, del_i) < array_get(add_ids, add_i)) > + align_add = true; > + else align_del = true; > + } > + else if (del_i < array_size(del_ids)) align_add = true; > + else align_del = true; > + > + if (align_add) { > + log_table_t* org = array_get(inp_remove, del_i); > + log_table_t* dummy = > + log_table_init(NULL, org->table_index, org->num_fields, > + 0, eng->m.glb_values); > + > + array_ins(inp_insert, del_i, dummy); > + array_ins(add_ids, del_i, NULL); > + add_i++; del_i++; > + } > + > + if (align_del) { > + log_table_t* org = array_get(inp_insert, add_i); > + log_table_t* dummy = > + log_table_init(NULL, org->table_index, org->num_fields, > + 0, eng->m.glb_values); > + dummy->is_remove = true; > + > + array_ins(inp_remove, add_i, dummy); > + array_ins(del_ids, add_i, NULL); > + add_i++; del_i++; > + } > + } > + > + log_array_free(del_ids); > + log_array_free(add_ids); > + > + if (LOG_COMP) { > + char buf[2048]; int pos; > + pos = 0; pos = log_array_print(buf, pos, inp_remove, false); > + buf[pos] = 0; printf("[LOG] align_tbl - %s\n", buf); > + pos = 0; pos = log_array_print(buf, pos, inp_insert, false); > + buf[pos] = 0; printf("[LOG] align_tbl + %s\n", buf); > + } > +} > + > +static bool > +eng_invoke_external(log_engine_t* eng, log_table_t* input, > + log_table_t* del_output, log_table_t* add_output) > +{ > + /* delOutput and addOutput have the same table index */ > + if (eng->ext_func == NULL) return false; > + return (*eng->ext_func)(eng, input, del_output, add_output); > +} > + > +void > +log_eng_set_ext_func(log_engine_t* eng, void* func) > +{ > + eng->ext_func = func; > +} > + > +log_table_t* > +log_get_table(log_engine_t* eng, const char* name) > +{ > + /* id is integer */ > + void* id = map_get(&eng->rule_set.rule_index_map, (char*)name); > + return map_get(&eng->tables, id); > +} > + > +log_table_t* > +log_get_org_table(log_engine_t* eng, log_table_t* t) > +{ > + return map_get(&eng->tables, i2ptr(t->table_index)); > +} > + > +log_tuple_t* > +log_query_on0(log_engine_t* eng, int32_t tid, log_value_t* value) > +{ > + /* USE INDEX_ALL to iterate on return value */ > + log_tuple_t* qt = log_tuple_init(1); > + qt->values[0] = value; > + > + TYPE(array_t*, int32_t) ints = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); > + array_add(ints, 0); > + log_int_tuple_t* key0 = log_int_tuple_init(ints); > + log_array_free(ints); > + > + log_table_t* orgt = map_get(&eng->tables, i2ptr(tid)); > + int32_t idx_i = log_table_add_index(orgt, key0); > + log_tuple_t* list = log_index_get_index(orgt, qt, idx_i); > + > + log_tuple_free(qt, eng->m.glb_values, true); > + log_int_tuple_free(key0); > + return list; > +} > + > +static TYPE(array_t*, log_table_t*) > +eng_query_table(log_engine_t* eng, log_table_t* table, > + TYPE2(map_t*, int32_t, log_table_t*) all) > +{ > + /* the original value is in form of =value */ > + > + set_t* gv = eng->m.glb_values; > + TYPE(array_t*, log_table_t*) res = > + log_array_init(NULL, 0 /* ENT_TABLE */, 0, gv); > + /* no type so free in log_eng_query will leave tables */ > + > + int i, j; > + log_value_t** val = calloc(table->num_fields, sizeof(void*)); > + log_table_t* org = map_get(all, i2ptr(table->table_index)); > + > + SET_ALL(&table->tuples, tval, log_tuple_t*) > + TYPE(array_t*, int32_t) param = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + > + for (i = j = 0;i < table->num_fields;i++) { > + /* NULL value only for query */ > + if (tval->values[i] == NULL) continue; > + array_add(param, i2ptr(i)); > + val[j++] = tval->values[i]; > + } > + > + log_table_t* res_tbl = tblopr_query_table(org, param, val); > + array_add(res, res_tbl); > + log_array_free(param); > + SET_END > + > + free(val); > + return res; > +} > + > +TYPE(array_t*, log_table_t*) > +log_eng_query(log_engine_t* eng, TYPE(array_t*, log_table_t*) input) > +{ > + TYPE(array_t*, log_table_t*) res = > + log_array_init(NULL, KEY_ID(ENT_TABLE), 0, eng->m.glb_values); > + > + TYPE(array_t*, log_table_t*) res1; > + > + ARRAY_ALL(input, table, log_table_t*) > + res1 = eng_query_table(eng, table, &eng->tables); > + ARRAY_ALL(res1, tbl, log_table_t*) > + array_add(res, tbl); > + ARRAY_END > + log_array_free(res1); > + ARRAY_END > + return res; > +} > + > +static int32_t > +eng_get_table_index(log_rule_t* rule, int32_t param) > +{ > + return log_array_look_for(&rule->rule, i2ptr(param)); > +} > + > +static int32_t > +eng_search_tbl_in_rules(log_table_t* tbl, > + TYPE(array_t*, log_table_t*) inp_tables) > +{ > + int idx = 0; > + ARRAY_ALL(inp_tables, t, log_table_t*) > + if (t->table_index == tbl->table_index) return idx; > + idx++; > + ARRAY_END > + return -1; > +} > + > +static log_table_t* > +eng_reset_count(log_table_t* input) > +{ > + log_table_t* output = log_table_init(NULL, input->table_index, > + input->num_fields, input->tuples.size, input->m.glb_values); > + output->is_remove = input->is_remove; > + > + SET_ALL(&input->tuples, t, log_tuple_t*) > + log_tuple_t* tup = log_tuple_clone(t); > + tup->count = 1; > + log_table_add0(output, tup); > + SET_END > + return output; > +} > + > +static bool > +eng_merge_output(log_engine_t* eng, TYPE(array_t*, int32_t) first_rule, > + log_table_t* out_d_del, log_table_t* out_d_add, > + TYPE(array_t*, log_table_t*) inp_del_tables, > + TYPE(array_t*, log_table_t*) inp_add_tables) > +{ > + /* returns true if input table will be used later */ > + > + if (LOG_COMP) { > + char buf[2048]; int pos; > + pos = 0; pos = log_table_print(buf, pos, out_d_del, false); > + buf[pos] = 0; printf("[LOG] merge_out - %s\n", buf); > + pos = 0; pos = log_table_print(buf, pos, out_d_add, false); > + buf[pos] = 0; printf("[LOG] merge_out + %s\n", buf); > + } > + > + if (table_size(out_d_del) == 0 && table_size(out_d_add) == 0) > + return false; > + > + int32_t ipos = eng_search_tbl_in_rules(out_d_del, inp_del_tables); > + if (ipos >= 0) { > + tblopr_merge_delta(out_d_del, > + array_get(inp_del_tables, ipos), > array_get(inp_add_tables, ipos)); > + tblopr_merge_delta(out_d_add, > + array_get(inp_add_tables, ipos), > array_get(inp_del_tables, ipos)); > + return false; > + } > + > + int32_t new_inp_no = > + map_has(&eng->rule_set.output_tables, i2ptr(out_d_del-> table_index)) > + ? 1000000 : /* just a big number */ > + array_get_int(map_get(&eng->rule_set.table_rule_map, > + i2ptr(out_d_del->table_index)), 0); > + log_insert_item(new_inp_no, first_rule, > + out_d_del, out_d_add, inp_del_tables, inp_add_tables); > + return true; > +} > + > +void > +log_eng_do_union(log_engine_t* eng, log_table_t* input, log_table_t* output) > +{ > + /* example: (0, 1, 2) > (2, 1, -, 'const', 0), (0, -, 'aa', 2, 1) */ > + > + log_rule_t* rule = map_get(&eng->rule_set.rules, > + i2ptr(output->table_index)); > + > + tblopr_match_reorder_and_merge(input, output, > + array_get(&rule->param, eng_get_table_index(rule, input-> table_index)), > + array_get(&rule->param, eng_get_table_index(rule, > output->table_index)), > + &rule->const_param); > +} > + > +void > +log_eng_do_join(log_engine_t* eng, log_table_t* input, log_table_t* output) > +{ > + /* > + * example: (7, 6, 2) : (7, 3, -1, -2, 2), (2, 3, -1, -3, 6) > + * initial process: (7, 3, -1, -2, 2) => (7, 3, 2) > + * drop 'ignored' param, match constants and merge identical; > + * repeated joins - with condition having a higher priority than > + * full join, possible with constants; the first join could beself join. > + * final process: reorder based on left (7, 6, 2) > + * intermediate tables are not hash table, just list? > + * > + * join in the order of table size, smallest first > + * table size -1 indicates it had been joined before > + */ > + set_t* gv = eng->m.glb_values; > + > + log_rule_t* rule = map_get(&eng->rule_set.rules, > + i2ptr(output->table_index)); > + > + TYPE(array_t*, int32_t) table_sz = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + TYPE(array_t*, int32_t) rule_reorder = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); /* remove later */ > + TYPE(array_t*, int32_t) natural_order = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + > + /* STEP 0: sort based on table size and get the first join pair */ > + array_add(table_sz, 0); > + array_add(natural_order, 0); > + array_add(rule_reorder, array_get(&rule->rule, 0)); > + > + int i; > + for (i = 1;i < array_size(&rule->rule);i++) { > + int32_t p = array_get_int(&rule->rule, i); > + array_add(table_sz, i2ptr( > + table_size((log_table_t*)map_get(&eng->tables, > + i2ptr(p))))); > + array_add(natural_order, i2ptr(i)); > + array_add(rule_reorder, i2ptr(p)); > + } > + > + /* 1 for skip left rule */ > + log_sort_array(1, table_sz, rule_reorder, natural_order); > + > + /* STEP 1: merge, reorder and merge for the input. */ > + log_table_t* table1 = NULL; > + TYPE(array_t*, int32_t) param1 = NULL; > + int32_t join1_idx = log_array_look_for(rule_reorder, > + i2ptr(input->table_index)); > + > + if (join1_idx < array_size(rule_reorder) - 1 && input->table_index == > + array_get_int(rule_reorder, join1_idx + 1)) { > + /* > + * do self join. only one self join is supported as more will need > + * full table join and less efficient - use intermediate > table instead > + * for this case. > + * Example: X(b, a, c) : x(a, b, 'v', -, c) x(b, 'w', -, a, c) > + */ > + > + array_set(table_sz, join1_idx, i2ptr(-1)); /* mark as used */ > + array_set(table_sz, join1_idx + 1, i2ptr(-1)); /* mark as used */ > + > + /* pos based on rule sequence */ > + int pos1 = log_array_look_for(&rule->rule, > i2ptr(input->table_index)); > + int pos2 = log_array_look_for(&rule->rule, > + i2ptr(input->table_index)) + 1; > + > + TYPE(array_t*, int32_t) param1Org = array_get(&rule->param, pos1); > + /* (a, b, 'v', -, c) */ > + TYPE(array_t*, int32_t) param2Org = array_get(&rule->param, pos2); > + /* (b, 'w', -, a, c) */ > + param1 = eng_get_cond_param(param1Org); > + /* (a, b, c) */ > + TYPE(array_t*, int32_t) param2 = eng_get_cond_param(param2Org); > + /* (b, a, c) */ > + > + log_table_t* table0 = map_get(&eng->tables, > + i2ptr(input->table_index)); > + > + /* original table */ > + table1 = log_table_init( > + NULL, -1, array_size(param1), 0, gv); > + log_table_t* table2 = log_table_init( > + NULL, -1, array_size(param1), 0, gv); > + > + tblopr_match_reorder_and_merge( > + input, table1, param1Org, param1, > &rule->const_param); > + tblopr_match_reorder_and_merge( > + input, table2, param2Org, param2, > &rule->const_param); > + > + log_join_param_t* joinp1 = eng_gen_join_param( > + param1, param2Org, table_sz, natural_order, rule); > + log_join_param_t* joinp2 = eng_gen_join_param( > + param2, param1Org, table_sz, natural_order, rule); > + > + log_table_t* out_tb1 = log_tblopr_join(table1, table0, joinp1); > + /* outp: (a, b, c) */ > + log_table_t* out_tb2 = log_tblopr_join(table1, input, joinp1); > + /* outp: (a, b, c) */ > + log_table_t* out_tb3 = log_tblopr_join(table2, table0, joinp2); > + /* outp: (b, a, c) */ > + > + /* reorder outTb3, it should only differ in order with outTb1, 2. */ > + TYPE(array_t*, int32_t) reorder = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + > + ARRAY_ALL_INT(&joinp1->out_param, pos) > + array_add(reorder, i2ptr( > + log_array_look_for(&joinp2->out_param, i2ptr (pos)))); > + ARRAY_END > + > + log_table_t* out_tb4 = > + log_table_init(NULL, -1, out_tb3->num_fields, 0, gv); > + tblopr_reorder_table(out_tb3, reorder, NULL, out_tb4); > + > + /* merge outTb1, 2, and 4 */ > + log_table_free(table1); > + table1 = log_table_init(NULL, -1, out_tb1->num_fields, 0, gv); > + > + tblopr_merge_table(out_tb1, table1, false); > + tblopr_merge_table(out_tb4, table1, false); > + tblopr_merge_table(out_tb2, table1, input->is_remove); > + > + log_array_free(param1); > + param1 = log_array_clone(&joinp1->out_param); > + > + log_table_free(table2); > + log_table_free(out_tb1); > + log_table_free(out_tb2); > + log_table_free(out_tb3); > + log_table_free(out_tb4); > + > + log_array_free(reorder); > + log_array_free(param2); > + log_join_param_free(joinp1); > + log_join_param_free(joinp2); > + } > + else { > + int32_t pos1 = log_array_look_for(&rule->rule, > + i2ptr(input->table_index)); > + TYPE(array_t*, int32_t) param1_org = array_get(&rule->param, pos1); > + param1 = eng_get_cond_param(param1_org); > + > + table1 = log_table_init(NULL, -1, array_size(param1), 0, gv); > + tblopr_match_reorder_and_merge( > + input, table1, param1_org, param1, &rule->const_param); > + array_set(table_sz, join1_idx, i2ptr(-1)); /* mark as used */ > + } > + > + /* STEP 2: repeated join, first condition join, then full join */ > + for (;;) { /* join loop, the iteration param is param1 and table1 */ > + > + int32_t join2_idx = eng_get_joinable( > + rule, param1, table_sz, natural_order); > + > + if (join2_idx < 0) break; /* nothing to join */ > + array_set(table_sz, join2_idx, i2ptr(-1)); /* mark as used */ > + > + log_table_t* table2 = map_get( > + &eng->tables, array_get(rule_reorder, join2_idx)); > + TYPE(array_t*, int32_t) param2 = array_get(&rule->param, > + array_get_int(natural_order, join2_idx)); > + > + log_join_param_t* joinp = > + eng_gen_join_param(param1, param2, table_sz, > natural_order, rule); > + log_table_t* table1n = log_tblopr_join(table1, table2, joinp); > + > + log_table_free(table1); > + log_array_free(param1); > + > + param1 = log_array_clone(&joinp->out_param); > + log_join_param_free(joinp); > + table1 = table1n; > + } > + > + /* STEP 3: reorder the final table */ > + TYPE(array_t*, int32_t) final_order = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + ARRAY_ALL_INT((array_t*)array_get(&rule->param, 0), p) > + if (p < -1) array_add(final_order, i2ptr(p)); > + else array_add(final_order, > + i2ptr(log_array_look_for(param1, i2ptr(p)))); > + ARRAY_END > + > + tblopr_reorder_table(table1, final_order, &rule->const_param, output); > + output->is_remove = input->is_remove; > + > + log_table_free(table1); > + log_array_free(param1); > + > + log_array_free(table_sz); > + log_array_free(rule_reorder); > + log_array_free(natural_order); > + log_array_free(final_order); > +} > + > +/* > + * the basic assumption about external function (and all) is that the result > + * of left side does not depend on the order of application of delta change > + * of the tables from right side. since the original input table could be > + * obtained in the function, the computation could rely on delta (preferable > + * for performance) or whole set, but C=A\B still could not be computed > + * because order of delta application is relevant. > + * > + * the computation order is based on topology sort, and guarantees that each > + * rule will only be applied once, e.g., C:A, B; D:C. Assume the input > + * contains A and B, D will be computed only if both A and B have been > + * applied on C. > + * > + * when input tables are checked (no adding to existing tuple, and > no removing > + * of none existing tuple, and adding does not overlap with removing), the > + * order of adding and removing is not relevant. inpDelTables applied before > + * inpAddTables is due to performance consideration. (not validated yet) > + * > + * tuples in right always regarded as having count 1 > + * tuples in left has actual count. > + * > + * inpDelTables and inpAddTables must be paired, i.e., > + * delTables[i].id == addTables[i].id holds for all i. > + */ > + > +static void > +eng_delta0(log_engine_t* eng, TYPE(array_t*, log_table_t*) inp_del_tables, > + TYPE(array_t*, log_table_t*) inp_add_tables, > + TYPE2(map_t*, int32_t, log_table_t*) del_out_tables, > + TYPE2(map_t*, int32_t, log_table_t*) add_out_tables) > +{ > + /* input tables will be destroyed, but still need to free it */ > + set_t* gv = eng->m.glb_values; > + > + TYPE(array_t*, int32_t) first_rule = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); > + > + ARRAY_ALL(inp_del_tables, table, log_table_t*) > + int32_t v = array_get_int( // item 0 is smallest in each set > + map_get(&eng->rule_set.table_rule_map, > + i2ptr(table->table_index)), 0); > + > + array_add(first_rule, i2ptr(v)); > + ARRAY_END > + > + /* there might be tie in sort. just pick from the initial order */ > + log_sort_array(0, first_rule, inp_del_tables, inp_add_tables); > + > + while (array_size(inp_del_tables) > 0) { > + /* inpDTables always has 1 as count but outDTable will has actual > + * count before turning into inpDTables. Final output tables will > + * have actual count. > + */ > + > + array_rmv(first_rule, 0); > + /* D stands for delta */ > + log_table_t* inp_d_del_table = array_rmv(inp_del_tables, 0); > + log_table_t* inp_d_add_table = array_rmv(inp_add_tables, 0); > + int32_t inp_table_id = inp_d_del_table->table_index; > + > + if (LOG_COMP) { > + char buf[2048]; int pos; > + pos = 0; pos = log_table_print(buf, pos, inp_d_del_table, false); > + buf[pos] = 0; printf("[LOG] delta - %s\n", buf); > + pos = 0; pos = log_table_print(buf, pos, inp_d_add_table, false); > + buf[pos] = 0; printf("[LOG] delta + %s\n", buf); > + } > + > + if (map_has(&eng->rule_set.output_tables, i2ptr(inp_table_id))) { > + if (table_size(inp_d_del_table) > 0) > + map_add(del_out_tables, i2ptr(inp_table_id), > inp_d_del_table); > + else log_table_free(inp_d_del_table); > + > + if (table_size(inp_d_add_table) > 0) > + map_add(add_out_tables, i2ptr(inp_table_id), > inp_d_add_table); > + else log_table_free(inp_d_add_table); > + continue; > + } > + > + if (!map_has(&eng->rule_set.input_tables, i2ptr(inp_table_id))) { > + tblopr_gen_delta(inp_d_del_table, > + map_get(&eng->tables, i2ptr (inp_table_id))); > + tblopr_gen_delta(inp_d_add_table, > + map_get(&eng->tables, i2ptr (inp_table_id))); > + } > + > + if (table_size(inp_d_del_table) == 0 && > + table_size(inp_d_add_table) == 0) { > + log_table_free(inp_d_del_table); > + log_table_free(inp_d_add_table); > + continue; > + } > + > + log_table_t* inp_d_del_table_c1 = eng_reset_count (inp_d_del_table); > + log_table_t* inp_d_add_table_c1 = eng_reset_count (inp_d_add_table); > + > + TYPE(array_t*, int32_t) rule_order = > + map_get(&eng->rule_set.table_rule_map, i2ptr (inp_table_id)); > + > + ARRAY_ALL_INT(rule_order, rule_no) > + log_rule_t* rule = map_get(&eng->rule_set.rules, i2ptr (rule_no)); > + > + log_table_t* out_d_del_table = log_table_init(0, rule_no, > + map_get_int(&eng->rule_set.param_size, > + i2ptr(rule_no)), 0, gv); > + > + log_table_t* out_d_add_table = log_table_init(0, rule_no, > + map_get_int(&eng->rule_set.param_size, > + i2ptr(rule_no)), 0, gv); > + > + out_d_del_table->is_remove = true; > + if (eng_invoke_external > + (eng, inp_d_del_table_c1, out_d_del_table, > out_d_add_table)) { > + bool keep = eng_merge_output(eng, first_rule, > out_d_del_table, > + out_d_add_table, inp_del_tables, inp_add_tables); > + > + if (!keep) { > + log_table_free(out_d_del_table); > + log_table_free(out_d_add_table); > + } > + > + out_d_del_table = log_table_init(0, rule_no, > + map_get_int(&eng->rule_set.param_size, > + i2ptr(rule_no)), 0, gv); > + > + out_d_add_table = log_table_init(0, rule_no, > + map_get_int(&eng->rule_set.param_size, > + i2ptr(rule_no)), 0, gv); > + > + out_d_del_table->is_remove = true; > + eng_invoke_external > + (eng, inp_d_add_table_c1, out_d_del_table, > out_d_add_table); > + /* will also merge below */ > + } > + else if (rule->is_union) { > + log_eng_do_union(eng, inp_d_del_table_c1, out_d_del_table); > + log_eng_do_union(eng, inp_d_add_table_c1, out_d_add_table); > + } > + else { > + log_eng_do_join(eng, inp_d_del_table_c1, out_d_del_table); > + log_eng_do_join(eng, inp_d_add_table_c1, out_d_add_table); > + } > + > + bool keep = eng_merge_output(eng, first_rule, out_d_del_table, > + out_d_add_table, inp_del_tables, inp_add_tables); > + > + if (!keep) { > + log_table_free(out_d_del_table); > + log_table_free(out_d_add_table); > + } > + ARRAY_END /* for all rules related to one table input */ > + > + /* merge input table */ > + tblopr_final_delta(inp_d_del_table, > + map_get(&eng->tables, i2ptr(inp_d_del_table->table_index))); > + tblopr_final_delta(inp_d_add_table, > + map_get(&eng->tables, i2ptr(inp_d_add_table->table_index))); > + > + log_table_free(inp_d_del_table_c1); > + log_table_free(inp_d_add_table_c1); > + log_table_free(inp_d_del_table); > + log_table_free(inp_d_add_table); > + } /* for all inputs */ > + > + log_array_free(first_rule); > +} > + > +TYPE(array_t*, log_table_t*) > +log_eng_delta(log_engine_t* eng, TYPE(array_t*, log_table_t*) inp_remove, > + TYPE(array_t*, log_table_t*) inp_insert) > +{ > + > + set_t* gv = eng->m.glb_values; > + /* no VALUE_ID so that tables will not be freed */ > + TYPE2(map_t*, int32_t, log_table_t*) del_output = > + map_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + TYPE2(map_t*, int32_t, log_table_t*) add_output = > + map_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + > + eng_align_tables(eng, inp_remove, inp_insert); > + eng_delta0(eng, inp_remove, inp_insert, del_output, add_output); > + > + TYPE(array_t*, log_table_t*) all = > + log_array_init(NULL, KEY_ID(ENT_TABLE), 0, gv); > + MAP_ALL(del_output, tn) array_add(all, tn->value); MAP_END > + MAP_ALL(add_output, tn) array_add(all, tn->value); MAP_END > + > + /* reset state */ > + eng_invoke_external(eng, NULL, NULL, NULL); > + map_free(del_output); > + map_free(add_output); > + return all; > +} > + > +/* ========================================================================== > + * SERIALIZATION > + * ========================================================================== > + */ > + > +log_table_t* > +log_io_table_name_reset(log_engine_t* eng, const char* name) > +{ > + set_t* gv = eng->m.glb_values; > + log_value_t* table_name = log_value_init(name, 0, gv); > + map_node_t* node = log_hash_get( > + &eng->rule_set.rule_index_map, table_name); > + > + log_value_free(table_name, eng->m.glb_values); > + if (node == NULL) return NULL; > + > + void* tbl_idx = node->value; /* int32_t */ > + int32_t tbl_fd = map_get_int(&eng->rule_set.param_size, tbl_idx); > + log_table_t* tbl = log_table_init(NULL, ptr2i(tbl_idx), tbl_fd, 0, gv); > + return tbl; > +} > + > +bool > +log_io_parse_line(const char* line, log_engine_t* eng, bool use_null, > + TYPE(array_t*, log_table_t*) inp_remove, > + TYPE(array_t*, log_table_t*) inp_insert) > +{ > + static int32_t line_type = 0; /* 1 for add; 2 for delete */ > + static int32_t line_rm = 0; > + static log_table_t* cur_tbl; > + > + if (line == NULL) { > + line_type = 0; > + return true; > + } > + > + set_t* gv = eng->tables.m.glb_values; > + if (line_type == 0) { > + char type; > + char buf[512]; > + > + int32_t f = sscanf(line, "%c:%d:%s", &type, &line_rm, buf); > + if (f != 3) return false; > + if (type == '+') line_type = 1; > + else if (type == '-') line_type = 2; > + else return false; > + if (line_rm <= 0) return false; > + > + log_table_t* tbl = log_io_table_name_reset(eng, buf); > + if (tbl == NULL) return false; > + > + if (line_type == 1) array_add(inp_insert, tbl); > + else array_add(inp_remove, tbl); > + if (line_type == 2) tbl->is_remove = true; > + cur_tbl = tbl; > + return true; > + } > + > + if (line_rm-- == 0) return false; > + if (line_rm == 0) line_type = 0; > + > + log_tuple_t* tpl = use_null ? > + log_tuple_init_str_null(line, ':', gv) : > + log_tuple_init_str(line, ':', gv); > + > + log_table_add(cur_tbl, tpl); > + return true; > +} > + > +int32_t > +log_io_gen_line(char* text, int pos, log_engine_t* eng, > + TYPE(array_t*, log_table_t*) all) > +{ > + ARRAY_ALL(all, tbl, log_table_t*) > + int32_t tbl_idx = tbl->table_index; > + log_value_t* tbl_name = map_get( > + &eng->rule_set.rule_name_map, > i2ptr(tbl_idx)); > + pos += sprintf(text + pos, "%c:%d:%s\n", tbl->is_remove ? '-' : '+', > + table_size(tbl), tbl_name->value.a); > + > + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) > + pos = log_tuple_print(text, pos, tuple); > + text[pos++] = '\n'; > + SET_END > + ARRAY_END > + return pos; > +} > + > +/* ========================================================================== > + * PUBLIC API > + * ========================================================================== > + */ > + > +void* > +datalog_init(const char* rules, void* func) > +{ > + /* will call exit in case rules are incorrect */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = log_eng_parse(rules, gv); > + log_eng_set_ext_func(eng, func); > + > + eng->io.inp_insert = NULL; > + eng->io.inp_remove = NULL; > + eng->io.res = NULL; > + return eng; > +} > + > +void > +datalog_free(void* e) > +{ > + log_engine_t* eng = e; > + set_t* gv = eng->tables.m.glb_values; > + log_engine_free(eng); > + set_free(gv); > +} > + > +bool > +datalog_put_table(void* e, bool is_remove, const char* name) > +{ > + /* each table must only appear at most twice - add / remove */ > + > + log_engine_t* eng = e; > + set_t* gv = eng->tables.m.glb_values; > + > + log_table_t* tbl = log_io_table_name_reset(eng, name); > + if (tbl == NULL) return false; > + > + if (eng->io.inp_insert == NULL || eng->io.inp_remove == NULL) { > + eng->io.inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); > + eng->io.inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); > + } > + > + tbl->is_remove = is_remove; > + eng->io.cur_tbl = tbl; > + eng->io.cur_tuple = NULL; > + if (is_remove) array_add(eng->io.inp_remove, tbl); > + else array_add(eng->io.inp_insert, tbl); > + return true; > +} > + > +void > +datalog_put_field(void* e, void* value, int32_t len) > +{ > + /* value is treated as c-str if len is zero */ > + > + log_engine_t* eng = e; > + set_t* gv = eng->tables.m.glb_values; > + int n_fd = eng->io.cur_tbl->num_fields; > + > + if (eng->io.cur_tuple == NULL) { > + eng->io.cur_tuple = log_tuple_init(n_fd); > + eng->io.cur_tuple->count = 1; > + eng->io.t_idx = 0; > + } > + > + eng->io.cur_tuple->values[eng->io.t_idx++] = > + value == NULL ? NULL : log_value_init(value, len, gv); > + > + if (eng->io.t_idx >= n_fd) { > + log_tuple_set_hash_code(eng->io.cur_tuple, n_fd); > + log_table_add(eng->io.cur_tbl, eng->io.cur_tuple); > + eng->io.cur_tuple = NULL; > + } > +} > + > +void > +datalog_opr(void* e, bool query) > +{ > + log_engine_t* eng = e; > + if (query) { > + /* must provide one tuple in inp_insert table */ > + ovs_assert(array_size(eng->io.inp_insert) == 1 && > + array_size(eng->io.inp_remove) == 0); > + > + eng->io.res = log_eng_query(eng, eng->io.inp_insert); > + } > + else { > + ovs_assert(eng->io.res == NULL && > + eng->io.inp_insert != NULL && eng->io.inp_remove != NULL); > + > + eng->io.res = log_eng_delta(eng, > + eng->io.inp_remove, eng->io.inp_insert); > + } > + > + log_array_free(eng->io.inp_remove); > + log_array_free(eng->io.inp_insert); > + eng->io.inp_remove = NULL; > + eng->io.inp_insert = NULL; > + eng->io.t_idx = 0; > +} > + > +bool > +datalog_get_table(void* e, bool* is_remove, const char** name, > + int32_t* n_tuples, int32_t* n_fields) > +{ > + /* return false if there is no more table */ > + log_engine_t* eng = e; > + > + if (eng->io.t_idx >= array_size(eng->io.res)) { > + log_array_free(eng->io.res); > + eng->io.res = NULL; > + return false; > + } > + > + eng->io.cur_tbl = array_get(eng->io.res, eng->io.t_idx++); > + *is_remove = eng->io.cur_tbl->is_remove; > + *n_tuples = table_size(eng->io.cur_tbl); > + *n_fields = eng->io.cur_tbl->num_fields; > + > + log_value_t* val = map_get( > + &eng->rule_set.rule_name_map, i2ptr(eng->io.cur_tbl-> table_index)); > + *name = val->value.a; > + > + eng->io.hash_node = NULL; > + eng->io.hash_b = 0; > + eng->io.f_idx = eng->io.cur_tbl->num_fields; > + return true; > +} > + > +bool > +datalog_get_field(void* e, void* res, int32_t* sz) > +{ > + /* return false if switches to another table */ > + > + void** v = (void**)res; > + log_engine_t* eng = e; > + int n_fd = eng->io.cur_tbl->num_fields; > + > + if (eng->io.f_idx < n_fd) { > + log_value_t* val = eng->io.cur_tuple->values[eng->io.f_idx++]; > + *v = val->value.a; > + *sz = val->size; > + return true; > + } > + > + bool r = log_hash_next(&eng->io.cur_tbl->tuples, > + &eng->io.hash_b, &eng->io.hash_node); > + if (!r) return false; > + > + eng->io.f_idx = 0; > + eng->io.cur_tuple = eng->io.hash_node->key; > + log_value_t* val = eng->io.cur_tuple->values[eng->io.f_idx++]; > + > + *v = val->value.a; > + *sz = val->size; > + return true; > +} > + > +/* > + * TODO: add stats to value, hash, and table operation. > + */ > diff --git a/ovn/lib/datalog.h b/ovn/lib/datalog.h > new file mode 100644 > index 0000000..c3d1a80 > --- /dev/null > +++ b/ovn/lib/datalog.h > @@ -0,0 +1,49 @@ > +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. > + * > + * Licensed under the Apache License, Version 2.0 (the "License"); > + * you may not use this file except in compliance with the License. > + * You may obtain a copy of the License at: > + * > + * http://www.apache.org/licenses/LICENSE-2.0 > + * > + * Unless required by applicable law or agreed to in writing, software > + * distributed under the License is distributed on an "AS IS" BASIS, > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > + * See the License for the specific language governing permissions and > + * limitations under the License. > + */ > + > +#ifndef OVN_DATALOG_H > +#define OVN_DATALOG_H 1 > + > +#include <stdbool.h> > +#include <stdint.h> > + > +/* > + * calling sequence: > + * init (put_table (put_field)*)* opr (get_table (get_field)*)* free > + * > + * init returns handle of the engine and will be used in all other calls. > + * put_table() returns false if table name is invalid. > + * size could be zero for put_field() and null-terminated c-str is assumed. > + * get_table() returns false if all tables have been retrieved. > + * get_field() returns false if the last tuple of a table has been retrieved. > + * > + * all values are assumed to be read only for both input and output. > + * see test_api for api examples. > + */ > + > +void* datalog_init(const char* rules, void* ext_func); > +void datalog_free(void* eng); > + > +/* true for query and false for delta change */ > +void datalog_opr(void* eng, bool query); > + > +bool datalog_put_table(void* eng, bool is_remove, const char* name); > +void datalog_put_field(void* eng, void* value, int32_t size); > + > +bool datalog_get_table(void* e, bool* is_remove, const char** name, > + int32_t* n_tuples, int32_t* n_fields); > +bool datalog_get_field(void* e, void* v, int32_t* size); > + > +#endif /* ovn_datalog.h */ > diff --git a/tests/automake.mk b/tests/automake.mk > index a5c6074..5f2ea58 100644 > --- a/tests/automake.mk > +++ b/tests/automake.mk > @@ -90,7 +90,8 @@ TESTSUITE_AT = \ > tests/ovn-nbctl.at \ > tests/ovn-sbctl.at \ > tests/ovn-controller.at \ > - tests/ovn-controller-vtep.at > + tests/ovn-controller-vtep.at \ > + tests/datalog.at > > SYSTEM_KMOD_TESTSUITE_AT = \ > tests/system-common-macros.at \ > @@ -337,7 +338,8 @@ tests_ovstest_SOURCES = \ > tests/test-uuid.c \ > tests/test-bitmap.c \ > tests/test-vconn.c \ > - tests/test-aa.c > + tests/test-aa.c \ > + tests/test-datalog.c > > if !WIN32 > tests_ovstest_SOURCES += \ > diff --git a/tests/datalog.at b/tests/datalog.at > new file mode 100644 > index 0000000..c12f308 > --- /dev/null > +++ b/tests/datalog.at > @@ -0,0 +1,11 @@ > +AT_BANNER([datalog]) > + > +AT_SETUP([datalog engine]) > +AT_KEYWORDS([ovn]) > + > +# test-datalog has a set of cases. run it alone to see its output. > +AT_CHECK([ovstest test-datalog test], [0], [ignore], [PASS > +]) > + > +AT_CLEANUP > + > diff --git a/tests/test-datalog.c b/tests/test-datalog.c > new file mode 100644 > index 0000000..f454a9b > --- /dev/null > +++ b/tests/test-datalog.c > @@ -0,0 +1,1662 @@ > +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. > + * > + * Licensed under the Apache License, Version 2.0 (the "License"); > + * you may not use this file except in compliance with the License. > + * You may obtain a copy of the License at: > + * > + * http://www.apache.org/licenses/LICENSE-2.0 > + * > + * Unless required by applicable law or agreed to in writing, software > + * distributed under the License is distributed on an "AS IS" BASIS, > + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > + * See the License for the specific language governing permissions and > + * limitations under the License. > + */ > + > +#include <config.h> > +#include "ovstest.h" > +#include "ovn/lib/datalog.h" > +#include "ovn/lib/datalog-private.h" > + > +static int32_t log_tst_no_cases_total = 0; > +static int32_t log_tst_no_cases_failed = 0; > + > +static void > +t_assert(bool r, const char* s) > +{ > + log_tst_no_cases_total++; > + if (r) return; > + log_tst_no_cases_failed++; > + printf("FAILED: %s\n",s); > +} > + > +static void > +t_assert1(bool r, const char* s) > +{ > + if (r) return; > + printf("FAILED: %s\n",s); > +} > + > +static void > +t_sum(void) > +{ > + printf(" test result %d/%d failed\n", > + log_tst_no_cases_failed, log_tst_no_cases_total); > +} > + > +/* ========================================================================== > + * TEST CASES > + * ========================================================================== > + */ > + > +struct test_align_s { > + int32_t a; > + /*char b; */ > + void* c[0]; > +}; > + > +static void > +gen_str(char* p, int32_t len) > +{ > + int32_t i; > + for (i = 0;i < len;i++) *p++ = (random() % 52) + 65; > + *p++ = 0; > +} > + > +static void > +test_collections(void) > +{ > + char buf[1024]; > + { > + /* null will be printed. */ > + char* p = NULL; > + printf("- hello log %s\n", p); > + > + struct test_align_s s; > + printf( > + "- size of types\n %p char(%lu) int(%lu) " > + "long(%lu) llong(%lu) %d %d\n", > + &s, sizeof(char), sizeof(int), sizeof(long), sizeof(long long), > + 0 /* (int)((char*)(&s.b) - (char*)(&s.a)) */, > + (int)((char*)s.c - (char*)(&s.a))); > + > + t_assert(sizeof(void*) >= sizeof(int32_t), "holding int in ptr"); > + /* nested variable with same name */ > + int i; for (i = 0;i < 5;i++) { > + int i; for (i = 0;i < 5;i++) { > + } > + } > + } > + > + { /* check integer could be stored as pointer */ > + void* p1; > + void* p2; > + > + int i; > + char* c1 = (char*)&p1; > + char* c2 = (char*)&p2; > + for (i = 0;i < sizeof(void*);i++) { > + c1[i] = 0x07; c2[i] = 0x70; > + } > + printf(" int as ptr %p %p\n", p1, p2); > + > + p1 = 0; p2 = 0; > + printf(" int as ptr %p %p\n", p1, p2); > + t_assert(p1 == p2, "int as ptr, compare 0 error"); > + t_assert(ptr2i(p1) == 0, "int as ptr, int 0 error"); > + > + p1 = i2ptr(-1); p2 = i2ptr(-1); > + printf(" int as ptr %p %p\n", p1, p2); > + t_assert(p1 == p2, "int as ptr, compare -1 error"); > + t_assert(ptr2i(p2) == -1, "int as ptr, int -1 error"); > + printf("- int as ptr\n"); > + } > + > + { /* check bitset */ > + bitset_t* bs = log_bitset_init(NULL); > + bitset_t* bs1 = log_bitset_init(NULL); > + > + t_assert(bs->size == 0, "bitmap, init size not 0"); > + bitset_set(bs, 0); > + bitset_set(bs, 31); > + t_assert(bs->size == 1, "bitmap, size not 1 after set bit 31"); > + bitset_set(bs, 32); > + t_assert(bs->size == 2, "bitmap, size not 2 after set bit 32"); > + > + t_assert(bitset_get( > + bs, 0) && bitset_get(bs, 31) && bitset_get(bs, 32), > + "bitmap, not set on bit 0, 31, 32"); > + > + bitset_set(bs1, 64); > + bitset_set(bs1, 31); > + bitset_and(bs, bs1); > + t_assert(!bitset_empty(bs) && !bitset_get(bs, 0) && > + bitset_get(bs, 31) && !bitset_get(bs, 32) && bs->size == 2, > + "bitmap, AND failed"); > + > + log_bitset_free(bs); > + log_bitset_free(bs1); > + printf("- bitmap\n"); > + } > + > + { /* check array */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + int i; > + array_t ary1; > + array_t* ary2; > + log_array_init(&ary1, 0, 0, gv); > + ary2 = log_array_init(NULL, 0, 5, gv); > + > + for (i = 0;i < 6;i++) { > + array_add(ary2, i2ptr(i)); > + array_add(&ary1, i2ptr(i * 10)); > + } > + t_assert(ary2->len == 10, "array realloc failed"); > + array_set(ary2, 2, i2ptr(20)); > + t_assert(ptr2i(array_get(ary2, 1)) == 1 && > + ptr2i(array_get(ary2, 5)) == 5 && > + ptr2i(array_get(ary2, 2)) == 20, > + "array get or set failed"); > + > + array_ins(&ary1, 0, i2ptr(1)); > + array_ins(&ary1, 1, i2ptr(2)); > + array_ins(&ary1, 8, i2ptr(100)); > + int32_t v = ptr2i(array_rmv(&ary1, 3)); > + t_assert(v == 10, "array insert and remove failed"); > + t_assert(array_size(&ary1) == 8, "array size failed"); > + t_assert(log_array_look_for(&ary1, i2ptr(100)) == 7, > + "array look for failed"); > + > + ARRAY_ALL_INT(&ary1, i) > + printf(" %d ", i); > + if (i == 40) ARRAY_EXIT; > + ARRAY_END > + log_array_free(&ary1); > + log_array_free(ary2); > + set_free(gv); > + printf("\n- array\n"); > + } > + > + { /* check hash code for string, int. check map */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + int32_t l1 = 0, l2 = 0; > + int32_t c1 = log_hash_code_byte("hello", &l1); > + int32_t c2 = log_hash_code_byte("world ", &l2); > + printf(" hash code len(%d) %x len(%d) %x\n", l1, c1, l2, c2); > + t_assert(l1 == 5 && l2 == 6, "hash code len"); > + > + map_t* map1 = map_init( > + NULL, KEY_ID(ENT_INT32) | VALUE_ID(ENT_INT32), 11, gv); > + /* all on same slot */ > + map_add(map1, i2ptr(5), i2ptr(6)); > + map_add(map1, i2ptr(16), i2ptr(17)); > + map_add(map1, i2ptr(27), i2ptr(28)); > + > + t_assert(ptr2i(map_get(map1, i2ptr(5))) == 6 && > + ptr2i(map_get(map1, i2ptr(16))) == 17 && > + ptr2i(map_get(map1, i2ptr(27))) == 28 && > + map_size(map1) == 3, > + "map size, get, or set"); > + > + t_assert(map1->len == 11, "map int 0"); > + int sz = 0; > + sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{27->28,16->17,5->6}") == 0, "map int 1"); > + > + /* check link operation */ > + t_assert(ptr2i(map_del(map1, i2ptr(16))) == 17, "map int 2"); > + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{27->28,5->6}") == 0, "map int 3"); > + > + t_assert(ptr2i(map_del(map1, i2ptr(5))) == 6, "map int 4"); > + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{27->28}") == 0, "map int 5"); > + > + map_add(map1, i2ptr(5), i2ptr(6)); > + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{5->6,27->28}") == 0, "map int 6"); > + t_assert(ptr2i(map_del(map1, i2ptr(5))) == 6, "map int 7"); > + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{27->28}") == 0, "map int 8"); > + > + t_assert(ptr2i(map_del(map1, i2ptr(27))) == 28, "map int 9"); > + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{}") == 0 && map_size(map1) == 0, > "map int 10"); > + > + /* break in map */ > + int32_t i = 0; > + map_add(map1, i2ptr(77), i2ptr(66)); > + map_add(map1, i2ptr(33), i2ptr(55)); > + map_add(map1, i2ptr(77), i2ptr(88)); > + t_assert(map_has(map1, i2ptr(33)), "map int 11a"); > + > + t_assert(ptr2i(map_get(map1, i2ptr(77))) == 88 && > + map_size(map1) == 2, "map int 11"); > + > + MAP_ALL(map1, node) > + node++; /* make no warning */ > + i++; > + MAP_EXIT; > + MAP_END > + t_assert(i == 1, "map int 12"); > + > + /* map rehash */ > + for (i = 0;i < 8192;i++) > + map_add(map1, i2ptr(i), i2ptr(i * 2)); > + > + printf(" map len = %d size = %d\n", map1->len, map1->size); > + t_assert(map_size(map1) == 8192, "map int 13"); > + for (i = 0;i < 8192;i++) > + t_assert1(ptr2i(map_get(map1, i2ptr(i))) == i * 2, > + "map set/get 14"); > + > + for (i = 0;i < 8192;i++) > + t_assert1(ptr2i(map_del(map1, i2ptr(i))) == i * 2, > + "map set/get 15"); > + printf(" map len = %d size = %d\n", map1->len, map1->size); > + t_assert(map_size(map1) == 0, "map int 16"); > + > + map_free(map1); > + set_free(gv); > + printf("- map, int and string\n"); > + } > + > + { /* check set on string */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + set_t set1; > + set_init(&set1, KEY_ID(ENT_STR), 0, gv); > + char* app1 = "apple"; > + char* app2 = malloc(strlen(app1) + 1); > + strcpy(app2, app1); > + t_assert(app1 != app2, "two apple string"); > + > + set_add(&set1, app1); > + set_add(&set1, app2); > + set_add(&set1, "cherry"); > + set_add(&set1, "kiwi"); > + // test_hash_dump(&set1); > + > + t_assert(set_has(&set1, app1) && set_has(&set1, app2) && > + set_get(&set1, app2) == app1 && > + set_size(&set1) == 3, "right apple"); > + > + int i = 0; > + SET_ALL(&set1, node, const char*) > + node++; /* make no warning */ > + i++; > + SET_EXIT; > + SET_END > + t_assert(i == 1, "set exit"); > + > + /* check set remove */ > + set_t* set2 = set_init(NULL, KEY_ID(ENT_INT32), 11, gv); > + t_assert(set2->len == 11, "set int 1"); > + > + set_add(set2, i2ptr(5)); > + set_add(set2, i2ptr(16)); > + set_add(set2, i2ptr(27)); > + // test_hash_dump(set2); > + int sz = 0; > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{27,16,5}") == 0, "set int 1"); > + > + SET_ALL_INT(set2, node) > + if (node == 16) SET_REMOVE_ITEM; > + SET_END > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{27,5}") == 0, "set int 3"); > + > + SET_ALL_INT(set2, node) > + if (node == 5) SET_REMOVE_ITEM; > + SET_END > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{27}") == 0, "set int 5"); > + > + set_add(set2, i2ptr(5)); > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{5,27}") == 0, "set int 6"); > + > + SET_ALL_INT(set2, node) > + if (node == 5) { SET_REMOVE_ITEM; SET_EXIT; } > + SET_END > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + sz = 0; t_assert(strcmp(buf, "{27}") == 0, "set int 8"); > + > + SET_ALL_INT(set2, node) > + if (node == 27) { SET_REMOVE_ITEM; SET_EXIT; } > + SET_END > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + t_assert(strcmp(buf, "{}") == 0 && set_size(set2) == 0, > "set int 10"); > + > + set_add(set2, 0); > + set_add(set2, i2ptr(5)); set_add(set2, i2ptr(16)); > + set_add(set2, i2ptr(27)); > + set_add(set2, i2ptr(6)); set_add(set2, i2ptr(17)); > + > + sz = 0; sz = log_hash_print(buf, sz, set2, true); buf[sz++] = 0; > + printf(" set: \n%s", buf); > + > + i = 0; > + SET_ALL_INT(set2, node) > + node++; /* make no warning */ > + if (i >= 2 && i <= 4) SET_REMOVE_ITEM; > + i++; > + SET_END > + > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + t_assert( > + strcmp(buf, "{0,27,6}") == 0 && set_size(set2) == 3, "set int 11"); > + > + SET_ALL_INT(set2, node) > + node++; /* make no warning */ > + SET_REMOVE_ITEM; > + SET_END > + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; > + t_assert( > + strcmp(buf, "{}") == 0 && set_size(set2) == 0, "set int 12"); > + > + free(app2); > + set_free(&set1); > + set_free(set2); > + set_free(gv); > + printf("- set, int and string\n"); > + } > + { /* check for hash code collision */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + map_t* map1 = map_init(NULL, KEY_ID(ENT_TST_INT32), 11, gv); > + map_add(map1, i2ptr(205), i2ptr(205)); > + map_add(map1, i2ptr(5), i2ptr(105)); > + map_add(map1, i2ptr(16), i2ptr(116)); > + map_add(map1, i2ptr(206), i2ptr(206)); > + > + int sz = 0; sz = log_hash_print(buf, sz, map1, true); buf[sz++] = 0; > + printf("%s", buf); > + > + t_assert(ptr2i(map_get(map1, i2ptr(205))) == 205 && > + ptr2i(map_get(map1, i2ptr(16))) == 116 && > + ptr2i(map_get(map1, i2ptr(5))) == 105, "hash, tst int32 1"); > + > + int i; > + for (i = 0;i < 1000;i++) > + map_add(map1, i2ptr(i), i2ptr((1000 + i))); > + > + printf(" map len = %d size = %d\n", map1->len, map_size(map1)); > + t_assert(map_size(map1) == 1000, "hash, tst int32 2"); > + > + for (i = 0;i < 1000;i++) > + t_assert1(ptr2i(map_get(map1, i2ptr(i))) == 1000 + i, > + "hash, tst int32 3"); > + > + for (i = 0;i < 1000;i++) > + t_assert1(ptr2i(map_del(map1, i2ptr(i))) == 1000 + i, > + "hash, tst int32 4"); > + > + printf(" map len = %d size = %d\n", map1->len, map_size(map1)); > + t_assert(map_size(map1) == 0, "hash, tst int32 5"); > + > + set_free(map1); > + set_free(gv); > + printf("- hash, code collision\n"); > + } > + { /* check large set */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + char* buf = malloc(20 * 100000); > + int i; > + for (i = 0;i < 100000;i++) > + gen_str(buf + i * 20, 19); > + > + set_t* set = set_init(NULL, KEY_ID(ENT_STR), 0, gv); > + printf(" gen set of strings\n"); > + for (i = 0;i < 100000;i++) set_add(set, buf + i * 20); > + printf(" add set of strings done\n"); > + > + t_assert(set_size(set) == 100000, "large set size"); > + printf(" set len = %d size = %d\n", set->len, set_size(set)); > + > + for (i = 0;i < 100000;i++) > + t_assert1(set_get(set, buf + i * 20) == buf + i * 20, > + "large set"); > + > + for (i = 0;i < 100000;i++) set_del(set, buf + i * 20); > + t_assert(set_size(set) == 0, "large set size 1"); > + > + free(buf); > + set_free(set); > + set_free(gv); > + printf("- set, str, large collection\n"); > + } > +} > + > +static void > +test_tables(void) > +{ > + char buf[1024]; > + { /* check value collection */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + log_value_t* nstr = log_value_init("", 0, gv); > + log_value_t* nhello = log_value_init("hello", 0, gv); > + char world[7] = "worlda"; > + log_value_t* nworld = log_value_init(world, 5, gv); > + > + char world1[5] = { 'a', 'b', 0, 'c', 'd' }; > + log_value_t* nabcd = log_value_init(world1, 5, gv); > + > + > + char hello1[5] = { 'h', 'e', 'l', 'l', 'o' }; > + log_value_t* nhello1 = log_value_init(hello1, 5, gv); > + log_value_t* nab = log_value_init("ab", 0, gv); > + > + char world2[5] = { 'a', 'b', 0, 'c', 'd' }; > + log_value_t* nabcd2 = log_value_init(world2, 5, gv); > + > + log_value_ref(nstr); > + t_assert(set_size(gv) == 5, "value set 1"); > + > + t_assert(nhello == nhello1 && nabcd == nabcd2 && > + nabcd != nab && strcmp(nworld->value.a, "world") == 0, > + "value set 2"); > + > + t_assert(nhello->ref_no == 2 && nabcd->ref_no == 2 && > + nab->ref_no == 1 && nworld->ref_no == 1 && nstr->ref_no == 2, > + "value set 3"); > + > + log_value_free(nhello, gv); > + log_value_free(nhello, gv); > + log_value_free(nabcd, gv); > + > + int sz = 0; sz = log_hash_print(buf, sz, gv, true); buf[sz++] = 0; > + printf("%s", buf); > + > + t_assert(set_size(gv) == 4 && > + set_has(gv, nabcd), "value set 4"); > + > + set_free(gv); > + printf("- value, ref_no\n"); > + } > + { /* check tuple */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + log_tuple_t* tuple1 = log_tuple_init_str("2:f0:f1:f2", ':', gv); > + log_tuple_t* tuple2 = log_tuple_init_str("10:f2:f0:f3:f1", ':', gv); > + > + int32_t sz = 0; > + sz = log_tuple_print(buf, sz, tuple1); > + buf[sz++] = '|'; > + sz = log_tuple_print(buf, sz, tuple2); > + buf[sz++] = 0; > + printf(" tuple %s %d\n", buf, sz); > + t_assert(strcmp("2:f0:f1:f2|10:f2:f0:f3:f1", buf) == 0 && sz == 26, > + "tuple str init"); > + > + sz = 0; sz = log_hash_print(buf, sz, gv, true); buf[sz++] = 0; > + printf("%s", buf); > + > + log_value_t* v_f2 = log_value_init("f2", 0, gv); > + t_assert(v_f2->ref_no == 3, "tuple ref"); > + log_tuple_free(tuple1, gv, true); > + log_tuple_free(tuple2, gv, true); > + > + sz = 0; sz = log_hash_print(buf, sz, gv, true); buf[sz++] = 0; > + printf("%s", buf); > + > + t_assert(v_f2->ref_no == 1 && set_size(gv) == 1, > + "tuple ref 1"); > + > + set_free(gv); > + printf("- tuple\n"); > + } > + { /* check table operation */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + array_t* key0 = log_array_init(NULL, 0, 0, gv); > + array_t* key01 = log_array_init(NULL, 0, 0, gv); > + array_add(key0, 0); > + array_add(key01, 0); array_add(key01, i2ptr(2)); > + > + log_int_tuple_t* ikey0 = log_int_tuple_init(key0); > + log_int_tuple_t* ikey01 = log_int_tuple_init(key01); > + log_array_free(key0); > + log_array_free(key01); > + > + log_value_t* val_a = log_value_init("a", 0, gv); > + log_table_t* tbl = log_table_init(NULL, 1, 3, 0, gv); > + > + log_tuple_t* tkab = log_tuple_init_str("0:a", ':', gv); > + log_tuple_t* tkab2 = log_tuple_init_str("0:a:b", ':', gv); > + > + log_tuple_t* tuple1 = log_tuple_init_str("1:a:c:b", ':', gv); > + log_tuple_t* tuple2 = log_tuple_init_str("2:a:e:b", ':', gv); > + log_tuple_t* tuple21 = log_tuple_init_str("2:a:e:b", ':', gv); > + log_tuple_t* tuple3 = log_tuple_init_str("3:a:f:d", ':', gv); > + log_tuple_t* tuple4 = log_tuple_init_str("4:b:g:d", ':', gv); > + > + /* create index later */ > + log_table_add(tbl, tuple1); log_table_add(tbl, tuple2); > + log_table_add(tbl, tuple3); log_table_add(tbl, tuple4); > + log_table_remove(tbl, tuple2); > + > + t_assert(val_a->ref_no == 6, "table index c0"); > + > + int sz = 0; > + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) > + sz = log_tuple_print(buf, sz, tuple); > + buf[sz++] = '|'; > + SET_END > + buf[sz++] = 0; > + printf(" table tuples: %s\n", buf); > + t_assert(strcmp(buf, "4:b:g:d|1:a:c:b|3:a:f:d|") == 0, > + "table index 0"); > + > + log_table_add(tbl, tuple21); > + int32_t i0 = log_table_add_index(tbl, ikey0); > + log_tuple_t* set = log_index_get_index(tbl, tkab, i0); > + > + sz = 0; > + INDEX_ALL(set, i0, t) > + sz = log_tuple_print(buf, sz, t); > + buf[sz++] = '|'; > + INDEX_END > + buf[sz++] = 0; > + > + printf(" search table on index0 (a): %s\n", buf); > + t_assert(strcmp(buf, "2:a:e:b|3:a:f:d|1:a:c:b|") == 0, > + "table index 1"); > + > + int32_t i01 = log_table_add_index(tbl, ikey01); > + log_int_tuple_free(ikey0); > + log_int_tuple_free(ikey01); > + > + set = log_index_get_index(tbl, tkab2, i01); > + sz = 0; sz = log_index_print(buf, sz, tbl); buf[sz++] = 0; > + printf("%s", buf); > + > + sz = 0; > + INDEX_ALL(set, i01, t) > + sz = log_tuple_print(buf, sz, t); > + buf[sz++] = '|'; > + INDEX_END > + buf[sz++] = 0; > + printf(" search table on index02 (a, b): %s\n", buf); > + t_assert(strcmp(buf, "2:a:e:b|1:a:c:b|") == 0, "table index 2"); > + > + printf(" add tuple when index is defined\n"); > + log_tuple_t* tuple5 = log_tuple_init_str("5:c:f:d", ':', gv); > + log_tuple_t* tuple6 = log_tuple_init_str("6:b:h:d", ':', gv); > + log_tuple_t* tuple7 = log_tuple_init_str("7:a:s:b", ':', gv); > + log_table_add(tbl, tuple5); log_table_add(tbl, tuple6); > + log_table_add(tbl, tuple7); > + > + sz = 0; sz = log_index_print(buf, sz, tbl); buf[sz++] = 0; > + printf("%s", buf); > + > + printf(" del tuple when index is defined\n"); > + log_table_remove(tbl, tuple1); > + log_table_remove(tbl, tuple21); > + log_table_remove(tbl, tuple7); > + t_assert(val_a->ref_no == 4, "table index c1"); > + > + sz = 0; sz = log_index_print(buf, sz, tbl); buf[sz++] = 0; > + printf("%s", buf); > + > + sz = 0; > + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) > + sz = log_tuple_print(buf, sz, tuple); > + buf[sz++] = '|'; > + SET_END > + buf[sz++] = 0; > + printf(" table tuples: %s\n", buf); > + t_assert(strcmp(buf, "4:b:g:d|6:b:h:d|3:a:f:d|5:c:f:d|") == 0, > + "table index 3"); > + > + log_table_free(tbl); > + log_tuple_free(tkab, gv, true); > + log_tuple_free(tkab2, gv, true); > + log_value_free(val_a, gv); > + > + t_assert(set_size(gv) == 0, "table values released"); > + set_free(gv); > + printf("- tables and tuples\n"); > + } > +} > + > +static void > +test_sort(void) > +{ > + char buf[1024]; > + > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + map_t right1, right2, rules; > + set_init(&right1, KEY_ID(ENT_VALUE), 0, gv); > + set_init(&right2, KEY_ID(ENT_VALUE), 0, gv); > + map_init(&rules, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_SET), 0, gv); > + > + set_add(&right1, log_value_init("1", 0, gv)); > + set_add(&right2, log_value_init("1", 0, gv)); > + set_add(&right2, log_value_init("2", 0, gv)); > + > + map_add(&rules, log_value_init("2", 0, gv), &right1); > + map_add(&rules, log_value_init("3", 0, gv), &right2); > + > + int sz = 0; sz = log_hash_print(buf, sz, &rules, false); buf[sz++] = 0; > + printf(" G=%s\n", buf); > + > + array_t order; > + map_t in_nodes, out_nodes; > + log_topo_sort(&rules, &order, &in_nodes, &out_nodes); > + > + sz = 0; > + sz = log_coll_print(buf, sz, &order, ENT_ARRAY, false); > + sz = log_coll_print(buf, sz, &in_nodes, ENT_SET, false); > + sz = log_coll_print(buf, sz, &out_nodes, ENT_SET, false); > + buf[sz++] = 0; > + > + t_assert(strcmp(buf, "[1,2,3]{1}{3}") == 0, "topo sort 1"); > + log_array_free(&order); > + set_free(&in_nodes); > + set_free(&out_nodes); > + > + array_t ary1, ary2, ary3; > + log_array_init(&ary1, KEY_ID(ENT_INT32), 0, NULL); > + log_array_init(&ary2, KEY_ID(ENT_INT32), 0, NULL); > + log_array_init(&ary3, KEY_ID(ENT_INT32), 0, NULL); > + > + int v1[] = { 9, 7, 9, 7, 5 }; > + int v2[] = { 1, 2, 3, 4, 5 }; > + int v3[] = { 5, 4, 3, 2, 1 }; > + > + int i; > + for (i = 0;i < 5;i++) { > + array_add(&ary1, i2ptr(v1[i])); > + array_add(&ary2, i2ptr(v2[i])); > + array_add(&ary3, i2ptr(v3[i])); > + } > + log_sort_array(0, &ary1, &ary2, &ary3); > + > + sz = 0; > + int iv1 = log_insert_item(8, &ary1, i2ptr(6), i2ptr(10), &ary2, &ary3); > + sz = log_array_print(buf, sz, &ary1, false); > + sz = log_array_print(buf, sz, &ary2, false); > + sz = log_array_print(buf, sz, &ary3, false); > + int iv2 = log_insert_item(8, &ary1, i2ptr(7), i2ptr(11), &ary2, &ary3); > + sz = log_array_print(buf, sz, &ary1, false); > + sz = log_array_print(buf, sz, &ary2, false); > + sz = log_array_print(buf, sz, &ary3, false); > + buf[sz++] = 0; > + printf(" sort: %s\n", buf); > + > + t_assert(iv1 == 3 && iv2 == 4 && strcmp(buf, > + "[5,7,7,8,9,9][5,2,4,6,1,3][1,4,2,10,5,3]" > + "[5,7,7,8,8,9,9][5,2,4,6,7,1,3][1,4,2,10,11,5,3]") == 0, "sort insert"); > + > + log_array_free(&ary1); log_array_free(&ary2); log_array_free(&ary3); > + set_free(gv); > + printf("- sort and insert\n"); > +} > + > +static void > +test_sync(void) > +{ > + char buf[1024]; > + { > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + log_sync_init( > + "table(a,b) : x(a,'aa')\n" > + "# comment \n" > + "y(b,-, -) ; z(aparam) > y(x) \n" > + ". # last line" > + ,gv); > + > + TYPE2(map_t, log_value_t*, array_t*) map; > + log_sync_parse(&map); > + > + int sz = 0; > + sz = log_hash_print(buf, sz, &map, false); buf[sz++] = 0; > + printf(" sync %s\n", buf); > + > + t_assert(0 == strcmp(buf, > + "{z->[[u,z,t,aparam],[,y,t,x]]," > + "table->[[j,table,t,a,t,b],[,x,t,a,s,aa],[,y,t,b,-,,-,]]}"), > + "sync check 1"); > + map_free(&map); > + set_free(gv); > + printf("- log syntax\n"); > + } > + { > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + log_sync_init( > + /* > + "Xx2(a) : x1(a,'aa'); X3(a) : x1(a, b) Xx2(b); _EXTERNAL(x1, func)." > + // input, output cannot be external > + "Xx2(a) : x1(a,'aa'); X3(a) : x1(a, b) Xx2(b); _EXTERNAL(X5, func)." > + // invalid external > + "X2(a) : X1(a,'aa'); X3(a) : X1(a, b, c) X2(b)." > + // upper / lower case wrong > + "X2(a, b) : x2(b) x1(a) x1(a) x1(a)." // join more than twice > + "X2(a) > x1(a) x1(a) x1(a)." // union more than twice > + "X2(a, b) > x1(a) x1(a, b)." // table size mismatch > + "X2(a, b) : x1(a) x2(a, b); X3(v) : x2(z)." // table size mismatch > + "X2(a, 'aa') > x1(a)." // constant not allowed in left > + "X2(a, -) : x1(a)." // ignored not allowed in left > + "X2(a, b) > x1(a, c)." // union param not found > + "X2(a) : x1(a, c)." // param not used should be 'ignored' > + "X2(a, z) : x1(a, -)." // param not defined > + */ > + "Xx2(a) : x1(a,'aa', -); X3(a) : Xx2(a) x1(a, -, b) > Xx2(b)."// correct > + /* > + " Span(ls_id, host_id) : ls(ls_id, vif) vif_place(host_id, vif); \n" > + " HOST_LS(host1, ls, host2) : Span(ls, host1) Span(ls, host2). " > + */ > + , gv); > + > + TYPE2(map_t, log_value_t*, array_t*) map; > + log_sync_parse(&map); > + > + int sz; > + sz = 0; sz = log_hash_print(buf, sz, &map, false); buf[sz++] = 0; > + printf(" sync %s\n", buf); > + > + log_rule_set_t rs; > + log_rule_set_init(&rs, gv); > + log_sem_process(&rs, &map); > + sz = 0; sz = log_rule_set_print(buf, sz, &rs); buf[sz++] = 0; > + printf(" sem=\n%s\n", buf); > + > + log_rule_set_free(&rs); > + map_free(&map); > + set_free(gv); > + printf("- log semantics\n"); > + } > +} > + > +static void > +test_join(void) > +{ > + char buf[1024]; > + { > + /* do join table 2 */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + log_table_t* tbl1 = log_table_init(NULL, 0, 3, 0, gv); > + log_table_t* tbl2 = log_table_init(NULL, 1, 4, 0, gv); > + > + log_table_add(tbl1, log_tuple_init_str("2 :1:2:7", ':', gv)); > + log_table_add(tbl1, log_tuple_init_str("1 :1:3:8", ':', gv)); > + log_table_add(tbl1, log_tuple_init_str("1 :1:5:9", ':', gv)); > + > + log_table_add(tbl2, log_tuple_init_str("1 :3:1:2:0", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1 :4:1:2:0", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("9 :4:1:5:0", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1 :4:1:5:1", ':', gv)); > + > + TYPE(array_t*, int32_t) ints = > + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); > + array_add(ints, i2ptr(1)); > + array_add(ints, i2ptr(2)); array_add(ints, i2ptr(3)); > + log_int_tuple_t* key0 = log_int_tuple_init(ints); > + log_array_free(ints); > + > + log_join_param_t jps; > + log_join_param_t* jp = log_join_param_init(&jps, key0, gv); > + > + array_add(&jp->select1i, 0); > + array_add(&jp->select1i, i2ptr(1)); > + array_add(&jp->select1i, 0); > + array_add(&jp->select1, NULL); > + array_add(&jp->select1, NULL); array_add(&jp->select1, > + log_value_init("0", 0, gv)); > + > + array_add(&jp->rem1, 0); > + array_add(&jp->rem2, 0); > + > + log_table_t* tbl3 = log_tblopr_join(tbl1, tbl2, jp); > + log_join_param_free(jp); > + > + int sz = 0; sz = log_table_print(buf, sz, tbl2, true); buf[sz++] = 0; > + printf(" cond join tbl2 index=%s\n", buf); > + sz = 0; sz = log_table_print(buf, sz, tbl3, false); buf[sz++] = 0; > + printf(" cond join result=%s\n", buf); > + t_assert(0 == strcmp(buf, "{2:1:3,3:1:4}"), "cond join 1"); > + > + log_table_free(tbl1); > + log_table_free(tbl2); > + log_table_free(tbl3); > + set_free(gv); > + } > + { > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + > + log_table_t* tbl1 = log_table_init(NULL, 0, 1, 0, gv); > + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); > + > + log_table_add(tbl1, log_tuple_init_str("2 :1", ':', gv)); > + log_table_add(tbl1, log_tuple_init_str("1 :2", ':', gv)); > + > + log_table_add(tbl2, log_tuple_init_str("8 :3:4:1", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1 :3:5:0", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1 :5:4:1", ':', gv)); > + > + TYPE(array_t*, int32_t) ints = log_array_init( > + NULL, KEY_ID(ENT_INT32), 0, gv); > + array_add(ints, i2ptr(2)); > + log_int_tuple_t* key0 = log_int_tuple_init(ints); > + log_array_free(ints); > + > + log_join_param_t jps; > + log_join_param_t* jp = log_join_param_init(&jps, key0, gv); > + > + array_add(&jp->select1i, 0); > + array_add(&jp->select1, log_value_init("1", 0, gv)); > + > + array_add(&jp->rem1, i2ptr(0)); > + array_add(&jp->rem2, i2ptr(1)); > + > + log_table_t* tbl3 = log_tblopr_join(tbl1, tbl2, jp); > + log_join_param_free(jp); > + > + int sz = 0; sz = log_table_print(buf, sz, tbl2, true); buf[sz++] = 0; > + printf(" join tbl2 index=%s\n", buf); > + sz = 0; sz = log_table_print(buf, sz, tbl3, false); buf[sz++] = 0; > + printf(" full join result=%s\n", buf); > + t_assert(0 == strcmp(buf, "{2:2:4,4:1:4}"), "full join 1"); > + > + log_table_free(tbl1); > + log_table_free(tbl2); > + log_table_free(tbl3); > + set_free(gv); > + } > + { > + /* do union */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value( gv); > + > + log_engine_t log; > + log_engine_init(&log, gv); > + > + log_sync_init("X(a0, a2, a1) : x(a2, a1, -, 'cst', a0) .", gv); > + TYPE2(map_t, log_value_t*, array_t*) map; > + > + log_sync_parse(&map); > + log_sem_process(&log.rule_set, &map); > + map_free(&map); > + > + log_table_t* tbl1 = log_table_init(NULL, 0, 5, 0, gv); > + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); > + > + log_table_add(tbl1, log_tuple_init_str("1:a:b:c:cst:d", ':', gv)); > + log_table_add(tbl1, log_tuple_init_str("1:a:b:c:dst:d", ':', gv)); > + log_table_add(tbl1, log_tuple_init_str("1:a:b:d:cst:d", ':', gv)); > + log_table_add(tbl1, log_tuple_init_str("1:a:b:d:cst:e", ':', gv)); > + > + log_eng_do_union(&log, tbl1, tbl2); > + > + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); > buf[sz++] = 0; > + printf(" union = %s\n", buf); > + t_assert(0 == strcmp(buf, "{1:e:a:b,2:d:a:b}"), "union 1"); > + > + log_table_free(tbl1); log_table_free(tbl2); > + log_engine_free(&log); > + set_free(gv); > + } > + { /* do self join 1 */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = > + log_eng_parse( > + "X(b, a, c) : x(a, b, 'v', -, c) x(b, 'w', -, a, c) .", gv); > + > + log_table_t* tbl0_org = map_get(&eng->tables, 0); > + log_table_add(tbl0_org, log_tuple_init_str("1:a:b:c:d:e", ':', gv)); > + log_table_add(tbl0_org, log_tuple_init_str("1:a:y:v:a:e", ':', gv)); > + > + log_table_t* tbl1 = log_table_init(NULL, 0, 5, 0, gv); > + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); > + log_table_add(tbl1, log_tuple_init_str("1:w:w:v:w:e", ':', gv)); > + > + log_eng_do_join(eng, tbl1, tbl2); > + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); > buf[sz++] = 0; > + printf(" self join simple match = %s\n", buf); > + t_assert(0 == strcmp(buf, "{1:w:w:e}"), "self join 1"); > + > + log_table_free(tbl1); log_table_free(tbl2); > + log_engine_free(eng); > + set_free(gv); > + } > + { > + // do self join 2 > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = > + log_eng_parse("X(a, b, c) : x(a, b) x(a, c) .", gv); > + > + log_table_t* tbl0_o = map_get(&eng->tables, 0); > + log_table_t* tbl0 = log_table_init(NULL, 0, 2, 0, gv); > + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); > + log_table_add(tbl0, log_tuple_init_str("1:1:1", ':', gv)); > + log_table_add(tbl0, log_tuple_init_str("1:1:2", ':', gv)); > + log_table_add(tbl0, log_tuple_init_str("1:2:1", ':', gv)); > + > + log_eng_do_join(eng, tbl0, tbl2); > + > + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); > buf[sz++] = 0; > + printf(" self join simple initial = %s\n", buf); > + t_assert(0 == strcmp(buf, > + "{1:2:1:1,1:1:1:2,1:1:1:1,1:1:2:2,1:1:2:1}"), > + "self join simple init 1"); > + > + log_table_add(tbl0_o, log_tuple_init_str("1:1:1", ':', gv)); > + log_table_add(tbl0_o, log_tuple_init_str("1:1:2", ':', gv)); > + log_table_add(tbl0_o, log_tuple_init_str("1:2:1", ':', gv)); > + > + log_table_t* tbl_0 = log_table_init(NULL, 0, 2, 0, gv); > + log_table_t* tbl_2 = log_table_init(NULL, 1, 3, 0, gv); > + log_table_add(tbl_0, log_tuple_init_str("1:1:3", ':', gv)); > + > + log_eng_do_join(eng, tbl_0, tbl_2); > + sz = 0; sz = log_table_print(buf, sz, tbl_2, false); buf[sz++] = 0; > + printf(" self join simple delta = %s\n", buf); > + t_assert(0 == strcmp(buf, > + "{1:1:1:3,1:1:3:2,1:1:3:3,1:1:2:3,1:1:3:1}"), > + "self join simple delta 1"); > + > + log_table_free(tbl_0); log_table_free(tbl_2); > + log_table_free(tbl0); log_table_free(tbl2); > + log_engine_free(eng); > + set_free(gv); > + } > + { > + // do join 3 > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = > + log_eng_parse( > + "R(h, b, c, d) : h(h, p1, p2) p1(p1, -, b) p2(p2, d, c) .",gv); > + > + log_table_t* tbl0_h = map_get(&eng->tables, i2ptr(0)); > + log_table_t* tbl0_p1 = map_get(&eng->tables, i2ptr(1)); > + log_table_t* tbl0_p2 = map_get(&eng->tables, i2ptr(2)); > + > + log_table_add(tbl0_h, log_tuple_init_str("1:h1:px:py", ':', gv)); > + log_table_add(tbl0_p1, log_tuple_init_str("1:p1:a1:b", ':', gv)); > + log_table_add(tbl0_p1, log_tuple_init_str("1:p1:a2:b", ':', gv)); > + log_table_add(tbl0_p2, log_tuple_init_str("1:p2:c:d", ':', gv)); > + > + log_table_t* tbl1 = log_table_init(NULL, 0, 3, 0, gv); > + log_table_t* tbl2 = log_table_init(NULL, 3, 4, 0, gv); > + log_table_add(tbl1, log_tuple_init_str("1:h1:p1:p2", ':', gv)); > + > + log_eng_do_join(eng, tbl1, tbl2); > + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); > buf[sz++] = 0; > + printf(" join 3 simple = %s\n", buf); > + t_assert(0 == strcmp(buf, "{2:h1:b:d:c}"), > + "join 3 simple"); > + > + log_table_free(tbl1); log_table_free(tbl2); > + log_engine_free(eng); > + set_free(gv); > + } > + printf("- join and union\n"); > +} > + > +static bool > +test_ext_func(log_engine_t* eng, log_table_t* inp, log_table_t* del_out, > + log_table_t* add_out) > +{ > + char buf[256]; > + > + if (inp == NULL) return true; /* reset state */ > + log_value_t* tn = map_get( > + &eng->rule_set.rule_name_map, i2ptr(del_out->table_index)); > + if (strcmp(tn->value.a, "Mm2") != 0) return false; > + > + int sz = 0; sz = log_table_print(buf, sz, inp, false); buf[sz] = 0; > + printf(" run ext inp = %s\n", buf); > + > + log_table_t* output = inp->is_remove ? del_out : add_out; > + log_value_t* tv[1]; > + > + SET_ALL(&inp->tuples, tuple, log_tuple_t*) > + int sz = sprintf(buf, "aa%scc", tuple->values[0]->value.a); > + log_value_t* nv = log_value_init(buf, sz, inp->m.glb_values); > + tv[0] = nv; > + > + log_tuple_t* nt = log_tuple_init_val(tv, 1); > + log_table_add(output, nt); > + SET_END > + > + sz = 0; sz = log_table_print(buf, sz, output, false); buf[sz] = 0; > + printf(" run ext inp remove = %s\n", inp->is_remove ? "true" :"false"); > + printf(" run ext out = %s\n", buf); > + return true; > +} > + > +static void > +test_delta(void) > +{ > + char buf[1024]; > + { > + /* delta with external function */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = > + log_eng_parse("M(a) : inp(a) Mm2(a); Mm2(a) : i(a) .", gv); > + /* {0=inp, 1=i, 2=Mm2, 3=M} */ > + > + log_eng_set_ext_func(eng, test_ext_func); > + log_table_t* tbl1 = map_get(&eng->tables, 0); > + log_table_t* tbl2 = log_table_init(NULL, 1, 1, 0, gv); > + > + log_table_add(tbl1, log_tuple_init_str("1:aaVbbcc", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1:bb", ':', gv)); > + > + TYPE(array_t*, log_table_t*) inp_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + array_add(inp_insert, tbl2); > + > + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); > + int sz = 0; sz = log_array_print(buf, sz, res, false); buf[sz] = 0; > + printf(" delta ext empty=%s\n", buf); > + t_assert(array_size(res) == 0, "delta ext func 0"); > + > + log_array_free(res); > + log_array_free(inp_remove); log_array_free(inp_insert); > + log_engine_free(eng); set_free(gv); > + } > + { > + /* delta with external function */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = > + log_eng_parse("M(a) : inp(a) Mm2(a); Mm2(a) : i(a) .", gv); > + /* {0=inp, 1=i, 2=Mm2, 3=M} */ > + > + log_eng_set_ext_func(eng, test_ext_func); > + log_table_t* tbl1 = map_get(&eng->tables, 0); > + log_table_t* tbl2 = log_table_init(NULL, 1, 1, 0, gv); > + > + log_table_add(tbl1, log_tuple_init_str("1:aabbcc", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1:bb", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1:ee", ':', gv)); > + > + TYPE(array_t*, log_table_t*) inp_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + array_add(inp_insert, tbl2); > + > + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); > + int sz = 0; sz = log_array_print(buf, sz, res, false); buf[sz] = 0; > + printf(" delta ext=%s\n", buf); > + t_assert(0 == strcmp(buf, "[{1:aabbcc}]"), "delta ext 1"); > + > + sz = 0; > + sz = log_hash_print(buf, sz, &eng->tables, false); buf[sz] = 0; > + printf(" all tables=%s\n", buf); > + > + log_table_t* tbl2d = log_table_init(NULL, 1, 1, 0, gv); > + tbl2d->is_remove = true; > + log_table_add(tbl2d, log_tuple_init_str("1:bb", ':', gv)); > + > + TYPE(array_t*, log_table_t*) inp1_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp1_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + array_add(inp1_remove, tbl2d); > + > + array_t* res1 = log_eng_delta(eng, inp1_remove, inp1_insert); > + sz = 0; sz = log_array_print(buf, sz, res1, false); buf[sz] = 0; > + printf(" delta ext rmv=%s\n", buf); > + t_assert(0 == strcmp(buf, "[{1:aabbcc}]") && > + ((log_table_t*)array_get(res1, 0))->is_remove == true, > + "delta ext rmv 1"); > + > + sz = 0; > + sz = log_hash_print(buf, sz, &eng->tables, false); buf[sz] = 0; > + > + log_array_free(res); log_array_free(inp_remove); > + log_array_free(inp_insert); > + log_array_free(res1); log_array_free(inp1_remove); > + log_array_free(inp1_insert); > + > + log_engine_free(eng); set_free(gv); > + printf(" all tables=%s\n", buf); > + } > + printf("- delta and ext func\n"); > +} > + > +static void > +test_delta_misc(void) > +{ > + char buf[1024]; > + { > + /* input's counter should be ignored */ > + int sz; > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = > + log_eng_parse("Xy(a) : x(a, -); Xy1(a) : Xy(a); Y(a) : > Xy1(a) .", gv); > + > + log_table_t* tbl1 = log_table_init(NULL, 0, 2, 0, gv); /* tbl x */ > + log_table_add(tbl1, log_tuple_init_str("1:1:1", ':', gv)); > + log_table_add(tbl1, log_tuple_init_str("1:1:2", ':', gv)); > + > + TYPE(array_t*, log_table_t*) inp_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + array_add(inp_insert, tbl1); > + > + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); > + sz = 0; > + sz = log_hash_print(buf, sz, &eng->tables, false); buf[sz] = 0; > + > + printf(" delta, counter %s\n", buf); > + t_assert(0 == strcmp(buf, "{0->{1:1:1,1:1:2},1->{2:1},2-> {1:1}}"), > + "delta, counter value 1"); > + > + log_array_free(res); log_array_free(inp_remove); > + log_array_free(inp_insert); > + > + log_table_t* tbl2 = log_table_init(NULL, 0, 2, 0, gv); /* tbl x */ > + tbl2->is_remove = true; > + log_table_add(tbl2, log_tuple_init_str("1:1:1", ':', gv)); > + log_table_add(tbl2, log_tuple_init_str("1:1:2", ':', gv)); > + > + TYPE(array_t*, log_table_t*) inp1_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp1_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + array_add(inp1_remove, tbl2); > + > + array_t* res1 = log_eng_delta(eng, inp1_remove, inp1_insert); > + /* no assert issue when minus */ > + sz = 0; sz = log_array_print(buf, sz, res1, false); buf[sz] = 0; > + > + t_assert(0 == strcmp(buf, "[{1:1}]"), "delta, counter value 2"); > + printf(" delta, counter minus%s\n", buf); > + > + log_array_free(res1); log_array_free(inp1_remove); > + log_array_free(inp1_insert); > + log_engine_free(eng); set_free(gv); > + } > + { > + /* merge add and remove for normal join */ > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = > + log_eng_parse("A(a) : b(a, b) c(b) .", gv); > + > + log_table_t* tbl1 = log_table_init(NULL, 1, 1, 0, gv); /* tbl c */ > + log_table_add(tbl1, log_tuple_init_str("1:1", ':', gv)); > + > + TYPE(array_t*, log_table_t*) inp_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + array_add(inp_insert, tbl1); > + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); > + > + /* when testing, should reorder the 2 doJoin in delta() */ > + log_table_t* tbl2 = log_table_init(NULL, 1, 1, 0, gv); /* tbl c */ > + log_table_t* tbl3 = log_table_init(NULL, 0, 2, 0, gv); /* tbl b */ > + > + tbl2->is_remove = true; > + log_table_add(tbl2, log_tuple_init_str("1:1", ':', gv)); > + log_table_add(tbl3, log_tuple_init_str("1:2:1", ':', gv)); > + > + TYPE(array_t*, log_table_t*) inp1_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp1_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + > + array_add(inp1_remove, tbl2); > + array_add(inp1_insert, tbl3); > + array_t* res1 = log_eng_delta(eng, inp1_remove, inp1_insert); > + > + int sz = 0; sz = log_hash_print(buf, sz, &eng->tables, false); > + buf[sz] = 0; > + printf(" merge add and remove %s\n", buf); > + t_assert(array_size(res1) == 0, "merge add and remove"); > + > + log_array_free(res); log_array_free(inp_remove); > + log_array_free(inp_insert); > + log_array_free(res1); log_array_free(inp1_remove); > + log_array_free(inp1_insert); > + log_engine_free(eng); set_free(gv); > + } > + printf("- delta, misc\n"); > +} > + > +static void > +test_delta_more(void) > +{ > + > + const char* rules = > + > +"LS_host_set(ls_id, host_id) : logical_switch(ls_id, port_id) " > +"port_bind(port_id, vif_id) dest_place(host_id, vif_id); " > +"LS_HOST_SET(host_id, ls_id, host_id_item) : LS_host_set(ls_id, host_id) " > +"LS_host_set(ls_id, host_id_item); " > +"LS_HOST_TUNNEL(host_id, host_id_item, tunnel_ip) : " > +"LS_host_set(ls_id, host_id) " > +"LS_host_set(ls_id, host_id_item) dest_tunnel(host_id_item,tunnel_ip) . "; > + > + const char* d1[] = { > + "+:1:logical_switch", "1:ls1:lp1", > + "+:1:port_bind", "1:lp1:vif1", > + "+:1:dest_place", "1:h1:vif1", > + "+:1:dest_tunnel", "1:h1:ip1" > + }; > + > + const char* d2[] = { > + "+:1:port_bind", "1:lp2:vif2", > + "+:1:dest_place", "1:h2:vif2", > + "+:1:logical_switch", "1:ls1:lp2", > + "+:1:dest_tunnel", "1:h2:ip2" > + }; > + > + const char* d3[] = { > + "-:1:dest_place", "1:h2:vif2", > + }; > + > + const char* d4[] = { > + "-:1:logical_switch", "1:ls1:lp1", > + }; > + > + const char* d5[] = { > + "-:1:logical_switch", "1:ls1:lp2", > + "+:2:port_bind", "1:lp1:vif1", "1:lp2:vif2", > + "+:1:dest_place", "1:h1:vif1", > + "+:2:dest_tunnel", "1:h1:ip1", "1:h2:ip2" > + }; > + > + char buf[1024]; > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = log_eng_parse(rules, gv); > + > + TYPE(array_t*, log_table_t*) inp_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + log_io_parse_line(NULL, NULL, false, NULL, NULL); > + > + int i; > + for (i = 0;i < sizeof d1 / sizeof(char*);i++) > + t_assert1(log_io_parse_line(d1[i], eng, false, inp_remove, inp_insert), > + "delta more 0"); > + > + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); > + int sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; > + > + t_assert(0 == strcmp(buf, > + "+:1:LS_HOST_TUNNEL\n1:h1:h1:ip1\n+:1:LS_HOST_SET\n1:h1:ls1:h1 \n"), > + "delta more 1"); > + > + log_array_free(inp_remove); log_array_free(inp_insert); > + log_array_free(res); > + > + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); > + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); > + log_io_parse_line(NULL, NULL, false, NULL, NULL); > + > + for (i = 0;i < sizeof d2 / sizeof(char*);i++) > + t_assert1(log_io_parse_line(d2[i], eng, false, inp_remove, inp_insert), > + "delta more 2"); > + > + res = log_eng_delta(eng, inp_remove, inp_insert); > + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; > + > + t_assert(0 == strcmp(buf, > + "+:3:LS_HOST_TUNNEL\n1:h2:h1:ip1\n1:h1:h2:ip2\n1:h2:h2:ip2\n" > + "+:3:LS_HOST_SET\n1:h2:ls1:h2\n1:h1:ls1:h2\n1:h2:ls1:h1\n"), > + "delta more 3"); > + > + log_array_free(inp_remove); log_array_free(inp_insert); > + log_array_free(res); > + > + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); > + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); > + log_io_parse_line(NULL, NULL, false, NULL, NULL); > + > + for (i = 0;i < sizeof d3 / sizeof(char*);i++) > + t_assert1(log_io_parse_line(d3[i], eng, false, inp_remove, inp_insert), > + "delta more 4"); > + > + res = log_eng_delta(eng, inp_remove, inp_insert); > + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; > + > + t_assert(0 == strcmp(buf, > + "-:3:LS_HOST_TUNNEL\n1:h2:h1:ip1\n1:h1:h2:ip2\n1:h2:h2:ip2\n" > + "-:3:LS_HOST_SET\n1:h2:ls1:h2\n1:h1:ls1:h2\n1:h2:ls1:h1\n" > + ), "delta more 5"); > + > + log_array_free(inp_remove); log_array_free(inp_insert); > + log_array_free(res); > + > + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); > + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); > + log_io_parse_line(NULL, NULL, false, NULL, NULL); > + > + for (i = 0;i < sizeof d4 / sizeof(char*);i++) > + t_assert1(log_io_parse_line(d4[i], eng, false, inp_remove, inp_insert), > + "delta more 5"); > + > + res = log_eng_delta(eng, inp_remove, inp_insert); > + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; > + > + t_assert(0 == strcmp(buf, > + "-:1:LS_HOST_TUNNEL\n1:h1:h1:ip1\n-:1:LS_HOST_SET\n1:h1:ls1:h1\n" > + ), "delta more 6"); > + > + log_array_free(inp_remove); log_array_free(inp_insert); > + log_array_free(res); > + > + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); > + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); > + log_io_parse_line(NULL, NULL, false, NULL, NULL); > + > + for (i = 0;i < sizeof d5 / sizeof(char*);i++) > + t_assert1(log_io_parse_line(d5[i], eng, false, inp_remove, inp_insert), > + "delta more 6"); > + > + res = log_eng_delta(eng, inp_remove, inp_insert); > + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; > + > + t_assert(0 == strcmp(buf,""), "delta more 7"); > + log_array_free(inp_remove); log_array_free(inp_insert); > + log_array_free(res); > + > + log_engine_free(eng); > + set_free(gv); > + printf("- delta, more\n"); > +} > + > +static int64_t > +calc_tm(struct timespec* ts0, struct timespec* ts1) > +{ > + int64_t t0 = ts0->tv_sec * 1000 * 1000L + ts0->tv_nsec / 1000; > + int64_t t1 = ts1->tv_sec * 1000 * 1000L + ts1->tv_nsec / 1000; > + return t1 - t0; > +} > + > +static void > +test_join_perf(int32_t sz1, int32_t sz2) > +{ > + /* correct value for full join is sz1 * sz2 * sz2 > + * correct value for delta join is sz2 * 2 + 1 > + */ > + > + struct timespec ts0, ts1; > + > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = log_eng_parse( > + "TABLE(b, a, c) : table1(a, b) table1(a, c).", gv); > + > + TYPE(array_t*, log_table_t*) inp_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + > + int i, j; > + char buf[1024]; > + > + log_table_t* tbl1 = log_table_init(NULL, 0, 2, 0, gv); > + for (i = 0;i < sz1;i++) > + for (j = 0;j < sz2;j++) { > + sprintf(buf, "1:ID_%d:XY_%d", i, j); > + log_tuple_t* tuple = log_tuple_init_str(buf, ':', gv); > + log_table_add(tbl1, tuple); > + } > + > + array_add(inp_insert, tbl1); > + > + clock_gettime(CLOCK_MONOTONIC, &ts0); > + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); > + clock_gettime(CLOCK_MONOTONIC, &ts1); > + log_table_t* res_tbl = array_get(res, 0); > + > + printf(" join sz1=%d sz2=%d\n", sz1, sz2); > + printf(" full run = %d buckets = %d\n", > + table_size(res_tbl), res_tbl->tuples.len); > + > + printf(" time = %" PRId64 " us\n", calc_tm(&ts0, &ts1)); > + log_array_free(inp_remove); > + log_array_free(inp_insert); > + log_array_free(res); > + > + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); > + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); > + > + tbl1 = log_table_init(NULL, 0, 2, 0, gv); > + log_table_add(tbl1, log_tuple_init_str("1:ID_0:XY_new_item", ':', gv)); > + array_add(inp_insert, tbl1); > + > + clock_gettime(CLOCK_MONOTONIC, &ts0); > + res = log_eng_delta(eng, inp_remove, inp_insert); > + clock_gettime(CLOCK_MONOTONIC, &ts1); > + > + res_tbl = array_get(res, 0); > + printf(" delta run = %d\n", table_size(res_tbl)); > + printf(" time = %" PRId64 " us\n", calc_tm(&ts0, &ts1)); > + log_array_free(inp_remove); > + log_array_free(inp_insert); > + log_array_free(res); > + > + log_engine_free(eng); > + set_free(gv); > + printf("- join perf\n"); > +} > + > +static void > +test_interactive(void) > +{ > + /* interactive engine for testing purpose */ > + log_set_sep(':', '\n'); > + printf("Input rules, e.g. R(a):r(a).\nuse EOF as end\n"); > + char inp[4096]; > + inp[0] = 0; > + > + for (;;) { > + char buf[1024]; > + if (scanf("%s", buf) != 1) continue; > + if (strcmp(buf, "EOF") == 0) break; > + strcat(inp, buf); > + } > + > + set_t* gv = set_init(NULL, 0, 0, NULL); > + log_set_global_value(gv); > + log_engine_t* eng = log_eng_parse(inp, gv); > + > + for (;;) { > + TYPE(array_t*, log_table_t*) inp_remove = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + TYPE(array_t*, log_table_t*) inp_insert = > + log_array_init(NULL, ENT_TABLE, 0, gv); > + log_io_parse_line(NULL, NULL, false, NULL, NULL); > + > + printf("Input changes, e.g. +:1:tbl_name|1:f0:f1|.\n" > + "use +, -, ? for add, remove, or query.\n" > + "could input multiple tables. the number in table name\n" > + "line indicates number of tuples to follow.\n" > + "'|' stands for new line. use '.' to exit\n" > + "for query, field value could be empty.\n"); > + > + bool is_query = false; > + for (;;) { > + if (scanf("%s", inp) != 1) continue; > + if (strcmp(inp, ".") == 0) break; > + > + if (is_query == false && strlen(inp) > 0 && inp[0] == '?') { > + inp[0] = '+'; > + is_query = true; > + } > + > + bool res = log_io_parse_line(inp, eng, > + is_query ? true : false, inp_remove, inp_insert); > + if (res == false) { > + printf("Error in format\n"); > + break; > + } > + } > + > + if (array_size(inp_remove) == 0 && array_size(inp_insert) == 0) { > + log_array_free(inp_remove); log_array_free(inp_insert); break; > + } > + > + array_t* res = is_query ? > + log_eng_query(eng, inp_insert) : > + log_eng_delta(eng, inp_remove, inp_insert); > + > + int sz = 0; sz = log_io_gen_line(inp, sz, eng, res); inp[sz] = 0; > + printf("Output\n%s\n", inp); > + > + log_array_free(inp_remove); log_array_free(inp_insert); > + log_array_free(res); > + } > + > + log_engine_free(eng); > + set_free(gv); > +} > + > +static void > +test_api(void) > +{ > + const char* p, *p1, *p2, *p3, *p4, *p5; > + int32_t sz, sz1, sz2; > + bool rmv, rmv1; > + > + void* eng = datalog_init("R2(a,b):r2(a,b); R1(a):r1(a).", > + /* ext func not provided */NULL); > + > + rmv = datalog_put_table(eng, /* adding */false, "r1"); > + /* first tuple for r1 */ > + datalog_put_field(eng, "r_1a", 0); > + > + rmv1 = datalog_put_table(eng, false, "r2"); > + /* first tuple for r2 */ > + datalog_put_field(eng, "r2_1a", /* c-str*/ 0); > + datalog_put_field(eng, "r2_1bx", /* len */5); > + /* second tuple r2 */ > + datalog_put_field(eng, "r2_2a", 0); > + datalog_put_field(eng, "r2_2b", 0); > + > + datalog_opr(eng, /* delta change */false); > + t_assert(rmv && rmv1, "api 0"); > + > + /* get output table R2 */ > + rmv1 = datalog_get_table(eng, &rmv, &p, &sz, &sz1); > + > + t_assert(0 == strcmp(p, "R2") && sz == 2 && sz1 ==2 && > + rmv == false && rmv1 == true, "api 1"); > + > + /* first tuple */ > + datalog_get_field(eng, &p1, &sz); > + datalog_get_field(eng, &p2, &sz1); > + /* second tuple */ > + datalog_get_field(eng, &p3, &sz); > + rmv = datalog_get_field(eng, &p4, &sz); > + /* indicates reaching next table */ > + rmv1 = datalog_get_field(eng, &p5, &sz); > + > + t_assert(0 == strcmp(p1, "r2_1a") && 0 == strcmp(p2, "r2_1b") && > + 0 == strcmp(p3, "r2_2a") && 0 == strcmp(p4, "r2_2b") > + && sz == 5 && sz1 == 5 && rmv == true > + && rmv1 == false, "api 2"); > + > + /* get output table R1 */ > + rmv1 = datalog_get_table(eng, &rmv, &p, &sz, &sz1); > + t_assert(0 == strcmp(p, "R1") && sz == 1 && sz1 == 1 && > + rmv == false && rmv1 == true, "api 3"); > + > + rmv = datalog_get_field(eng, &p1, &sz); > + /* indicates reaching next table */ > + rmv1 = datalog_get_field(eng, &p2, &sz); > + > + t_assert(0 == strcmp(p1, "r_1a") && > + sz == 4 && rmv == true && rmv1 == false, "api 4"); > + > + /* no more table returned */ > + rmv = datalog_get_table(eng, &rmv, &p, &sz, &sz1); > + t_assert(rmv == false, "api 5"); > + > + datalog_put_table(eng, false, "r2"); > + datalog_put_field(eng, "r2_1a0", 0); > + datalog_put_field(eng, "r2_1b", 0); > + datalog_opr(eng, false); > + datalog_get_table(eng, &rmv, &p, &sz, &sz1); > + /* it is ok to skip tuples of a table */ > + datalog_get_table(eng, &rmv, &p, &sz, &sz1); > + > + datalog_put_table(eng, false, "r2"); > + datalog_put_field(eng, NULL, 0); > + datalog_put_field(eng, "r2_1b", 0); > + datalog_opr(eng, /* query */true); > + > + rmv = datalog_get_table(eng, &rmv, &p, &sz2, &sz1); > + datalog_get_field(eng, &p1, &sz); > + datalog_get_field(eng, &p2, &sz); > + datalog_get_field(eng, &p3, &sz); > + datalog_get_field(eng, &p4, &sz); > + rmv1 = datalog_get_table(eng, &rmv, &p, &sz, &sz1); > + > + t_assert(0 == strcmp(p1, "r2_1a") && sz2 == 2 && > + sz1 == 2 && rmv == true && rmv1 == false, "api 5"); > + > + datalog_free(eng); > + printf("- api\n"); > +} > + > +static void > +test_datalog(int argc, char** argv) > +{ > + if (argc == 2 && !strcmp(argv[1], "test")) { > + log_set_sep(':', '\n'); > + > + test_collections(); > + test_tables(); > + test_sort(); > + test_sync(); > + test_join(); > + test_delta(); > + test_delta_misc(); > + test_delta_more(); > + test_join_perf(100, 100); > + test_api(); > + > + t_sum(); > + fprintf(stderr, "%s\n", /* for at script */ > + log_tst_no_cases_failed == 0 ? "PASS" : "FAIL"); > + } > + else if (argc == 2 && !strcmp(argv[1], "run")) > + test_interactive(); > + else printf("usage: test-datalog test|run\nrun for interactive mode"); > +} > + > +/*int main(int argc, char** argv) { test_datalog(argc, argv); return 0; }*/ > + > +OVSTEST_REGISTER("test-datalog", test_datalog); > diff --git a/tests/testsuite.at b/tests/testsuite.at > index 7ac74df..33c196f 100644 > --- a/tests/testsuite.at > +++ b/tests/testsuite.at > @@ -72,3 +72,4 @@ m4_include([tests/ovn-nbctl.at]) > m4_include([tests/ovn-sbctl.at]) > m4_include([tests/ovn-controller.at]) > m4_include([tests/ovn-controller-vtep.at]) > +m4_include([tests/datalog.at]) > -- > 2.7.4 > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev

diff --git a/ovn/lib/automake.mk b/ovn/lib/automake.mk index 4870505..5e5b04f 100644 --- a/ovn/lib/automake.mk +++ b/ovn/lib/automake.mk @@ -10,7 +10,10 @@ ovn_lib_libovn_la_SOURCES = \ ovn/lib/expr.h \ ovn/lib/lex.c \ ovn/lib/lex.h \ - ovn/lib/logical-fields.h + ovn/lib/logical-fields.h \ + ovn/lib/datalog.h \ + ovn/lib/datalog-private.h \ + ovn/lib/datalog.c nodist_ovn_lib_libovn_la_SOURCES = \ ovn/lib/ovn-nb-idl.c \ ovn/lib/ovn-nb-idl.h \ diff --git a/ovn/lib/datalog-private.h b/ovn/lib/datalog-private.h new file mode 100644 index 0000000..4a1acfb --- /dev/null +++ b/ovn/lib/datalog-private.h @@ -0,0 +1,808 @@ +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef OVN_DATALOG_PRIV_H +#define OVN_DATALOG_PRIV_H 1 + +#include <string.h> +#include <stdbool.h> +#include <stdlib.h> +#include <stdio.h> +#include <ctype.h> +#include <stdint.h> +#include <inttypes.h> +#include <time.h> + +#include "util.h" + +/* -------------------------------------------------------------------------- + * CONSTANTS + * -------------------------------------------------------------------------- + */ + +#define LOG_COMP 0 /* 0 for not enable logging */ + +#define SZ_INIT_BITSET 16 /* initial capacity of bitset */ +#define SZ_INIT_ARRAY 16 /* initial capacity of array */ +#define SZ_INIT_HASH 11 /* initial capacity of hash map */ + +#define LOG_TOKEN_SZ 512 /* max length of token or string literal */ + +/* ENTITY TYPES */ + +#define ENT_UNKNOWN 0 +#define ENT_INT32 1 +#define ENT_TST_INT32 2 /* for testing only */ +#define ENT_STR 3 /* null terminated string */ +#define ENT_VALUE 4 /* log_value_t* */ +#define ENT_INT_TUPLE 5 /* log_int_tuple_t* */ +#define ENT_TUPLE 6 /* log_tuple_t* */ +#define ENT_TABLE 7 /* log_table_t* */ +#define ENT_RULE 8 /* log_rule_t* */ +#define ENT_RULE_SET 9 /* log_rule_set_t* */ +#define ENT_LOG_ENG 0x0a/* log_engine_t* */ +#define ENT_JOIN_PARAM 0x0b/* log_join_param_t* */ + +#define ENT_ARRAY 0x0f/* array */ +#define ENT_MAP 0x0e/* map */ +#define ENT_SET 0x0d/* set */ +#define ENT_INDEX 0x0c/* index of tuple */ + /* the collection type must be */ + /* (ENT_UNKNOWN, ENT_UNKNOWN, ENT_INDEX) */ +#define ENT_BITSET 0x10 + +#define KEY_TYPE(t) ((t) & 0xff) +#define KEY_ID(id) ((id)) +#define VALUE_TYPE(t) (((t) >> 8) & 0xff) +#define VALUE_ID(id) ((id) << 8) +#define COLL_TYPE(t) (((t) >> 16) & 0xff) /* collection type */ +#define COLL_ID(id) ((id) << 16) + +#define MAP_INT32_VALUE (KEY_ID(ENT_INT32) | VALUE_ID(ENT_VALUE)) + +/* conversion between pointer and int32_t. type cast will produce warning */ +#define ptr2i(v) ((int32_t)(((union { intptr_t i; void* p; })(v)).i)) +#define i2ptr(v) (((union { intptr_t i; void* p; })(intptr_t)(v)).p) + +#define TYPE(t, def) t /* showing type of object in array, set */ +#define TYPE2(t, key, val) t /* showing type of object in map */ + +/* -------------------------------------------------------------------------- + * BITSET, ARRAY, HASH MAP, AND HASH SET + * -------------------------------------------------------------------------- + */ + +/* when type is known for a collection, it could be freed using [coll]_free, + * and all nested structure will be freed. items never cross reference by + * default, except for log_values. when there is cross reference, the 'free' + * function must be defined. 'global value' could be regarded as global + * context for the object. + * + * naming convention for collection function: collection_method: + * collection could be: bitset, array, set, map + * method could be: (init, free), (add, del), (ins, rmv), (get, set). + */ + +struct hash_s; + +typedef struct meta_s { + int32_t type; /* must be the first field */ + bool alloc; /* indicates if the structure is from malloc */ + struct hash_s* glb_values; +} meta_t; + +typedef struct bitset_s { + meta_t m; + int32_t size; /* size of items in bytes */ + int32_t len; /* length of allocated items */ + uint32_t* items; +} bitset_t; + +typedef struct array_s { + meta_t m; + void** item; + int32_t size; /* size of the array */ + int32_t len; /* length of allocated space */ +} array_t; + +typedef struct map_node_s { + struct map_node_s* next; + void* key; + void* value; +} map_node_t; + +/* set_node_s, index_node_s must have the same structure as map_node_s + * except the last field. assume they have the same alignment, and + * map_node_s is used to access both. + */ +typedef struct set_node_s { + struct set_node_s* next; + void* key; +} set_node_t; + +/* for index map, the aux points to log_int_tuple which defines the + * index. the key of each entry points to one tuple of the tuple set + * whose elements share the same tuple key. the set is represented by + * double link presented in the tuple. + */ +typedef struct index_node_s { + struct index_node_s* next; + void* key; /* points to one tuple of the set */ + int32_t hash_code; /* hash_code of the tuple key */ +} index_node_t; + +typedef struct hash_s { + meta_t m; + map_node_t** bucket; + int32_t size; /* size of items */ + int32_t len; /* length of bucket array */ + void* aux; /* extra data */ +} hash_t; + +typedef hash_t map_t; +typedef hash_t set_t; + +/* -------------------------------------------------------------------------- + * DATA STRUCTURE OF LOG ENGINE + * -------------------------------------------------------------------------- + */ + +typedef struct log_config_s { + char sep1; /* field separator */ + char sep2; /* record separator */ +} log_config_t; + +typedef struct log_value_s {/* variable size structure */ + int32_t hash_code; /* must be first field */ + int32_t ref_no; /* number of references */ + /* the actual size is abs(size) and it does not count terminating 0 */ + /* 0 is always padded no matter .a or .p is used for printf */ + int32_t size; /* size < 0 indicates using value.p */ + /* size >= 0 indicates using value.a */ + union { + char a[0]; /* byte array, need not terminate with null */ + char* p; /* only used in populating the value set */ + } value; +} log_value_t; + +/* + * log_tuple_s.indexes is a log_tuple_t pointer array, which represents + * an array of double linked list. for index i, indexes[i * 2] points to + * previous node and index[i * 2 + 1] points to success node. + */ +typedef struct log_tuple_s {/* variable size structure */ + /* do not have size due to memory consideration */ + int32_t hash_code; /* must be first field */ + int32_t n_values; /* duplicated, see table.num_fields */ + int64_t count; /* valid only in table, when tuple is outside */ + /* table, it has different meaning */ + struct log_tuple_s** indexes; /* valid only in table */ + log_value_t* values[0]; /* field array of the tuple */ +} log_tuple_t; + +typedef struct log_int_tuple_s {/* variable size structure */ + int32_t hash_code; /* must be first field */ + int32_t n_items; /* number of integers */ + int32_t values[0]; +} log_int_tuple_t; + +struct log_engine_s; + +/* index_map, index_def and tuple.indexes must have the same size, + * and aligns to each other, i.e. assume there is N indexes defined, + * tuple.indexes has N * 2 items, and index_map and index_def + * has N items. For index j: + * (1) index_def[key_def]->j, defines the index, e.g., '0:3:4'->1. + * (2) index_map[j] defines hash map from key tuples to tuple set. + * The aux of this map points to corresponding index_def's key. + * (3) tuple.indexes[j * 2] and tuple.indexes[j * 2 + 1] defines tuple set. + */ +typedef struct log_table_s { + meta_t m; + int32_t table_index; /* id of the table */ + int32_t num_fields; /* also present in each tuple */ + bool is_remove; /* valid if this is delta */ + + TYPE2(map_t, log_int_tuple_t*, int32_t) index_def; + TYPE(array_t, hash_t*) index_map; + TYPE(set_t, log_tuple_t*) tuples; +} log_table_t; + +typedef struct log_rule_s { + meta_t m; + + bool is_union; + /* + * table index start from 0. item 0 is left side + * example X1 : X2, X3 -> 7, 3, 6 + * rule and param has the same sequence. + */ + TYPE(array_t, int32_t) rule; + + /* + * item 0 is left side. param index starts from 0. + * example: X1(a,b) : X2(a, -), X3('c', b) -> ((0, 1), (0, -1), (-2, 1)) + * -1 indicates 'ignore', -2 is the index for first constant + */ + TYPE(array_t, array_t*) param; /* array of array of integer */ + /* example: -2 -> 'c', map integer to value */ + TYPE2(map_t, int32_t, log_value_t*) const_param; + + /* example: 0->'a', 1->'b' */ + TYPE2(map_t, int32_t, log_value_t*) param_name_map; +} log_rule_t; + +typedef struct log_rule_set_s { + meta_t m; + + /* example: 7-> 'X1' */ + TYPE2(map_t, int32_t, log_value_t*) rule_name_map; + TYPE2(map_t, log_value_t*, int32_t) rule_index_map; + + /* table index -> rule. table index starts from 0 */ + TYPE2(map_t, int32_t, log_rule_s*) rules; + /* table index -> table index set */ + TYPE2(map_t, int32_t, array_t*) table_rule_map; /* array of integer */ + /* example: 3->(7), 6->(7), list is ordered */ + + TYPE(set_t, int32_t) input_tables; + TYPE(set_t, int32_t) output_tables; + TYPE2(map_t, int32_t, int32_t) param_size; +} log_rule_set_t; + +typedef struct log_io_s { + TYPE(array_t*, log_table_t*) inp_remove; + TYPE(array_t*, log_table_t*) inp_insert; + TYPE(array_t*, log_table_t*) res; + + log_table_t* cur_tbl; + log_tuple_t* cur_tuple; + int32_t t_idx; /* table index */ + int32_t f_idx; /* field index */ + + map_node_t* hash_node; /* current node of iterator */ + int32_t hash_b; /* current bucket of iterator */ +} log_io_t; + +typedef struct log_engine_s { + meta_t m; + + log_rule_set_t rule_set; + /* map table index to log_table_t */ + TYPE2(map_t, int32_t, log_table_t*) tables; + + bool (*ext_func)( /* to reset state if last 3 param are NULL */ + struct log_engine_s* eng, log_table_t*, log_table_t*, log_table_t*); + + log_io_t io; /* temporary data for calling (put|get)_(table|field) */ +} log_engine_t; + +typedef struct log_join_param_s { + meta_t m; + + log_int_tuple_t* index2; + TYPE(array_t, log_value_t*) select1; + TYPE(array_t, int32_t) select1i; /* match select1 in size */ + TYPE(array_t, int32_t) rem1; + TYPE(array_t, int32_t) rem2; + TYPE(array_t, int32_t) out_param; + bool full_join; +} log_join_param_t; + +/* -------------------------------------------------------------------------- + * PROTOTYPES + * -------------------------------------------------------------------------- + */ + +void log_topo_sort( + TYPE2(map_t*, log_value_t*, set_t*) g, /* set of value */ + TYPE(array_t*, log_value_t*) order, + TYPE(set_t*, log_value_t*) in_nodes, + TYPE(set_t*, log_value_t*) out_nodes); +void log_sort_array( + int start, array_t* list, array_t* sem1, array_t* sem2); +int log_insert_item( + int val, array_t* list, void* obj1, + void* obj2, array_t* sem1, array_t* sem2); + +void log_set_sep(char sep1, char sep2); +void log_set_global_value(set_t* s); + +void log_sync_init(const char* log, set_t* gv); +void log_sync_parse(map_t* sem); +void log_sem_process(log_rule_set_t* rule_set, map_t* sem); +void log_eng_set_ext_func(log_engine_t* eng, void* func); +log_engine_t* log_eng_parse(const char* rules, set_t* gv); + +int32_t log_hash_code(hash_t* m, const void* v); +int32_t log_hash_code_byte(const void* v, int32_t* size); +bool log_key_equal(hash_t* m, const void* k1, const void* k2); +void log_coll_free(void* coll, int32_t type, set_t* gv); + +map_t* log_hash_init(map_t* m, int type, int sz_init, hash_t*); +void log_hash_free(hash_t*); +void log_hash_add(map_t* m, void* k, void* v); +void* log_hash_del(map_t* m, void* k); +void log_hash_rehash(map_t* m); +bool log_hash_next(map_t* m, int32_t* b, map_node_t** cur); +void* log_hash_get_one(map_t* m); + +bitset_t* log_bitset_init(bitset_t* set); +void log_bitset_free(bitset_t*); + +array_t* log_array_init( + array_t* a, int32_t type, int32_t i_size, set_t* gv); +void log_array_free(array_t*); +array_t* log_array_clone(array_t* a); +int32_t log_array_look_for(array_t* a, void* v); + +log_value_t* log_value_init(const char* v, int32_t size, set_t* gv); +log_int_tuple_t* log_int_tuple_init(TYPE(array_t*, int32_t) a); +void log_int_tuple_free(log_int_tuple_t*); + +log_tuple_t* log_tuple_init(int32_t n_values); +void log_tuple_free(log_tuple_t*, set_t*, bool); +log_tuple_t* log_tuple_init_val(log_value_t** val, int32_t n_values); +log_tuple_t* log_tuple_init_str(const char* t, char sep, set_t* gv); +log_tuple_t* log_tuple_init_str_null( + const char* t, char sep, set_t* gv); + +log_table_t* log_table_init( + log_table_t* tbl, int32_t n, int32_t f, + int32_t size, set_t* gv); +void log_table_free(log_table_t* tbl); +void log_table_add(log_table_t* tbl, log_tuple_t* t); +void log_table_remove(log_table_t* tbl, log_tuple_t* t); + +int32_t log_table_add_index( + log_table_t* tbl, log_int_tuple_t* index_key); +log_tuple_t* log_index_get_index( + log_table_t* tbl, log_tuple_t* t, int32_t i_idx); + +void log_rule_free(log_rule_t* rule); +void log_rule_set_init(log_rule_set_t* rs, set_t* gv); +void log_rule_set_free(log_rule_set_t*); + +log_engine_t* log_engine_init(log_engine_t* log, set_t* gv); +void log_engine_free(log_engine_t*); + +log_join_param_t* log_join_param_init( + log_join_param_t* jp, log_int_tuple_t* i2, set_t* gv); +void log_join_param_free(log_join_param_t*); + +log_table_t* log_tblopr_join( + log_table_t* t1, log_table_t* t2, + log_join_param_t* joinp); +void log_eng_do_join( + log_engine_t* eng, + log_table_t* input, log_table_t* output); +void log_eng_do_union(log_engine_t* eng, + log_table_t* input, log_table_t* output); + +TYPE(array_t*, log_table_t*) log_eng_delta( + log_engine_t* eng, + TYPE(array_t*, log_table_t*) inp_remove, + TYPE(array_t*, log_table_t*) inp_insert); +TYPE(array_t*, log_table_t*) log_eng_query( + log_engine_t* eng, + TYPE(array_t*, log_table_t*) input); + +log_table_t* log_get_table(log_engine_t* eng, const char* name); +log_table_t* log_get_org_table(log_engine_t* eng, log_table_t* t); +log_tuple_t* log_query_on0(log_engine_t* eng, + int32_t tid, log_value_t* value); + +log_table_t* log_io_table_name_reset( + log_engine_t* eng, const char* name); +bool log_io_parse_line( + const char* line, log_engine_t* eng, bool use_null, + TYPE(array_t*, log_table_t*) inp_remove, + TYPE(array_t*, log_table_t*) inp_insert); +int32_t log_io_gen_line(char* text, int pos, log_engine_t* eng, + TYPE(array_t*, log_table_t*) all); + +int32_t log_coll_print( + char* buf, int32_t pos, void* item, int32_t type, bool verbose); +int32_t log_array_print(char* buf, int32_t pos, array_t* a, bool verbose); +int32_t log_hash_print(char* buf, int32_t pos, hash_t* m, bool verbose); +int32_t log_table_print(char* buf, int32_t pos, log_table_t* t, bool verbose); +int32_t log_index_print(char* buf, int32_t pos, log_table_t* t); +int32_t log_tuple_print(char* buf, int32_t pos, log_tuple_t* t); +int32_t log_rule_set_print(char* buf, int32_t pos, log_rule_set_t* rs); + +/* -------------------------------------------------------------------------- + * MACROS AND INLINE IMPLEMENTATIONS + * -------------------------------------------------------------------------- + */ + +#define map_size(m) ((m)->size) +#define map_has(m, k) (log_hash_get(m, k) != NULL) +#define map_free(m) log_hash_free(m) +#define map_del(m, k) log_hash_del(m, k) + +#define set_size(m) ((m)->size) +#define set_has(m, k) (log_hash_get(m, k) != NULL) +#define set_free(m) log_hash_free(m) +#define set_del(m, k) log_hash_del(m, k) + +#define array_size(a) ((a)->size) +#define table_size(t) (set_size(&(t)->tuples)) + +#define map_get_int(m, k) ptr2i(map_get(m, k)) +#define array_get_int(a, i) ptr2i(array_get(a, i)) + +#define log_index_i_pre(tuple, i) ((tuple)->indexes[(i) * 2]) +#define log_index_i_suc(tuple, i) ((tuple)->indexes[(i) * 2 + 1]) + +/* -------------------------------------------------------------------------- + * META DATA + * -------------------------------------------------------------------------- + */ + +static inline void* +c_realloc(void* ptr, size_t old_sz, size_t new_sz) +{ + /* ptr could be NULL and expanded area will be zeroed */ + void* n = realloc(ptr, new_sz); + memset(((char*)n) + old_sz, 0, new_sz - old_sz); + return n; +} + +static inline void +hash_code_array_init(int32_t* c) +{ + /* sequence of 0 will have different hash code depending on its length */ + *c = 1; +} + +static inline void +hash_code_array_add(int32_t* c, int32_t i) +{ + *c = (*c) * 31 + i; +} + +static inline void +hash_code_array_final(int32_t* c) +{ + if (*c < 0) *c = -(*c); +} + +static inline void +coll_alloc(void* ptr, int32_t size, int32_t type, void* global_values) +{ + meta_t** m = (meta_t**)ptr; + if (*m == NULL) { + *m = malloc(size); + (*m)->alloc = true; + } + else { + (*m)->alloc = false; + } + + (*m)->type = COLL_ID(type); + (*m)->glb_values = global_values; +} + +static inline void +coll_free_ptr(void* ptr) +{ + meta_t* m = ptr; + if (m->alloc) free(m); +} + +static inline void +log_value_ref(log_value_t* v) +{ + v->ref_no++; +} + +static inline void +log_value_free(log_value_t* v, set_t* gv) +{ + if (v->ref_no > 1) --v->ref_no; + else { + int32_t sv_type = gv->m.type; + gv->m.type = 0; + log_hash_del(gv, v); + gv->m.type = sv_type; + free(v); + } +} + +static inline void +check_value_ref(void* v, int32_t type, set_t* gv, bool add) +{ + if (v == NULL) return; + if (!add) log_coll_free(v, type, gv); + else if (type == ENT_VALUE) log_value_ref((log_value_t*)v); +} + +/* -------------------------------------------------------------------------- + * BITSET + * -------------------------------------------------------------------------- + */ + +static inline void +bitset_resize(bitset_t* set, int32_t len) +{ + /* since there is no reset operation, it always expands */ + set->items = c_realloc(set->items, + set->len * sizeof(int32_t), len * sizeof(int32_t)); + /* assume align to 4 bytes for int32_t */ + set->len = len; +} + +static inline void +bitset_set(bitset_t* set, int32_t b) +{ + int32_t p = b >> 5; + if (p >= set->size) { + set->size = p + 1; + if (set->size > set->len) bitset_resize(set, set->size); + } + + set->items[p] |= 1 << (b % 32); +} + +static inline bool +bitset_get(bitset_t* set, int b) +{ + int p = b >> 5; + if (p >= set->size) return false; + return (set->items[p] & (1 << (b % 32))) != 0; +} + +static inline bool +bitset_empty(bitset_t* set) +{ + int i; + for (i = 0;i < set->size;i++) + if (set->items[i] != 0) return false; + return true; +} + +static inline void +bitset_and(bitset_t* dest, bitset_t* src) +{ + + int32_t m_size = dest->size > src->size ? src->size : dest->size; + + int i; + for (i = 0;i < m_size;i++) dest->items[i] &= src->items[i]; + for (i = m_size;i < dest->size;i++) dest->items[i] = 0; +} + +/* -------------------------------------------------------------------------- + * ARRAY + * -------------------------------------------------------------------------- + */ + +static inline void +array_add(array_t* a, void* i) +{ + ovs_assert(a->size <= a->len); + + if (a->size == a->len) { + a->item = c_realloc(a->item, + a->len * sizeof(void*), a->len * 2 * sizeof(void*)); + a->len = a->len * 2; + } + a->item[a->size++] = i; + check_value_ref(i, KEY_TYPE(a->m.type), a->m.glb_values, true); +} + +static inline void +array_ins(array_t* a, int32_t pos, void* i) +{ + array_add(a, NULL); /* make room for new item */ + memmove(&a->item[pos + 1], &a->item[pos], + (a->size - pos - 1) * sizeof(void*)); /* size has increased by 1 */ + + a->item[pos] = i; + check_value_ref(i, KEY_TYPE(a->m.type), a->m.glb_values, true); +} + +static inline void* +array_rmv(array_t* a, int32_t pos) +{ + /* no change to ref */ + void* org = a->item[pos]; + memmove(&a->item[pos], &a->item[pos + 1], + (a->size - pos - 1) * sizeof(void*)); + a->size--; + return org; +} + +static inline void* +array_get(array_t* a, int32_t i) +{ + ovs_assert(i >= 0 && i < a->size); + return a->item[i]; +} + +static inline void +array_set(array_t* a, int i, void* v) +{ + ovs_assert(i >= 0 && i < a->size); + check_value_ref(a->item[i], KEY_TYPE(a->m.type), a->m.glb_values, false); + + a->item[i] = v; + check_value_ref(v, KEY_TYPE(a->m.type), a->m.glb_values, true); +} + +/* -------------------------------------------------------------------------- + * HASH TABLE + * -------------------------------------------------------------------------- + */ + +static inline map_node_t* +log_hash_get(map_t* m, void* k) +{ + /* no change to reference count */ + + int32_t code = log_hash_code(m, k); + int32_t slot = code % m->len; + map_node_t* head = m->bucket[slot]; + + while (head != NULL) { + if (log_key_equal(m, head->key, k)) return head; + head = head->next; + } + return NULL; +} + +/* -------------------------------------------------------------------------- + * MAP + * -------------------------------------------------------------------------- + */ + +static inline map_t* +map_init(map_t* m, int type, int sz_init, set_t* gv) +{ + return log_hash_init(m, type | COLL_ID(ENT_MAP), sz_init, gv); +} + +static inline void* +map_get(map_t* m, void* k) +{ + map_node_t* node = log_hash_get(m, k); + if (node == NULL) return NULL; + else return node->value; +} + +static inline void +map_add(map_t* m, void* k, void* v) +{ + map_node_t* node = log_hash_get(m, k); + + if (node != NULL) { + void* old = node->value; + check_value_ref(old, VALUE_ID(m->m.type), m->m.glb_values, false); + + node->value = v; + check_value_ref(v, VALUE_ID(m->m.type), m->m.glb_values, true); + + } else log_hash_add(m, k, v); +} + +/* -------------------------------------------------------------------------- + * SET + * -------------------------------------------------------------------------- + */ + +static inline set_t* +set_init(set_t* m, int type, int sz_init, set_t* gv) +{ + return log_hash_init(m, type | COLL_ID(ENT_SET), sz_init, gv); +} + +static inline void* +set_get(set_t* m, void* k) +{ + map_node_t* node = log_hash_get(m, k); + if (node == NULL) return NULL; + else return node->key; +} + +static inline void +set_add(set_t* m, void* k) +{ + map_node_t* node = log_hash_get(m, k); + if (node != NULL) return; + log_hash_add(m, k, NULL); +} + +/* -------------------------------------------------------------------------- + * ITERATIONS + * -------------------------------------------------------------------------- + */ + +/* nested usage is supported. + * 'continue' does not work with set. + * 'if (c) [TYPE]_EXIT; else {}' does not work with map and set. + */ + +#define ARRAY_ALL_0(array, node, type, typec) \ + { int __array_i = 0; \ + for (;__array_i < (array)->size;__array_i++) { \ + type node = typec((array)->item[__array_i]); { + +#define ARRAY_EXIT break +#define ARRAY_END }}} + +#define MAP_ALL(map, node) \ + { bool __map_exit = false; \ + map_t* __map = (map_t*)(map); \ + int __map_i = 0; \ + for (;__map_i < __map->len;__map_i++) { \ + if (__map_exit) break; \ + struct map_node_s* __map_head = __map->bucket[__map_i]; \ + while (__map_head != NULL) { \ + struct map_node_s* node = __map_head; \ + __map_head = __map_head->next; { + +#define MAP_EXIT {__map_exit = true; break; } +#define MAP_END }}}} + +#define SET_ALL_0(set, node, type, typec) \ + { bool __set_exit = false, __set_rmv; \ + struct set_node_s* __set_save = NULL, *__set_pre; \ + set_t* __set = (set_t*)(set); \ + int32_t __set_i = 0; \ + for (;__set_i < __set->len;__set_i++) { \ + if (__set_exit) break; \ + struct set_node_s* __set_this = (struct set_node_s*) \ + (__set->bucket[__set_i]); \ + __set_pre = NULL; \ + while (__set_this != NULL) { \ + type node = typec(__set_this->key); \ + __set_rmv = false; { + +#define SET_REMOVE_ITEM /* will not rehash */ \ + (__set_rmv = true, __set->size--, __set_save = __set_this, \ + __set_pre == NULL ? (__set->bucket[__set_i] = \ + (map_node_t*)__set_this->next) : \ + (map_node_t*)(__set_pre->next = __set_this->next), \ + __set_this = __set_this->next, free(__set_save)) + +#define SET_EXIT {__set_exit = true; break; } + +#define SET_END } \ + if (!__set_rmv) { __set_pre = __set_this; \ + __set_this = __set_this->next;} }} \ + __set_save++; __set_pre++; } /* make no warning */ + +#define INDEX_ALL(tuple, i_idx, node) \ + { bool __index_first = true; \ + log_tuple_t* __index_head = tuple; \ + log_tuple_t* node; \ + if (__index_head != NULL) { for (;;) { \ + if (__index_first) { \ + node = __index_head; __index_first = false; } \ + else { \ + (node) = (node)->indexes[i_idx * 2 + 1]; \ + if (__index_head == node) break; \ + } + +#define INDEX_EXIT break +#define INDEX_END }}} + +#define ARRAY_ALL(array, node, type) ARRAY_ALL_0(array, node, type, (type)) +#define ARRAY_ALL_INT(array, node) ARRAY_ALL_0(array, node, int32_t, ptr2i) + +#define SET_ALL(set, node, type) SET_ALL_0(set, node, type, (type)) +#define SET_ALL_INT(set, node) SET_ALL_0(set, node, int32_t, ptr2i) + +#endif /* datalog-private.h */ diff --git a/ovn/lib/datalog.c b/ovn/lib/datalog.c new file mode 100644 index 0000000..3354512 --- /dev/null +++ b/ovn/lib/datalog.c @@ -0,0 +1,3392 @@ +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <config.h> +#include "datalog.h" +#include "datalog-private.h" + +/* ========================================================================== + * BASIC COLLECTIONS + * ========================================================================== + */ + +void +log_coll_free(void* coll, int32_t type, set_t* gv) +{ + switch (type) { + case ENT_BITSET: log_bitset_free(coll); break; + case ENT_ARRAY: log_array_free(coll); break; + + case ENT_MAP: + case ENT_SET: + case ENT_INDEX: log_hash_free(coll); break; + + case ENT_TUPLE: log_tuple_free(coll, gv, true); break; + case ENT_VALUE: log_value_free(coll, gv); break; + case ENT_TABLE: log_table_free(coll); break; + case ENT_INT_TUPLE: log_int_tuple_free(coll); break; + case ENT_RULE: log_rule_free(coll); break; + case ENT_RULE_SET: log_rule_set_free(coll); break; + case ENT_LOG_ENG: log_engine_free(coll); break; + case ENT_JOIN_PARAM:log_join_param_free(coll); break; + } +} + +/* -------------------------------------------------------------------------- + * BITSET + * -------------------------------------------------------------------------- + */ + +bitset_t* +log_bitset_init(bitset_t* set) +{ + coll_alloc(&set, sizeof(bitset_t), ENT_BITSET, NULL); + set->len = SZ_INIT_BITSET; + set->items = calloc(set->len, sizeof(void*)); + set->size = 0; + return set; +} + +void +log_bitset_free(bitset_t* dest) +{ + if (dest->items != NULL) free(dest->items); + coll_free_ptr(dest); +} + +/* -------------------------------------------------------------------------- + * ARRAY + * -------------------------------------------------------------------------- + */ + +void +log_set_global_value(set_t* s) +{ + s->m.type = KEY_ID(ENT_VALUE) | COLL_ID(ENT_SET); + s->m.glb_values = s; +} + +array_t* +log_array_init(array_t* a, int32_t type, int32_t i_size, set_t* gv) +{ + coll_alloc(&a, sizeof(array_t), ENT_ARRAY, gv); + + a->len = i_size; + a->m.type |= type; + if (a->len == 0) a->len = SZ_INIT_ARRAY; + a->item = calloc(a->len, sizeof(void*)); + a->size = 0; + return a; +} + +array_t* +log_array_clone(array_t* a) +{ + int i; + array_t* na = log_array_init(NULL, a->m.type, a->size, a->m.glb_values); + memcpy(na->item, a->item, a->size * sizeof(void*)); + na->size = a->size; + + for (i = 0;i < a->size;i++) + check_value_ref(na->item[i], KEY_TYPE(na->m.type), + na->m.glb_values, true); + return na; +} + +void +log_array_free(array_t* a) +{ + int i; + for (i = 0;i < a->size;i++) + check_value_ref(a->item[i], KEY_TYPE(a->m.type), + a->m.glb_values, false); + free(a->item); + coll_free_ptr(a); +} + +int32_t +log_array_look_for(array_t* a, void* v) +{ + int i = 0; + bool found = false; + + ARRAY_ALL(a, item, const void*) + if (item == v) { + found = true; + ARRAY_EXIT; + } + i++; + ARRAY_END + + if (found) return i; + else return -1; +} + +int32_t +log_array_print(char* buf, int32_t pos, array_t* a, _Bool verbose) { + int i = 0; + buf[pos++] = '['; + int32_t ktype = KEY_TYPE(a->m.type); + for (i = 0; i < a->size; i++) { + if (i > 0) + buf[pos++] = ','; + + pos = log_coll_print(buf, pos, a->item[i], ktype, verbose); + } + buf[pos++] = ']'; + return pos; +} + +/* -------------------------------------------------------------------------- + * HASH CODE + * -------------------------------------------------------------------------- + */ + +int32_t +log_hash_code_byte(const void* v, int32_t* size) +{ + /* + * size of 0 indicates null terminated string, and actual size not + * including null is returned in size. + */ + + const unsigned char* p = v; + uint32_t hash = 0; + int32_t i = 0; + + /* Jenkins's Hash */ + if (*size == 0) { + unsigned char c; + while ((c = *p++) != 0) { + i++; + hash += c; + hash += (hash << 10); + hash ^= (hash >> 6); + } + *size = i; + } + else { + int32_t s = *size; + for (;i < s;i++) { + hash += *p++; + hash += (hash << 10); + hash ^= (hash >> 6); + } + } + + hash += (hash << 3); + hash ^= (hash >> 11); + hash += (hash << 15); + return (int32_t) hash < 0 ? -hash : hash; +} + +bool +log_key_equal(hash_t* m, const void* k1, const void* k2) +{ + /* if only one key is from hash_table, it must be k1 */ + int type = KEY_TYPE(m->m.type); + + if (type == ENT_VALUE) { + return k1 == k2; + } + else if (type == ENT_TUPLE) { + const log_tuple_t* t1 = k1; + const log_tuple_t* t2 = k2; + + if (t1->hash_code != t2->hash_code) return false; + int32_t sz_tuples = + (((log_table_t*)(m->aux))->num_fields) * sizeof(void*); + return memcmp(&t1->values, &t2->values, sz_tuples) == 0; + } + + else if (COLL_TYPE(m->m.type) == ENT_INDEX) { + const log_tuple_t* t1 = k1; + const log_tuple_t* t2 = k2; + + int i; + log_int_tuple_t* index_def = m->aux; + + if (t1 == t2) return true; + if (t2->count == 0) { /* compact form */ + for (i = 0;i < index_def->n_items;i++) + if (t1->values[index_def->values[i]] != t2->values[i]) + return false; + } + else { + for (i = 0;i < index_def->n_items;i++) + if (t1->values[index_def->values[i]] != + t2->values[index_def->values[i]]) { + return false; + } + } + return true; + } + + else if (m == m->m.glb_values) { + const log_value_t* v1 = k1; + const log_value_t* v2 = k2; + + if (v1->size >= 0 && v2->size >= 0) return k1 == k2; + if (v1->hash_code != v2->hash_code) return false; + if (v1->size == 0 && v2->size == 0) return true; + + const void* p1 = v1->size > 0 ? v1->value.a : v1->value.p; + const void* p2 = v2->size > 0 ? v2->value.a : v2->value.p; + + int32_t s1 = v1->size > 0 ? v1->size : (-v1->size); + int32_t s2 = v2->size > 0 ? v2->size : (-v2->size); + if (s1 != s2) return false; + return memcmp(p1, p2, s1) == 0; + } + + else if (type == ENT_INT_TUPLE) { + const log_int_tuple_t* t1 = k1; + const log_int_tuple_t* t2 = k2; + + if (t1->hash_code != t2->hash_code) return false; + if (t1->n_items != t2->n_items) return false; + + int s = t1->n_items * sizeof(int32_t); + return memcmp(&t1->values, &t2->values, s) == 0; + } + + + else if (type == ENT_INT32 || type == ENT_TST_INT32) + return ptr2i((void*)k1) == ptr2i((void*)k2); + else if (type == ENT_STR) return strcmp(k1, k2) == 0; + + ovs_assert(false); + return false; +} + +int32_t +log_hash_code(hash_t* m, const void* v) +{ + + int32_t type = KEY_TYPE(m->m.type); + if (type == ENT_STR) { + int32_t size0 = 0; + return log_hash_code_byte(v, &size0); + } + + else if (COLL_TYPE(m->m.type) == ENT_INDEX) { + int32_t i, code; + log_tuple_t* t = (log_tuple_t*)v; + + log_int_tuple_t* index_def = m->aux; + hash_code_array_init(&code); + + if (t->count == 0) { /* compact form */ + for (i = 0; i < index_def->n_items; i++) { + log_value_t* v = t->values[i]; + hash_code_array_add(&code, v->hash_code); + } + } else { + for (i = 0; i < index_def->n_items; i++) { + log_value_t* v = t->values[index_def->values[i]]; + hash_code_array_add(&code, v->hash_code); + } + } + + hash_code_array_final(&code); + return code; + } + + else if (type == ENT_INT32) { + int32_t n = ptr2i((void*)v); + if (n < 0) + n = -n; + return n; + } else if (type == ENT_TST_INT32) + return ptr2i((void*)v) % 100; + else + return *(const int32_t*) v; +} + +/* -------------------------------------------------------------------------- + * HASH TABLE + * -------------------------------------------------------------------------- + */ + +map_t* +log_hash_init(map_t* m, int type, int sz_init, struct hash_s* values) +{ + coll_alloc(&m, sizeof(map_t), 0, values); + m->m.type = type; + m->size = 0; + m->aux = NULL; + m->len = sz_init == 0 ? SZ_INIT_HASH : sz_init; + m->bucket = calloc(m->len, sizeof(void*)); + return m; +} + +void +log_hash_free(map_t* m) +{ + int i; + int32_t k_type = KEY_TYPE(m->m.type); + int32_t v_type = VALUE_TYPE(m->m.type); + int32_t c_type = COLL_TYPE(m->m.type); + + for (i = 0;i < m->len;i++) { + map_node_t* head = m->bucket[i]; + while (head != NULL) { + if (m == m->m.glb_values) free((void*)head->key); + else { + void* key = c_type != ENT_INDEX ? head->key : NULL; + void* value = c_type == ENT_MAP ? head->value : NULL; + check_value_ref(key, k_type, m->m.glb_values, false); + check_value_ref(value, v_type, m->m.glb_values, false); + } + + map_node_t* next = head->next; + free(head); + head = next; + } + } + free(m->bucket); + coll_free_ptr(m); +} + +void +log_hash_rehash(map_t* m) +{ + int nlen; + if (m->size > m->len * 2 / 3) nlen = m->len * 2; + else if (m->size < m->len / 5 && m->size > 50) nlen = m->len / 2; + else return; /* 50 is arbitrary number */ + + int i; + map_node_t** nb = calloc(nlen, sizeof(void*)); + + for (i = 0;i < m->len;i++) { + map_node_t* head = m->bucket[i]; + while (head != NULL) { + map_node_t* next = head->next; + + int b; + if (COLL_TYPE(m->m.type) == ENT_INDEX) + b = ptr2i(head->value); + else b = log_hash_code(m, head->key); + + /* the list is reversed in some sense */ + map_node_t* nhead = nb[b % nlen]; + if (nhead == NULL) head->next = NULL; + else head->next = nhead; + + nb[b % nlen] = head; + head = next; + } + } + + free(m->bucket); + m->bucket = nb; + m->len = nlen; +} + +bool +log_hash_next(map_t* m, int32_t* b, map_node_t** cur) +{ + /* return false if there is no next */ + int32_t i = *b; + for (;;) { + if (*cur == NULL) { + while (i < m->len && m->bucket[i] == NULL) + i++; + if (i >= m->len) + return false; + + *b = i; + *cur = m->bucket[i]; + return true; + } + *cur = (*cur)->next; + if (*cur != NULL) + return true; + i++; + } + return false; /* never reach here */ +} + +void +log_hash_add(map_t* m, void* k, void* v) +{ + /* check key existence before calling this */ + + int32_t type = COLL_TYPE(m->m.type); + int32_t code = log_hash_code(m, k); + int32_t slot = code % m->len; + + map_node_t* head = m->bucket[slot]; + map_node_t* item; + + if (type == ENT_MAP) { + item = malloc(sizeof (map_node_t)); + item->value = v; + } + else if (type == ENT_SET) { + item = malloc(sizeof (set_node_t)); + } + else if (type == ENT_INDEX) { + item = malloc(sizeof (index_node_t)); + item->value = i2ptr(code); + } + else ovs_assert(false); + + item->key = k; + item->next = head; + m->bucket[slot] = item; + m->size++; + + check_value_ref(k, KEY_TYPE(m->m.type), m->m.glb_values, true); + check_value_ref(v, VALUE_TYPE(m->m.type), m->m.glb_values, true); + log_hash_rehash(m); +} + +void* +log_hash_del(map_t* m, void* k) +{ + /* reference counter of value will NOT be changed, key is freed. */ + int32_t slot = log_hash_code(m, k) % m->len; + + map_node_t* head = m->bucket[slot]; + map_node_t* pre = NULL; + + while (head != NULL) { + if (log_key_equal(m, head->key, k)) { + if (pre == NULL) m->bucket[slot] = head->next; + else pre->next = head->next; + + const void* value = COLL_TYPE(m->m.type) == ENT_MAP ? + head->value : NULL; + free(head); + + check_value_ref(k, KEY_TYPE(m->m.type), m->m.glb_values, false); + /* value ref not changed, so that caller may still use + * value, even only for free. + */ + m->size--; + log_hash_rehash(m); + return (void*)value; + } + pre = head; + head = head->next; + } + return NULL; +} + +void* +log_hash_get_one(map_t* m) +{ + if (m->size == 0) return NULL; + MAP_ALL(m, node) + return node->key; + MAP_END + return NULL; /* not reachable */ +} + +int32_t +log_hash_print(char* buf, int32_t pos, hash_t* m, bool verbose) +{ + int i, j = 0; + int32_t ktype = KEY_TYPE(m->m.type); + int32_t vtype = VALUE_TYPE(m->m.type); + int32_t htype = COLL_TYPE(m->m.type); + + if (verbose) + pos += sprintf(buf + pos, " hash %s, size=%d, len=%d\n", + htype == ENT_SET ? "set" : "map", m->size, m->len); + else buf[pos++] = '{'; + + for (i = 0;i < m->len;i++) { + map_node_t* head = m->bucket[i]; + if (head == NULL) continue; + if (verbose) pos += sprintf(buf + pos, " [%d] ", i); + + while (head != NULL) { + if (j > 0 && !verbose) buf[pos++] = ','; + if (verbose) + pos += sprintf(buf + pos, " (%x)", + log_hash_code(m, head->key)); + + pos = log_coll_print(buf, pos, head->key, ktype, verbose); + if (htype == ENT_MAP) { + pos += sprintf(buf + pos, "->"); + pos = log_coll_print(buf, pos, head->value, vtype, verbose); + } + + head = head->next; + if (++j > 200) { + pos += sprintf(buf + pos, " ..."); + if (verbose) pos += sprintf(buf + pos, "\n"); + return pos; + } + } + if (verbose) pos += sprintf(buf + pos, "\n"); + } + + if (!verbose) buf[pos++] = '}'; + return pos; +} + +/* ========================================================================== + * LOG VALUES, INDEXES, AND TABLES + * ========================================================================== + */ + +/* + * for tuple used as key passed to hash_* function, tuple->count: + * == 0 indicates compact form, i.e., tuple only contains key fields, and + * hash_code has not been provisioned. + * != 0 indicates tuple from table and hash_code is not the hash code + * for the key fields (is the hash code for tuple) + */ + +log_int_tuple_t* +log_int_tuple_init(TYPE(array_t*, int32_t) a) +{ + log_int_tuple_t* it = calloc(1, + sizeof(log_int_tuple_t) + sizeof(int32_t) * array_size(a)); + it->n_items = array_size(a); + + int32_t code; + hash_code_array_init(&code); + int i = 0; + + ARRAY_ALL_INT(a, v) + it->values[i++] = v; + hash_code_array_add(&code, v); + ARRAY_END + + hash_code_array_final(&code); + it->hash_code = code; + return it; +} + +static log_int_tuple_t* +log_int_tuple_clone(log_int_tuple_t* i) +{ + int32_t sz = sizeof(log_int_tuple_t) + sizeof(int32_t) * i->n_items; + log_int_tuple_t* it = calloc(1, sz); + memcpy(it, i, sz); + return it; +} + +void +log_int_tuple_free(log_int_tuple_t* t) +{ + free(t); +} + +static int32_t +log_int_tuple_print(char* buf, int32_t pos, log_int_tuple_t* t) +{ + int i; + buf[pos++] = '['; + for (i = 0;i < t->n_items;i++) { + if (i > 0) buf[pos++] = ','; + pos += sprintf(buf + pos, "%d", t->values[i]); + } + buf[pos++] = ']'; + return pos; +} + +log_value_t* +log_value_init(const char* v, int32_t size, set_t* gv) +{ + /* two types of values: null terminating string (size == 0) + * and byte array (size indicates the size of byte array) + * data will be copied and saved in global value set. + * the ref_no is increased after calling this. + */ + + log_value_t inp; + inp.value.p = (char*)v; + inp.size = size; + inp.hash_code = log_hash_code_byte(v, &inp.size); + inp.size = -inp.size; /* using value.p */ + + int32_t sv_type = gv->m.type; + gv->m.type = 0; + + log_value_t* setv = set_get(gv, &inp); + gv->m.type = sv_type; + + if (setv != NULL) { + setv->ref_no++; + return setv; + } + + inp.size = -inp.size; + int32_t offset = inp.value.a - ((char*)&inp); + setv = calloc(1, offset + inp.size + 1); + memcpy(setv, &inp, offset); + + memcpy(setv->value.a, v, inp.size); + /* make it null terminate for printing when debugging */ + *((char*)setv->value.a + inp.size) = 0; + + setv->ref_no = 0; + log_hash_add(gv, setv, NULL); + return setv; +} + +static int32_t +log_value_print(char* buf, int32_t pos, log_value_t* value, bool verbose) +{ + if (value == NULL) return pos + sprintf(buf + pos, "<null>"); + else if (verbose) + return pos + sprintf(buf + pos, "%s<r%d,s%d>", + value->value.a, value->ref_no, value->size); + else return pos + sprintf(buf + pos, "%s", value->value.a); +} + +static log_rule_t* +log_rule_init(log_rule_t* rule, set_t* gv) +{ + coll_alloc(&rule, sizeof(log_rule_t), ENT_RULE, gv); + rule->is_union = false; + log_array_init(&rule->rule, ENT_INT32, 0, gv); + log_array_init(&rule->param, ENT_ARRAY, 0, gv); + map_init(&rule->const_param, MAP_INT32_VALUE, 0, gv); + map_init(&rule->param_name_map, MAP_INT32_VALUE, 0, gv); + return rule; +} + +void +log_rule_free(log_rule_t* rule) +{ + log_array_free(&rule->rule); + log_array_free(&rule->param); + map_free(&rule->const_param); + map_free(&rule->param_name_map); + coll_free_ptr(rule); +} + +void +log_rule_set_init(log_rule_set_t* rs, set_t* gv) +{ + coll_alloc(&rs, sizeof(log_rule_set_t), ENT_RULE_SET, gv); + map_init(&rs->rule_name_map, MAP_INT32_VALUE, 0, gv); + map_init(&rs->rule_index_map, + KEY_ID(ENT_VALUE) | VALUE_ID(ENT_INT32), 0, gv); + map_init(&rs->rules, KEY_ID(ENT_INT32) | VALUE_ID(ENT_RULE), 0, gv); + map_init(&rs->table_rule_map, + KEY_ID(ENT_INT32) | VALUE_ID(ENT_ARRAY), 0, gv); + set_init(&rs->input_tables, KEY_ID(ENT_INT32), 0, gv); + set_init(&rs->output_tables, KEY_ID(ENT_INT32), 0, gv); + map_init(&rs->param_size, KEY_ID(ENT_INT32) | VALUE_ID(ENT_INT32), 0, gv); +} + +void +log_rule_set_free(log_rule_set_t* rs) +{ + map_free(&rs->rule_name_map); + map_free(&rs->rule_index_map); + map_free(&rs->rules); + map_free(&rs->table_rule_map); + set_free(&rs->input_tables); + set_free(&rs->output_tables); + map_free(&rs->param_size); + coll_free_ptr(rs); +} + +log_engine_t* +log_engine_init(log_engine_t* log, set_t* gv) +{ + coll_alloc(&log, sizeof(log_engine_t), ENT_LOG_ENG, gv); + map_init(&log->tables, KEY_ID(ENT_INT32) | VALUE_ID(ENT_TABLE), 0, gv); + log_rule_set_init(&log->rule_set, gv); + log->ext_func = NULL; + return log; +} + +void +log_engine_free(log_engine_t* log) +{ + map_free(&log->tables); + log_rule_set_free(&log->rule_set); + coll_free_ptr(log); +} + +/* -------------------------------------------------------------------------- + * TUPLES + * -------------------------------------------------------------------------- + */ + +static log_config_t log_config; + +void log_set_sep(char sep1, char sep2) +{ + log_config.sep1 = sep1; + log_config.sep2 = sep2; +} + +/* indexes will never be manipulated directly. use log_table_add|remove */ + +void +log_tuple_free(log_tuple_t* t, set_t* values, bool free_val) +{ + /* free indicates if value is freed. */ + + if (free_val) { + int i; + for (i = 0;i < t->n_values;i++) + if (t->values[i] != NULL) + log_value_free(t->values[i], values); + } + + if (t->indexes != NULL) free(t->indexes); + free(t); +} + +static void +log_tuple_set_hash_code(log_tuple_t* t, int32_t n_values) +{ + int32_t i, code; + hash_code_array_init(&code); + + for (i = 0;i < n_values;i++) { + /* NULL only for specifying query condition */ + if (t->values[i] != NULL) + hash_code_array_add(&code, t->values[i]->hash_code); + } + + hash_code_array_final(&code); + t->hash_code = code; +} + +log_tuple_t* +log_tuple_init(int32_t n_values) +{ + log_tuple_t* tuple = + calloc(1, sizeof(log_tuple_t) + sizeof(void*) * n_values); + tuple->n_values = n_values; + return tuple; +} + +log_tuple_t* +log_tuple_init_val(log_value_t** val, int32_t n_values) +{ + /* ref will not be updated */ + log_tuple_t* tuple = log_tuple_init(n_values); + if (val != NULL) { + memcpy(tuple->values, val, sizeof(void*) * n_values); + log_tuple_set_hash_code(tuple, n_values); + } + return tuple; +} + +static log_tuple_t* +log_tuple_clone(log_tuple_t* tuple) +{ + int i; + log_tuple_t* nt = log_tuple_init_val(tuple->values, tuple->n_values); + for (i = 0;i < tuple->n_values;i++) log_value_ref(tuple->values[i]); + nt->count = tuple->count; + return nt; +} + +static log_tuple_t* +log_tuple_init_str0(const char* t, char sep, bool use_null, set_t* gv) +{ + /* the value's reference will be added */ + /* the input is in the form of n:f0: ... : fn */ + int64_t count = atoll(t); + if (t[0] < '0' || t[0] > '9') return NULL; + + TYPE(array_t, char*) pos; + log_array_init(&pos, 0, 0, gv); + + char* p = (char*)t; + for (;*p != 0;p++) + if (*p == sep) array_add(&pos, p); + + int32_t size = array_size(&pos); + array_add(&pos, p); + + if (size == 0) { + log_array_free(&pos); + return NULL; + } + + log_tuple_t* tuple = log_tuple_init(size); + tuple->count = count; + + int i; + for (i = 0;i < size;i++) { + if (use_null) { + char c = ((char*)array_get(&pos, i))[1]; + if (c == 0 || c == sep) { + tuple->values[i] = NULL; + continue; + } + } + + log_value_t* val = log_value_init((char*)array_get(&pos, i) + 1, + (char*)array_get(&pos, i + 1) - + ((char*)array_get(&pos, i) + 1), gv); + tuple->values[i] = val; + } + + log_array_free(&pos); + log_tuple_set_hash_code(tuple, size); + return tuple; +} + +log_tuple_t* +log_tuple_init_str_null(const char* t, char sep, set_t* gv) +{ + return log_tuple_init_str0(t, sep, true, gv); +} + +log_tuple_t* +log_tuple_init_str(const char* t, char sep, set_t* gv) +{ + return log_tuple_init_str0(t, sep, false, gv); +} + +int32_t +log_tuple_print(char* buf, int32_t pos, log_tuple_t* t) +{ + pos += sprintf(buf + pos, "%" PRId64, t->count); + buf[pos++] = log_config.sep1; + + int32_t i; + for (i = 0;i < t->n_values;i++) { + if (i > 0) buf[pos++] = log_config.sep1; + pos = log_value_print(buf, pos, t->values[i], false); + } + return pos; +} + +/* -------------------------------------------------------------------------- + * TABLES + * -------------------------------------------------------------------------- + */ + +log_table_t* +log_table_init(log_table_t* tbl, int32_t n, int32_t f, + int32_t size, set_t* gv) +{ + coll_alloc(&tbl, sizeof(log_table_t), ENT_TABLE, gv); + + tbl->table_index = n; + tbl->num_fields = f; + tbl->is_remove = false; + + map_init(&tbl->index_def, + KEY_ID(ENT_INT_TUPLE) | VALUE_ID(ENT_INT32), 0, gv); + log_array_init(&tbl->index_map, KEY_ID(ENT_INDEX), 0, gv); + set_init(&tbl->tuples, KEY_ID(ENT_TUPLE), size, gv); + + tbl->tuples.aux = tbl; + return tbl; +} + +void +log_table_free(log_table_t* tbl) +{ + map_free(&tbl->index_def); + log_array_free(&tbl->index_map); + set_free(&tbl->tuples); + coll_free_ptr(tbl); +} + +static void +log_index_add_node0(log_tuple_t* t, int32_t i_idx) +{ + /* add first tuple for the key of index */ + log_index_i_pre(t, i_idx) = log_index_i_suc(t, i_idx) = t; +} + +static void +log_index_add_node1(log_tuple_t* t, log_tuple_t* head, int32_t i_idx) +{ + log_index_i_suc(t, i_idx) = head; + log_index_i_pre(t, i_idx) = log_index_i_pre(head, i_idx); + log_index_i_suc(log_index_i_pre(head, i_idx), i_idx) = t; + log_index_i_pre(head, i_idx) = t; +} + +static bool +log_index_del_node(log_tuple_t* t, int32_t i_idx) +{ + /* returns true if this is the last node in link */ + if (log_index_i_pre(t, i_idx) == t) return true; + + log_index_i_suc(log_index_i_pre(t, i_idx), i_idx) = + log_index_i_suc(t, i_idx); + + log_index_i_pre(log_index_i_suc(t, i_idx), i_idx) = + log_index_i_pre(t, i_idx); + return false; +} + +static void +log_index_add_tuple(hash_t* index, int32_t i, log_tuple_t* t) +{ + map_node_t* node = log_hash_get(index, t); + if (node == NULL) { + log_index_add_node0(t, i); + log_hash_add(index, t, NULL); + } + else { + log_index_add_node1(t, (log_tuple_t*)node->key, i); + node->key = t; + } +} + +log_tuple_t* +log_index_get_index(log_table_t* tbl, log_tuple_t* t, int32_t i_idx) +{ + /* t is key tuple */ + hash_t* index = array_get(&tbl->index_map, i_idx); + map_node_t* node = log_hash_get(index, t); + + if (node == NULL) return NULL; + return (log_tuple_t*)log_hash_get(index, t)->key; +} + +static void +log_table_add0(log_table_t* tbl, log_tuple_t* t) +{ + /* assume tuple is not present in table, need not free t afterwards */ + /* check where tuple count is set to 1 */ + log_hash_add(&tbl->tuples, t, NULL); + int n_idx = array_size(&tbl->index_def); + if (n_idx > 0) t->indexes = calloc(n_idx * 2, sizeof(void*)); + + /* update index */ + int i; + for (i = 0;i < n_idx;i++) { + hash_t* index = array_get(&tbl->index_map, i); + log_index_add_tuple(index, i, t); + } +} + +static void +log_table_remove0(log_table_t* tbl, log_tuple_t* t) +{ + /* assume tuple is present in table, t has been freed afterwards */ + int i; + /* update index */ + int n_idx = array_size(&tbl->index_def); + + for (i = 0;i < n_idx;i++) { + bool last = log_index_del_node(t, i); + hash_t* index = array_get(&tbl->index_map, i); + map_node_t* node = log_hash_get(index, t); + + if (last) log_hash_del(index, t); + else if (node->key == t) node->key = log_index_i_suc(t, i); + } + + log_hash_del(&tbl->tuples, t); +} + +static void +log_table_add_extra(log_table_t* tbl, log_tuple_t* t) +{ + /* add or merge count, will be referred (add) or freed (merge count) */ + log_tuple_t* et = set_get(&tbl->tuples, t); + if (et == NULL) log_table_add0(tbl, t); + else { + et->count += t->count; + log_tuple_free(t, tbl->m.glb_values, true); + } +} + +void +log_table_add(log_table_t* tbl, log_tuple_t* t) +{ + /* validation, t not in table */ + ovs_assert(!set_has(&tbl->tuples, t) && tbl->num_fields == t->n_values); + log_table_add0(tbl, t); +} + +void +log_table_remove(log_table_t* tbl, log_tuple_t* t) +{ + /* validation, must have this and t is from table */ + ovs_assert(set_get(&tbl->tuples, t) == t); + log_table_remove0(tbl, t); +} + +int32_t +log_table_add_index(log_table_t* tbl, log_int_tuple_t* index_key) +{ + /* add new index for table or returning existing one */ + /* index_key will be cloned */ + + map_node_t* index = log_hash_get(&tbl->index_def, index_key); + if (index != NULL) return ptr2i(index->value); + + log_int_tuple_t* ikey = log_int_tuple_clone(index_key); + int32_t index_id = map_size(&tbl->index_def); + map_add(&tbl->index_def, ikey, i2ptr(index_id)); + + map_t* new_i = log_hash_init(NULL, COLL_ID(ENT_INDEX), + SZ_INIT_HASH, tbl->m.glb_values); + array_add(&tbl->index_map, new_i); + new_i->aux = ikey; + + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) + tuple->indexes = realloc( + tuple->indexes, (index_id + 1) * 2 * sizeof(void*)); + log_index_add_tuple(new_i, index_id, tuple); + SET_END + + return index_id; +} + +int32_t +log_index_print(char* buf, int32_t pos, log_table_t* t) +{ + int i = 0, j = 0; + ARRAY_ALL(&t->index_map, index, hash_t*) + log_int_tuple_t* def = index->aux; + pos += sprintf(buf + pos, " index="); + pos = log_int_tuple_print(buf, pos, def); + buf[pos++] = '\n'; + + SET_ALL(index, key, log_tuple_t*) + pos += sprintf(buf + pos, " key set: "); + INDEX_ALL(key, i, t1) + pos = log_tuple_print(buf, pos, t1); + buf[pos++] = ' '; + + if (++j > 200) { + pos += sprintf(buf + pos, " ...\n"); + return pos; + } + INDEX_END + buf[pos++] = '\n'; + SET_END + i++; + ARRAY_END + return pos; +} + +int32_t +log_table_print(char* buf, int32_t pos, log_table_t* t, bool verbose) +{ + if (verbose) { + pos += sprintf(buf + pos, " tbl id=%d fd=%d sz=%d\n", + t->table_index, t->num_fields, table_size(t)); + } + pos = log_hash_print(buf, pos, &t->tuples, verbose); + if (verbose) pos = log_index_print(buf, pos, t); + return pos; +} + +int32_t +log_rule_set_print(char* buf, int32_t pos, log_rule_set_t* rs) +{ + pos += sprintf(buf + pos, "rule_name_map\n"); + pos = log_hash_print(buf, pos, &rs->rule_name_map, false); + + pos += sprintf(buf + pos, "\nrule_index_map\n"); + pos = log_hash_print(buf, pos, &rs->rule_index_map, false); + + pos += sprintf(buf + pos, "\ntable_rule_map\n"); + pos = log_hash_print(buf, pos, &rs->table_rule_map, false); + + pos += sprintf(buf + pos, "\nparam_size\n"); + pos = log_hash_print(buf, pos, &rs->param_size, false); + + pos += sprintf(buf + pos, "\ninput_tables\n"); + pos = log_hash_print(buf, pos, &rs->input_tables, false); + + pos += sprintf(buf + pos, "\noutput_tables\n"); + pos = log_hash_print(buf, pos, &rs->output_tables, false); + + pos += sprintf(buf + pos, "\nrules\n"); + pos = log_hash_print(buf, pos, &rs->rules, false); + return pos; +} + +static int32_t +log_rule_print(char* buf, int32_t pos, log_rule_t* r) +{ + pos += sprintf(buf + pos, r->is_union ? "(union," : "(join,"); + pos += sprintf(buf + pos, "rule="); + pos = log_array_print(buf, pos, &r->rule, false); + pos += sprintf(buf + pos, ",param="); + pos = log_array_print(buf, pos, &r->param, false); + pos += sprintf(buf + pos, ",name="); + pos = log_hash_print(buf, pos, &r->param_name_map, false); + pos += sprintf(buf + pos, ",const="); + pos = log_hash_print(buf, pos, &r->const_param, false); + pos += sprintf(buf + pos, ")\n"); + return pos; +} + +int32_t +log_coll_print(char* buf, int pos, void* item, int32_t type, bool verbose) +{ + /* do not check the buf limit */ + if (item == NULL && type != ENT_INT32 && type != ENT_TST_INT32) + return pos; + + else if (type == ENT_INT32 || type == ENT_TST_INT32) + pos += sprintf(buf + pos, "%d", ptr2i(item)); + else if (type == ENT_STR) + pos += sprintf(buf + pos, "%s", (const char*)item); + else if (type == ENT_VALUE) + pos = log_value_print(buf, pos, item, verbose); + else if (type == ENT_INT_TUPLE) + pos = log_int_tuple_print(buf, pos, item); + else if (type == ENT_TUPLE) + pos = log_tuple_print(buf, pos, item); + else if (type == ENT_ARRAY) + pos = log_array_print(buf, pos, item, verbose); + else if (type == ENT_SET || type == ENT_MAP) + pos = log_hash_print(buf, pos, item, verbose); + else if (type == ENT_TABLE) + pos = log_table_print(buf, pos, item, verbose); + else if (type == ENT_RULE) + pos = log_rule_print(buf, pos, item); + else if (type == ENT_RULE_SET) + pos = log_rule_set_print(buf, pos, item); + return pos; +} + +/* ========================================================================== + * SYNTAX PROCESSING + * ========================================================================== + */ + +/* + * token of id: [a-zA-Z_][a-zA-Z0-9_]* + * all upper case: output table; all lower case: input table; others: + * intermediate. + * <table_name> ( <param_name>, ... ) ':'|'>' <table_name> ( + * <param_name> | 'value' | - ), ... ; + * ':' is for join. '>' is for union. order is always important + * special table: not used now, e.g., could be used to specify language + * parameters. join is preferred with external function as there is more + * flexibility in param. comments start with # document started with ## + */ + +struct log_sync_s { + const char* text; + int len; + + int curpos; + char curchar; + char curtoken; + + char token[LOG_TOKEN_SZ]; + int token_pos; + set_t* gv; +}; + +static struct log_sync_s sync; + +void +log_sync_init(const char* log, set_t* gv) +{ + sync.len = strlen(log); + sync.text = log; + sync.curpos = 0; + sync.token_pos = 0; + sync.gv = gv; +} + +static void +sync_getc(void) +{ + sync.curchar = sync.text[sync.curpos++]; +} + +static bool +sync_eof(void) +{ + return sync.curpos >= sync.len; +} + +static void +sync_error(const char* s0, const log_value_t* s1) +{ + printf("syntax error: %s %s \n%s\n", s0, + s1 == NULL ? "" : s1->value.a, + sync.text + sync.curpos); + exit(1); +} + +static void +sync_init_token(void) +{ + sync.token_pos = 0; +} + +static log_value_t* +sync_get_value(void) +{ + log_value_t* token = log_value_init(sync.token, sync.token_pos, sync.gv); + return token; +} + +static log_value_t* +sync_get_const(const char* c) +{ + log_value_t* token = log_value_init(c, 0, sync.gv); + return token; +} + +static void +sync_append_c(void) +{ + sync.token[sync.token_pos++] = sync.curchar; + sync_getc(); + if (sync.token_pos >= LOG_TOKEN_SZ) sync_error("token too long", 0); +} + +static void +sync_gett(void) +{ + bool in_comment = false; + bool in_literal = false; + bool in_ident = false; + + while (!sync_eof()) { + if (in_comment) { + if (sync.curchar == '\n') { + in_comment = false; + } + sync_append_c(); + } + else if (in_literal) { + if (sync.curchar == '\'') { + sync_getc(); + sync.curtoken = 's'; + return; + } + else { + sync_append_c(); + } + } + else if (in_ident) { + if (sync.curchar != '_' && !isalpha(sync.curchar) + && !isdigit(sync.curchar)) { + sync.curtoken = 't'; + return; + } + sync_append_c(); + } + else { + if (sync.curchar == '#') { + in_comment = true; + sync_getc(); + } + else if (sync.curchar == '\'') { + in_literal = true; + sync_init_token(); + sync_getc(); + } + else if (sync.curchar == '_' || isalpha(sync.curchar)) { + sync_init_token(); + sync_append_c(); + in_ident = true; + } + else if (strchr(":>().,-;", sync.curchar) != NULL) { + sync.curtoken = sync.curchar; + sync_getc(); + return; + } + else if (isspace(sync.curchar)) sync_getc(); + else if (sync.curchar == '\n') sync_getc(); + else sync_error("unknown char near", 0); + } + } + + /* for last period without following chars */ + sync.curtoken = sync.curchar; +} + +static void +sync_nt_params(array_t* list) +{ + /* ( ( param | 'literal' | - )* , ) */ + if (sync.curtoken != '(') sync_error("expecting (", 0); + sync_gett(); + + for (;;) { + if (sync.curtoken == 't') { + array_add(list, sync_get_const("t")); + array_add(list, sync_get_value()); + } + else if (sync.curtoken == '-') { + array_add(list, sync_get_const("-")); + array_add(list, NULL); + } + else if (sync.curtoken == 's') { + array_add(list, sync_get_const("s")); + array_add(list, sync_get_value()); + } + else sync_error("expecting param, literal, or -", NULL); + + sync_gett(); + if (sync.curtoken == ',') { + sync_gett(); + continue; + } + else if (sync.curtoken == ')') { + sync_gett(); + return; + } + else sync_error("expecting , or )", NULL); + } +} + +static void +sync_nt_table(array_t* list) +{ + if (sync.curtoken != 't') sync_error("table name expected", NULL); + log_value_t* table_name = sync_get_value(); + + sync_gett(); + array_add(list, NULL); /* will be overrided later */ + array_add(list, table_name); + sync_nt_params(list); +} + +void +log_sync_parse(map_t* sem) +{ + /* + * tableName -> (left side, right side table 0, right side table 1, ...) + * each table contains (table name, param0, param1, ...) + */ + + map_init(sem, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_ARRAY), 0, sync.gv); + sync.gv = sem->m.glb_values; + + sync_getc(); + sync_gett(); + + for (;;) { + array_t* tables = log_array_init(NULL, KEY_ID(ENT_ARRAY), 0, sync.gv); + array_t* tbl = log_array_init(NULL, KEY_ID(ENT_VALUE), 0, sync.gv); + + sync_nt_table(tbl); + array_add(tables, tbl); + log_value_t* table_name = array_get(tbl, 1); + + if (sync.curtoken != ':' && sync.curtoken != '>') { + sync_error("expecting : or >", NULL); + } + + array_set(tbl, 0, sync_get_const(sync.curtoken == '>' ? "u" : "j")); + sync_gett(); + + for (;;) { + tbl = log_array_init(NULL, KEY_ID(ENT_VALUE), 0, sync.gv); + sync_nt_table(tbl); + array_add(tables, tbl); + if (sync.curtoken == ';' || sync.curtoken == '.') break; + } + + if (map_get(sem, table_name) != NULL) { + sync_error("definition existed: ", table_name); + } + + map_add(sem, table_name, tables); + if (sync.curtoken == ';') sync_gett(); + else if (sync.curtoken == '.') return; + } +} + +/* ========================================================================== + * SYNTAX PROCESSING + * ========================================================================== + */ + +void +log_sort_array(int start, array_t* list, array_t* sem1, array_t* sem2) +{ + /* this must be stable sort. see the sort for table size */ + + int i, j; + for (i = start;i < list->size;i++) { + int index = i; + + for (j = i + 1;j < list->size;j++) + if (ptr2i(list->item[j]) < ptr2i(list->item[index])) + index = j; + + void* newi = (void*)list->item[index]; + void* newv1 = sem1 == NULL ? NULL : (void*)sem1->item[index]; + void* newv2 = sem2 == NULL ? NULL : (void*)sem2->item[index]; + + memmove(&list->item[i + 1], &list->item[i], + (index - i) * sizeof(void*)); + + if (sem1 != NULL) + memmove(&sem1->item[i + 1], &sem1->item[i], + (index - i) * sizeof(void*)); + if (sem2 != NULL) + memmove(&sem2->item[i + 1], &sem2->item[i], + (index - i) * sizeof(void*)); + + list->item[i] = newi; + if (sem1 != NULL) sem1->item[i] = newv1; + if (sem2 != NULL) sem2->item[i] = newv2; + } +} + +int +log_insert_item(int val, array_t* list, void* obj1, + void* obj2, array_t* sem1, array_t* sem2) +{ + /* + * returns -1 if there is no equal value in the list, or the position to + * insert. If there is tie, the position is after all items of equal + * value. will insert only if there is no equal value. + */ + + int count; + for (count = 0;count < list->size;count++) { + if (val < ptr2i(list->item[count])) break; + } + + array_ins(list, count, i2ptr(val)); + if (sem1 != NULL) array_ins(sem1, count, obj1); + if (sem2 != NULL) array_ins(sem2, count, obj2); + return count; +} + +void +log_topo_sort(TYPE2(map_t*, log_value_t*, set_t*) g, /* set of value */ + TYPE(array_t* order, log_value_t*), + TYPE(set_t*, log_value_t*) in_nodes, + TYPE(set_t*, log_value_t*) out_nodes) +{ + /* input g will be destroyed after sort */ + + set_t* gv = g->m.glb_values; + log_array_init(order, KEY_ID(ENT_VALUE), 0, gv); + set_init(in_nodes, KEY_ID(ENT_VALUE), 0, gv); + set_init(out_nodes, KEY_ID(ENT_VALUE), 0, gv); + + set_t right_nodes, all_nodes, to_remove; + set_init(&right_nodes, KEY_ID(ENT_VALUE), 0, gv); + set_init(&all_nodes, KEY_ID(ENT_VALUE), 0, gv); + + MAP_ALL(g, node) + set_add(&all_nodes, node->key); + + SET_ALL(node->value, node1, log_value_t*) + set_add(&all_nodes, node1); + set_add(&right_nodes, node1); + SET_END /* end of iteration */ + MAP_END /* end of iteration */ + + SET_ALL(&all_nodes, key, log_value_t*) + if (!map_has(g, key)) set_add(in_nodes, key); + else if (!set_has(&right_nodes, key)) + set_add(out_nodes, key); + SET_END /* end of iteration */ + + while (set_size(&all_nodes) > 0) { + log_value_t* next = NULL; + + SET_ALL(&all_nodes, key, log_value_t*) + if (!map_has(g, key)) { + next = key; + SET_EXIT; + } + SET_END + + if (next == NULL) + sync_error("circular graph, check around ", + log_hash_get_one(&all_nodes)); + + array_add(order, next); + set_del(&all_nodes, next); + set_init(&to_remove, KEY_ID(ENT_VALUE), 0, gv); + + MAP_ALL(g, node) + set_del(node->value, next); + if (set_size((set_t*)node->value) == 0) + set_add(&to_remove, node->key); + MAP_END + + SET_ALL(&to_remove, key, log_value_t*) + set_free(map_del(g, key)); + SET_END + set_free(&to_remove); + } + + set_free(&right_nodes); + set_free(&all_nodes); + map_free(g); +} + +static int +check_name(log_value_t* t) +{ + /* check if string are in all lower case (> 0), all upper case (<0) + * or mixed (== 0) + */ + const char* s = t->value.a; + bool lower = false; + bool upper = false; + char c; + + while ((c = *s++) != 0) { + if (islower(c)) lower = true; + else if (isupper(c)) upper = true; + } + + if (lower && upper) return 0; + else if (lower) return 1; + else if (upper) return -1; + else return 0; +} + +/* ========================================================================== + * SEMANTICS DEFINITION + * ========================================================================== + */ + +void +log_sem_process(log_rule_set_t* rule_set, map_t* sem) +{ + /* string -> array of array of string */ + set_t* gv = sem->m.glb_values; + /* STEP 1: check table dependency and assign digit table / rule index */ + map_t rules; /* map string to (set of value) */ + map_init(&rules, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_SET), 0, gv); + + log_value_t* rname; + MAP_ALL(sem, node) + rname = NULL; + set_t* dep = set_init(NULL, KEY_ID(ENT_VALUE), 0, gv); + + ARRAY_ALL((array_t*)node->value, t, array_t*) + log_value_t* nm = array_get(t, 1); + if (rname == NULL) rname = nm; + else set_add(dep, nm); + ARRAY_END + map_add(&rules, rname, dep); + MAP_END + + array_t topo_order; + set_t topo_in, topo_out; + log_topo_sort(&rules, &topo_order, &topo_in, &topo_out); + + int32_t rule_index = 0; + ARRAY_ALL(&topo_order, name, log_value_t*) + map_add(&rule_set->rule_name_map, i2ptr(rule_index), name); + map_add(&rule_set->rule_index_map, name, i2ptr(rule_index++)); + ARRAY_END + + /* STEP 2: check upper case, lower case and mixed */ + SET_ALL(&topo_in, name, log_value_t*) + if (check_name(name) <= 0) + sync_error("input must be all lower case: ", name); + + set_add(&rule_set->input_tables, + map_get(&rule_set->rule_index_map, name)); + SET_END + + SET_ALL(&topo_out, name, log_value_t*) + if (check_name(name) >= 0) + sync_error("output must be all upper case: ", name); + + set_add(&rule_set->output_tables, + map_get(&rule_set->rule_index_map, name)); + SET_END + + set_t intr; + set_init(&intr, KEY_ID(ENT_VALUE), 0, gv); + + MAP_ALL(sem, name) + if (map_has(&topo_out, i2ptr(name->key))) continue; + if (check_name(i2ptr(name->key)) != 0) + sync_error("intermediate must be mixed: ", name->key); + MAP_END + + /* + * STEP 3: check table size consistency and assign digit index. + * and if union / join on itself, it could only perform twice on + * each param, i.e., X : A, A, A is not supported. + */ + map_t table_size; + map_init(&table_size, KEY_ID(ENT_VALUE), 0, gv); /* map string to int */ + + MAP_ALL(sem, name) + array_t* val = name->value; /* array of array of value */ + log_rule_t* rule = log_rule_init(NULL, gv); + + /* rule_id is integer */ + void* rule_id = map_get(&rule_set->rule_index_map, name->key); + map_add(&rule_set->rules, rule_id, rule); + + log_value_t* rule_t = array_get(array_get(val, 0), 0); + rule->is_union = strcmp("u", rule_t->value.a) == 0; + + int const_value = -2; /* -1 is for ignore */ + + /* reorder table sequence */ + ARRAY_ALL(val, t, array_t*) + log_value_t* table_name = array_get(t, 1); + void* rule_id = map_get(&rule_set->rule_index_map, table_name); + array_add(&rule->rule, rule_id); + ARRAY_END + + int i; + log_sort_array(1, &rule->rule, val, NULL); + + for (i = 2;i < rule->rule.size - 1;i++) { + if (array_get(&rule->rule, i - 1) == array_get(&rule->rule, i) && + array_get(&rule->rule, i + 1) == array_get(&rule->rule, i)) + sync_error( + "cannot join / union on itself for more than twice: ", + name->key); + } + + int param_size = 0; + bool is_left = true; + bool left_has_param = false; + bool left_has_value = false; + + map_t param_map; /* map string to integer */ + map_init(&param_map, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_INT32), 0, gv); + + ARRAY_ALL(val, t, array_t*) + array_t* param_list = log_array_init(NULL, KEY_ID(ENT_INT32), + 0, gv); + array_add(&rule->param, param_list); + log_value_t* table_name = array_get(t, 1); + + int32_t size = ((array_t*)t)->size / 2 - 1; + int32_t table_id = map_get_int( + &rule_set->rule_index_map, table_name); + map_add(&rule_set->param_size, i2ptr(table_id), i2ptr(size)); + + /* check table size */ + if (rule->is_union) { + if (is_left) param_size = size; + else if (size != param_size) + sync_error("table param size mismatch in union rule: ", + name->key); + } + else { + struct map_node_s* ksize = log_hash_get( + &table_size, table_name); + if (ksize == NULL) + map_add(&table_size, table_name, i2ptr(size)); + + else if (ptr2i(ksize->value) != size) + sync_error("table param size mismatch in join rule: " , + name->key); + } /* if check table size */ + + /* assign index to each param */ + for (i = 0; i < size;i++) { + log_value_t* param_type = array_get(t, i * 2 + 2); + log_value_t* param_value = array_get(t, i * 2 + 3); + + if (is_left && strcmp(param_type->value.a, "-") == 0) + sync_error("left cannot have 'ignore': ", name->key); + + if (strcmp(param_type->value.a, "-") == 0) { + array_add(param_list, i2ptr(-1)); + } + + else if (strcmp(param_type->value.a, "t") == 0) { + if (is_left) left_has_param = true; + map_node_t* param_no = log_hash_get( + &param_map, param_value); + + void* no; /* type is int */ + if (param_no == NULL) { + no = i2ptr(param_map.size); + map_add(&param_map, param_value, no); + map_add(&rule->param_name_map, no, param_value); + } else no = param_no->value; + array_add(param_list, no); + } + + else if (strcmp(param_type->value.a, "s") == 0) { + if (is_left) left_has_value = true; + void* c_value = param_value; + map_add(&rule->const_param, i2ptr(const_value), c_value); + array_add(param_list, i2ptr(const_value--)); + } + } /* for each table param */ + is_left = false; + ARRAY_END /* for each table */ + + map_free(&param_map); + if (!left_has_param) + sync_error("left must have param: ", name->key); + if (rule->is_union && left_has_value) + sync_error("left cannot have const: ", name->key); + MAP_END /* for each rule */ + + /* STEP 4: check param reference. */ + MAP_ALL(&rule_set->rules, rule_no) + array_t* left_param = NULL; /* array of int, for checking union */ + + log_rule_t* rule = map_get(&rule_set->rules, rule_no->key); + log_value_t* rule_name = map_get( + &rule_set->rule_name_map, rule_no->key); + + /* add table rule map */ + int32_t i; + for (i = 1;i < rule->rule.size;i++) { + array_t* set = map_get(&rule_set->table_rule_map, + array_get(&rule->rule, i)); + + if (set == NULL) { + set = log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); + map_add(&rule_set->table_rule_map, + array_get(&rule->rule, i), set); + } + + int32_t found = log_array_look_for(set, rule_no->key); + if (found < 0) array_add(set, rule_no->key); + } + + ARRAY_ALL(&rule->param, param, array_t*) + /* param is array of int */ + if (left_param == NULL) left_param = param; + + if (rule->is_union) { + if (left_param != NULL) { + bool not_found = false; + + ARRAY_ALL(left_param, item, void*) + if (log_array_look_for(param, item) < 0) { + not_found = true; + ARRAY_EXIT; + } + ARRAY_END + + if (not_found) + sync_error("union param not found in ", rule_name); + } + continue; + } + + /* + * for right side param, it must also appear either in left + * side or right side or being 'ignored'; for left side param, + * it must appear in right side + */ + ARRAY_ALL((array_t*)param, p0no, void*) + int32_t p0 = ptr2i(p0no); + if (p0 < 0) continue; + + bool found = false; + ARRAY_ALL(&rule->param, param1, array_t*) + if (param == param1) continue; + + if (log_array_look_for(param1, p0no) >= 0) { + found = true; + ARRAY_EXIT; + } + ARRAY_END + + if (!found) + sync_error("not used / undefined param ", rule_name); + ARRAY_END + ARRAY_END /* for each table */ + MAP_END /* for each rule */ + + /* STEP 5: sort table_rule_map */ + MAP_ALL(&rule_set->table_rule_map, node) + array_t* list = node->value; + log_sort_array(0, list, NULL, NULL); + MAP_END + + set_free(&intr); + set_free(&topo_in); + set_free(&topo_out); + map_free(&table_size); + log_array_free(&topo_order); +} + +/* ========================================================================== + * TABLE OPERATION + * ========================================================================== + */ + +static log_tuple_t* +tblopr_reorder_tuple(log_tuple_t* t, TYPE(array_t*, int32_t) order, + TYPE2(map_t*, int32_t, log_value_t*) const_map) +{ + /* input and output table could be different. order is sequence + * instead of param index. + * (a, b, c, d, e) + (2, -2, 1, 4) => (c, C[-2], b, e) + */ + log_tuple_t* newt = log_tuple_init(array_size(order)); + int i; + for (i = 0;i < array_size(order);i++) { + log_value_t* value; + int32_t order_i = array_get_int(order, i); + + if (order_i >= 0) value = t->values[order_i]; + else if (order_i < -1) value = map_get(const_map, i2ptr(order_i)); + else ovs_assert(false); /* reorder sees constants */ + + log_value_ref(value); + newt->values[i] = value; + } + + newt->count = t->count; + log_tuple_set_hash_code(newt, array_size(order)); + return newt; +} + +static void +tblopr_reorder_table(log_table_t* input, TYPE(array_t*, int32_t) order, + TYPE2(map_t*, int32_t, log_value_t*) const_map, + log_table_t* output) +{ + /* input and output table could be different */ + SET_ALL(&input->tuples, t, log_tuple_t*) + log_table_add(output, tblopr_reorder_tuple(t, order, const_map)); + SET_END +} + +static bool +tblopr_match_const(log_tuple_t* t, TYPE(array_t*, int32_t) cpos, + log_value_t** cval) +{ + int i; + for (i = 0;i < array_size(cpos);i++) { + int32_t ci = array_get_int(cpos, i); + if (t->values[ci] != cval[i]) return false; + } + return true; +} + +static log_table_t* +tblopr_query_table(log_table_t* input, TYPE(array_t*, int32_t) param, + log_value_t** val) +{ + /* input only used for get table param */ + + log_table_t* output = + log_table_init(NULL, input->table_index, input->num_fields, + 0, input->m.glb_values); + + if (array_size(param) == 0) { + SET_ALL(&input->tuples, t, log_tuple_t*) + log_tuple_t* nt = log_tuple_clone(t); + log_table_add(output, nt); + SET_END + return output; + } + + log_int_tuple_t* index = log_int_tuple_init(param); + int32_t idx = log_table_add_index(input, index); + log_int_tuple_free(index); + + log_tuple_t* key = log_tuple_init_val(val, array_size(param)); + log_tuple_t* set = log_index_get_index(input, key, idx); + + /* check set is null? */ + INDEX_ALL(set, idx, t) + log_tuple_t* nt = log_tuple_clone(t); + log_table_add(output, nt); + INDEX_END + + log_tuple_free(key, input->m.glb_values, false); + return output; +} + +static void +tblopr_merge_tuple(log_table_t* tbl, log_tuple_t* tup, + bool negative, bool free_t) +{ + /* tuple untouched */ + log_tuple_t* org_tuple = set_get(&tbl->tuples, tup); + + if (org_tuple == NULL) { + log_tuple_t* nt = free_t ? tup : log_tuple_clone(tup); + log_table_add0(tbl, nt); + } + else { + if (negative) org_tuple->count -= tup->count; + else org_tuple->count += tup->count; + if (free_t) log_tuple_free(tup, tbl->m.glb_values, true); + } +} + +static void +tblopr_merge_table(log_table_t* src, log_table_t* dst, bool negative) +{ + SET_ALL(&src->tuples, t, log_tuple_t*) + tblopr_merge_tuple(dst, t, negative, false); + SET_END +} + +static void +tblopr_final_delta(log_table_t* source, log_table_t* dest) +{ + if (source->is_remove) { + SET_ALL(&source->tuples, t, log_tuple_t*) + log_tuple_t* ot = set_get(&dest->tuples, t); + log_table_remove0(dest, ot); + SET_END + } + else { + SET_ALL(&source->tuples, t, log_tuple_t*) + ovs_assert(set_get(&dest->tuples, t) == NULL); + log_tuple_t* nt = log_tuple_clone(t); + log_table_add0(dest, nt); + SET_END + } +} + +static void +tblopr_match_reorder_and_merge(log_table_t* input, log_table_t* output, + TYPE(array_t*, int32_t) parami, + TYPE(array_t*, int32_t) paramo, + TYPE2(map_t*, int32_t, log_value_t*) const_map) +{ + /* example: output(0, 1, 2) input(2, 1, -, 'const', 0) + * input count is ignored. parami/o is param number, not + * sequence number. + */ + + set_t* gv = input->m.glb_values; + TYPE(array_t*, int32_t) cpos = log_array_init( + NULL, KEY_ID(ENT_INT32), 0, gv); + TYPE(array_t*, int32_t) order = log_array_init( + NULL, KEY_ID(ENT_INT32), 0, gv); + int pos = 0, i; + + for (i = 0;i < array_size(parami);i++) { + int32_t c = array_get_int(parami, i); + if (c < -1) array_add(cpos, i2ptr(i)); + else if (c == -1) continue; + else { + int32_t op = log_array_look_for(parami, array_get(paramo, pos++)); + if (op >= 0) array_add(order, i2ptr(op)); + } + } + + log_value_t** cval = calloc(array_size(cpos), sizeof(void*)); + for (i = 0;i < array_size(cpos);i++) + cval[i] = (log_value_t*) + map_get(const_map, array_get(parami, + array_get_int(cpos, i))); + + SET_ALL(&input->tuples, t, log_tuple_t*) + if (tblopr_match_const(t, cpos, cval)) { + log_tuple_t* nt = tblopr_reorder_tuple(t, order, NULL); + log_table_add_extra(output, nt); + } + SET_END + + free(cval); + log_array_free(cpos); + log_array_free(order); +} + +static void +tblopr_combine_tuple(log_table_t* res, int32_t tuple1_count, + log_join_param_t* joinp, int r1_fnum, + log_tuple_t* tuple2, log_value_t** tuple_values) +{ + int i; + for (i = 0;i < array_size(&joinp->rem2);i++) + tuple_values[i + r1_fnum] = tuple2->values[ + ptr2i(array_get(&joinp->rem2, i))]; + + int32_t res_num_fields = res->num_fields; + log_tuple_t* nt = log_tuple_init_val(tuple_values, res->num_fields); + nt->count = tuple1_count; + + for (i = 0;i < res_num_fields;i++) log_value_ref(nt->values[i]); + tblopr_merge_tuple(res, nt, false, true); +} + +static log_table_t* +tblopr_cond_join(log_table_t* t1, log_table_t* t2, log_join_param_t* joinp) +{ + /* + * t1 is intermediate table during join; t2 is original table. + * select values joinp.select1 from t1 and const, and match that with + * params indicated by joinp.index2. + */ + set_t* gv = t1->m.glb_values; + + int32_t r1_fnum = array_size(&joinp->rem1); + int32_t r2_fnum = array_size(&joinp->rem2); + + log_table_t* res = log_table_init(NULL, -1, r1_fnum + r2_fnum, 0, gv); + int32_t t2_index = map_get_int(&t2->index_def, joinp->index2); + + int t1val_sz = array_size(&joinp->select1); + TYPE(array_t*, int32_t) t1param = log_array_init( + NULL, KEY_ID(ENT_INT32), t1val_sz, gv); + log_tuple_t* key_tuple = log_tuple_init_val(NULL, t1val_sz); + + int i; + for (i = 0;i < t1val_sz;i++) { + log_value_t* obj = array_get(&joinp->select1, i); + if (obj != NULL) { + array_add(t1param, i2ptr(-1)); /* mark as not set */ + key_tuple->values[i] = obj; + } + else array_add(t1param, array_get(&joinp->select1i, i)); + } + + /* loop over t1 and join */ + log_value_t** tuple_values = calloc(sizeof(void*), res->num_fields); + SET_ALL(&t1->tuples, tuple1, log_tuple_t*) + + for (i = 0;i < array_size(t1param);i++) { + int32_t t1p = ptr2i(array_get(t1param, i)); + if (t1p < 0) continue; + key_tuple->values[i] = tuple1->values[t1p]; + } + + log_tuple_set_hash_code(key_tuple, t1val_sz); + log_tuple_t* match_tuples = + log_index_get_index(t2, key_tuple, t2_index); + + if (match_tuples != NULL) { /* do not use continue */ + for (i = 0;i < r1_fnum;i++) + tuple_values[i] = + tuple1->values[ptr2i(array_get(&joinp->rem1, i))]; + + /* join the value */ + INDEX_ALL(match_tuples, t2_index, tuple2) + tblopr_combine_tuple(res, tuple1->count, + joinp, r1_fnum, tuple2, tuple_values); + INDEX_END + } + SET_END + + log_tuple_free(key_tuple, t1->m.glb_values, false); + log_array_free(t1param); + free(tuple_values); + return res; +} + +static void +tblopr_gen_delta(log_table_t* source, log_table_t* dest) +{ + bool is_remove = source->is_remove; + set_t* gv = dest->m.glb_values; + + SET_ALL(&source->tuples, st, log_tuple_t*) + ovs_assert(st->indexes == NULL); + + int64_t st_count = st->count; + log_tuple_t* dt = set_get(&dest->tuples, st); + + if (is_remove) { + ovs_assert(dt != NULL && dt->count >= st_count); + if (dt->count > st_count) { + dt->count -= st_count; + SET_REMOVE_ITEM; + log_tuple_free(st, gv, true); + } + } + else { + if (dt != NULL) { + dt->count += st_count; + SET_REMOVE_ITEM; + log_tuple_free(st, gv, true); + } + } + SET_END +} + +static void +tblopr_merge_delta(log_table_t* source, log_table_t* dest, + log_table_t* dest_ivt) +{ + + /* dest is of the same operation as source, while destIvt is opposite. */ + ovs_assert(source->table_index == dest->table_index); + set_t* gv = dest->m.glb_values; + + SET_ALL(&source->tuples, st, log_tuple_t*) + ovs_assert(st->indexes == NULL); + log_tuple_t* dt = set_get(&dest->tuples, st); + + if (dt != NULL) { + SET_REMOVE_ITEM; + dt->count += st->count; /* count will not change hash */ + log_tuple_free(st, gv, true); + /* will not match opposite */ + } + else { + log_tuple_t* dt_ivt = set_get(&dest_ivt->tuples, st); + /* will be added later for dt_ivt == NULL */ + if (dt_ivt != NULL) { + /* cross merge */ + long st_count = st->count; + dt_ivt->count -= st_count; + + if (dt_ivt->count >= 0) { + SET_REMOVE_ITEM; + log_tuple_free(st, gv, true); + } + + if (dt_ivt->count == 0) log_table_remove0(dest_ivt, dt_ivt); + else if (dt_ivt->count < 0) { /* move to opposite table */ + dt_ivt->count = -dt_ivt->count; + log_tuple_t* nt = log_tuple_clone(dt_ivt); + log_table_add(dest, nt); + log_table_remove0(dest_ivt, dt_ivt); + } + } + } + SET_END /* for source tuple */ + + /* add remaining */ + SET_ALL(&source->tuples, t, log_tuple_t*) + log_tuple_t* nt = log_tuple_clone(t); + log_table_add0(dest, nt); + SET_END +} + +static log_table_t* +tblopr_full_join(log_table_t* t1, log_table_t* t2, log_join_param_t* joinp) +{ + /* select1 contains only constants */ + + int32_t r1_fnum = array_size(&joinp->rem1); + log_table_t* res = log_table_init( + NULL, -1, r1_fnum + array_size(&joinp->rem2), 0, t1->m.glb_values); + + int32_t i; + log_value_t** tuple_values = calloc(sizeof(void*), res->num_fields); + + if (joinp->index2 == NULL) { + SET_ALL(&t1->tuples, tuple1, log_tuple_t*) + for (i = 0;i < r1_fnum;i++) + tuple_values[i] = tuple1->values[ + ptr2i(array_get(&joinp->rem1, i))]; + + SET_ALL(&t2->tuples, tuple2, log_tuple_t*) + tblopr_combine_tuple(res, tuple1->count, + joinp, r1_fnum, tuple2, tuple_values); + SET_END + SET_END + } + + else { + int32_t t2_index = map_get_int(&t2->index_def, joinp->index2); + int t1val_sz = array_size(&joinp->select1); + log_value_t** t1val = calloc(sizeof(void*), t1val_sz); + + for (i = 0;i < t1val_sz;i++) + t1val[i] = array_get(&joinp->select1, i); + + log_tuple_t* key_tuple = log_tuple_init_val(t1val, t1val_sz); + log_tuple_t* match_tuples = log_index_get_index( + t2, key_tuple, t2_index); + + free(t1val); + log_tuple_free(key_tuple, t1->m.glb_values, false); + + SET_ALL(&t1->tuples, tuple1, log_tuple_t*) + for (i = 0;i < r1_fnum;i++) + tuple_values[i] = tuple1->values[ + ptr2i(array_get(&joinp->rem1, i))]; + + /* join the value */ + INDEX_ALL(match_tuples, t2_index, tuple2) + tblopr_combine_tuple(res, tuple1->count, + joinp, r1_fnum, tuple2, tuple_values); + INDEX_END + SET_END + } + + free(tuple_values); + return res; /* isEmpty? */ +} + +log_table_t* +log_tblopr_join(log_table_t* t1, log_table_t* t2, log_join_param_t* joinp) +{ + if (joinp->index2 != NULL && + !map_has(&t2->index_def, joinp->index2)) + log_table_add_index(t2, joinp->index2); + + if (joinp->full_join) return tblopr_full_join(t1, t2, joinp); + else return tblopr_cond_join(t1, t2, joinp); +} + +/* ========================================================================== + * LOG ENGINE + * ========================================================================== + */ + +log_join_param_t* +log_join_param_init(log_join_param_t* jp, log_int_tuple_t* i2, set_t* gv) +{ + coll_alloc(&jp, sizeof(log_join_param_t), ENT_JOIN_PARAM, gv); + + jp->full_join = false; + jp->index2 = i2; + log_array_init(&jp->select1, KEY_ID(ENT_VALUE), 0, gv); + log_array_init(&jp->select1i, KEY_ID(ENT_INT32), 0, gv); + log_array_init(&jp->rem1, KEY_ID(ENT_INT32), 0, gv); + log_array_init(&jp->rem2, KEY_ID(ENT_INT32), 0, gv); + log_array_init(&jp->out_param, KEY_ID(ENT_INT32), 0, gv); + return jp; +} + +void +log_join_param_free(log_join_param_t* jp) +{ + log_array_free(&jp->select1); + log_array_free(&jp->select1i); + log_array_free(&jp->rem1); + log_array_free(&jp->rem2); + log_array_free(&jp->out_param); + if (jp->index2 != NULL) log_int_tuple_free(jp->index2); + coll_free_ptr(jp); +} + +static array_t* +eng_get_cond_param(array_t* param) +{ + /* input and output are array of integer */ + + int i; + array_t* res = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, param->m.glb_values); + for (i = 0;i < array_size(param);i++) { + int32_t p = array_get_int(param, i); + if (p >= 0) array_add(res, i2ptr(p)); + } + return res; +} + +static bool +eng_check_will_use(int32_t p, TYPE(array_t*, int32_t) not_used, + TYPE(array_t*, int32_t) reorder_list, log_rule_t* rule) +{ + int i; + for (i = 0;i < array_size(not_used);i++) { + if (ptr2i(array_get(not_used, i)) < 0) continue; + + array_t* a = (array_t*)array_get(&rule->param, + array_get_int(reorder_list, i)); + if (log_array_look_for(a, i2ptr(p)) >= 0) return true; + } + return false; +} + +static bitset_t* +eng_gen_bitset(TYPE(array_t*, int32_t) inp) +{ + int i; + bitset_t* b = log_bitset_init(NULL); + + for (i = 0;i < array_size(inp);i++) { + int32_t idx = array_get_int(inp, i); + if (idx >= 0) bitset_set(b, idx); + } + return b; +} + +static int32_t +eng_get_joinable(log_rule_t* rule, TYPE(array_t*, int32_t) cur_param, + TYPE(array_t*, int32_t) table_sz, + TYPE(array_t*, int32_t) reorder) +{ + /* returns the seq id of the table to be joined. */ + + int32_t i, tb_index; + bitset_t* bitset1 = eng_gen_bitset(cur_param); + + for (i = 1;i < array_size(table_sz);i++) { + + if (ptr2i(array_get(table_sz, i)) < 0) continue; + /* had joined */ + + tb_index = ptr2i(array_get(reorder, i)); + + bitset_t* bitset2 = eng_gen_bitset(array_get(&rule->param, tb_index)); + bitset_and(bitset2, bitset1); + + bool empty = bitset_empty(bitset2); + log_bitset_free(bitset2); + + if (!empty) { + log_bitset_free(bitset1); + return i; /* cond join */ + } + } + + log_bitset_free(bitset1); + for (i = 1;i < array_size(table_sz);i++) + if (ptr2i(array_get(table_sz, i)) >= 0) return i; + /* full join */ + return -1; +} + +static log_join_param_t* +eng_gen_join_param(TYPE(array_t*, int32_t) param1, + TYPE(array_t*, int32_t) param2, + TYPE(array_t*, int32_t) not_used, + TYPE(array_t*, int32_t) reorder, + log_rule_t* rule) +{ + /* + * full join is false: + * p1(7, 3, 2) p2(2, 3, -1, -3, 6) => outParam(7, 2, 6) + * for join: select1(2, 1, v[-3]), index2(0, 1, 3) + * for keep: rem1(0, 2), rem2(4) + * + * full join is true: + * p1(7, 2, 6) p2(-4, 9, -1) => outParam(7, 2, 6, 9) + * for join: select1(v[-4]), index2(0) + * for keep: rem1(0, 1), rem2(1) + */ + + log_join_param_t* joinp = log_join_param_init( + NULL, /*idx2*/NULL, rule->m.glb_values); + + TYPE(array_t*, int32_t) index2 = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, param1->m.glb_values); + + bitset_t* join_set = eng_gen_bitset(param1); + bitset_t* param2_set = eng_gen_bitset(param2); + + int32_t i; + bitset_and(join_set, param2_set); + joinp->full_join = true; + + /* set select1 and index2 */ + for (i = 0;i < array_size(param2);i++) { + int32_t p2 = array_get_int(param2, i); + + if (p2 < -1) { + array_add(index2, i2ptr(i)); + log_value_t* v = (log_value_t*) + map_get(&rule->const_param, i2ptr(p2)); + array_add(&joinp->select1, v); + array_add(&joinp->select1i, 0); + } + else if (p2 >= 0 && bitset_get(join_set, p2)) { + int32_t pos1 = log_array_look_for(param1, i2ptr(p2)); + array_add(index2, i2ptr(i)); + + array_add(&joinp->select1, NULL); + array_add(&joinp->select1i, i2ptr(pos1)); + joinp->full_join = false; + } + } + + /* set rem1 */ + for (i = 0;i < array_size(param1);i++) { + // if in left or right excluding processed tables, keep. + int32_t p1 = array_get_int(param1, i); + if (!eng_check_will_use(p1, not_used, reorder, rule)) continue; + array_add(&joinp->rem1, i2ptr(i)); + array_add(&joinp->out_param, i2ptr(p1)); + } + + /* set rem2 */ + for (i = 0;i < array_size(param2);i++) { + /* + * excluding all join set (because it is included in param1) + * if in left or right excluding processed tables, keep. + */ + int32_t p2 = array_get_int(param2, i); + if (p2 <= -1) continue; + if (bitset_get(join_set, p2)) continue; + if (!eng_check_will_use(p2, not_used, reorder, rule)) continue; + + array_add(&joinp->rem2, i2ptr(i)); + array_add(&joinp->out_param, i2ptr(p2)); + } + + /* void* ->int array not correct */ + int32_t sz2 = array_size(index2); + joinp->index2 = sz2 > 0 ? log_int_tuple_init(index2) : NULL; + + log_array_free(index2); + log_bitset_free(join_set); + log_bitset_free(param2_set); + return joinp; +} + +log_engine_t* +log_eng_parse(const char* rules, set_t* gv) +{ + log_engine_t* eng = log_engine_init(NULL, gv); + TYPE(map_t, array_t*) sem; + + log_sync_init(rules, gv); + log_sync_parse(&sem); + log_sem_process(&eng->rule_set, &sem); + map_free(&sem); + + /* create tables */ + MAP_ALL(&eng->rule_set.param_size, rule) + int32_t tsize = ptr2i(rule->value); + if (map_has(&eng->rule_set.input_tables, rule->key) || + !map_has(&eng->rule_set.output_tables, rule->key)) + + map_add(&eng->tables, rule->key, + log_table_init(NULL, ptr2i(rule->key), tsize, 0, gv)); + MAP_END + return eng; +} + +static void +eng_check_tuples(log_engine_t* eng, log_table_t* table) +{ + /* check if tuple already present for add; or does not exist for remove. + * the check is not good if there are multiple same items in the input. + */ + + bool is_remove = table->is_remove; + log_table_t* org_table = map_get(&eng->tables, i2ptr(table->table_index)); + TYPE(array_t*, log_tuple_t*) to_be_removed = /* no ID to prevent free */ + log_array_init(NULL, 0, 0, table->m.glb_values); + + SET_ALL(&table->tuples, t, log_tuple_t*) + log_tuple_t* ot = set_get(&org_table->tuples, t); + if ((is_remove && ot == NULL) || (!is_remove && ot != NULL)) + array_add(to_be_removed, t); + /*array_add(to_be_removed, ot); */ + SET_END + + ARRAY_ALL(to_be_removed, t, log_tuple_t*) + /* log_table_remove(org_table, t); ?? */ + log_table_remove(table, t); + ARRAY_END + log_array_free(to_be_removed); +} + +static void +eng_align_tables(log_engine_t* eng, + TYPE(array_t*, log_table_t*) inp_remove, + TYPE(array_t*, log_table_t*) inp_insert) +{ + TYPE(array_t*, int32_t) del_ids = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); + TYPE(array_t*, int32_t) add_ids = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); + + ARRAY_ALL(inp_remove, tbl, log_table_t*) + eng_check_tuples(eng, tbl); + array_add(del_ids, i2ptr(tbl->table_index)); + ARRAY_END + + ARRAY_ALL(inp_insert, tbl, log_table_t*) + eng_check_tuples(eng, tbl); + array_add(add_ids, i2ptr(tbl->table_index)); + ARRAY_END + + log_sort_array(0, del_ids, inp_remove, NULL); + log_sort_array(0, add_ids, inp_insert, NULL); + + int del_i, add_i; + for (del_i = 0, add_i = 0; + del_i < array_size(del_ids) || add_i < array_size(add_ids);) { + + bool align_add = false; + bool align_del = false; + + if (del_i < array_size(del_ids) && add_i < array_size(add_ids)) { + if (array_get(del_ids, del_i) == + array_get(add_ids, add_i)) { + add_i++; del_i++; continue; + } + else if (array_get(del_ids, del_i) < array_get(add_ids, add_i)) + align_add = true; + else align_del = true; + } + else if (del_i < array_size(del_ids)) align_add = true; + else align_del = true; + + if (align_add) { + log_table_t* org = array_get(inp_remove, del_i); + log_table_t* dummy = + log_table_init(NULL, org->table_index, org->num_fields, + 0, eng->m.glb_values); + + array_ins(inp_insert, del_i, dummy); + array_ins(add_ids, del_i, NULL); + add_i++; del_i++; + } + + if (align_del) { + log_table_t* org = array_get(inp_insert, add_i); + log_table_t* dummy = + log_table_init(NULL, org->table_index, org->num_fields, + 0, eng->m.glb_values); + dummy->is_remove = true; + + array_ins(inp_remove, add_i, dummy); + array_ins(del_ids, add_i, NULL); + add_i++; del_i++; + } + } + + log_array_free(del_ids); + log_array_free(add_ids); + + if (LOG_COMP) { + char buf[2048]; int pos; + pos = 0; pos = log_array_print(buf, pos, inp_remove, false); + buf[pos] = 0; printf("[LOG] align_tbl - %s\n", buf); + pos = 0; pos = log_array_print(buf, pos, inp_insert, false); + buf[pos] = 0; printf("[LOG] align_tbl + %s\n", buf); + } +} + +static bool +eng_invoke_external(log_engine_t* eng, log_table_t* input, + log_table_t* del_output, log_table_t* add_output) +{ + /* delOutput and addOutput have the same table index */ + if (eng->ext_func == NULL) return false; + return (*eng->ext_func)(eng, input, del_output, add_output); +} + +void +log_eng_set_ext_func(log_engine_t* eng, void* func) +{ + eng->ext_func = func; +} + +log_table_t* +log_get_table(log_engine_t* eng, const char* name) +{ + /* id is integer */ + void* id = map_get(&eng->rule_set.rule_index_map, (char*)name); + return map_get(&eng->tables, id); +} + +log_table_t* +log_get_org_table(log_engine_t* eng, log_table_t* t) +{ + return map_get(&eng->tables, i2ptr(t->table_index)); +} + +log_tuple_t* +log_query_on0(log_engine_t* eng, int32_t tid, log_value_t* value) +{ + /* USE INDEX_ALL to iterate on return value */ + log_tuple_t* qt = log_tuple_init(1); + qt->values[0] = value; + + TYPE(array_t*, int32_t) ints = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); + array_add(ints, 0); + log_int_tuple_t* key0 = log_int_tuple_init(ints); + log_array_free(ints); + + log_table_t* orgt = map_get(&eng->tables, i2ptr(tid)); + int32_t idx_i = log_table_add_index(orgt, key0); + log_tuple_t* list = log_index_get_index(orgt, qt, idx_i); + + log_tuple_free(qt, eng->m.glb_values, true); + log_int_tuple_free(key0); + return list; +} + +static TYPE(array_t*, log_table_t*) +eng_query_table(log_engine_t* eng, log_table_t* table, + TYPE2(map_t*, int32_t, log_table_t*) all) +{ + /* the original value is in form of =value */ + + set_t* gv = eng->m.glb_values; + TYPE(array_t*, log_table_t*) res = + log_array_init(NULL, 0 /* ENT_TABLE */, 0, gv); + /* no type so free in log_eng_query will leave tables */ + + int i, j; + log_value_t** val = calloc(table->num_fields, sizeof(void*)); + log_table_t* org = map_get(all, i2ptr(table->table_index)); + + SET_ALL(&table->tuples, tval, log_tuple_t*) + TYPE(array_t*, int32_t) param = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); + + for (i = j = 0;i < table->num_fields;i++) { + /* NULL value only for query */ + if (tval->values[i] == NULL) continue; + array_add(param, i2ptr(i)); + val[j++] = tval->values[i]; + } + + log_table_t* res_tbl = tblopr_query_table(org, param, val); + array_add(res, res_tbl); + log_array_free(param); + SET_END + + free(val); + return res; +} + +TYPE(array_t*, log_table_t*) +log_eng_query(log_engine_t* eng, TYPE(array_t*, log_table_t*) input) +{ + TYPE(array_t*, log_table_t*) res = + log_array_init(NULL, KEY_ID(ENT_TABLE), 0, eng->m.glb_values); + + TYPE(array_t*, log_table_t*) res1; + + ARRAY_ALL(input, table, log_table_t*) + res1 = eng_query_table(eng, table, &eng->tables); + ARRAY_ALL(res1, tbl, log_table_t*) + array_add(res, tbl); + ARRAY_END + log_array_free(res1); + ARRAY_END + return res; +} + +static int32_t +eng_get_table_index(log_rule_t* rule, int32_t param) +{ + return log_array_look_for(&rule->rule, i2ptr(param)); +} + +static int32_t +eng_search_tbl_in_rules(log_table_t* tbl, + TYPE(array_t*, log_table_t*) inp_tables) +{ + int idx = 0; + ARRAY_ALL(inp_tables, t, log_table_t*) + if (t->table_index == tbl->table_index) return idx; + idx++; + ARRAY_END + return -1; +} + +static log_table_t* +eng_reset_count(log_table_t* input) +{ + log_table_t* output = log_table_init(NULL, input->table_index, + input->num_fields, input->tuples.size, input->m.glb_values); + output->is_remove = input->is_remove; + + SET_ALL(&input->tuples, t, log_tuple_t*) + log_tuple_t* tup = log_tuple_clone(t); + tup->count = 1; + log_table_add0(output, tup); + SET_END + return output; +} + +static bool +eng_merge_output(log_engine_t* eng, TYPE(array_t*, int32_t) first_rule, + log_table_t* out_d_del, log_table_t* out_d_add, + TYPE(array_t*, log_table_t*) inp_del_tables, + TYPE(array_t*, log_table_t*) inp_add_tables) +{ + /* returns true if input table will be used later */ + + if (LOG_COMP) { + char buf[2048]; int pos; + pos = 0; pos = log_table_print(buf, pos, out_d_del, false); + buf[pos] = 0; printf("[LOG] merge_out - %s\n", buf); + pos = 0; pos = log_table_print(buf, pos, out_d_add, false); + buf[pos] = 0; printf("[LOG] merge_out + %s\n", buf); + } + + if (table_size(out_d_del) == 0 && table_size(out_d_add) == 0) + return false; + + int32_t ipos = eng_search_tbl_in_rules(out_d_del, inp_del_tables); + if (ipos >= 0) { + tblopr_merge_delta(out_d_del, + array_get(inp_del_tables, ipos), array_get(inp_add_tables, ipos)); + tblopr_merge_delta(out_d_add, + array_get(inp_add_tables, ipos), array_get(inp_del_tables, ipos)); + return false; + } + + int32_t new_inp_no = + map_has(&eng->rule_set.output_tables, i2ptr(out_d_del->table_index)) + ? 1000000 : /* just a big number */ + array_get_int(map_get(&eng->rule_set.table_rule_map, + i2ptr(out_d_del->table_index)), 0); + log_insert_item(new_inp_no, first_rule, + out_d_del, out_d_add, inp_del_tables, inp_add_tables); + return true; +} + +void +log_eng_do_union(log_engine_t* eng, log_table_t* input, log_table_t* output) +{ + /* example: (0, 1, 2) > (2, 1, -, 'const', 0), (0, -, 'aa', 2, 1) */ + + log_rule_t* rule = map_get(&eng->rule_set.rules, + i2ptr(output->table_index)); + + tblopr_match_reorder_and_merge(input, output, + array_get(&rule->param, eng_get_table_index(rule, input->table_index)), + array_get(&rule->param, eng_get_table_index(rule, output->table_index)), + &rule->const_param); +} + +void +log_eng_do_join(log_engine_t* eng, log_table_t* input, log_table_t* output) +{ + /* + * example: (7, 6, 2) : (7, 3, -1, -2, 2), (2, 3, -1, -3, 6) + * initial process: (7, 3, -1, -2, 2) => (7, 3, 2) + * drop 'ignored' param, match constants and merge identical; + * repeated joins - with condition having a higher priority than + * full join, possible with constants; the first join could be self join. + * final process: reorder based on left (7, 6, 2) + * intermediate tables are not hash table, just list? + * + * join in the order of table size, smallest first + * table size -1 indicates it had been joined before + */ + set_t* gv = eng->m.glb_values; + + log_rule_t* rule = map_get(&eng->rule_set.rules, + i2ptr(output->table_index)); + + TYPE(array_t*, int32_t) table_sz = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); + TYPE(array_t*, int32_t) rule_reorder = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); /* remove later */ + TYPE(array_t*, int32_t) natural_order = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); + + /* STEP 0: sort based on table size and get the first join pair */ + array_add(table_sz, 0); + array_add(natural_order, 0); + array_add(rule_reorder, array_get(&rule->rule, 0)); + + int i; + for (i = 1;i < array_size(&rule->rule);i++) { + int32_t p = array_get_int(&rule->rule, i); + array_add(table_sz, i2ptr( + table_size((log_table_t*)map_get(&eng->tables, + i2ptr(p))))); + array_add(natural_order, i2ptr(i)); + array_add(rule_reorder, i2ptr(p)); + } + + /* 1 for skip left rule */ + log_sort_array(1, table_sz, rule_reorder, natural_order); + + /* STEP 1: merge, reorder and merge for the input. */ + log_table_t* table1 = NULL; + TYPE(array_t*, int32_t) param1 = NULL; + int32_t join1_idx = log_array_look_for(rule_reorder, + i2ptr(input->table_index)); + + if (join1_idx < array_size(rule_reorder) - 1 && input->table_index == + array_get_int(rule_reorder, join1_idx + 1)) { + /* + * do self join. only one self join is supported as more will need + * full table join and less efficient - use intermediate table instead + * for this case. + * Example: X(b, a, c) : x(a, b, 'v', -, c) x(b, 'w', -, a, c) + */ + + array_set(table_sz, join1_idx, i2ptr(-1)); /* mark as used */ + array_set(table_sz, join1_idx + 1, i2ptr(-1)); /* mark as used */ + + /* pos based on rule sequence */ + int pos1 = log_array_look_for(&rule->rule, i2ptr(input->table_index)); + int pos2 = log_array_look_for(&rule->rule, + i2ptr(input->table_index)) + 1; + + TYPE(array_t*, int32_t) param1Org = array_get(&rule->param, pos1); + /* (a, b, 'v', -, c) */ + TYPE(array_t*, int32_t) param2Org = array_get(&rule->param, pos2); + /* (b, 'w', -, a, c) */ + param1 = eng_get_cond_param(param1Org); + /* (a, b, c) */ + TYPE(array_t*, int32_t) param2 = eng_get_cond_param(param2Org); + /* (b, a, c) */ + + log_table_t* table0 = map_get(&eng->tables, + i2ptr(input->table_index)); + + /* original table */ + table1 = log_table_init( + NULL, -1, array_size(param1), 0, gv); + log_table_t* table2 = log_table_init( + NULL, -1, array_size(param1), 0, gv); + + tblopr_match_reorder_and_merge( + input, table1, param1Org, param1, &rule->const_param); + tblopr_match_reorder_and_merge( + input, table2, param2Org, param2, &rule->const_param); + + log_join_param_t* joinp1 = eng_gen_join_param( + param1, param2Org, table_sz, natural_order, rule); + log_join_param_t* joinp2 = eng_gen_join_param( + param2, param1Org, table_sz, natural_order, rule); + + log_table_t* out_tb1 = log_tblopr_join(table1, table0, joinp1); + /* outp: (a, b, c) */ + log_table_t* out_tb2 = log_tblopr_join(table1, input, joinp1); + /* outp: (a, b, c) */ + log_table_t* out_tb3 = log_tblopr_join(table2, table0, joinp2); + /* outp: (b, a, c) */ + + /* reorder outTb3, it should only differ in order with outTb1, 2. */ + TYPE(array_t*, int32_t) reorder = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); + + ARRAY_ALL_INT(&joinp1->out_param, pos) + array_add(reorder, i2ptr( + log_array_look_for(&joinp2->out_param, i2ptr(pos)))); + ARRAY_END + + log_table_t* out_tb4 = + log_table_init(NULL, -1, out_tb3->num_fields, 0, gv); + tblopr_reorder_table(out_tb3, reorder, NULL, out_tb4); + + /* merge outTb1, 2, and 4 */ + log_table_free(table1); + table1 = log_table_init(NULL, -1, out_tb1->num_fields, 0, gv); + + tblopr_merge_table(out_tb1, table1, false); + tblopr_merge_table(out_tb4, table1, false); + tblopr_merge_table(out_tb2, table1, input->is_remove); + + log_array_free(param1); + param1 = log_array_clone(&joinp1->out_param); + + log_table_free(table2); + log_table_free(out_tb1); + log_table_free(out_tb2); + log_table_free(out_tb3); + log_table_free(out_tb4); + + log_array_free(reorder); + log_array_free(param2); + log_join_param_free(joinp1); + log_join_param_free(joinp2); + } + else { + int32_t pos1 = log_array_look_for(&rule->rule, + i2ptr(input->table_index)); + TYPE(array_t*, int32_t) param1_org = array_get(&rule->param, pos1); + param1 = eng_get_cond_param(param1_org); + + table1 = log_table_init(NULL, -1, array_size(param1), 0, gv); + tblopr_match_reorder_and_merge( + input, table1, param1_org, param1, &rule->const_param); + array_set(table_sz, join1_idx, i2ptr(-1)); /* mark as used */ + } + + /* STEP 2: repeated join, first condition join, then full join */ + for (;;) { /* join loop, the iteration param is param1 and table1 */ + + int32_t join2_idx = eng_get_joinable( + rule, param1, table_sz, natural_order); + + if (join2_idx < 0) break; /* nothing to join */ + array_set(table_sz, join2_idx, i2ptr(-1)); /* mark as used */ + + log_table_t* table2 = map_get( + &eng->tables, array_get(rule_reorder, join2_idx)); + TYPE(array_t*, int32_t) param2 = array_get(&rule->param, + array_get_int(natural_order, join2_idx)); + + log_join_param_t* joinp = + eng_gen_join_param(param1, param2, table_sz, natural_order, rule); + log_table_t* table1n = log_tblopr_join(table1, table2, joinp); + + log_table_free(table1); + log_array_free(param1); + + param1 = log_array_clone(&joinp->out_param); + log_join_param_free(joinp); + table1 = table1n; + } + + /* STEP 3: reorder the final table */ + TYPE(array_t*, int32_t) final_order = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); + ARRAY_ALL_INT((array_t*)array_get(&rule->param, 0), p) + if (p < -1) array_add(final_order, i2ptr(p)); + else array_add(final_order, + i2ptr(log_array_look_for(param1, i2ptr(p)))); + ARRAY_END + + tblopr_reorder_table(table1, final_order, &rule->const_param, output); + output->is_remove = input->is_remove; + + log_table_free(table1); + log_array_free(param1); + + log_array_free(table_sz); + log_array_free(rule_reorder); + log_array_free(natural_order); + log_array_free(final_order); +} + +/* + * the basic assumption about external function (and all) is that the result + * of left side does not depend on the order of application of delta change + * of the tables from right side. since the original input table could be + * obtained in the function, the computation could rely on delta(preferable + * for performance) or whole set, but C=A\B still could not be computed + * because order of delta application is relevant. + * + * the computation order is based on topology sort, and guarantees that each + * rule will only be applied once, e.g., C:A, B; D:C. Assume the input + * contains A and B, D will be computed only if both A and B have been + * applied on C. + * + * when input tables are checked (no adding to existing tuple, and no removing + * of none existing tuple, and adding does not overlap with removing), the + * order of adding and removing is not relevant. inpDelTables applied before + * inpAddTables is due to performance consideration. (not validated yet) + * + * tuples in right always regarded as having count 1 + * tuples in left has actual count. + * + * inpDelTables and inpAddTables must be paired, i.e., + * delTables[i].id == addTables[i].id holds for all i. + */ + +static void +eng_delta0(log_engine_t* eng, TYPE(array_t*, log_table_t*) inp_del_tables, + TYPE(array_t*, log_table_t*) inp_add_tables, + TYPE2(map_t*, int32_t, log_table_t*) del_out_tables, + TYPE2(map_t*, int32_t, log_table_t*) add_out_tables) +{ + /* input tables will be destroyed, but still need to free it */ + set_t* gv = eng->m.glb_values; + + TYPE(array_t*, int32_t) first_rule = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, eng->m.glb_values); + + ARRAY_ALL(inp_del_tables, table, log_table_t*) + int32_t v = array_get_int( // item 0 is smallest in each set + map_get(&eng->rule_set.table_rule_map, + i2ptr(table->table_index)), 0); + + array_add(first_rule, i2ptr(v)); + ARRAY_END + + /* there might be tie in sort. just pick from the initial order */ + log_sort_array(0, first_rule, inp_del_tables, inp_add_tables); + + while (array_size(inp_del_tables) > 0) { + /* inpDTables always has 1 as count but outDTable will has actual + * count before turning into inpDTables. Final output tables will + * have actual count. + */ + + array_rmv(first_rule, 0); + /* D stands for delta */ + log_table_t* inp_d_del_table = array_rmv(inp_del_tables, 0); + log_table_t* inp_d_add_table = array_rmv(inp_add_tables, 0); + int32_t inp_table_id = inp_d_del_table->table_index; + + if (LOG_COMP) { + char buf[2048]; int pos; + pos = 0; pos = log_table_print(buf, pos, inp_d_del_table, false); + buf[pos] = 0; printf("[LOG] delta - %s\n", buf); + pos = 0; pos = log_table_print(buf, pos, inp_d_add_table, false); + buf[pos] = 0; printf("[LOG] delta + %s\n", buf); + } + + if (map_has(&eng->rule_set.output_tables, i2ptr(inp_table_id))) { + if (table_size(inp_d_del_table) > 0) + map_add(del_out_tables, i2ptr(inp_table_id), inp_d_del_table); + else log_table_free(inp_d_del_table); + + if (table_size(inp_d_add_table) > 0) + map_add(add_out_tables, i2ptr(inp_table_id), inp_d_add_table); + else log_table_free(inp_d_add_table); + continue; + } + + if (!map_has(&eng->rule_set.input_tables, i2ptr(inp_table_id))) { + tblopr_gen_delta(inp_d_del_table, + map_get(&eng->tables, i2ptr(inp_table_id))); + tblopr_gen_delta(inp_d_add_table, + map_get(&eng->tables, i2ptr(inp_table_id))); + } + + if (table_size(inp_d_del_table) == 0 && + table_size(inp_d_add_table) == 0) { + log_table_free(inp_d_del_table); + log_table_free(inp_d_add_table); + continue; + } + + log_table_t* inp_d_del_table_c1 = eng_reset_count(inp_d_del_table); + log_table_t* inp_d_add_table_c1 = eng_reset_count(inp_d_add_table); + + TYPE(array_t*, int32_t) rule_order = + map_get(&eng->rule_set.table_rule_map, i2ptr(inp_table_id)); + + ARRAY_ALL_INT(rule_order, rule_no) + log_rule_t* rule = map_get(&eng->rule_set.rules, i2ptr(rule_no)); + + log_table_t* out_d_del_table = log_table_init(0, rule_no, + map_get_int(&eng->rule_set.param_size, + i2ptr(rule_no)), 0, gv); + + log_table_t* out_d_add_table = log_table_init(0, rule_no, + map_get_int(&eng->rule_set.param_size, + i2ptr(rule_no)), 0, gv); + + out_d_del_table->is_remove = true; + if (eng_invoke_external + (eng, inp_d_del_table_c1, out_d_del_table, out_d_add_table)) { + bool keep = eng_merge_output(eng, first_rule, out_d_del_table, + out_d_add_table, inp_del_tables, inp_add_tables); + + if (!keep) { + log_table_free(out_d_del_table); + log_table_free(out_d_add_table); + } + + out_d_del_table = log_table_init(0, rule_no, + map_get_int(&eng->rule_set.param_size, + i2ptr(rule_no)), 0, gv); + + out_d_add_table = log_table_init(0, rule_no, + map_get_int(&eng->rule_set.param_size, + i2ptr(rule_no)), 0, gv); + + out_d_del_table->is_remove = true; + eng_invoke_external + (eng, inp_d_add_table_c1, out_d_del_table, out_d_add_table); + /* will also merge below */ + } + else if (rule->is_union) { + log_eng_do_union(eng, inp_d_del_table_c1, out_d_del_table); + log_eng_do_union(eng, inp_d_add_table_c1, out_d_add_table); + } + else { + log_eng_do_join(eng, inp_d_del_table_c1, out_d_del_table); + log_eng_do_join(eng, inp_d_add_table_c1, out_d_add_table); + } + + bool keep = eng_merge_output(eng, first_rule, out_d_del_table, + out_d_add_table, inp_del_tables, inp_add_tables); + + if (!keep) { + log_table_free(out_d_del_table); + log_table_free(out_d_add_table); + } + ARRAY_END /* for all rules related to one table input */ + + /* merge input table */ + tblopr_final_delta(inp_d_del_table, + map_get(&eng->tables, i2ptr(inp_d_del_table->table_index))); + tblopr_final_delta(inp_d_add_table, + map_get(&eng->tables, i2ptr(inp_d_add_table->table_index))); + + log_table_free(inp_d_del_table_c1); + log_table_free(inp_d_add_table_c1); + log_table_free(inp_d_del_table); + log_table_free(inp_d_add_table); + } /* for all inputs */ + + log_array_free(first_rule); +} + +TYPE(array_t*, log_table_t*) +log_eng_delta(log_engine_t* eng, TYPE(array_t*, log_table_t*) inp_remove, + TYPE(array_t*, log_table_t*) inp_insert) +{ + + set_t* gv = eng->m.glb_values; + /* no VALUE_ID so that tables will not be freed */ + TYPE2(map_t*, int32_t, log_table_t*) del_output = + map_init(NULL, KEY_ID(ENT_INT32), 0, gv); + TYPE2(map_t*, int32_t, log_table_t*) add_output = + map_init(NULL, KEY_ID(ENT_INT32), 0, gv); + + eng_align_tables(eng, inp_remove, inp_insert); + eng_delta0(eng, inp_remove, inp_insert, del_output, add_output); + + TYPE(array_t*, log_table_t*) all = + log_array_init(NULL, KEY_ID(ENT_TABLE), 0, gv); + MAP_ALL(del_output, tn) array_add(all, tn->value); MAP_END + MAP_ALL(add_output, tn) array_add(all, tn->value); MAP_END + + /* reset state */ + eng_invoke_external(eng, NULL, NULL, NULL); + map_free(del_output); + map_free(add_output); + return all; +} + +/* ========================================================================== + * SERIALIZATION + * ========================================================================== + */ + +log_table_t* +log_io_table_name_reset(log_engine_t* eng, const char* name) +{ + set_t* gv = eng->m.glb_values; + log_value_t* table_name = log_value_init(name, 0, gv); + map_node_t* node = log_hash_get( + &eng->rule_set.rule_index_map, table_name); + + log_value_free(table_name, eng->m.glb_values); + if (node == NULL) return NULL; + + void* tbl_idx = node->value; /* int32_t */ + int32_t tbl_fd = map_get_int(&eng->rule_set.param_size, tbl_idx); + log_table_t* tbl = log_table_init(NULL, ptr2i(tbl_idx), tbl_fd, 0, gv); + return tbl; +} + +bool +log_io_parse_line(const char* line, log_engine_t* eng, bool use_null, + TYPE(array_t*, log_table_t*) inp_remove, + TYPE(array_t*, log_table_t*) inp_insert) +{ + static int32_t line_type = 0; /* 1 for add; 2 for delete */ + static int32_t line_rm = 0; + static log_table_t* cur_tbl; + + if (line == NULL) { + line_type = 0; + return true; + } + + set_t* gv = eng->tables.m.glb_values; + if (line_type == 0) { + char type; + char buf[512]; + + int32_t f = sscanf(line, "%c:%d:%s", &type, &line_rm, buf); + if (f != 3) return false; + if (type == '+') line_type = 1; + else if (type == '-') line_type = 2; + else return false; + if (line_rm <= 0) return false; + + log_table_t* tbl = log_io_table_name_reset(eng, buf); + if (tbl == NULL) return false; + + if (line_type == 1) array_add(inp_insert, tbl); + else array_add(inp_remove, tbl); + if (line_type == 2) tbl->is_remove = true; + cur_tbl = tbl; + return true; + } + + if (line_rm-- == 0) return false; + if (line_rm == 0) line_type = 0; + + log_tuple_t* tpl = use_null ? + log_tuple_init_str_null(line, ':', gv) : + log_tuple_init_str(line, ':', gv); + + log_table_add(cur_tbl, tpl); + return true; +} + +int32_t +log_io_gen_line(char* text, int pos, log_engine_t* eng, + TYPE(array_t*, log_table_t*) all) +{ + ARRAY_ALL(all, tbl, log_table_t*) + int32_t tbl_idx = tbl->table_index; + log_value_t* tbl_name = map_get( + &eng->rule_set.rule_name_map, i2ptr(tbl_idx)); + pos += sprintf(text + pos, "%c:%d:%s\n", tbl->is_remove ? '-' : '+', + table_size(tbl), tbl_name->value.a); + + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) + pos = log_tuple_print(text, pos, tuple); + text[pos++] = '\n'; + SET_END + ARRAY_END + return pos; +} + +/* ========================================================================== + * PUBLIC API + * ========================================================================== + */ + +void* +datalog_init(const char* rules, void* func) +{ + /* will call exit in case rules are incorrect */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = log_eng_parse(rules, gv); + log_eng_set_ext_func(eng, func); + + eng->io.inp_insert = NULL; + eng->io.inp_remove = NULL; + eng->io.res = NULL; + return eng; +} + +void +datalog_free(void* e) +{ + log_engine_t* eng = e; + set_t* gv = eng->tables.m.glb_values; + log_engine_free(eng); + set_free(gv); +} + +bool +datalog_put_table(void* e, bool is_remove, const char* name) +{ + /* each table must only appear at most twice - add / remove */ + + log_engine_t* eng = e; + set_t* gv = eng->tables.m.glb_values; + + log_table_t* tbl = log_io_table_name_reset(eng, name); + if (tbl == NULL) return false; + + if (eng->io.inp_insert == NULL || eng->io.inp_remove == NULL) { + eng->io.inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); + eng->io.inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); + } + + tbl->is_remove = is_remove; + eng->io.cur_tbl = tbl; + eng->io.cur_tuple = NULL; + if (is_remove) array_add(eng->io.inp_remove, tbl); + else array_add(eng->io.inp_insert, tbl); + return true; +} + +void +datalog_put_field(void* e, void* value, int32_t len) +{ + /* value is treated as c-str if len is zero */ + + log_engine_t* eng = e; + set_t* gv = eng->tables.m.glb_values; + int n_fd = eng->io.cur_tbl->num_fields; + + if (eng->io.cur_tuple == NULL) { + eng->io.cur_tuple = log_tuple_init(n_fd); + eng->io.cur_tuple->count = 1; + eng->io.t_idx = 0; + } + + eng->io.cur_tuple->values[eng->io.t_idx++] = + value == NULL ? NULL : log_value_init(value, len, gv); + + if (eng->io.t_idx >= n_fd) { + log_tuple_set_hash_code(eng->io.cur_tuple, n_fd); + log_table_add(eng->io.cur_tbl, eng->io.cur_tuple); + eng->io.cur_tuple = NULL; + } +} + +void +datalog_opr(void* e, bool query) +{ + log_engine_t* eng = e; + if (query) { + /* must provide one tuple in inp_insert table */ + ovs_assert(array_size(eng->io.inp_insert) == 1 && + array_size(eng->io.inp_remove) == 0); + + eng->io.res = log_eng_query(eng, eng->io.inp_insert); + } + else { + ovs_assert(eng->io.res == NULL && + eng->io.inp_insert != NULL && eng->io.inp_remove != NULL); + + eng->io.res = log_eng_delta(eng, + eng->io.inp_remove, eng->io.inp_insert); + } + + log_array_free(eng->io.inp_remove); + log_array_free(eng->io.inp_insert); + eng->io.inp_remove = NULL; + eng->io.inp_insert = NULL; + eng->io.t_idx = 0; +} + +bool +datalog_get_table(void* e, bool* is_remove, const char** name, + int32_t* n_tuples, int32_t* n_fields) +{ + /* return false if there is no more table */ + log_engine_t* eng = e; + + if (eng->io.t_idx >= array_size(eng->io.res)) { + log_array_free(eng->io.res); + eng->io.res = NULL; + return false; + } + + eng->io.cur_tbl = array_get(eng->io.res, eng->io.t_idx++); + *is_remove = eng->io.cur_tbl->is_remove; + *n_tuples = table_size(eng->io.cur_tbl); + *n_fields = eng->io.cur_tbl->num_fields; + + log_value_t* val = map_get( + &eng->rule_set.rule_name_map, i2ptr(eng->io.cur_tbl->table_index)); + *name = val->value.a; + + eng->io.hash_node = NULL; + eng->io.hash_b = 0; + eng->io.f_idx = eng->io.cur_tbl->num_fields; + return true; +} + +bool +datalog_get_field(void* e, void* res, int32_t* sz) +{ + /* return false if switches to another table */ + + void** v = (void**)res; + log_engine_t* eng = e; + int n_fd = eng->io.cur_tbl->num_fields; + + if (eng->io.f_idx < n_fd) { + log_value_t* val = eng->io.cur_tuple->values[eng->io.f_idx++]; + *v = val->value.a; + *sz = val->size; + return true; + } + + bool r = log_hash_next(&eng->io.cur_tbl->tuples, + &eng->io.hash_b, &eng->io.hash_node); + if (!r) return false; + + eng->io.f_idx = 0; + eng->io.cur_tuple = eng->io.hash_node->key; + log_value_t* val = eng->io.cur_tuple->values[eng->io.f_idx++]; + + *v = val->value.a; + *sz = val->size; + return true; +} + +/* + * TODO: add stats to value, hash, and table operation. + */ diff --git a/ovn/lib/datalog.h b/ovn/lib/datalog.h new file mode 100644 index 0000000..c3d1a80 --- /dev/null +++ b/ovn/lib/datalog.h @@ -0,0 +1,49 @@ +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef OVN_DATALOG_H +#define OVN_DATALOG_H 1 + +#include <stdbool.h> +#include <stdint.h> + +/* + * calling sequence: + * init (put_table (put_field)*)* opr (get_table (get_field)*)* free + * + * init returns handle of the engine and will be used in all other calls. + * put_table() returns false if table name is invalid. + * size could be zero for put_field() and null-terminated c-str is assumed. + * get_table() returns false if all tables have been retrieved. + * get_field() returns false if the last tuple of a table has been retrieved. + * + * all values are assumed to be read only for both input and output. + * see test_api for api examples. + */ + +void* datalog_init(const char* rules, void* ext_func); +void datalog_free(void* eng); + +/* true for query and false for delta change */ +void datalog_opr(void* eng, bool query); + +bool datalog_put_table(void* eng, bool is_remove, const char* name); +void datalog_put_field(void* eng, void* value, int32_t size); + +bool datalog_get_table(void* e, bool* is_remove, const char** name, + int32_t* n_tuples, int32_t* n_fields); +bool datalog_get_field(void* e, void* v, int32_t* size); + +#endif /* ovn_datalog.h */ diff --git a/tests/automake.mk b/tests/automake.mk index a5c6074..5f2ea58 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -90,7 +90,8 @@ TESTSUITE_AT = \ tests/ovn-nbctl.at \ tests/ovn-sbctl.at \ tests/ovn-controller.at \ - tests/ovn-controller-vtep.at + tests/ovn-controller-vtep.at \ + tests/datalog.at SYSTEM_KMOD_TESTSUITE_AT = \ tests/system-common-macros.at \ @@ -337,7 +338,8 @@ tests_ovstest_SOURCES = \ tests/test-uuid.c \ tests/test-bitmap.c \ tests/test-vconn.c \ - tests/test-aa.c + tests/test-aa.c \ + tests/test-datalog.c if !WIN32 tests_ovstest_SOURCES += \ diff --git a/tests/datalog.at b/tests/datalog.at new file mode 100644 index 0000000..c12f308 --- /dev/null +++ b/tests/datalog.at @@ -0,0 +1,11 @@ +AT_BANNER([datalog]) + +AT_SETUP([datalog engine]) +AT_KEYWORDS([ovn]) + +# test-datalog has a set of cases. run it alone to see its output. +AT_CHECK([ovstest test-datalog test], [0], [ignore], [PASS +]) + +AT_CLEANUP + diff --git a/tests/test-datalog.c b/tests/test-datalog.c new file mode 100644 index 0000000..f454a9b --- /dev/null +++ b/tests/test-datalog.c @@ -0,0 +1,1662 @@ +/* Copyright (c) 2016 VMware, Inc. All Rights Reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include <config.h> +#include "ovstest.h" +#include "ovn/lib/datalog.h" +#include "ovn/lib/datalog-private.h" + +static int32_t log_tst_no_cases_total = 0; +static int32_t log_tst_no_cases_failed = 0; + +static void +t_assert(bool r, const char* s) +{ + log_tst_no_cases_total++; + if (r) return; + log_tst_no_cases_failed++; + printf("FAILED: %s\n",s); +} + +static void +t_assert1(bool r, const char* s) +{ + if (r) return; + printf("FAILED: %s\n",s); +} + +static void +t_sum(void) +{ + printf(" test result %d/%d failed\n", + log_tst_no_cases_failed, log_tst_no_cases_total); +} + +/* ========================================================================== + * TEST CASES + * ========================================================================== + */ + +struct test_align_s { + int32_t a; + /*char b; */ + void* c[0]; +}; + +static void +gen_str(char* p, int32_t len) +{ + int32_t i; + for (i = 0;i < len;i++) *p++ = (random() % 52) + 65; + *p++ = 0; +} + +static void +test_collections(void) +{ + char buf[1024]; + { + /* null will be printed. */ + char* p = NULL; + printf("- hello log %s\n", p); + + struct test_align_s s; + printf( + "- size of types\n %p char(%lu) int(%lu) " + "long(%lu) llong(%lu) %d %d\n", + &s, sizeof(char), sizeof(int), sizeof(long), sizeof(long long), + 0 /* (int)((char*)(&s.b) - (char*)(&s.a)) */, + (int)((char*)s.c - (char*)(&s.a))); + + t_assert(sizeof(void*) >= sizeof(int32_t), "holding int in ptr"); + /* nested variable with same name */ + int i; for (i = 0;i < 5;i++) { + int i; for (i = 0;i < 5;i++) { + } + } + } + + { /* check integer could be stored as pointer */ + void* p1; + void* p2; + + int i; + char* c1 = (char*)&p1; + char* c2 = (char*)&p2; + for (i = 0;i < sizeof(void*);i++) { + c1[i] = 0x07; c2[i] = 0x70; + } + printf(" int as ptr %p %p\n", p1, p2); + + p1 = 0; p2 = 0; + printf(" int as ptr %p %p\n", p1, p2); + t_assert(p1 == p2, "int as ptr, compare 0 error"); + t_assert(ptr2i(p1) == 0, "int as ptr, int 0 error"); + + p1 = i2ptr(-1); p2 = i2ptr(-1); + printf(" int as ptr %p %p\n", p1, p2); + t_assert(p1 == p2, "int as ptr, compare -1 error"); + t_assert(ptr2i(p2) == -1, "int as ptr, int -1 error"); + printf("- int as ptr\n"); + } + + { /* check bitset */ + bitset_t* bs = log_bitset_init(NULL); + bitset_t* bs1 = log_bitset_init(NULL); + + t_assert(bs->size == 0, "bitmap, init size not 0"); + bitset_set(bs, 0); + bitset_set(bs, 31); + t_assert(bs->size == 1, "bitmap, size not 1 after set bit 31"); + bitset_set(bs, 32); + t_assert(bs->size == 2, "bitmap, size not 2 after set bit 32"); + + t_assert(bitset_get( + bs, 0) && bitset_get(bs, 31) && bitset_get(bs, 32), + "bitmap, not set on bit 0, 31, 32"); + + bitset_set(bs1, 64); + bitset_set(bs1, 31); + bitset_and(bs, bs1); + t_assert(!bitset_empty(bs) && !bitset_get(bs, 0) && + bitset_get(bs, 31) && !bitset_get(bs, 32) && bs->size == 2, + "bitmap, AND failed"); + + log_bitset_free(bs); + log_bitset_free(bs1); + printf("- bitmap\n"); + } + + { /* check array */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + int i; + array_t ary1; + array_t* ary2; + log_array_init(&ary1, 0, 0, gv); + ary2 = log_array_init(NULL, 0, 5, gv); + + for (i = 0;i < 6;i++) { + array_add(ary2, i2ptr(i)); + array_add(&ary1, i2ptr(i * 10)); + } + t_assert(ary2->len == 10, "array realloc failed"); + array_set(ary2, 2, i2ptr(20)); + t_assert(ptr2i(array_get(ary2, 1)) == 1 && + ptr2i(array_get(ary2, 5)) == 5 && + ptr2i(array_get(ary2, 2)) == 20, + "array get or set failed"); + + array_ins(&ary1, 0, i2ptr(1)); + array_ins(&ary1, 1, i2ptr(2)); + array_ins(&ary1, 8, i2ptr(100)); + int32_t v = ptr2i(array_rmv(&ary1, 3)); + t_assert(v == 10, "array insert and remove failed"); + t_assert(array_size(&ary1) == 8, "array size failed"); + t_assert(log_array_look_for(&ary1, i2ptr(100)) == 7, + "array look for failed"); + + ARRAY_ALL_INT(&ary1, i) + printf(" %d ", i); + if (i == 40) ARRAY_EXIT; + ARRAY_END + log_array_free(&ary1); + log_array_free(ary2); + set_free(gv); + printf("\n- array\n"); + } + + { /* check hash code for string, int. check map */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + int32_t l1 = 0, l2 = 0; + int32_t c1 = log_hash_code_byte("hello", &l1); + int32_t c2 = log_hash_code_byte("world ", &l2); + printf(" hash code len(%d) %x len(%d) %x\n", l1, c1, l2, c2); + t_assert(l1 == 5 && l2 == 6, "hash code len"); + + map_t* map1 = map_init( + NULL, KEY_ID(ENT_INT32) | VALUE_ID(ENT_INT32), 11, gv); + /* all on same slot */ + map_add(map1, i2ptr(5), i2ptr(6)); + map_add(map1, i2ptr(16), i2ptr(17)); + map_add(map1, i2ptr(27), i2ptr(28)); + + t_assert(ptr2i(map_get(map1, i2ptr(5))) == 6 && + ptr2i(map_get(map1, i2ptr(16))) == 17 && + ptr2i(map_get(map1, i2ptr(27))) == 28 && + map_size(map1) == 3, + "map size, get, or set"); + + t_assert(map1->len == 11, "map int 0"); + int sz = 0; + sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{27->28,16->17,5->6}") == 0, "map int 1"); + + /* check link operation */ + t_assert(ptr2i(map_del(map1, i2ptr(16))) == 17, "map int 2"); + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{27->28,5->6}") == 0, "map int 3"); + + t_assert(ptr2i(map_del(map1, i2ptr(5))) == 6, "map int 4"); + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{27->28}") == 0, "map int 5"); + + map_add(map1, i2ptr(5), i2ptr(6)); + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{5->6,27->28}") == 0, "map int 6"); + t_assert(ptr2i(map_del(map1, i2ptr(5))) == 6, "map int 7"); + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{27->28}") == 0, "map int 8"); + + t_assert(ptr2i(map_del(map1, i2ptr(27))) == 28, "map int 9"); + sz = 0; sz = log_hash_print(buf, sz, map1, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{}") == 0 && map_size(map1) == 0, "map int 10"); + + /* break in map */ + int32_t i = 0; + map_add(map1, i2ptr(77), i2ptr(66)); + map_add(map1, i2ptr(33), i2ptr(55)); + map_add(map1, i2ptr(77), i2ptr(88)); + t_assert(map_has(map1, i2ptr(33)), "map int 11a"); + + t_assert(ptr2i(map_get(map1, i2ptr(77))) == 88 && + map_size(map1) == 2, "map int 11"); + + MAP_ALL(map1, node) + node++; /* make no warning */ + i++; + MAP_EXIT; + MAP_END + t_assert(i == 1, "map int 12"); + + /* map rehash */ + for (i = 0;i < 8192;i++) + map_add(map1, i2ptr(i), i2ptr(i * 2)); + + printf(" map len = %d size = %d\n", map1->len, map1->size); + t_assert(map_size(map1) == 8192, "map int 13"); + for (i = 0;i < 8192;i++) + t_assert1(ptr2i(map_get(map1, i2ptr(i))) == i * 2, + "map set/get 14"); + + for (i = 0;i < 8192;i++) + t_assert1(ptr2i(map_del(map1, i2ptr(i))) == i * 2, + "map set/get 15"); + printf(" map len = %d size = %d\n", map1->len, map1->size); + t_assert(map_size(map1) == 0, "map int 16"); + + map_free(map1); + set_free(gv); + printf("- map, int and string\n"); + } + + { /* check set on string */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + set_t set1; + set_init(&set1, KEY_ID(ENT_STR), 0, gv); + char* app1 = "apple"; + char* app2 = malloc(strlen(app1) + 1); + strcpy(app2, app1); + t_assert(app1 != app2, "two apple string"); + + set_add(&set1, app1); + set_add(&set1, app2); + set_add(&set1, "cherry"); + set_add(&set1, "kiwi"); + // test_hash_dump(&set1); + + t_assert(set_has(&set1, app1) && set_has(&set1, app2) && + set_get(&set1, app2) == app1 && + set_size(&set1) == 3, "right apple"); + + int i = 0; + SET_ALL(&set1, node, const char*) + node++; /* make no warning */ + i++; + SET_EXIT; + SET_END + t_assert(i == 1, "set exit"); + + /* check set remove */ + set_t* set2 = set_init(NULL, KEY_ID(ENT_INT32), 11, gv); + t_assert(set2->len == 11, "set int 1"); + + set_add(set2, i2ptr(5)); + set_add(set2, i2ptr(16)); + set_add(set2, i2ptr(27)); + // test_hash_dump(set2); + int sz = 0; + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{27,16,5}") == 0, "set int 1"); + + SET_ALL_INT(set2, node) + if (node == 16) SET_REMOVE_ITEM; + SET_END + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{27,5}") == 0, "set int 3"); + + SET_ALL_INT(set2, node) + if (node == 5) SET_REMOVE_ITEM; + SET_END + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{27}") == 0, "set int 5"); + + set_add(set2, i2ptr(5)); + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{5,27}") == 0, "set int 6"); + + SET_ALL_INT(set2, node) + if (node == 5) { SET_REMOVE_ITEM; SET_EXIT; } + SET_END + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + sz = 0; t_assert(strcmp(buf, "{27}") == 0, "set int 8"); + + SET_ALL_INT(set2, node) + if (node == 27) { SET_REMOVE_ITEM; SET_EXIT; } + SET_END + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + t_assert(strcmp(buf, "{}") == 0 && set_size(set2) == 0, "set int 10"); + + set_add(set2, 0); + set_add(set2, i2ptr(5)); set_add(set2, i2ptr(16)); + set_add(set2, i2ptr(27)); + set_add(set2, i2ptr(6)); set_add(set2, i2ptr(17)); + + sz = 0; sz = log_hash_print(buf, sz, set2, true); buf[sz++] = 0; + printf(" set: \n%s", buf); + + i = 0; + SET_ALL_INT(set2, node) + node++; /* make no warning */ + if (i >= 2 && i <= 4) SET_REMOVE_ITEM; + i++; + SET_END + + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + t_assert( + strcmp(buf, "{0,27,6}") == 0 && set_size(set2) == 3, "set int 11"); + + SET_ALL_INT(set2, node) + node++; /* make no warning */ + SET_REMOVE_ITEM; + SET_END + sz = 0; sz = log_hash_print(buf, sz, set2, false); buf[sz++] = 0; + t_assert( + strcmp(buf, "{}") == 0 && set_size(set2) == 0, "set int 12"); + + free(app2); + set_free(&set1); + set_free(set2); + set_free(gv); + printf("- set, int and string\n"); + } + { /* check for hash code collision */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + map_t* map1 = map_init(NULL, KEY_ID(ENT_TST_INT32), 11, gv); + map_add(map1, i2ptr(205), i2ptr(205)); + map_add(map1, i2ptr(5), i2ptr(105)); + map_add(map1, i2ptr(16), i2ptr(116)); + map_add(map1, i2ptr(206), i2ptr(206)); + + int sz = 0; sz = log_hash_print(buf, sz, map1, true); buf[sz++] = 0; + printf("%s", buf); + + t_assert(ptr2i(map_get(map1, i2ptr(205))) == 205 && + ptr2i(map_get(map1, i2ptr(16))) == 116 && + ptr2i(map_get(map1, i2ptr(5))) == 105, "hash, tst int32 1"); + + int i; + for (i = 0;i < 1000;i++) + map_add(map1, i2ptr(i), i2ptr((1000 + i))); + + printf(" map len = %d size = %d\n", map1->len, map_size(map1)); + t_assert(map_size(map1) == 1000, "hash, tst int32 2"); + + for (i = 0;i < 1000;i++) + t_assert1(ptr2i(map_get(map1, i2ptr(i))) == 1000 + i, + "hash, tst int32 3"); + + for (i = 0;i < 1000;i++) + t_assert1(ptr2i(map_del(map1, i2ptr(i))) == 1000 + i, + "hash, tst int32 4"); + + printf(" map len = %d size = %d\n", map1->len, map_size(map1)); + t_assert(map_size(map1) == 0, "hash, tst int32 5"); + + set_free(map1); + set_free(gv); + printf("- hash, code collision\n"); + } + { /* check large set */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + char* buf = malloc(20 * 100000); + int i; + for (i = 0;i < 100000;i++) + gen_str(buf + i * 20, 19); + + set_t* set = set_init(NULL, KEY_ID(ENT_STR), 0, gv); + printf(" gen set of strings\n"); + for (i = 0;i < 100000;i++) set_add(set, buf + i * 20); + printf(" add set of strings done\n"); + + t_assert(set_size(set) == 100000, "large set size"); + printf(" set len = %d size = %d\n", set->len, set_size(set)); + + for (i = 0;i < 100000;i++) + t_assert1(set_get(set, buf + i * 20) == buf + i * 20, + "large set"); + + for (i = 0;i < 100000;i++) set_del(set, buf + i * 20); + t_assert(set_size(set) == 0, "large set size 1"); + + free(buf); + set_free(set); + set_free(gv); + printf("- set, str, large collection\n"); + } +} + +static void +test_tables(void) +{ + char buf[1024]; + { /* check value collection */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + log_value_t* nstr = log_value_init("", 0, gv); + log_value_t* nhello = log_value_init("hello", 0, gv); + char world[7] = "worlda"; + log_value_t* nworld = log_value_init(world, 5, gv); + + char world1[5] = { 'a', 'b', 0, 'c', 'd' }; + log_value_t* nabcd = log_value_init(world1, 5, gv); + + + char hello1[5] = { 'h', 'e', 'l', 'l', 'o' }; + log_value_t* nhello1 = log_value_init(hello1, 5, gv); + log_value_t* nab = log_value_init("ab", 0, gv); + + char world2[5] = { 'a', 'b', 0, 'c', 'd' }; + log_value_t* nabcd2 = log_value_init(world2, 5, gv); + + log_value_ref(nstr); + t_assert(set_size(gv) == 5, "value set 1"); + + t_assert(nhello == nhello1 && nabcd == nabcd2 && + nabcd != nab && strcmp(nworld->value.a, "world") == 0, + "value set 2"); + + t_assert(nhello->ref_no == 2 && nabcd->ref_no == 2 && + nab->ref_no == 1 && nworld->ref_no == 1 && nstr->ref_no == 2, + "value set 3"); + + log_value_free(nhello, gv); + log_value_free(nhello, gv); + log_value_free(nabcd, gv); + + int sz = 0; sz = log_hash_print(buf, sz, gv, true); buf[sz++] = 0; + printf("%s", buf); + + t_assert(set_size(gv) == 4 && + set_has(gv, nabcd), "value set 4"); + + set_free(gv); + printf("- value, ref_no\n"); + } + { /* check tuple */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + log_tuple_t* tuple1 = log_tuple_init_str("2:f0:f1:f2", ':', gv); + log_tuple_t* tuple2 = log_tuple_init_str("10:f2:f0:f3:f1", ':', gv); + + int32_t sz = 0; + sz = log_tuple_print(buf, sz, tuple1); + buf[sz++] = '|'; + sz = log_tuple_print(buf, sz, tuple2); + buf[sz++] = 0; + printf(" tuple %s %d\n", buf, sz); + t_assert(strcmp("2:f0:f1:f2|10:f2:f0:f3:f1", buf) == 0 && sz == 26, + "tuple str init"); + + sz = 0; sz = log_hash_print(buf, sz, gv, true); buf[sz++] = 0; + printf("%s", buf); + + log_value_t* v_f2 = log_value_init("f2", 0, gv); + t_assert(v_f2->ref_no == 3, "tuple ref"); + log_tuple_free(tuple1, gv, true); + log_tuple_free(tuple2, gv, true); + + sz = 0; sz = log_hash_print(buf, sz, gv, true); buf[sz++] = 0; + printf("%s", buf); + + t_assert(v_f2->ref_no == 1 && set_size(gv) == 1, + "tuple ref 1"); + + set_free(gv); + printf("- tuple\n"); + } + { /* check table operation */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + array_t* key0 = log_array_init(NULL, 0, 0, gv); + array_t* key01 = log_array_init(NULL, 0, 0, gv); + array_add(key0, 0); + array_add(key01, 0); array_add(key01, i2ptr(2)); + + log_int_tuple_t* ikey0 = log_int_tuple_init(key0); + log_int_tuple_t* ikey01 = log_int_tuple_init(key01); + log_array_free(key0); + log_array_free(key01); + + log_value_t* val_a = log_value_init("a", 0, gv); + log_table_t* tbl = log_table_init(NULL, 1, 3, 0, gv); + + log_tuple_t* tkab = log_tuple_init_str("0:a", ':', gv); + log_tuple_t* tkab2 = log_tuple_init_str("0:a:b", ':', gv); + + log_tuple_t* tuple1 = log_tuple_init_str("1:a:c:b", ':', gv); + log_tuple_t* tuple2 = log_tuple_init_str("2:a:e:b", ':', gv); + log_tuple_t* tuple21 = log_tuple_init_str("2:a:e:b", ':', gv); + log_tuple_t* tuple3 = log_tuple_init_str("3:a:f:d", ':', gv); + log_tuple_t* tuple4 = log_tuple_init_str("4:b:g:d", ':', gv); + + /* create index later */ + log_table_add(tbl, tuple1); log_table_add(tbl, tuple2); + log_table_add(tbl, tuple3); log_table_add(tbl, tuple4); + log_table_remove(tbl, tuple2); + + t_assert(val_a->ref_no == 6, "table index c0"); + + int sz = 0; + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) + sz = log_tuple_print(buf, sz, tuple); + buf[sz++] = '|'; + SET_END + buf[sz++] = 0; + printf(" table tuples: %s\n", buf); + t_assert(strcmp(buf, "4:b:g:d|1:a:c:b|3:a:f:d|") == 0, + "table index 0"); + + log_table_add(tbl, tuple21); + int32_t i0 = log_table_add_index(tbl, ikey0); + log_tuple_t* set = log_index_get_index(tbl, tkab, i0); + + sz = 0; + INDEX_ALL(set, i0, t) + sz = log_tuple_print(buf, sz, t); + buf[sz++] = '|'; + INDEX_END + buf[sz++] = 0; + + printf(" search table on index0 (a): %s\n", buf); + t_assert(strcmp(buf, "2:a:e:b|3:a:f:d|1:a:c:b|") == 0, + "table index 1"); + + int32_t i01 = log_table_add_index(tbl, ikey01); + log_int_tuple_free(ikey0); + log_int_tuple_free(ikey01); + + set = log_index_get_index(tbl, tkab2, i01); + sz = 0; sz = log_index_print(buf, sz, tbl); buf[sz++] = 0; + printf("%s", buf); + + sz = 0; + INDEX_ALL(set, i01, t) + sz = log_tuple_print(buf, sz, t); + buf[sz++] = '|'; + INDEX_END + buf[sz++] = 0; + printf(" search table on index02 (a, b): %s\n", buf); + t_assert(strcmp(buf, "2:a:e:b|1:a:c:b|") == 0, "table index 2"); + + printf(" add tuple when index is defined\n"); + log_tuple_t* tuple5 = log_tuple_init_str("5:c:f:d", ':', gv); + log_tuple_t* tuple6 = log_tuple_init_str("6:b:h:d", ':', gv); + log_tuple_t* tuple7 = log_tuple_init_str("7:a:s:b", ':', gv); + log_table_add(tbl, tuple5); log_table_add(tbl, tuple6); + log_table_add(tbl, tuple7); + + sz = 0; sz = log_index_print(buf, sz, tbl); buf[sz++] = 0; + printf("%s", buf); + + printf(" del tuple when index is defined\n"); + log_table_remove(tbl, tuple1); + log_table_remove(tbl, tuple21); + log_table_remove(tbl, tuple7); + t_assert(val_a->ref_no == 4, "table index c1"); + + sz = 0; sz = log_index_print(buf, sz, tbl); buf[sz++] = 0; + printf("%s", buf); + + sz = 0; + SET_ALL(&tbl->tuples, tuple, log_tuple_t*) + sz = log_tuple_print(buf, sz, tuple); + buf[sz++] = '|'; + SET_END + buf[sz++] = 0; + printf(" table tuples: %s\n", buf); + t_assert(strcmp(buf, "4:b:g:d|6:b:h:d|3:a:f:d|5:c:f:d|") == 0, + "table index 3"); + + log_table_free(tbl); + log_tuple_free(tkab, gv, true); + log_tuple_free(tkab2, gv, true); + log_value_free(val_a, gv); + + t_assert(set_size(gv) == 0, "table values released"); + set_free(gv); + printf("- tables and tuples\n"); + } +} + +static void +test_sort(void) +{ + char buf[1024]; + + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + map_t right1, right2, rules; + set_init(&right1, KEY_ID(ENT_VALUE), 0, gv); + set_init(&right2, KEY_ID(ENT_VALUE), 0, gv); + map_init(&rules, KEY_ID(ENT_VALUE) | VALUE_ID(ENT_SET), 0, gv); + + set_add(&right1, log_value_init("1", 0, gv)); + set_add(&right2, log_value_init("1", 0, gv)); + set_add(&right2, log_value_init("2", 0, gv)); + + map_add(&rules, log_value_init("2", 0, gv), &right1); + map_add(&rules, log_value_init("3", 0, gv), &right2); + + int sz = 0; sz = log_hash_print(buf, sz, &rules, false); buf[sz++] = 0; + printf(" G=%s\n", buf); + + array_t order; + map_t in_nodes, out_nodes; + log_topo_sort(&rules, &order, &in_nodes, &out_nodes); + + sz = 0; + sz = log_coll_print(buf, sz, &order, ENT_ARRAY, false); + sz = log_coll_print(buf, sz, &in_nodes, ENT_SET, false); + sz = log_coll_print(buf, sz, &out_nodes, ENT_SET, false); + buf[sz++] = 0; + + t_assert(strcmp(buf, "[1,2,3]{1}{3}") == 0, "topo sort 1"); + log_array_free(&order); + set_free(&in_nodes); + set_free(&out_nodes); + + array_t ary1, ary2, ary3; + log_array_init(&ary1, KEY_ID(ENT_INT32), 0, NULL); + log_array_init(&ary2, KEY_ID(ENT_INT32), 0, NULL); + log_array_init(&ary3, KEY_ID(ENT_INT32), 0, NULL); + + int v1[] = { 9, 7, 9, 7, 5 }; + int v2[] = { 1, 2, 3, 4, 5 }; + int v3[] = { 5, 4, 3, 2, 1 }; + + int i; + for (i = 0;i < 5;i++) { + array_add(&ary1, i2ptr(v1[i])); + array_add(&ary2, i2ptr(v2[i])); + array_add(&ary3, i2ptr(v3[i])); + } + log_sort_array(0, &ary1, &ary2, &ary3); + + sz = 0; + int iv1 = log_insert_item(8, &ary1, i2ptr(6), i2ptr(10), &ary2, &ary3); + sz = log_array_print(buf, sz, &ary1, false); + sz = log_array_print(buf, sz, &ary2, false); + sz = log_array_print(buf, sz, &ary3, false); + int iv2 = log_insert_item(8, &ary1, i2ptr(7), i2ptr(11), &ary2, &ary3); + sz = log_array_print(buf, sz, &ary1, false); + sz = log_array_print(buf, sz, &ary2, false); + sz = log_array_print(buf, sz, &ary3, false); + buf[sz++] = 0; + printf(" sort: %s\n", buf); + + t_assert(iv1 == 3 && iv2 == 4 && strcmp(buf, + "[5,7,7,8,9,9][5,2,4,6,1,3][1,4,2,10,5,3]" + "[5,7,7,8,8,9,9][5,2,4,6,7,1,3][1,4,2,10,11,5,3]") == 0, "sort insert"); + + log_array_free(&ary1); log_array_free(&ary2); log_array_free(&ary3); + set_free(gv); + printf("- sort and insert\n"); +} + +static void +test_sync(void) +{ + char buf[1024]; + { + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + log_sync_init( + "table(a,b) : x(a,'aa')\n" + "# comment \n" + "y(b,-, -) ; z(aparam) > y(x) \n" + ". # last line" + ,gv); + + TYPE2(map_t, log_value_t*, array_t*) map; + log_sync_parse(&map); + + int sz = 0; + sz = log_hash_print(buf, sz, &map, false); buf[sz++] = 0; + printf(" sync %s\n", buf); + + t_assert(0 == strcmp(buf, + "{z->[[u,z,t,aparam],[,y,t,x]]," + "table->[[j,table,t,a,t,b],[,x,t,a,s,aa],[,y,t,b,-,,-,]]}"), + "sync check 1"); + map_free(&map); + set_free(gv); + printf("- log syntax\n"); + } + { + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + log_sync_init( + /* + "Xx2(a) : x1(a,'aa'); X3(a) : x1(a, b) Xx2(b); _EXTERNAL(x1, func)." + // input, output cannot be external + "Xx2(a) : x1(a,'aa'); X3(a) : x1(a, b) Xx2(b); _EXTERNAL(X5, func)." + // invalid external + "X2(a) : X1(a,'aa'); X3(a) : X1(a, b, c) X2(b)." + // upper / lower case wrong + "X2(a, b) : x2(b) x1(a) x1(a) x1(a)." // join more than twice + "X2(a) > x1(a) x1(a) x1(a)." // union more than twice + "X2(a, b) > x1(a) x1(a, b)." // table size mismatch + "X2(a, b) : x1(a) x2(a, b); X3(v) : x2(z)." // table size mismatch + "X2(a, 'aa') > x1(a)." // constant not allowed in left + "X2(a, -) : x1(a)." // ignored not allowed in left + "X2(a, b) > x1(a, c)." // union param not found + "X2(a) : x1(a, c)." // param not used should be 'ignored' + "X2(a, z) : x1(a, -)." // param not defined + */ + "Xx2(a) : x1(a,'aa', -); X3(a) : Xx2(a) x1(a, -, b) Xx2(b)."// correct + /* + " Span(ls_id, host_id) : ls(ls_id, vif) vif_place(host_id, vif); \n" + " HOST_LS(host1, ls, host2) : Span(ls, host1) Span(ls, host2). " + */ + , gv); + + TYPE2(map_t, log_value_t*, array_t*) map; + log_sync_parse(&map); + + int sz; + sz = 0; sz = log_hash_print(buf, sz, &map, false); buf[sz++] = 0; + printf(" sync %s\n", buf); + + log_rule_set_t rs; + log_rule_set_init(&rs, gv); + log_sem_process(&rs, &map); + sz = 0; sz = log_rule_set_print(buf, sz, &rs); buf[sz++] = 0; + printf(" sem=\n%s\n", buf); + + log_rule_set_free(&rs); + map_free(&map); + set_free(gv); + printf("- log semantics\n"); + } +} + +static void +test_join(void) +{ + char buf[1024]; + { + /* do join table 2 */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + log_table_t* tbl1 = log_table_init(NULL, 0, 3, 0, gv); + log_table_t* tbl2 = log_table_init(NULL, 1, 4, 0, gv); + + log_table_add(tbl1, log_tuple_init_str("2 :1:2:7", ':', gv)); + log_table_add(tbl1, log_tuple_init_str("1 :1:3:8", ':', gv)); + log_table_add(tbl1, log_tuple_init_str("1 :1:5:9", ':', gv)); + + log_table_add(tbl2, log_tuple_init_str("1 :3:1:2:0", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1 :4:1:2:0", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("9 :4:1:5:0", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1 :4:1:5:1", ':', gv)); + + TYPE(array_t*, int32_t) ints = + log_array_init(NULL, KEY_ID(ENT_INT32), 0, gv); + array_add(ints, i2ptr(1)); + array_add(ints, i2ptr(2)); array_add(ints, i2ptr(3)); + log_int_tuple_t* key0 = log_int_tuple_init(ints); + log_array_free(ints); + + log_join_param_t jps; + log_join_param_t* jp = log_join_param_init(&jps, key0, gv); + + array_add(&jp->select1i, 0); + array_add(&jp->select1i, i2ptr(1)); + array_add(&jp->select1i, 0); + array_add(&jp->select1, NULL); + array_add(&jp->select1, NULL); array_add(&jp->select1, + log_value_init("0", 0, gv)); + + array_add(&jp->rem1, 0); + array_add(&jp->rem2, 0); + + log_table_t* tbl3 = log_tblopr_join(tbl1, tbl2, jp); + log_join_param_free(jp); + + int sz = 0; sz = log_table_print(buf, sz, tbl2, true); buf[sz++] = 0; + printf(" cond join tbl2 index=%s\n", buf); + sz = 0; sz = log_table_print(buf, sz, tbl3, false); buf[sz++] = 0; + printf(" cond join result=%s\n", buf); + t_assert(0 == strcmp(buf, "{2:1:3,3:1:4}"), "cond join 1"); + + log_table_free(tbl1); + log_table_free(tbl2); + log_table_free(tbl3); + set_free(gv); + } + { + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + + log_table_t* tbl1 = log_table_init(NULL, 0, 1, 0, gv); + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); + + log_table_add(tbl1, log_tuple_init_str("2 :1", ':', gv)); + log_table_add(tbl1, log_tuple_init_str("1 :2", ':', gv)); + + log_table_add(tbl2, log_tuple_init_str("8 :3:4:1", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1 :3:5:0", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1 :5:4:1", ':', gv)); + + TYPE(array_t*, int32_t) ints = log_array_init( + NULL, KEY_ID(ENT_INT32), 0, gv); + array_add(ints, i2ptr(2)); + log_int_tuple_t* key0 = log_int_tuple_init(ints); + log_array_free(ints); + + log_join_param_t jps; + log_join_param_t* jp = log_join_param_init(&jps, key0, gv); + + array_add(&jp->select1i, 0); + array_add(&jp->select1, log_value_init("1", 0, gv)); + + array_add(&jp->rem1, i2ptr(0)); + array_add(&jp->rem2, i2ptr(1)); + + log_table_t* tbl3 = log_tblopr_join(tbl1, tbl2, jp); + log_join_param_free(jp); + + int sz = 0; sz = log_table_print(buf, sz, tbl2, true); buf[sz++] = 0; + printf(" join tbl2 index=%s\n", buf); + sz = 0; sz = log_table_print(buf, sz, tbl3, false); buf[sz++] = 0; + printf(" full join result=%s\n", buf); + t_assert(0 == strcmp(buf, "{2:2:4,4:1:4}"), "full join 1"); + + log_table_free(tbl1); + log_table_free(tbl2); + log_table_free(tbl3); + set_free(gv); + } + { + /* do union */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value( gv); + + log_engine_t log; + log_engine_init(&log, gv); + + log_sync_init("X(a0, a2, a1) : x(a2, a1, -, 'cst', a0) .", gv); + TYPE2(map_t, log_value_t*, array_t*) map; + + log_sync_parse(&map); + log_sem_process(&log.rule_set, &map); + map_free(&map); + + log_table_t* tbl1 = log_table_init(NULL, 0, 5, 0, gv); + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); + + log_table_add(tbl1, log_tuple_init_str("1:a:b:c:cst:d", ':', gv)); + log_table_add(tbl1, log_tuple_init_str("1:a:b:c:dst:d", ':', gv)); + log_table_add(tbl1, log_tuple_init_str("1:a:b:d:cst:d", ':', gv)); + log_table_add(tbl1, log_tuple_init_str("1:a:b:d:cst:e", ':', gv)); + + log_eng_do_union(&log, tbl1, tbl2); + + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); buf[sz++] = 0; + printf(" union = %s\n", buf); + t_assert(0 == strcmp(buf, "{1:e:a:b,2:d:a:b}"), "union 1"); + + log_table_free(tbl1); log_table_free(tbl2); + log_engine_free(&log); + set_free(gv); + } + { /* do self join 1 */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = + log_eng_parse( + "X(b, a, c) : x(a, b, 'v', -, c) x(b, 'w', -, a, c) .", gv); + + log_table_t* tbl0_org = map_get(&eng->tables, 0); + log_table_add(tbl0_org, log_tuple_init_str("1:a:b:c:d:e", ':', gv)); + log_table_add(tbl0_org, log_tuple_init_str("1:a:y:v:a:e", ':', gv)); + + log_table_t* tbl1 = log_table_init(NULL, 0, 5, 0, gv); + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); + log_table_add(tbl1, log_tuple_init_str("1:w:w:v:w:e", ':', gv)); + + log_eng_do_join(eng, tbl1, tbl2); + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); buf[sz++] = 0; + printf(" self join simple match = %s\n", buf); + t_assert(0 == strcmp(buf, "{1:w:w:e}"), "self join 1"); + + log_table_free(tbl1); log_table_free(tbl2); + log_engine_free(eng); + set_free(gv); + } + { + // do self join 2 + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = + log_eng_parse("X(a, b, c) : x(a, b) x(a, c) .", gv); + + log_table_t* tbl0_o = map_get(&eng->tables, 0); + log_table_t* tbl0 = log_table_init(NULL, 0, 2, 0, gv); + log_table_t* tbl2 = log_table_init(NULL, 1, 3, 0, gv); + log_table_add(tbl0, log_tuple_init_str("1:1:1", ':', gv)); + log_table_add(tbl0, log_tuple_init_str("1:1:2", ':', gv)); + log_table_add(tbl0, log_tuple_init_str("1:2:1", ':', gv)); + + log_eng_do_join(eng, tbl0, tbl2); + + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); buf[sz++] = 0; + printf(" self join simple initial = %s\n", buf); + t_assert(0 == strcmp(buf, + "{1:2:1:1,1:1:1:2,1:1:1:1,1:1:2:2,1:1:2:1}"), + "self join simple init 1"); + + log_table_add(tbl0_o, log_tuple_init_str("1:1:1", ':', gv)); + log_table_add(tbl0_o, log_tuple_init_str("1:1:2", ':', gv)); + log_table_add(tbl0_o, log_tuple_init_str("1:2:1", ':', gv)); + + log_table_t* tbl_0 = log_table_init(NULL, 0, 2, 0, gv); + log_table_t* tbl_2 = log_table_init(NULL, 1, 3, 0, gv); + log_table_add(tbl_0, log_tuple_init_str("1:1:3", ':', gv)); + + log_eng_do_join(eng, tbl_0, tbl_2); + sz = 0; sz = log_table_print(buf, sz, tbl_2, false); buf[sz++] = 0; + printf(" self join simple delta = %s\n", buf); + t_assert(0 == strcmp(buf, + "{1:1:1:3,1:1:3:2,1:1:3:3,1:1:2:3,1:1:3:1}"), + "self join simple delta 1"); + + log_table_free(tbl_0); log_table_free(tbl_2); + log_table_free(tbl0); log_table_free(tbl2); + log_engine_free(eng); + set_free(gv); + } + { + // do join 3 + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = + log_eng_parse( + "R(h, b, c, d) : h(h, p1, p2) p1(p1, -, b) p2(p2, d, c) .",gv); + + log_table_t* tbl0_h = map_get(&eng->tables, i2ptr(0)); + log_table_t* tbl0_p1 = map_get(&eng->tables, i2ptr(1)); + log_table_t* tbl0_p2 = map_get(&eng->tables, i2ptr(2)); + + log_table_add(tbl0_h, log_tuple_init_str("1:h1:px:py", ':', gv)); + log_table_add(tbl0_p1, log_tuple_init_str("1:p1:a1:b", ':', gv)); + log_table_add(tbl0_p1, log_tuple_init_str("1:p1:a2:b", ':', gv)); + log_table_add(tbl0_p2, log_tuple_init_str("1:p2:c:d", ':', gv)); + + log_table_t* tbl1 = log_table_init(NULL, 0, 3, 0, gv); + log_table_t* tbl2 = log_table_init(NULL, 3, 4, 0, gv); + log_table_add(tbl1, log_tuple_init_str("1:h1:p1:p2", ':', gv)); + + log_eng_do_join(eng, tbl1, tbl2); + int sz = 0; sz = log_table_print(buf, sz, tbl2, false); buf[sz++] = 0; + printf(" join 3 simple = %s\n", buf); + t_assert(0 == strcmp(buf, "{2:h1:b:d:c}"), + "join 3 simple"); + + log_table_free(tbl1); log_table_free(tbl2); + log_engine_free(eng); + set_free(gv); + } + printf("- join and union\n"); +} + +static bool +test_ext_func(log_engine_t* eng, log_table_t* inp, log_table_t* del_out, + log_table_t* add_out) +{ + char buf[256]; + + if (inp == NULL) return true; /* reset state */ + log_value_t* tn = map_get( + &eng->rule_set.rule_name_map, i2ptr(del_out->table_index)); + if (strcmp(tn->value.a, "Mm2") != 0) return false; + + int sz = 0; sz = log_table_print(buf, sz, inp, false); buf[sz] = 0; + printf(" run ext inp = %s\n", buf); + + log_table_t* output = inp->is_remove ? del_out : add_out; + log_value_t* tv[1]; + + SET_ALL(&inp->tuples, tuple, log_tuple_t*) + int sz = sprintf(buf, "aa%scc", tuple->values[0]->value.a); + log_value_t* nv = log_value_init(buf, sz, inp->m.glb_values); + tv[0] = nv; + + log_tuple_t* nt = log_tuple_init_val(tv, 1); + log_table_add(output, nt); + SET_END + + sz = 0; sz = log_table_print(buf, sz, output, false); buf[sz] = 0; + printf(" run ext inp remove = %s\n", inp->is_remove ? "true" : "false"); + printf(" run ext out = %s\n", buf); + return true; +} + +static void +test_delta(void) +{ + char buf[1024]; + { + /* delta with external function */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = + log_eng_parse("M(a) : inp(a) Mm2(a); Mm2(a) : i(a) .", gv); + /* {0=inp, 1=i, 2=Mm2, 3=M} */ + + log_eng_set_ext_func(eng, test_ext_func); + log_table_t* tbl1 = map_get(&eng->tables, 0); + log_table_t* tbl2 = log_table_init(NULL, 1, 1, 0, gv); + + log_table_add(tbl1, log_tuple_init_str("1:aaVbbcc", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1:bb", ':', gv)); + + TYPE(array_t*, log_table_t*) inp_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + array_add(inp_insert, tbl2); + + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); + int sz = 0; sz = log_array_print(buf, sz, res, false); buf[sz] = 0; + printf(" delta ext empty=%s\n", buf); + t_assert(array_size(res) == 0, "delta ext func 0"); + + log_array_free(res); + log_array_free(inp_remove); log_array_free(inp_insert); + log_engine_free(eng); set_free(gv); + } + { + /* delta with external function */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = + log_eng_parse("M(a) : inp(a) Mm2(a); Mm2(a) : i(a) .", gv); + /* {0=inp, 1=i, 2=Mm2, 3=M} */ + + log_eng_set_ext_func(eng, test_ext_func); + log_table_t* tbl1 = map_get(&eng->tables, 0); + log_table_t* tbl2 = log_table_init(NULL, 1, 1, 0, gv); + + log_table_add(tbl1, log_tuple_init_str("1:aabbcc", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1:bb", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1:ee", ':', gv)); + + TYPE(array_t*, log_table_t*) inp_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + array_add(inp_insert, tbl2); + + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); + int sz = 0; sz = log_array_print(buf, sz, res, false); buf[sz] = 0; + printf(" delta ext=%s\n", buf); + t_assert(0 == strcmp(buf, "[{1:aabbcc}]"), "delta ext 1"); + + sz = 0; + sz = log_hash_print(buf, sz, &eng->tables, false); buf[sz] = 0; + printf(" all tables=%s\n", buf); + + log_table_t* tbl2d = log_table_init(NULL, 1, 1, 0, gv); + tbl2d->is_remove = true; + log_table_add(tbl2d, log_tuple_init_str("1:bb", ':', gv)); + + TYPE(array_t*, log_table_t*) inp1_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp1_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + array_add(inp1_remove, tbl2d); + + array_t* res1 = log_eng_delta(eng, inp1_remove, inp1_insert); + sz = 0; sz = log_array_print(buf, sz, res1, false); buf[sz] = 0; + printf(" delta ext rmv=%s\n", buf); + t_assert(0 == strcmp(buf, "[{1:aabbcc}]") && + ((log_table_t*)array_get(res1, 0))->is_remove == true, + "delta ext rmv 1"); + + sz = 0; + sz = log_hash_print(buf, sz, &eng->tables, false); buf[sz] = 0; + + log_array_free(res); log_array_free(inp_remove); + log_array_free(inp_insert); + log_array_free(res1); log_array_free(inp1_remove); + log_array_free(inp1_insert); + + log_engine_free(eng); set_free(gv); + printf(" all tables=%s\n", buf); + } + printf("- delta and ext func\n"); +} + +static void +test_delta_misc(void) +{ + char buf[1024]; + { + /* input's counter should be ignored */ + int sz; + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = + log_eng_parse("Xy(a) : x(a, -); Xy1(a) : Xy(a); Y(a) : Xy1(a) .", gv); + + log_table_t* tbl1 = log_table_init(NULL, 0, 2, 0, gv); /* tbl x */ + log_table_add(tbl1, log_tuple_init_str("1:1:1", ':', gv)); + log_table_add(tbl1, log_tuple_init_str("1:1:2", ':', gv)); + + TYPE(array_t*, log_table_t*) inp_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + array_add(inp_insert, tbl1); + + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); + sz = 0; + sz = log_hash_print(buf, sz, &eng->tables, false); buf[sz] = 0; + + printf(" delta, counter %s\n", buf); + t_assert(0 == strcmp(buf, "{0->{1:1:1,1:1:2},1->{2:1},2->{1:1}}"), + "delta, counter value 1"); + + log_array_free(res); log_array_free(inp_remove); + log_array_free(inp_insert); + + log_table_t* tbl2 = log_table_init(NULL, 0, 2, 0, gv); /* tbl x */ + tbl2->is_remove = true; + log_table_add(tbl2, log_tuple_init_str("1:1:1", ':', gv)); + log_table_add(tbl2, log_tuple_init_str("1:1:2", ':', gv)); + + TYPE(array_t*, log_table_t*) inp1_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp1_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + array_add(inp1_remove, tbl2); + + array_t* res1 = log_eng_delta(eng, inp1_remove, inp1_insert); + /* no assert issue when minus */ + sz = 0; sz = log_array_print(buf, sz, res1, false); buf[sz] = 0; + + t_assert(0 == strcmp(buf, "[{1:1}]"), "delta, counter value 2"); + printf(" delta, counter minus%s\n", buf); + + log_array_free(res1); log_array_free(inp1_remove); + log_array_free(inp1_insert); + log_engine_free(eng); set_free(gv); + } + { + /* merge add and remove for normal join */ + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = + log_eng_parse("A(a) : b(a, b) c(b) .", gv); + + log_table_t* tbl1 = log_table_init(NULL, 1, 1, 0, gv); /* tbl c */ + log_table_add(tbl1, log_tuple_init_str("1:1", ':', gv)); + + TYPE(array_t*, log_table_t*) inp_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + array_add(inp_insert, tbl1); + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); + + /* when testing, should reorder the 2 doJoin in delta() */ + log_table_t* tbl2 = log_table_init(NULL, 1, 1, 0, gv); /* tbl c */ + log_table_t* tbl3 = log_table_init(NULL, 0, 2, 0, gv); /* tbl b */ + + tbl2->is_remove = true; + log_table_add(tbl2, log_tuple_init_str("1:1", ':', gv)); + log_table_add(tbl3, log_tuple_init_str("1:2:1", ':', gv)); + + TYPE(array_t*, log_table_t*) inp1_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp1_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + + array_add(inp1_remove, tbl2); + array_add(inp1_insert, tbl3); + array_t* res1 = log_eng_delta(eng, inp1_remove, inp1_insert); + + int sz = 0; sz = log_hash_print(buf, sz, &eng->tables, false); + buf[sz] = 0; + printf(" merge add and remove %s\n", buf); + t_assert(array_size(res1) == 0, "merge add and remove"); + + log_array_free(res); log_array_free(inp_remove); + log_array_free(inp_insert); + log_array_free(res1); log_array_free(inp1_remove); + log_array_free(inp1_insert); + log_engine_free(eng); set_free(gv); + } + printf("- delta, misc\n"); +} + +static void +test_delta_more(void) +{ + + const char* rules = + +"LS_host_set(ls_id, host_id) : logical_switch(ls_id, port_id) " +"port_bind(port_id, vif_id) dest_place(host_id, vif_id); " +"LS_HOST_SET(host_id, ls_id, host_id_item) : LS_host_set(ls_id, host_id) " +"LS_host_set(ls_id, host_id_item); " +"LS_HOST_TUNNEL(host_id, host_id_item, tunnel_ip) : " +"LS_host_set(ls_id, host_id) " +"LS_host_set(ls_id, host_id_item) dest_tunnel(host_id_item,tunnel_ip) . "; + + const char* d1[] = { + "+:1:logical_switch", "1:ls1:lp1", + "+:1:port_bind", "1:lp1:vif1", + "+:1:dest_place", "1:h1:vif1", + "+:1:dest_tunnel", "1:h1:ip1" + }; + + const char* d2[] = { + "+:1:port_bind", "1:lp2:vif2", + "+:1:dest_place", "1:h2:vif2", + "+:1:logical_switch", "1:ls1:lp2", + "+:1:dest_tunnel", "1:h2:ip2" + }; + + const char* d3[] = { + "-:1:dest_place", "1:h2:vif2", + }; + + const char* d4[] = { + "-:1:logical_switch", "1:ls1:lp1", + }; + + const char* d5[] = { + "-:1:logical_switch", "1:ls1:lp2", + "+:2:port_bind", "1:lp1:vif1", "1:lp2:vif2", + "+:1:dest_place", "1:h1:vif1", + "+:2:dest_tunnel", "1:h1:ip1", "1:h2:ip2" + }; + + char buf[1024]; + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = log_eng_parse(rules, gv); + + TYPE(array_t*, log_table_t*) inp_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + log_io_parse_line(NULL, NULL, false, NULL, NULL); + + int i; + for (i = 0;i < sizeof d1 / sizeof(char*);i++) + t_assert1(log_io_parse_line(d1[i], eng, false, inp_remove, inp_insert), + "delta more 0"); + + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); + int sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; + + t_assert(0 == strcmp(buf, + "+:1:LS_HOST_TUNNEL\n1:h1:h1:ip1\n+:1:LS_HOST_SET\n1:h1:ls1:h1\n"), + "delta more 1"); + + log_array_free(inp_remove); log_array_free(inp_insert); + log_array_free(res); + + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); + log_io_parse_line(NULL, NULL, false, NULL, NULL); + + for (i = 0;i < sizeof d2 / sizeof(char*);i++) + t_assert1(log_io_parse_line(d2[i], eng, false, inp_remove, inp_insert), + "delta more 2"); + + res = log_eng_delta(eng, inp_remove, inp_insert); + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; + + t_assert(0 == strcmp(buf, + "+:3:LS_HOST_TUNNEL\n1:h2:h1:ip1\n1:h1:h2:ip2\n1:h2:h2:ip2\n" + "+:3:LS_HOST_SET\n1:h2:ls1:h2\n1:h1:ls1:h2\n1:h2:ls1:h1\n"), + "delta more 3"); + + log_array_free(inp_remove); log_array_free(inp_insert); + log_array_free(res); + + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); + log_io_parse_line(NULL, NULL, false, NULL, NULL); + + for (i = 0;i < sizeof d3 / sizeof(char*);i++) + t_assert1(log_io_parse_line(d3[i], eng, false, inp_remove, inp_insert), + "delta more 4"); + + res = log_eng_delta(eng, inp_remove, inp_insert); + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; + + t_assert(0 == strcmp(buf, + "-:3:LS_HOST_TUNNEL\n1:h2:h1:ip1\n1:h1:h2:ip2\n1:h2:h2:ip2\n" + "-:3:LS_HOST_SET\n1:h2:ls1:h2\n1:h1:ls1:h2\n1:h2:ls1:h1\n" + ), "delta more 5"); + + log_array_free(inp_remove); log_array_free(inp_insert); + log_array_free(res); + + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); + log_io_parse_line(NULL, NULL, false, NULL, NULL); + + for (i = 0;i < sizeof d4 / sizeof(char*);i++) + t_assert1(log_io_parse_line(d4[i], eng, false, inp_remove, inp_insert), + "delta more 5"); + + res = log_eng_delta(eng, inp_remove, inp_insert); + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; + + t_assert(0 == strcmp(buf, + "-:1:LS_HOST_TUNNEL\n1:h1:h1:ip1\n-:1:LS_HOST_SET\n1:h1:ls1:h1\n" + ), "delta more 6"); + + log_array_free(inp_remove); log_array_free(inp_insert); + log_array_free(res); + + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); + log_io_parse_line(NULL, NULL, false, NULL, NULL); + + for (i = 0;i < sizeof d5 / sizeof(char*);i++) + t_assert1(log_io_parse_line(d5[i], eng, false, inp_remove, inp_insert), + "delta more 6"); + + res = log_eng_delta(eng, inp_remove, inp_insert); + sz = 0; sz = log_io_gen_line(buf, sz, eng, res); buf[sz] = 0; + + t_assert(0 == strcmp(buf,""), "delta more 7"); + log_array_free(inp_remove); log_array_free(inp_insert); + log_array_free(res); + + log_engine_free(eng); + set_free(gv); + printf("- delta, more\n"); +} + +static int64_t +calc_tm(struct timespec* ts0, struct timespec* ts1) +{ + int64_t t0 = ts0->tv_sec * 1000 * 1000L + ts0->tv_nsec / 1000; + int64_t t1 = ts1->tv_sec * 1000 * 1000L + ts1->tv_nsec / 1000; + return t1 - t0; +} + +static void +test_join_perf(int32_t sz1, int32_t sz2) +{ + /* correct value for full join is sz1 * sz2 * sz2 + * correct value for delta join is sz2 * 2 + 1 + */ + + struct timespec ts0, ts1; + + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = log_eng_parse( + "TABLE(b, a, c) : table1(a, b) table1(a, c).", gv); + + TYPE(array_t*, log_table_t*) inp_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + + int i, j; + char buf[1024]; + + log_table_t* tbl1 = log_table_init(NULL, 0, 2, 0, gv); + for (i = 0;i < sz1;i++) + for (j = 0;j < sz2;j++) { + sprintf(buf, "1:ID_%d:XY_%d", i, j); + log_tuple_t* tuple = log_tuple_init_str(buf, ':', gv); + log_table_add(tbl1, tuple); + } + + array_add(inp_insert, tbl1); + + clock_gettime(CLOCK_MONOTONIC, &ts0); + array_t* res = log_eng_delta(eng, inp_remove, inp_insert); + clock_gettime(CLOCK_MONOTONIC, &ts1); + log_table_t* res_tbl = array_get(res, 0); + + printf(" join sz1=%d sz2=%d\n", sz1, sz2); + printf(" full run = %d buckets = %d\n", + table_size(res_tbl), res_tbl->tuples.len); + + printf(" time = %" PRId64 " us\n", calc_tm(&ts0, &ts1)); + log_array_free(inp_remove); + log_array_free(inp_insert); + log_array_free(res); + + inp_remove = log_array_init(NULL, ENT_TABLE, 0, gv); + inp_insert = log_array_init(NULL, ENT_TABLE, 0, gv); + + tbl1 = log_table_init(NULL, 0, 2, 0, gv); + log_table_add(tbl1, log_tuple_init_str("1:ID_0:XY_new_item", ':', gv)); + array_add(inp_insert, tbl1); + + clock_gettime(CLOCK_MONOTONIC, &ts0); + res = log_eng_delta(eng, inp_remove, inp_insert); + clock_gettime(CLOCK_MONOTONIC, &ts1); + + res_tbl = array_get(res, 0); + printf(" delta run = %d\n", table_size(res_tbl)); + printf(" time = %" PRId64 " us\n", calc_tm(&ts0, &ts1)); + log_array_free(inp_remove); + log_array_free(inp_insert); + log_array_free(res); + + log_engine_free(eng); + set_free(gv); + printf("- join perf\n"); +} + +static void +test_interactive(void) +{ + /* interactive engine for testing purpose */ + log_set_sep(':', '\n'); + printf("Input rules, e.g. R(a):r(a).\nuse EOF as end\n"); + char inp[4096]; + inp[0] = 0; + + for (;;) { + char buf[1024]; + if (scanf("%s", buf) != 1) continue; + if (strcmp(buf, "EOF") == 0) break; + strcat(inp, buf); + } + + set_t* gv = set_init(NULL, 0, 0, NULL); + log_set_global_value(gv); + log_engine_t* eng = log_eng_parse(inp, gv); + + for (;;) { + TYPE(array_t*, log_table_t*) inp_remove = + log_array_init(NULL, ENT_TABLE, 0, gv); + TYPE(array_t*, log_table_t*) inp_insert = + log_array_init(NULL, ENT_TABLE, 0, gv); + log_io_parse_line(NULL, NULL, false, NULL, NULL); + + printf("Input changes, e.g. +:1:tbl_name|1:f0:f1|.\n" + "use +, -, ? for add, remove, or query.\n" + "could input multiple tables. the number in table name\n" + "line indicates number of tuples to follow.\n" + "'|' stands for new line. use '.' to exit\n" + "for query, field value could be empty.\n"); + + bool is_query = false; + for (;;) { + if (scanf("%s", inp) != 1) continue; + if (strcmp(inp, ".") == 0) break; + + if (is_query == false && strlen(inp) > 0 && inp[0] == '?') { + inp[0] = '+'; + is_query = true; + } + + bool res = log_io_parse_line(inp, eng, + is_query ? true : false, inp_remove, inp_insert); + if (res == false) { + printf("Error in format\n"); + break; + } + } + + if (array_size(inp_remove) == 0 && array_size(inp_insert) == 0) { + log_array_free(inp_remove); log_array_free(inp_insert); break; + } + + array_t* res = is_query ? + log_eng_query(eng, inp_insert) : + log_eng_delta(eng, inp_remove, inp_insert); + + int sz = 0; sz = log_io_gen_line(inp, sz, eng, res); inp[sz] = 0; + printf("Output\n%s\n", inp); + + log_array_free(inp_remove); log_array_free(inp_insert); + log_array_free(res); + } + + log_engine_free(eng); + set_free(gv); +} + +static void +test_api(void) +{ + const char* p, *p1, *p2, *p3, *p4, *p5; + int32_t sz, sz1, sz2; + bool rmv, rmv1; + + void* eng = datalog_init("R2(a,b):r2(a,b); R1(a):r1(a).", + /* ext func not provided */NULL); + + rmv = datalog_put_table(eng, /* adding */false, "r1"); + /* first tuple for r1 */ + datalog_put_field(eng, "r_1a", 0); + + rmv1 = datalog_put_table(eng, false, "r2"); + /* first tuple for r2 */ + datalog_put_field(eng, "r2_1a", /* c-str*/ 0); + datalog_put_field(eng, "r2_1bx", /* len */5); + /* second tuple r2 */ + datalog_put_field(eng, "r2_2a", 0); + datalog_put_field(eng, "r2_2b", 0); + + datalog_opr(eng, /* delta change */false); + t_assert(rmv && rmv1, "api 0"); + + /* get output table R2 */ + rmv1 = datalog_get_table(eng, &rmv, &p, &sz, &sz1); + + t_assert(0 == strcmp(p, "R2") && sz == 2 && sz1 ==2 && + rmv == false && rmv1 == true, "api 1"); + + /* first tuple */ + datalog_get_field(eng, &p1, &sz); + datalog_get_field(eng, &p2, &sz1); + /* second tuple */ + datalog_get_field(eng, &p3, &sz); + rmv = datalog_get_field(eng, &p4, &sz); + /* indicates reaching next table */ + rmv1 = datalog_get_field(eng, &p5, &sz); + + t_assert(0 == strcmp(p1, "r2_1a") && 0 == strcmp(p2, "r2_1b") && + 0 == strcmp(p3, "r2_2a") && 0 == strcmp(p4, "r2_2b") + && sz == 5 && sz1 == 5 && rmv == true + && rmv1 == false, "api 2"); + + /* get output table R1 */ + rmv1 = datalog_get_table(eng, &rmv, &p, &sz, &sz1); + t_assert(0 == strcmp(p, "R1") && sz == 1 && sz1 == 1 && + rmv == false && rmv1 == true, "api 3"); + + rmv = datalog_get_field(eng, &p1, &sz); + /* indicates reaching next table */ + rmv1 = datalog_get_field(eng, &p2, &sz); + + t_assert(0 == strcmp(p1, "r_1a") && + sz == 4 && rmv == true && rmv1 == false, "api 4"); + + /* no more table returned */ + rmv = datalog_get_table(eng, &rmv, &p, &sz, &sz1); + t_assert(rmv == false, "api 5"); + + datalog_put_table(eng, false, "r2"); + datalog_put_field(eng, "r2_1a0", 0); + datalog_put_field(eng, "r2_1b", 0); + datalog_opr(eng, false); + datalog_get_table(eng, &rmv, &p, &sz, &sz1); + /* it is ok to skip tuples of a table */ + datalog_get_table(eng, &rmv, &p, &sz, &sz1); + + datalog_put_table(eng, false, "r2"); + datalog_put_field(eng, NULL, 0); + datalog_put_field(eng, "r2_1b", 0); + datalog_opr(eng, /* query */true); + + rmv = datalog_get_table(eng, &rmv, &p, &sz2, &sz1); + datalog_get_field(eng, &p1, &sz); + datalog_get_field(eng, &p2, &sz); + datalog_get_field(eng, &p3, &sz); + datalog_get_field(eng, &p4, &sz); + rmv1 = datalog_get_table(eng, &rmv, &p, &sz, &sz1); + + t_assert(0 == strcmp(p1, "r2_1a") && sz2 == 2 && + sz1 == 2 && rmv == true && rmv1 == false, "api 5"); + + datalog_free(eng); + printf("- api\n"); +} + +static void +test_datalog(int argc, char** argv) +{ + if (argc == 2 && !strcmp(argv[1], "test")) { + log_set_sep(':', '\n'); + + test_collections(); + test_tables(); + test_sort(); + test_sync(); + test_join(); + test_delta(); + test_delta_misc(); + test_delta_more(); + test_join_perf(100, 100); + test_api(); + + t_sum(); + fprintf(stderr, "%s\n", /* for at script */ + log_tst_no_cases_failed == 0 ? "PASS" : "FAIL"); + } + else if (argc == 2 && !strcmp(argv[1], "run")) + test_interactive(); + else printf("usage: test-datalog test|run\nrun for interactive mode"); +} + +/*int main(int argc, char** argv) { test_datalog(argc, argv); return 0; }*/ + +OVSTEST_REGISTER("test-datalog", test_datalog); diff --git a/tests/testsuite.at b/tests/testsuite.at index 7ac74df..33c196f 100644 --- a/tests/testsuite.at +++ b/tests/testsuite.at @@ -72,3 +72,4 @@ m4_include([tests/ovn-nbctl.at]) m4_include([tests/ovn-sbctl.at]) m4_include([tests/ovn-controller.at]) m4_include([tests/ovn-controller-vtep.at]) +m4_include([tests/datalog.at])

[ovs-dev] OVN: initial patch of datalog engine

Commit Message

Comments

Patch