From patchwork Thu Jul 4 00:45:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lucas Bates X-Patchwork-Id: 1127283 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=mojatatu.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=mojatatu-com.20150623.gappssmtp.com header.i=@mojatatu-com.20150623.gappssmtp.com header.b="J+Piyr8r"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 45fK604lzrz9sPB for ; Thu, 4 Jul 2019 10:46:00 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727095AbfGDAp7 (ORCPT ); Wed, 3 Jul 2019 20:45:59 -0400 Received: from mail-io1-f68.google.com ([209.85.166.68]:45891 "EHLO mail-io1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726550AbfGDAp7 (ORCPT ); Wed, 3 Jul 2019 20:45:59 -0400 Received: by mail-io1-f68.google.com with SMTP id g20so1525858ioc.12 for ; Wed, 03 Jul 2019 17:45:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mojatatu-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=f8fcnM09WvnMgREWvby4Pkdfwla53pclgGTRiqTO+eE=; b=J+Piyr8rXnls530bCEVtD8i1WycCEigz9uO2IyXXJXVSdFohmClkBePiVVqMJdI8fD FxL9XYDSKHFxfGpPh2zbMM5M2l/Ux/RKdJvcrSJnlplfRL24pMrEJPqBbIG5laU9/Hs3 Z1uKhUXpXtJjsILAmYeZSD+yFyYHf5MpQvdd1aSSSeq7afGuMthy5G1I5mh7LEplVICC hIf7K1xTEsb3B5sanXhDxMUgaRpdoQ0zDjaeiF/f1KreICA9tRxbavD9RR5JFNMVnfEH BfN55YC40ZbI6br5VrNlzWwKdCcxUcIgVAfx7xn3LNaNjwWknhXULVd3Lw5my79piY4S c55Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=f8fcnM09WvnMgREWvby4Pkdfwla53pclgGTRiqTO+eE=; b=W52kepY3oCfnDy01OUtDn6XmWoW+slAm4Xxnvwn7AwQkUOCaCrA8G3d/JyUAhw1ab1 1wWhuviHn+GRwx+entJYldYDzqF7RUH21iM0SBl8R8HoDmHEY2Iq0UNGk2Aq02w+5lt6 ag+t8W1ZfUUxk+iE++iOgibgC+a3sPoXRn9rrQsY0ZIUDeR/ArzwEtwbc6XEL3UUwdb3 pQJZm7WNo8ciwbJwz+xb8mozGrYD8EOeGmYis3j16xy38wMCnbt58dT1jlVFCDLy/2Ow TNqE6gGd6ZkeN8s4bSAL0aaekztw3GuKZK+uM+wjg+5+XCIzQ8xG1l1yXB9qkjAcnzD7 pWFQ== X-Gm-Message-State: APjAAAVP2U32lMHGo/HNjaLHgA/1FjxjobUnrhBUXkg+WQ15BAKJ9VEA /8ZMgwT4HmOZjWrzcSOGQZvPWA== X-Google-Smtp-Source: APXvYqz/3mpMW+1iAdECDMY5YHdy1dQAKp8Jg1dhOC9fiYe/ZxalnETbOgcE25dXEfEkrdvpakRirw== X-Received: by 2002:a5e:9e0a:: with SMTP id i10mr16745548ioq.44.1562201157501; Wed, 03 Jul 2019 17:45:57 -0700 (PDT) Received: from mojatatu.com ([2607:f2c0:e4b2:adf:4494:a7b3:9aab:d513]) by smtp.gmail.com with ESMTPSA id l5sm5619776ioq.83.2019.07.03.17.45.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jul 2019 17:45:56 -0700 (PDT) From: Lucas Bates To: davem@davemloft.net Cc: netdev@vger.kernel.org, jhs@mojatatu.com, xiyou.wangcong@gmail.com, jiri@resnulli.us, mleitner@redhat.com, vladbu@mellanox.com, dcaratti@redhat.com, kernel@mojatatu.com, Lucas Bates Subject: [PATCH v2 net-next 1/3] tc-testing: Add JSON verification to tdc Date: Wed, 3 Jul 2019 20:45:00 -0400 Message-Id: <1562201102-4332-2-git-send-email-lucasb@mojatatu.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1562201102-4332-1-git-send-email-lucasb@mojatatu.com> References: <1562201102-4332-1-git-send-email-lucasb@mojatatu.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch allows tdc to process JSON output to perform secondary verification of the command under test. If the verifyCmd generates JSON, one can provide the 'matchJSON' key to process it instead of a regex. matchJSON has two elements: 'path' and 'value'. The 'path' key is a list of integers and strings that provide the key values for tdc to navigate the JSON information. The value is an integer or string that tdc will compare against what it finds in the provided path. If the numerical position of an element can vary, it's possible to substitute an asterisk as a wildcard. tdc will search all possible entries in the array. Multiple matches are possible, but everything specified must match for the test to pass. If both matchPattern and matchJSON are present, tdc will only operate on matchPattern. If neither are present, verification is skipped. Example: "cmdUnderTest": "$TC actions add action pass index 8", "verifyCmd": "$TC actions list action gact", "matchJSON": [ { "path": [ 0, "actions", 0, "control action", "type" ], "value": "gact" }, { "path": [ 0, "actions", 0, "index" ], "value": 8 } ] Signed-off-by: Lucas Bates --- tools/testing/selftests/tc-testing/tdc.py | 136 +++++++++++++++++++++++++++--- 1 file changed, 123 insertions(+), 13 deletions(-) diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py index 678182a..1afa803 100755 --- a/tools/testing/selftests/tc-testing/tdc.py +++ b/tools/testing/selftests/tc-testing/tdc.py @@ -35,6 +35,10 @@ class PluginMgrTestFail(Exception): self.output = output self.message = message +class ElemNotFound(Exception): + def __init__(self, path_element): + self.path_element = path_element + class PluginMgr: def __init__(self, argparser): super().__init__() @@ -167,6 +171,40 @@ class PluginMgr: self.argparser = argparse.ArgumentParser( description='Linux TC unit tests') +def find_in_json(jsonobj, path): + if type(jsonobj) == list: + if type(path[0]) == int: + if len(jsonobj) > path[0]: + return find_in_json(jsonobj[path[0]], path[1:]) + else: + raise ElemNotFound(path[0]) + elif path[0] == '*': + res = [] + for index in jsonobj: + try: + res.append(find_in_json(index, path[1:])) + except ElemNotFound: + continue + if len(res) == 0: + raise ElemNotFound(path[0]) + else: + return res + elif type(jsonobj) == dict: + if path[0] in jsonobj: + if len(path) > 1: + return find_in_json(jsonobj[path[0]], path[1:]) + return jsonobj[path[0]] + else: + raise ElemNotFound(path[0]) + else: + # Assume we have found the correct depth in the object + if len(path) >= 1: + print('The remainder of the specified path cannot be found!') + print('Path values: {}'.format(path)) + raise ElemNotFound(path[0]) + return jsonobj + + def replace_keywords(cmd): """ For a given executable command, substitute any known @@ -246,6 +284,86 @@ def prepare_env(args, pm, stage, prefix, cmdlist, output = None): stage, output, '"{}" did not complete successfully'.format(prefix)) +def verify_by_regex(res, tidx, args, pm): + if 'matchCount' not in tidx: + res.set_result(ResultState.skip) + fmsg = 'matchCount was not provided in the test case. ' + fmsg += 'Unable to complete pattern match.' + res.set_failmsg(fmsg) + print(fmsg) + return res + (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"]) + match_pattern = re.compile( + str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE) + if procout: + match_index = re.findall(match_pattern, procout) + if len(match_index) != int(tidx["matchCount"]): + res.set_result(ResultState.fail) + fmsg = 'Verify stage failed because the output did not match ' + fmsg += 'the pattern in the test case.\nMatch pattern is:\n' + fmsg += '\t{}\n'.format(tidx["matchPattern"]) + fmsg += 'Output generated by the verify command:\n' + fmsg += '{}\n'.format(procout) + res.set_failmsg(fmsg) + else: + res.set_result(ResultState.success) + elif int(tidx["matchCount"]) != 0: + res.set_result(ResultState.fail) + res.set_failmsg('No output generated by verify command.') + else: + res.set_result(ResultState.success) + return res + +def verify_by_json(res, tidx, args, pm): + # Validate the matchJSON struct + for match in tidx['matchJSON']: + if 'path' in match and 'value' in match: + pass + else: + res.set_result(ResultState.skip) + res.set_failmsg('matchJSON missing required keys for this case.') + return res + (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"]) + # Run procout through the JSON decoder + try: + jsonobj = json.loads(procout) + except json.JSONDecodeError: + if len(tidx['matchJSON']) > 0: + res.set_result(ResultState.fail) + res.set_failmsg('Cannot decode verify command\'s output. Is it JSON?') + return res + # Then recurse through the object + valuesmatch = True + for match in tidx['matchJSON']: + try: + value = find_in_json(jsonobj, match['path']) + except ElemNotFound as ENF: + fmsg = 'Could not find the element {} specified in the path.'.format(ENF.path_element) + valuesmatch = False + break + if type(value) == list: + if match['value'] not in value: + valuesmatch = False + fmsg = 'Verify stage failed because the value specified in the path\n' + fmsg += '{}\n'.format(match['path']) + fmsg += 'Expected value: {}\nReceived value: {}'.format( + match['value'], value) + break + elif match['value'] != value: + valuesmatch = False + fmsg = 'Verify stage failed because the value specified in the path\n' + fmsg += '{}\n'.format(match['path']) + fmsg += 'Expected value: {}\nReceived value: {}'.format( + match['value'], value) + break + if valuesmatch: + res.set_result(ResultState.success) + else: + res.set_result(ResultState.fail) + res.set_failmsg(fmsg) + print(fmsg) + return res + def run_one_test(pm, args, index, tidx): global NAMES result = True @@ -292,21 +410,13 @@ def run_one_test(pm, args, index, tidx): else: if args.verbose > 0: print('-----> verify stage') - match_pattern = re.compile( - str(tidx["matchPattern"]), re.DOTALL | re.MULTILINE) - (p, procout) = exec_cmd(args, pm, 'verify', tidx["verifyCmd"]) - if procout: - match_index = re.findall(match_pattern, procout) - if len(match_index) != int(tidx["matchCount"]): - res.set_result(ResultState.fail) - res.set_failmsg('Could not match regex pattern. Verify command output:\n{}'.format(procout)) - else: - res.set_result(ResultState.success) - elif int(tidx["matchCount"]) != 0: - res.set_result(ResultState.fail) - res.set_failmsg('No output generated by verify command.') + if 'matchPattern' in tidx: + res = verify_by_regex(res, tidx, args, pm) + elif 'matchJSON' in tidx: + res = verify_by_json(res, tidx, args, pm) else: res.set_result(ResultState.success) + print('No match method defined in current test case, skipping verify') prepare_env(args, pm, 'teardown', '-----> teardown stage', tidx['teardown'], procout) pm.call_post_case() From patchwork Thu Jul 4 00:45:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lucas Bates X-Patchwork-Id: 1127284 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=mojatatu.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=mojatatu-com.20150623.gappssmtp.com header.i=@mojatatu-com.20150623.gappssmtp.com header.b="pz/1pZEp"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 45fK620dv8z9s8m for ; Thu, 4 Jul 2019 10:46:02 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727113AbfGDAqB (ORCPT ); Wed, 3 Jul 2019 20:46:01 -0400 Received: from mail-io1-f65.google.com ([209.85.166.65]:38693 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727056AbfGDAp7 (ORCPT ); Wed, 3 Jul 2019 20:45:59 -0400 Received: by mail-io1-f65.google.com with SMTP id j6so9270832ioa.5 for ; Wed, 03 Jul 2019 17:45:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mojatatu-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RHK7SlJqiYa2Rq9rqHsf5d7QBoTdkJldScxSF9BOtuU=; b=pz/1pZEpqZKzvjCwryblYlm5TzBXTZ79bh34dYtgekZlnc/frPKC+bMOoqra881RMO DBG/c5XYpQi4vKAdcYu2R1s/lj9v8id2Vv2pzcglRW4DOs816EJULPdd2lQQCMITBpnz 3LMLIX3om3uBXyv+cnbsQkAj3HT1ky1PPwlzo3fvSKv+eaNlwWwL7IuvQJSGzQAH48Nm FHhDtbE1W4cNG1gRgdAmHmPhcyOKma1VfHyF+I4JDh/QmFYVO4M/0/DDgoGythnPA+c1 OTZfizSQJppTMZkhvxSFnO1pPnf/6R6mCr5pHrGABv9jcXQi4A64yvjPvFxliK8Qr+zx RHFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RHK7SlJqiYa2Rq9rqHsf5d7QBoTdkJldScxSF9BOtuU=; b=ry3aXhl8IDCTfS8EMxPD2qlgKeSkZzbA/S0ri1cGb7Y3izSYUGfcFau2Mb94pgk8wQ FCnFfrlItJ6CCJtnRi5l/oCF78x0mMk8vy3Di5JWKXDt5Kg6Al45Cy5HABGZQLnhipuX oAJysIe6/mpmLXmsb/B/8lDm6CcHVJ3g2cyTfNhvkmfUF+AcmyJNNyNTxm91mUXHiWrU W6r7PQX+QYyFImKD5B/Fl/J8cgr2S95rB1WH2Buplw6hV5RmIPua2Y6ZA4pX1pIk2nQc e7sjBV1KaojqUe/59ZskYPWr/GAeTi41DVHihi6NcRyoTT5yvrc4U2D4NmRtHZ2VZZEy IjDg== X-Gm-Message-State: APjAAAVPBfNue09dwu+NSrWAyaGVrwGRfVrvc9gCC7ZkoSwGBsE+gEs8 9P0frx93vSdz24/g3K+aKz2vhg== X-Google-Smtp-Source: APXvYqzdQYa+cczENRt6f03cb+9WnCIMchrkDSBeUtBG2vpc3iGNeT1Wbvubj+MUsnfXJydCzhxOcA== X-Received: by 2002:a5d:8195:: with SMTP id u21mr12086889ion.260.1562201158884; Wed, 03 Jul 2019 17:45:58 -0700 (PDT) Received: from mojatatu.com ([2607:f2c0:e4b2:adf:4494:a7b3:9aab:d513]) by smtp.gmail.com with ESMTPSA id l5sm5619776ioq.83.2019.07.03.17.45.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jul 2019 17:45:58 -0700 (PDT) From: Lucas Bates To: davem@davemloft.net Cc: netdev@vger.kernel.org, jhs@mojatatu.com, xiyou.wangcong@gmail.com, jiri@resnulli.us, mleitner@redhat.com, vladbu@mellanox.com, dcaratti@redhat.com, kernel@mojatatu.com, Lucas Bates Subject: [PATCH v2 net-next 2/3] tc-testing: Allow tdc plugins to see test case data Date: Wed, 3 Jul 2019 20:45:01 -0400 Message-Id: <1562201102-4332-3-git-send-email-lucasb@mojatatu.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1562201102-4332-1-git-send-email-lucasb@mojatatu.com> References: <1562201102-4332-1-git-send-email-lucasb@mojatatu.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Instead of only passing the test case name and ID, pass the entire current test case down to the plugins. This change allows plugins to start accepting commands and directives from the test cases themselves, for greater flexibility in testing. Signed-off-by: Lucas Bates --- tools/testing/selftests/tc-testing/TdcPlugin.py | 5 ++--- tools/testing/selftests/tc-testing/tdc.py | 10 +++++----- 2 files changed, 7 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/tc-testing/TdcPlugin.py b/tools/testing/selftests/tc-testing/TdcPlugin.py index b980a56..79f3ca8 100644 --- a/tools/testing/selftests/tc-testing/TdcPlugin.py +++ b/tools/testing/selftests/tc-testing/TdcPlugin.py @@ -18,12 +18,11 @@ class TdcPlugin: if self.args.verbose > 1: print(' -- {}.post_suite'.format(self.sub_class)) - def pre_case(self, testid, test_name, test_skip): + def pre_case(self, caseinfo, test_skip): '''run commands before test_runner does one test''' if self.args.verbose > 1: print(' -- {}.pre_case'.format(self.sub_class)) - self.args.testid = testid - self.args.test_name = test_name + self.args.caseinfo = caseinfo self.args.test_skip = test_skip def post_case(self): diff --git a/tools/testing/selftests/tc-testing/tdc.py b/tools/testing/selftests/tc-testing/tdc.py index 1afa803..de7da9a 100755 --- a/tools/testing/selftests/tc-testing/tdc.py +++ b/tools/testing/selftests/tc-testing/tdc.py @@ -126,15 +126,15 @@ class PluginMgr: for pgn_inst in reversed(self.plugin_instances): pgn_inst.post_suite(index) - def call_pre_case(self, testid, test_name, *, test_skip=False): + def call_pre_case(self, caseinfo, *, test_skip=False): for pgn_inst in self.plugin_instances: try: - pgn_inst.pre_case(testid, test_name, test_skip) + pgn_inst.pre_case(caseinfo, test_skip) except Exception as ee: print('exception {} in call to pre_case for {} plugin'. format(ee, pgn_inst.__class__)) print('test_ordinal is {}'.format(test_ordinal)) - print('testid is {}'.format(testid)) + print('testid is {}'.format(caseinfo['id'])) raise def call_post_case(self): @@ -379,14 +379,14 @@ def run_one_test(pm, args, index, tidx): res = TestResult(tidx['id'], tidx['name']) res.set_result(ResultState.skip) res.set_errormsg('Test case designated as skipped.') - pm.call_pre_case(tidx['id'], tidx['name'], test_skip=True) + pm.call_pre_case(tidx, test_skip=True) pm.call_post_execute() return res # populate NAMES with TESTID for this test NAMES['TESTID'] = tidx['id'] - pm.call_pre_case(tidx['id'], tidx['name']) + pm.call_pre_case(tidx) prepare_env(args, pm, 'setup', "-----> prepare stage", tidx["setup"]) if (args.verbose > 0): From patchwork Thu Jul 4 00:45:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lucas Bates X-Patchwork-Id: 1127285 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=mojatatu.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=mojatatu-com.20150623.gappssmtp.com header.i=@mojatatu-com.20150623.gappssmtp.com header.b="bWdnYakQ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 45fK636hsFz9s8m for ; Thu, 4 Jul 2019 10:46:03 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727123AbfGDAqC (ORCPT ); Wed, 3 Jul 2019 20:46:02 -0400 Received: from mail-io1-f65.google.com ([209.85.166.65]:35414 "EHLO mail-io1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727102AbfGDAqB (ORCPT ); Wed, 3 Jul 2019 20:46:01 -0400 Received: by mail-io1-f65.google.com with SMTP id m24so9288268ioo.2 for ; Wed, 03 Jul 2019 17:46:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mojatatu-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=WF+Cz7KLSMn8XUGDuGnkx5caNFaJT5HONH8aacQS2u0=; b=bWdnYakQumM1ify3A8qCzJXjTeaTJouBv2oOkNPE1dW4uTbK7JEfg1FEN6cTHZF2XH TN5lL6YwXc1LoPdDnMIKAOaj5vNxVbfuab6ydIjbjiT9ezmK1rHgTobY5J1Gvg8czROD J0GEehilZ4Tod63+USwQpJSNvEmOwu2BlA0/+0cuW/JAEXNzDcVh1QbPLgpYwhELc+Ou hV5QxL8YIOK81BGnJFyOQyO5I/4wJlSlhfQKqDmlqDEBBl9/R+BO7NxGG9bZAgkjhW2Y JlrmHA2VtsZYAUiKFU8/nbTXSkMJS5je6OnHheRCnSeoWrg+5UFlOPbl9ya3t4GmAoNO NL5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=WF+Cz7KLSMn8XUGDuGnkx5caNFaJT5HONH8aacQS2u0=; b=DXP6CDUQKHge5Lg8YvwhSBkllI23vOBXsyEatb7dwFMOv/eMFdsokvvQW4wg9a9z11 2QW1AvbiThoxRMeLZ330gRguPVBAXvFdgAcX42Rr+aR4HcbJkZSQ/8RLBbPHW77jfEES Ul1keKV960s55tpsC4izjW783Wm6X8bE5IJq/K6KGyIBeGY4tc2NW9khNvaV4Z9ubPzB b3uTnwwrNSSoULy2WRwEM1Hn6MTrwKY2Dw+vnEVdlHEWPXFnOhC3ZmnQ9cX7zvprBwgu 9BNgKvw+eOI2Mr78WgVMyBbHwxE2fxLMd/3l4CNHbw77MKBvECycvKyEyUCDKD258epx +HUA== X-Gm-Message-State: APjAAAUoXwJAvNSU0fEznGba6VbAObRjYNx1FG65uelPxOan3CHnkmvm SRZ6XHd72D6fnZxtyglrUwsCjw== X-Google-Smtp-Source: APXvYqyvdjjAANqvAuViQ/RBAbu9NB5XTZ3gd12mXCpNRul8/IN5sMe+qck65YghmwrhzUOEsOxU6g== X-Received: by 2002:a5e:a712:: with SMTP id b18mr41599226iod.220.1562201160201; Wed, 03 Jul 2019 17:46:00 -0700 (PDT) Received: from mojatatu.com ([2607:f2c0:e4b2:adf:4494:a7b3:9aab:d513]) by smtp.gmail.com with ESMTPSA id l5sm5619776ioq.83.2019.07.03.17.45.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 03 Jul 2019 17:45:59 -0700 (PDT) From: Lucas Bates To: davem@davemloft.net Cc: netdev@vger.kernel.org, jhs@mojatatu.com, xiyou.wangcong@gmail.com, jiri@resnulli.us, mleitner@redhat.com, vladbu@mellanox.com, dcaratti@redhat.com, kernel@mojatatu.com, Lucas Bates Subject: [PATCH v2 net-next 3/3] tc-testing: introduce scapyPlugin for basic traffic Date: Wed, 3 Jul 2019 20:45:02 -0400 Message-Id: <1562201102-4332-4-git-send-email-lucasb@mojatatu.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1562201102-4332-1-git-send-email-lucasb@mojatatu.com> References: <1562201102-4332-1-git-send-email-lucasb@mojatatu.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The scapyPlugin allows for simple traffic generation in tdc to test various tc features. It was tested with scapy v2.4.2, but should work with any successive version. In order to use the plugin's functionality, scapy must be installed. This can be done with: pip3 install scapy or to install 2.4.2: pip3 install scapy==2.4.2 If the plugin is unable to import the scapy module, it will terminate the tdc run. The plugin makes use of a new key in the test case data, 'scapy'. This block contains three other elements: 'iface', 'count', and 'packet': "scapy": { "iface": "$DEV0", "count": 1, "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()" }, * iface is the name of the device on the host machine from which the packet(s) will be sent. Values contained within tdc_config.py's NAMES dict can be used here - this is useful if paired with nsPlugin * count is the number of copies of this packet to be sent * packet is a string detailing the different layers of the packet to be sent. If a property isn't explicitly set, scapy will set default values for you. Layers in the packet info are separated by slashes. For info about common TCP and IP properties, see: https://blogs.sans.org/pen-testing/files/2016/04/ScapyCheatSheet_v0.2.pdf Caution is advised when running tests using the scapy functionality, since the plugin blindly sends the packet as defined in the test case data. See creating-testcases/scapy-example.json for sample test cases; the first test is intended to pass while the second is intended to fail. Consider using the matchJSON functionality for verification when using scapy. Signed-off-by: Lucas Bates --- .../creating-testcases/scapy-example.json | 98 ++++++++++++++++++++++ .../selftests/tc-testing/plugin-lib/scapyPlugin.py | 51 +++++++++++ 2 files changed, 149 insertions(+) create mode 100644 tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json create mode 100644 tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py diff --git a/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json new file mode 100644 index 0000000..5a9377b --- /dev/null +++ b/tools/testing/selftests/tc-testing/creating-testcases/scapy-example.json @@ -0,0 +1,98 @@ +[ + { + "id": "b1e9", + "name": "Test matching of source IP", + "category": [ + "actions", + "scapy" + ], + "plugins": { + "requires": [ + "nsPlugin", + "scapyPlugin" + ] + }, + "setup": [ + [ + "$TC qdisc del dev $DEV1 ingress", + 0, + 1, + 2, + 255 + ], + "$TC qdisc add dev $DEV1 ingress" + ], + "cmdUnderTest": "$TC filter add dev $DEV1 parent ffff: prio 3 protocol ip flower src_ip 16.61.16.61 flowid 1:1 action ok", + "scapy": { + "iface": "$DEV0", + "count": 1, + "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()" + }, + "expExitCode": "0", + "verifyCmd": "$TC -s -j filter ls dev $DEV1 ingress prio 3", + "matchJSON": [ + { + "path": [ + 1, + "options", + "actions", + 0, + "stats", + "packets" + ], + "value": 1 + } + ], + "teardown": [ + "$TC qdisc del dev $DEV1 ingress" + ] + }, + { + "id": "e9c4", + "name": "Test matching of source IP with wrong count", + "category": [ + "actions", + "scapy" + ], + "plugins": { + "requires": [ + "nsPlugin", + "scapyPlugin" + ] + }, + "setup": [ + [ + "$TC qdisc del dev $DEV1 ingress", + 0, + 1, + 2, + 255 + ], + "$TC qdisc add dev $DEV1 ingress" + ], + "cmdUnderTest": "$TC filter add dev $DEV1 parent ffff: prio 3 protocol ip flower src_ip 16.61.16.61 flowid 1:1 action ok", + "scapy": { + "iface": "$DEV0", + "count": 3, + "packet": "Ether(type=0x800)/IP(src='16.61.16.61')/ICMP()" + }, + "expExitCode": "0", + "verifyCmd": "$TC -s -j filter ls dev $DEV1 parent ffff:", + "matchJSON": [ + { + "path": [ + 1, + "options", + "actions", + 0, + "stats", + "packets" + ], + "value": 1 + } + ], + "teardown": [ + "$TC qdisc del dev $DEV1 ingress" + ] + } +] diff --git a/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py new file mode 100644 index 0000000..db57916 --- /dev/null +++ b/tools/testing/selftests/tc-testing/plugin-lib/scapyPlugin.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 + +import os +import signal +from string import Template +import subprocess +import time +from TdcPlugin import TdcPlugin + +from tdc_config import * + +try: + from scapy.all import * +except ImportError: + print("Unable to import the scapy python module.") + print("\nIf not already installed, you may do so with:") + print("\t\tpip3 install scapy==2.4.2") + exit(1) + +class SubPlugin(TdcPlugin): + def __init__(self): + self.sub_class = 'scapy/SubPlugin' + super().__init__() + + def post_execute(self): + if 'scapy' not in self.args.caseinfo: + if self.args.verbose: + print('{}.post_execute: no scapy info in test case'.format(self.sub_class)) + return + + # Check for required fields + scapyinfo = self.args.caseinfo['scapy'] + scapy_keys = ['iface', 'count', 'packet'] + missing_keys = [] + keyfail = False + for k in scapy_keys: + if k not in scapyinfo: + keyfail = True + missing_keys.add(k) + if keyfail: + print('{}: Scapy block present in the test, but is missing info:' + .format(self.sub_class)) + print('{}'.format(missing_keys)) + + pkt = eval(scapyinfo['packet']) + if '$' in scapyinfo['iface']: + tpl = Template(scapyinfo['iface']) + scapyinfo['iface'] = tpl.safe_substitute(NAMES) + for count in range(scapyinfo['count']): + sendp(pkt, iface=scapyinfo['iface']) +