Message ID | 20231103182901.3795263-1-thaller@redhat.com |
---|---|
Headers | show |
Series | add and check dump files for JSON in tests/shell | expand |
On Fri, 2023-11-03 at 19:26 +0100, Thomas Haller wrote: > Generate and add ".json-nft" files. These files contain the output of > `nft -j list ruleset` after the test. Also, "test-wrapper.sh" will > compare the current ruleset against the ".json-nft" files and test > them > with "nft -j --check -f $FILE`. These are useful extra tests, that we > almost get for free. > > Note that for some JSON dumps, `nft -f --check` fails (or prints > something). For those tests no *.json-nft file is added. The bugs > needs > to be fixed first. > > An example of such an issue is: > > $ DUMPGEN=all ./tests/shell/run-tests.sh > tests/shell/testcases/maps/nat_addr_port > > which gives a file "rc-failed-chkdump" with > > Command `./tests/shell/../../src/nft -j --check -f > "tests/shell/testcases/maps/dumps/nat_addr_port.json-nft"` failed > >>>> > internal:0:0-0: Error: Invalid map type 'ipv4_addr . > inet_service'. > > internal:0:0-0: Error: Parsing command array at index 3 failed. > > internal:0:0-0: Error: unqualified type integer specified in map > definition. Try "typeof expression" instead of "type datatype". > > <<<< > > Tests like "tests/shell/testcases/nft-f/0012different_defines_0" and > "tests/shell/testcases/nft-f/0024priority_0" also don't get a .json- > nft > dump yet, because their output is not stable. That needs fixing too. > > Cc: Pablo Neira Ayuso <pablo@netfilter.org> > Cc: Florian Westphal <fw@strlen.de> > Signed-off-by: Thomas Haller <thaller@redhat.com> > --- ... > tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft | ... > tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft > create mode 100644 "tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft" need to be dropped from this patch. Otherwise, make && ./tests/shell/run-tests.sh tests/shell/testcases/sets/0062set_connlimit_0 -V fails (in valgrind mode). Thomas
On Mon, Nov 06, 2023 at 02:56:18PM +0100, Thomas Haller wrote: > On Fri, 2023-11-03 at 19:26 +0100, Thomas Haller wrote: > > Generate and add ".json-nft" files. These files contain the output of > > `nft -j list ruleset` after the test. Also, "test-wrapper.sh" will > > compare the current ruleset against the ".json-nft" files and test > > them > > with "nft -j --check -f $FILE`. These are useful extra tests, that we > > almost get for free. > > > > Note that for some JSON dumps, `nft -f --check` fails (or prints > > something). For those tests no *.json-nft file is added. The bugs > > needs > > to be fixed first. > > > > An example of such an issue is: > > > > $ DUMPGEN=all ./tests/shell/run-tests.sh > > tests/shell/testcases/maps/nat_addr_port > > > > which gives a file "rc-failed-chkdump" with > > > > Command `./tests/shell/../../src/nft -j --check -f > > "tests/shell/testcases/maps/dumps/nat_addr_port.json-nft"` failed > > >>>> > > internal:0:0-0: Error: Invalid map type 'ipv4_addr . > > inet_service'. > > > > internal:0:0-0: Error: Parsing command array at index 3 failed. > > > > internal:0:0-0: Error: unqualified type integer specified in map > > definition. Try "typeof expression" instead of "type datatype". > > > > <<<< > > > > Tests like "tests/shell/testcases/nft-f/0012different_defines_0" and > > "tests/shell/testcases/nft-f/0024priority_0" also don't get a .json- > > nft > > dump yet, because their output is not stable. That needs fixing too. > > > > Cc: Pablo Neira Ayuso <pablo@netfilter.org> > > Cc: Florian Westphal <fw@strlen.de> > > Signed-off-by: Thomas Haller <thaller@redhat.com> > > --- > ... > > tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft | > ... > > tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft > > create mode 100644 > > > "tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft" need to > be dropped from this patch. > > Otherwise, > > make && ./tests/shell/run-tests.sh tests/shell/testcases/sets/0062set_connlimit_0 -V > > fails (in valgrind mode). I have to fix 0062set_connlimit_0, the listing fails because the GC is fast enough to remove the entry that just got added, because it has no conntrack entries. Maybe valgrind is just getting things slowier there to trigger what I can already reproduce here on a VM?
On Mon, 2023-11-06 at 15:04 +0100, Pablo Neira Ayuso wrote: > On Mon, Nov 06, 2023 at 02:56:18PM +0100, Thomas Haller wrote: > > On Fri, 2023-11-03 at 19:26 +0100, Thomas Haller wrote: > > > Generate and add ".json-nft" files. These files contain the > > > output of > > > `nft -j list ruleset` after the test. Also, "test-wrapper.sh" > > > will > > > compare the current ruleset against the ".json-nft" files and > > > test > > > them > > > with "nft -j --check -f $FILE`. These are useful extra tests, > > > that we > > > almost get for free. > > > > > > Note that for some JSON dumps, `nft -f --check` fails (or prints > > > something). For those tests no *.json-nft file is added. The bugs > > > needs > > > to be fixed first. > > > > > > An example of such an issue is: > > > > > > $ DUMPGEN=all ./tests/shell/run-tests.sh > > > tests/shell/testcases/maps/nat_addr_port > > > > > > which gives a file "rc-failed-chkdump" with > > > > > > Command `./tests/shell/../../src/nft -j --check -f > > > "tests/shell/testcases/maps/dumps/nat_addr_port.json-nft"` failed > > > >>>> > > > internal:0:0-0: Error: Invalid map type 'ipv4_addr . > > > inet_service'. > > > > > > internal:0:0-0: Error: Parsing command array at index 3 > > > failed. > > > > > > internal:0:0-0: Error: unqualified type integer specified in > > > map > > > definition. Try "typeof expression" instead of "type datatype". > > > > > > <<<< > > > > > > Tests like "tests/shell/testcases/nft-f/0012different_defines_0" > > > and > > > "tests/shell/testcases/nft-f/0024priority_0" also don't get a > > > .json- > > > nft > > > dump yet, because their output is not stable. That needs fixing > > > too. > > > > > > Cc: Pablo Neira Ayuso <pablo@netfilter.org> > > > Cc: Florian Westphal <fw@strlen.de> > > > Signed-off-by: Thomas Haller <thaller@redhat.com> > > > --- > > ... > > > tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft > > > | > > ... > > > tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft > > > create mode 100644 > > > > > > "tests/shell/testcases/sets/dumps/0062set_connlimit_0.json-nft" > > need to > > be dropped from this patch. > > > > Otherwise, > > > > make && ./tests/shell/run-tests.sh > > tests/shell/testcases/sets/0062set_connlimit_0 -V > > > > fails (in valgrind mode). > > I have to fix 0062set_connlimit_0, the listing fails because the GC > is > fast enough to remove the entry that just got added, because it has > no > conntrack entries. > > Maybe valgrind is just getting things slowier there to trigger what > I can already reproduce here on a VM? that sounds very plausible. By adding a `sleep 1` at the end of "tests/shell/testcases/sets/0062set_connlimit_0", the dump "tests/shell/testcases/sets/dumps/0062set_connlimit_0.nft" also no longer matches. Thomas
Hi, I am splitting a few tests and extending features to allow to run this in 5.4 (and older kernels). Let me submit these then let me ask you to rebase this series on top. Thanks.