| Message ID | 1387050197-3183-1-git-send-email-luca@coelho.fi |
|---|---|
| State | Superseded |
| Headers | show |
All of these test cases fail with older versions of mac80211/cfg80211/hwsim (e.g., with the version I'm running on my build server). As such, I cannot apply this yet. Would there be any way for hostapd to detect whether CSA functionality is properly supported (I sure hope so..) and then make these test case report SKIP rather than FAIL?
Hi Jouni, Sorry for the late reply. I've been on vacations and just got back to action now. ;) On Sat, 2013-12-14 at 21:55 -0800, Jouni Malinen wrote: > All of these test cases fail with older versions of > mac80211/cfg80211/hwsim (e.g., with the version I'm running on my build > server). As such, I cannot apply this yet. I see. So we should keep all the tests backward-compatible? Seems like a lot of extra work, but makes sense since the kernel version you're using to run the tests is independent of the hostap version... > Would there be any way for hostapd to detect whether CSA functionality > is properly supported (I sure hope so..) and then make these test case > report SKIP rather than FAIL? It can be done, but AFAICT there's currently no way to get this information via the control interface. To do this, I we have to export the capa.flags information via the control API. -- Luca.
On Fri, Jan 10, 2014 at 04:26:33PM +0200, Luca Coelho wrote: > On Sat, 2013-12-14 at 21:55 -0800, Jouni Malinen wrote: > > All of these test cases fail with older versions of > > mac80211/cfg80211/hwsim (e.g., with the version I'm running on my build > > server). As such, I cannot apply this yet. > > I see. So we should keep all the tests backward-compatible? Seems like > a lot of extra work, but makes sense since the kernel version you're > using to run the tests is independent of the hostap version... Depends on what you mean with backward-compatible.. There is no need to be backwards compatible with older wpa_supplicant/hostapd versions, but there is need for the full setup to work with number of different kernel versions. This does not need to include everything, but in general, it would be nice to be able to at minimum report SKIP instead of FAIL if the kernel does not support something. In general, I think something went wrong with the kernel interface design or hostapd/wpa_supplicant implementation if that cannot be done easily since hostapd/wpa_supplicant themselves will obviously need to work with older kernel versions and it is useful helpful to know what functionality is available. At minimum, there will be a hard blocker on the new test cases not failing on my test server since I try not to commit anything that breaks those tests. I may be convinced to update the Backports version there when proposing a new test case, but in general, there better be a good reason for this and why the test case could not be made to report SKIP instead. > > Would there be any way for hostapd to detect whether CSA functionality > > is properly supported (I sure hope so..) and then make these test case > > report SKIP rather than FAIL? > > It can be done, but AFAICT there's currently no way to get this > information via the control interface. To do this, I we have to export > the capa.flags information via the control API. That has already been done.. However, the current wireless-testing.git snapshot is still disabling CSA and if I re-enable it in cfg80211 and use the latest mac80211_hwsim contribution to claim support for it, some of these test cases fail due to kernel issues: START ap_csa_1_switch 1/6 PASS ap_csa_1_switch 4.358744 2014-01-14 15:02:09.299603 START ap_csa_1_switch_count_0 2/6 FAIL ap_csa_1_switch_count_0 3.481473 2014-01-14 15:02:12.795904 START ap_csa_1_switch_count_1 3/6 PASS ap_csa_1_switch_count_1 3.312774 2014-01-14 15:02:16.222982 START ap_csa_2_switches 4/6 PASS ap_csa_2_switches 5.610565 2014-01-14 15:02:21.848812 START ap_csa_2_switches_count_0 5/6 FAIL ap_csa_2_switches_count_0 3.788668 2014-01-14 15:02:25.652828 START ap_csa_2_switches_count_1 6/6 PASS ap_csa_2_switches_count_1 3.759073 2014-01-14 15:02:29.531521 failed tests: ap_csa_1_switch_count_0 ap_csa_2_switches_count_0 With both failures due to this one: WARNING: CPU: 0 PID: 0 at net/mac80211/tx.c:2443 ieee80211_update_csa+0x221/0x2a0() There needs to be a reasonable expectation of being able to run these tests with wireless-testing.git successfully before it is justifiable to add the test cases into hostap.git.
On Tue, 2014-01-14 at 17:05 +0200, Jouni Malinen wrote: > On Fri, Jan 10, 2014 at 04:26:33PM +0200, Luca Coelho wrote: > > On Sat, 2013-12-14 at 21:55 -0800, Jouni Malinen wrote: > > > All of these test cases fail with older versions of > > > mac80211/cfg80211/hwsim (e.g., with the version I'm running on my build > > > server). As such, I cannot apply this yet. > > > > I see. So we should keep all the tests backward-compatible? Seems like > > a lot of extra work, but makes sense since the kernel version you're > > using to run the tests is independent of the hostap version... > > Depends on what you mean with backward-compatible.. There is no need to > be backwards compatible with older wpa_supplicant/hostapd versions, but > there is need for the full setup to work with number of different kernel > versions. That's what I meant. Backward-compatible with old kernels. > This does not need to include everything, but in general, it > would be nice to be able to at minimum report SKIP instead of FAIL if > the kernel does not support something. In general, I think something > went wrong with the kernel interface design or hostapd/wpa_supplicant > implementation if that cannot be done easily since > hostapd/wpa_supplicant themselves will obviously need to work with older > kernel versions and it is useful helpful to know what functionality is > available. I totally agree. :) > At minimum, there will be a hard blocker on the new test cases not > failing on my test server since I try not to commit anything that breaks > those tests. I may be convinced to update the Backports version there > when proposing a new test case, but in general, there better be a good > reason for this and why the test case could not be made to report SKIP > instead. Fair enough. I tend to forget that these tests are mostly for hostap and not the for the kernel itself. They just do such a good job in testing the kernel that I get carried away. :P > > > Would there be any way for hostapd to detect whether CSA functionality > > > is properly supported (I sure hope so..) and then make these test case > > > report SKIP rather than FAIL? > > > > It can be done, but AFAICT there's currently no way to get this > > information via the control interface. To do this, I we have to export > > the capa.flags information via the control API. > > That has already been done.. However, the current wireless-testing.git > snapshot is still disabling CSA and if I re-enable it in cfg80211 and > use the latest mac80211_hwsim contribution to claim support for it, some > of these test cases fail due to kernel issues: > > START ap_csa_1_switch 1/6 > PASS ap_csa_1_switch 4.358744 2014-01-14 15:02:09.299603 > START ap_csa_1_switch_count_0 2/6 > FAIL ap_csa_1_switch_count_0 3.481473 2014-01-14 15:02:12.795904 > START ap_csa_1_switch_count_1 3/6 > PASS ap_csa_1_switch_count_1 3.312774 2014-01-14 15:02:16.222982 > START ap_csa_2_switches 4/6 > PASS ap_csa_2_switches 5.610565 2014-01-14 15:02:21.848812 > START ap_csa_2_switches_count_0 5/6 > FAIL ap_csa_2_switches_count_0 3.788668 2014-01-14 15:02:25.652828 > START ap_csa_2_switches_count_1 6/6 > PASS ap_csa_2_switches_count_1 3.759073 2014-01-14 15:02:29.531521 > failed tests: ap_csa_1_switch_count_0 ap_csa_2_switches_count_0 > > With both failures due to this one: > WARNING: CPU: 0 PID: 0 at net/mac80211/tx.c:2443 ieee80211_update_csa+0x221/0x2a0() Right there is still some stuff missing in wireless-testing. And, in any case, even when all the latest submitted patches get into wireless-testing, the tests with count 0 and count 1 will still fail because the CSA action frame implementation for BSS is still missing. > There needs to be a reasonable expectation of being able to run these > tests with wireless-testing.git successfully before it is justifiable to > add the test cases into hostap.git. Right, I understand your point. I was looking from a different perspective, a bit more like test-driven development. As I see, all the tests *are* valid and hostap's implementation supports them all. So the tests could be added to hostap already. If we did this, we would have a clearer visibility of the status of CSA in the kernel. The fact that the current kernel still fails some tests is a bug in the kernel; not in testcases nor in hostap. Anyway, I don't want to change your existing process. And there's no hurry to get these tests in the mainline, so we can wait. :) -- Cheers, Luca.
diff --git a/tests/hwsim/test_ap_csa.py b/tests/hwsim/test_ap_csa.py new file mode 100644 index 0000000..9b2feb0 --- /dev/null +++ b/tests/hwsim/test_ap_csa.py @@ -0,0 +1,85 @@ +#!/usr/bin/python +# +# AP CSA tests +# Copyright (c) 2013, Luciano Coelho <luciano.coelho@intel.com> +# +# This software may be distributed under the terms of the BSD license. +# See README for more details. + +import time +import subprocess +import logging +logger = logging.getLogger() + +import hwsim_utils +import hostapd + +def connect(dev, apdev): + params = { "ssid": "ap-csa", + "channel": "1" } + ap = hostapd.add_ap(apdev[0]['ifname'], params) + dev.connect("ap-csa", key_mgmt="NONE") + return ap + +def switch_channel(ap, count, freq): + ap.request("CHAN_SWITCH " + str(count) + " " + str(freq)) + ev = ap.wait_event(["AP-CSA-FINISHED"], timeout=10) + if ev is None: + raise Exception("CSA finished event timed out") + if "freq=" + str(freq) not in ev: + raise Exception("Unexpected channel in CSA finished event") + time.sleep(0.1) + +def test_ap_csa_1_switch(dev, apdev): + """AP Channel Switch, one switch""" + ap = connect(dev[0], apdev) + + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 10, 2462) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + +def test_ap_csa_2_switches(dev, apdev): + """AP Channel Switch, two switches""" + ap = connect(dev[0], apdev) + + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 10, 2462) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 10, 2412) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + +def test_ap_csa_1_switch_count_0(dev, apdev): + """AP Channel Switch, one switch with count 0""" + ap = connect(dev[0], apdev) + + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 0, 2462) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + +def test_ap_csa_2_switches_count_0(dev, apdev): + """AP Channel Switch, two switches with count 0""" + ap = connect(dev[0], apdev) + + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 0, 2462) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 0, 2412) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + +def test_ap_csa_1_switch_count_1(dev, apdev): + """AP Channel Switch, one switch with count 1""" + ap = connect(dev[0], apdev) + + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 1, 2462) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + +def test_ap_csa_2_switches_count_1(dev, apdev): + """AP Channel Switch, two switches with count 1""" + ap = connect(dev[0], apdev) + + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 1, 2462) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname']) + switch_channel(ap, 1, 2412) + hwsim_utils.test_connectivity(dev[0].ifname, apdev[0]['ifname'])
From: Luciano Coelho <luciano.coelho@intel.com> Signed-hostap: Luciano Coelho <luciano.coelho@intel.com> --- In v2: * fixed copy and paste bug in the description of the testcases with count 1 (thanks Ilan). tests/hwsim/test_ap_csa.py | 85 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 tests/hwsim/test_ap_csa.py