[v3,00/17] Support seamless PTK rekeys with Extended Key ID

Extended Key ID for Individually Addressed Frames has been added to the
IEEE 802.11 - 2012 standard to allow STAs to rekey the pairwise key
without an interruption for ongoing data traffic but so far there are no
implementations using it.

A good and easy to follow introduction to Extended Key ID can be found
here:
https://mentor.ieee.org/802.11/dcn/10/11-10-0313-01-000m-rekeying-protocol-fix.ppt
In a nutshell Extended Key ID is just allowing to also use keyid 1 for
pairwise keys: Therefore we can have two pairwise keys in paralell,
allowing to switch between them comparable how it's already done for GTK
and IGTK keys.

All implementations of IEEE 802.11 without Extended Key ID only use the
keyid 0 for unicast MPDUs and since each STA can only have a single
unicast key active at any time without a window able to handle both old
and the new key at the same time. When rekeying the unicast key under
load MPDU losses are unavoidable: Each STA will get MPDUs encrypted with
a key either no longer installed or not installed, yet.
In both cases the remote STA will be unable to decrypt the MPDUs and
drop them. (Since only the decryption fails and the frames are still
acknowledged these are truly lost and it's up to e.g. tcp to retransmit
them.)

Especially on high speed connections this can easily affect a few dozen
MPDUs, causing tcp to drastically throttle the speed or causing dropouts
in voice or video transmissions even for "correct" implementations of the
standard. (Most implementations are not correct and make matter much
worse, but that's not really relevant for this patch series.)

This patch series aims to implement Extended Key ID for both hostapd and
wpa_supplicant and finally allow us to rekey unicast keys with virtually
no MPDU loss and no (noticeable) performance impact at all.
While there are still ways to extend or improve this patch set it's
already quite complex and I believe this to be good enough to be merged
as it is. Or in other words: It's high time I get some feedback on what
I'm cooking here:-)

Since Extended Key ID breaks the long-standing agreement that there can
be only one active pairwise key using keyid 0 and also has to delay the
Tx usage of a key from the key install this must be supported by the
driver. So far Extended Key ID is only supported for linux kernels >=5.2
and cards only supporting SW crypto. This patch series has been developed
using hwsim (works out of the box with a kernel >= 5.2.0) and two
iwlwifi cards. (All cards prior below the current 22000 should work with a
trivial patch as of today.)
Some early versions were also tested with ath9, but the required
mac80211 to support that card were not merged upstream and I abandoned
that approach for now.

The patch series consist of basically five parts:
 1) The first 8 patches are addressing issues not directly tied to the
 Extended Key ID support. More details are directly in the individual patch
 descriptions, but they are primarily related to deprecating the set_tx
 boolean used in various set_key() functions and replace it with
 something more flexible and able to handle also Extended Key ID key
 installs:
  01-nl80211: Migrate to current netlink key message format
  02-Driver: Introduce key_types and Extended Key ID driver flag
  03-Add new argument key_type to all set_key() functions
  04-hostapd: Set the correct key_type for key installs
  05-wpa_supplicant: Set the correct key_type for key installs
  06-drivers: Migrate drivers from set_tx to key_type API
  07-nl80211: Switch to the new key_type API & cleanup
  08-nl80211,wpa_supplicant: Drop outdated tdls hack

 2) The patches 9-11 are finally the Extended Key ID support for
 hostapd/wpa_supplicant and fix/amend to the existing tests to be
 able to work properly with and without Extended Key ID:
  09-hostapd: Add support for Extended Key ID
  10-wpa_supplicant: Extended Key ID support for AP connections
  11-tests: Extended Key ID tests

 3) Patches 12 and 13 are trying to close a gap not covered in the
 official standard. But I'm far from sure that this is the correct way
 and hope we can have a discussion here how to best handle FILS and
 probably also FT. (The standard is not having any guidance here...)
  12-hostapd: FILS Extended Key ID support
  13-wpa_supplicant: FILS Extended Key ID support

 4) The patches 14+15 are finally adding Extended Key ID support to
 nl80211, basically activating all the previous Extended Key ID patches.
 But only patch 14 is really to be consider for merge: Patch 15 is just
 a hack to work around overly restrictive key install check in the linux
 kernels currently supporting Extended Key ID. I'm currently trying to
 fix that upstream and hope we can simply drop the patch.
  14-nl80211: Extended Key ID support
  15-nl80211: Hack for keyidx=1 installs

 5) The last two patches, (16+17) are optional. Patch 16 is finishing
 the work the first 8 patches started. But since redesigning the set_tx
 API is the most invasive part of the patch series it is sometimes
 interesting to see how the old set_tx key calls would have performed.
 Patch 17 is so far just a proof that everything still works when we
 use keyid 1 at the initial connect instead of the more traditional
 keyid 0. I have some plans to extend that, allowing hostapd to detect
 broken STAs claiming to be compatible with Extended Key ID due to a
 broken capability handling and automatically stop using it. (At least
 the Samsung Galaxy Tab S3 is setting the RSN capabilities wrong and
 chances are there are others...)
  16-Drop set_tx from all set_key() functions
  17-hostapd: Extended Key ID stress test

Besides the test cases I'm also using it on real hardware with iwlwifi
with rekeys all 30s.

To use Extended Key ID or see how the tests in hostapd are interacting
with it you need at least a 5.2 linux kernel. Using a kernel without
Extended Key ID support is useful to check that the changes are backward
compatible. (I guess I could extend the tests to simulate a run without
Extended Key ID, but then that would basically double the runtime... 
So for now there are just some generic Extended Key ID tests checking
rekey is working with and without it and all other tests are just
switching over to Extended Key ID when they do not actively opt out.)

To make it a bit simpler to see what the patches are changing I've
uploaded you a tgz containing three full test runs:

 1) head-2019-08-17
    Tests run with a unpatched hostapd (542913943)

 2) patched-2019-08-17
    Test run with hwsim supporting Extended Key ID and all patches of
    the series applied.

 3) patched-legacy-2019-08-17
    Also all patches applied, but Extended Key ID detection sabotaged,
    so it's using the classical unicast rekey procedure.

You can download the roughly 560MB big file here:
https://www.awhome.eu/index.php/s/8HQXpY4qf4CmHGs

All tests were executed running wt-2019-08-06 from wireless-testing.
(The kernel has three additional patched on top of the official version
to use Extended Key ID with iwlwifi, my proposed patch to fix the
Extended Key ID install checks in cfg80211 and a debug patch with
printk's informing me when installing or activating any PTK.

Most of the failed tests are related to a wrong Python call to AES
encrypt data, missing one argument. (I'm using Python 3.6.9)

I've added two extra files in each test directory:
 - failed.log
   All tests failed in this run. With a comment when it worked when
   trying the test a second time. (But without adding the output of the
   rerun.)
 - script.log
   The output of the actual test run.

Compared to version 2 of the patch series the following changes have
been made:
 - use key_type (enum) instead of key_flag (bit array)
 - migrate set_tx to the new key_type API and clean up related
   workarounds and bugs.
 - also support Extended Key ID for OSEN, FT and FILS
 - wpa_supplicant is now checking RSN capabilities instead of the
   presence of the KeyID KDE to enable Extended Key ID support to follow
   the wording of the standard
 - detect and prevent dropping or adding Extended Key ID support on
   rekey. (Initial connection decides if we can use it.)
 - Don't enable Extended Key ID for TKIP. (Not allowed in the standard.)
 - also flip KeyIDs for WPA_REAUTH and WPA_REAUTH_EAPOL
 - fixed many "test-only" bugs (both tests and test code)
 - split fixes and extensions in different patches
 - dropped "GET drv_flags" and handle it differently
 - better description of the patches

Alexander Wetzel (17):
  nl80211: Migrate to current netlink key message format
  Driver: Introduce key_types and Extended Key ID driver flag
  Add new argument key_type to all set_key() functions
  hostapd: Set the correct key_type for key installs
  wpa_supplicant: Set the correct key_type for key installs
  drivers: Migrate drivers from set_tx to key_type API
  nl80211: Switch to the new key_type API & cleanup
  nl80211,wpa_supplicant: Drop outdated tdls hack
  hostapd: Add support for Extended Key ID
  wpa_supplicant: Extended Key ID support for AP connections
  tests: Extended Key ID tests
  hostapd: FILS Extended Key ID support
  wpa_supplicant: FILS Extended Key ID support
  nl80211: Extended Key ID support
  nl80211: Hack for keyidx=1 installs
  Drop set_tx from all set_key() functions
  hostapd: Extended Key ID stress test

Alexander Wetzel (17):
  nl80211: Migrate to current netlink key message format
  Driver: Introduce key_types and Extended Key ID driver flag
  Add new argument key_type to all set_key() functions
  hostapd: Set the correct key_type for key installs
  wpa_supplicant: Set the correct key_type for key installs
  drivers: Migrate drivers from set_tx to key_type API
  nl80211: Switch to the new key_type API & cleanup
  nl80211,wpa_supplicant: Drop outdated tdls hack
  hostapd: Add support for Extended Key ID
  wpa_supplicant: Extended Key ID support for AP connections
  tests: Extended Key ID tests
  hostapd: FILS Extended Key ID support
  wpa_supplicant: FILS Extended Key ID support
  nl80211: Extended Key ID support
  nl80211: Hack for keyidx=1 installs
  Drop set_tx from all set_key() functions
  hostapd: Extended Key ID stress test

 hostapd/config_file.c              |   2 +
 hostapd/ctrl_iface.c               |  67 ++++++----
 hostapd/hostapd.conf               |  10 ++
 src/ap/ap_config.c                 |   1 +
 src/ap/ap_config.h                 |   1 +
 src/ap/ap_drv_ops.c                |   9 +-
 src/ap/ap_drv_ops.h                |   5 +-
 src/ap/hostapd.c                   |  17 ++-
 src/ap/hs20.c                      |   2 +
 src/ap/ieee802_11.c                |   6 +-
 src/ap/ieee802_1x.c                |  14 +-
 src/ap/wpa_auth.c                  |  99 +++++++++++---
 src/ap/wpa_auth.h                  |   4 +-
 src/ap/wpa_auth_ft.c               |  13 +-
 src/ap/wpa_auth_glue.c             |  38 +++++-
 src/ap/wpa_auth_i.h                |   3 +
 src/ap/wpa_auth_ie.c               |  43 +++++-
 src/common/wpa_common.c            |   1 +
 src/common/wpa_common.h            |   9 ++
 src/drivers/driver.h               |  33 +++--
 src/drivers/driver_atheros.c       |  10 +-
 src/drivers/driver_bsd.c           |  12 +-
 src/drivers/driver_hostap.c        |   9 +-
 src/drivers/driver_ndis.c          |  23 ++--
 src/drivers/driver_nl80211.c       | 204 ++++++++++++++++++-----------
 src/drivers/driver_nl80211_capa.c  |   4 +
 src/drivers/driver_openbsd.c       |   4 +-
 src/drivers/driver_privsep.c       |  12 +-
 src/drivers/driver_wext.c          |  27 ++--
 src/drivers/driver_wext.h          |   4 +-
 src/rsn_supp/tdls.c                |   9 +-
 src/rsn_supp/wpa.c                 | 156 +++++++++++++++++++---
 src/rsn_supp/wpa.h                 |   9 +-
 src/rsn_supp/wpa_ft.c              |  17 ++-
 src/rsn_supp/wpa_i.h               |  12 +-
 src/rsn_supp/wpa_ie.c              |  11 ++
 src/rsn_supp/wpa_ie.h              |   1 +
 tests/hwsim/hostapd.py             |  19 ++-
 tests/hwsim/test_ap_ciphers.py     |   2 +-
 tests/hwsim/test_ap_eap.py         |  20 ++-
 tests/hwsim/test_ap_psk.py         |  76 ++++++++++-
 tests/hwsim/test_ocv.py            |  13 +-
 tests/hwsim/test_rrm.py            |   7 +-
 tests/hwsim/wpasupplicant.py       |   7 +-
 wlantest/bss.c                     |   4 +-
 wlantest/rx_eapol.c                |   4 +
 wpa_supplicant/config.c            |   2 +
 wpa_supplicant/config_file.c       |   1 +
 wpa_supplicant/config_ssid.h       |  10 ++
 wpa_supplicant/ctrl_iface.c        |  34 +++--
 wpa_supplicant/driver_i.h          |  22 +++-
 wpa_supplicant/ibss_rsn.c          |  19 +--
 wpa_supplicant/mesh_mpm.c          |  12 +-
 wpa_supplicant/mesh_rsn.c          |  14 +-
 wpa_supplicant/preauth_test.c      |   2 +-
 wpa_supplicant/wpa_cli.c           |   3 +-
 wpa_supplicant/wpa_supplicant.c    |  40 ++++--
 wpa_supplicant/wpa_supplicant.conf |   5 +
 wpa_supplicant/wpas_glue.c         |  24 ++--
 59 files changed, 915 insertions(+), 326 deletions(-)

Am 17.08.19 um 23:14 schrieb Alexander Wetzel:
> Extended Key ID for Individually Addressed Frames has been added to the
> IEEE 802.11 - 2012 standard to allow STAs to rekey the pairwise key
> without an interruption for ongoing data traffic but so far there are no
> implementations using it.

I somehow missed some "int key_type" references in the series which 
should of course be "enum key_type key_type" instead. Also the comment 
in "nl80211: Hack for keyidx=1 installs" was wrong and I still had some 
pointless newlines in the patch series.

I've fixed all that now locally but if it's ok for you I'll wait some 
days for more issues or feedback prior to sending out v4 of the series.


Alexander