| Message ID | 20170919003904.5124-1-tom@quantonium.net |
|---|---|
| Headers | show |
| Series | gtp: Additional feature support | expand |
Hi Tom, first of all, thanks a lot for your patch series. It makes me happy to see contributions on the GTP code :) On Mon, Sep 18, 2017 at 05:38:50PM -0700, Tom Herbert wrote: > - IPv6 support see my detailed comments in other mails. It's unfortunately only support for the already "deprecated" IPv6-only PDP contexts, not the more modern v4v6 type. In order to interoperate with old and new approach, all three cases (v4, v6 and v4v6) should be supported from one code base. > - Configurable networking interfaces so that GTP kernel can be used > and tested without needing GSN network emulation (i.e. no user space > daemon needed). We have some pretty decent userspace utilities for configuring the GTP interfaces and tunnels in the libgtpnl repository, but if it helps people to have another way of configuration, I won't be against it. What we have to keep in mind is that the current model of 1:1 mapping of a "UDP socket' to a GTP netdevice is conceptually broken and needs to be refactored soon (without breaking backwards compatibility). See related earlier discussions with patches submitted by Andreas Schultz. Summary: In real-world GGSNs you often want to host multiple virtual GGSNs on a single GGSN (= UDP socket). Each virtual GGSN terminates into one external PDN (packet data network), which can be a private corporate vpn or any other IP network, with no routing between those networks. Naively one would assume you "simply" run another virtual GGSN instance on another IP address, and then differentiate like that. However, the problem is that adding a new GGSN IP address will require manual configuration changes at each of your roaming partners (easily hundreds of operators!) and hence it is avoided at all cost due to the related long schedule, requirement for interop testing with each of them, etc. So what you do in reality at operators is that you operate many of those virtual GGSNs on the same IP:Port combination (and hence UDP socket), which means you have PDP contexts for vGGSN A which terminate on e.g. gtp0 and PDP contexts for vGGSN B on gtp1, and so on. The decision which gtp-device a given PDP context is a member is made by the GTP-C instance. In the kenel we'll have to decouple net-devices from sockets. So whatever new configuration mechanism or architectural changes we introduce, we need to make sure that those will accomodate the "new model" rather than introducing further dependencies for which we will have to maintain backwards compatibility workaronds later on. > - Port numbers are configurable I'm not sure if this is a useful feature. GTP is used only in operator-controlled networks and only on standard ports. It's not possible to negotiate any non-standard ports on the signaling plane either. > - Addition of a dst_cache in the GTP structure and other cleanup looks fine to me. > - GSO,GRO > - Control of zero UDP checksums [...] > Additionally, this patch set also includes a couple of general support > capabilities: > > - A facility that allows application specific GSO callbacks > - Common functions to get a route fo for an IP tunnel This is where the "core netdev" folks will have to comment. I'm too remote from mainline kernel development these days and will focus on reviewing the GTP specific bits of your patch series. > For IPv6 support, the mobile subscriber needs to allow IPv6 addresses, > and the remote enpoint can be IPv6. Minor correction: The mobile subscriber specifically requests a PDP Type when establishing the PDP context via Session Management related signaling from MS/UE to SGSN. The SGSN simply translates this to GTP and then forwards it to the GGSN. So it's acutally not "allow" but "specifically request". > Configured the matrix of IPv4/IPv6 mobile subscriber, IPv4/IPv6 remote > peer, and GTP version 0 and 1 (eight combinations). Observed > connectivity and proper GSO/GRO. Also, tested VXLAN for > regression. I presume those tests were done with manually configured GTP-devices and PDP contexts to the (patched) kernel GTP module? If so, I would like to strongly suggest interop testing with a different implementation, such as real phones on the MS/UE side and e.g. OsmoSGSN. That would, however, of course mean that the netlink related bits would have to be added to libgtpnl and OsmoGGSN (or ergw) so that you have a daemon for the control plane. For IPv6 (and v4v6) PDP contexts there is quite a bit of extra headache related to the way how router solicitation/advertisements are modified in the 3GPP world. The address allocation in v4 is simple: * MS/UE requests dynamic or fixed IPv4 address via EUA IE of PDP context activation * GGSN responds with IPv4 address in EUA of Activate PDP context response (and then uses netlink to tell the kernel about that IPv4 address) In v6 or the v6 portion of v4v6 it works differently: * MS/UE requests dynamic or fixed IPv4 address in EUA IE of PDP context activation * GGSN responds with an IPv6 address, but that address is *not* used for communication, but simply used as an "interface identifier" to build a link-local address. * MS then uses router solicitation using that link-local address * GGSN responds with router advertisement, allocating a single /64 prefix, from which the MS then generates a fully-qualified IPv6 source address for communication. How did you envision this to be done with the v6 support you just added? At the very least, the /64 prefix matching would have to be implemented so that in fact all addresses within that /64 prefix are matched + encapsulated for a given PDP context in the downlink (to phone) direction. Also, I think the responsibility for the router advertisements would be in the kernel, too. Otherwise, a GTP-C userspace implementation would have to inject packets into the user plane (which is otherwise handled completely inside the kernel). Injecting packets would mean that in caes GTP sequence numbers are used, that userspace implementation would have to alter the sequence numbers of the kernel gtp.ko code using netlink, but therre would be race conditions, ... The router advertisements and neighbor advertisements basically have the semantics of one link per PDP context. Each of them is a point-to-point link, and it's not one router advertisement that's sent to all of the PDP contexts on that gtp-device. I know it all sucks. I'm still happy to see somebody tackling v6 support in gtp.c :) Regards, Harald
On Tue, Sep 19, 2017 at 5:43 AM, Harald Welte <laforge@gnumonks.org> wrote: > Hi Tom, > > first of all, thanks a lot for your patch series. It makes me happy to > see contributions on the GTP code :) > > On Mon, Sep 18, 2017 at 05:38:50PM -0700, Tom Herbert wrote: >> - IPv6 support > > see my detailed comments in other mails. It's unfortunately only > support for the already "deprecated" IPv6-only PDP contexts, not the > more modern v4v6 type. In order to interoperate with old and new > approach, all three cases (v4, v6 and v4v6) should be supported from one > code base. > It sounds like something that can be subsequently added. Do you have a reference to the spec? >> - Configurable networking interfaces so that GTP kernel can be used >> and tested without needing GSN network emulation (i.e. no user space >> daemon needed). > > We have some pretty decent userspace utilities for configuring the GTP > interfaces and tunnels in the libgtpnl repository, but if it helps > people to have another way of configuration, I won't be against it. > AFAIK those userspace utilities don't support IPv6. Being able to configure GTP like any other encapsulation will facilitate development of IPv6 and other features. > What we have to keep in mind is that the current model of 1:1 mapping of > a "UDP socket' to a GTP netdevice is conceptually broken and needs to be > refactored soon (without breaking backwards compatibility). See related > earlier discussions with patches submitted by Andreas Schultz. > I don't think I changed the model, so this can evolve. > Summary: > > In real-world GGSNs you often want to host multiple virtual GGSNs on a > single GGSN (= UDP socket). Each virtual GGSN terminates into one > external PDN (packet data network), which can be a private corporate vpn > or any other IP network, with no routing between those networks. > Sounds like network virtualization and VNIs. > Naively one would assume you "simply" run another virtual GGSN > instance on another IP address, and then differentiate like that. > > However, the problem is that adding a new GGSN IP address will require > manual configuration changes at each of your roaming partners (easily > hundreds of operators!) and hence it is avoided at all cost due to the > related long schedule, requirement for interop testing with each of them, > etc. > > So what you do in reality at operators is that you operate many of those > virtual GGSNs on the same IP:Port combination (and hence UDP socket), > which means you have PDP contexts for vGGSN A which terminate on e.g. > gtp0 and PDP contexts for vGGSN B on gtp1, and so on. The decision > which gtp-device a given PDP context is a member is made by the GTP-C > instance. In the kenel we'll have to decouple net-devices from sockets. > > So whatever new configuration mechanism or architectural changes we > introduce, we need to make sure that those will accomodate the "new > model" rather than introducing further dependencies for which we will > have to maintain backwards compatibility workaronds later on. > >> - Port numbers are configurable > > I'm not sure if this is a useful feature. GTP is used only in > operator-controlled networks and only on standard ports. It's not > possible to negotiate any non-standard ports on the signaling plane > either. > Bear in mind that we're not required to do everything the GTP spec says. Adding port configuration is another one of those things that gives us flexibility and and better capability to test without needing a full blown GSN network. One feature I didn't implement was UDP source for flow entropy-- as we've seen with other encapsulation protocols this helps significantly to get good ECMP in the network. My impression is GTP designers probably didn't think in terms of getting best performance. But we can ;-) >> - Addition of a dst_cache in the GTP structure and other cleanup > > looks fine to me. > >> - GSO,GRO >> - Control of zero UDP checksums > > [...] > >> Additionally, this patch set also includes a couple of general support >> capabilities: >> >> - A facility that allows application specific GSO callbacks >> - Common functions to get a route fo for an IP tunnel > > This is where the "core netdev" folks will have to comment. I'm too > remote from mainline kernel development these days and will focus on > reviewing the GTP specific bits of your patch series. > Thanks. Obviously, I and many on this list have more expertise on the core networking side than GTP, so your review is quite welcome. >> For IPv6 support, the mobile subscriber needs to allow IPv6 addresses, >> and the remote enpoint can be IPv6. > > Minor correction: The mobile subscriber specifically requests a PDP Type > when establishing the PDP context via Session Management related > signaling from MS/UE to SGSN. The SGSN simply translates this to GTP > and then forwards it to the GGSN. So it's acutally not "allow" but > "specifically request". > Okay. >> Configured the matrix of IPv4/IPv6 mobile subscriber, IPv4/IPv6 remote >> peer, and GTP version 0 and 1 (eight combinations). Observed >> connectivity and proper GSO/GRO. Also, tested VXLAN for >> regression. > > I presume those tests were done with manually configured GTP-devices and > PDP contexts to the (patched) kernel GTP module? If so, I would like to > strongly suggest interop testing with a different implementation, such > as real phones on the MS/UE side and e.g. OsmoSGSN. That would, > however, of course mean that the netlink related bits would have to be > added to libgtpnl and OsmoGGSN (or ergw) so that you have a daemon for > the control plane. > I also brought up open_ggsn. ggsn to sgsn. > For IPv6 (and v4v6) PDP contexts there is quite a bit of extra headache > related to the way how router solicitation/advertisements are modified > in the 3GPP world. > > The address allocation in v4 is simple: > * MS/UE requests dynamic or fixed IPv4 address via EUA IE of PDP context > activation > * GGSN responds with IPv4 address in EUA of Activate PDP context > response (and then uses netlink to tell the kernel about that > IPv4 address) > > In v6 or the v6 portion of v4v6 it works differently: > * MS/UE requests dynamic or fixed IPv4 address in EUA IE of PDP context > activation > * GGSN responds with an IPv6 address, but that address is *not* used > for communication, but simply used as an "interface identifier" to > build a link-local address. > * MS then uses router solicitation using that link-local address > * GGSN responds with router advertisement, allocating a single /64 > prefix, from which the MS then generates a fully-qualified IPv6 > source address for communication. > > How did you envision this to be done with the v6 support you just added? > At the very least, the /64 prefix matching would have to be implemented > so that in fact all addresses within that /64 prefix are matched + > encapsulated for a given PDP context in the downlink (to phone) > direction. > > Also, I think the responsibility for the router advertisements would be > in the kernel, too. Otherwise, a GTP-C userspace implementation would > have to inject packets into the user plane (which is otherwise handled > completely inside the kernel). Injecting packets would mean that in caes > GTP sequence numbers are used, that userspace implementation would have > to alter the sequence numbers of the kernel gtp.ko code using netlink, > but therre would be race conditions, ... > > The router advertisements and neighbor advertisements basically have the > semantics of one link per PDP context. Each of them is a point-to-point > link, and it's not one router advertisement that's sent to all of the > PDP contexts on that gtp-device. > > I know it all sucks. I'm still happy to see somebody tackling v6 > support in gtp.c :) > I would hope all the above you're describing is mostly control plane matters. At least a good design decouples data palne and control plane. I know that GTP is a bit convoluted in this regard. Tom
Hi Tom, On Tue, Sep 19, 2017 at 08:59:28AM -0700, Tom Herbert wrote: > On Tue, Sep 19, 2017 at 5:43 AM, Harald Welte <laforge@gnumonks.org> > wrote: > > On Mon, Sep 18, 2017 at 05:38:50PM -0700, Tom Herbert wrote: > >> - IPv6 support > > > > see my detailed comments in other mails. It's unfortunately only > > support for the already "deprecated" IPv6-only PDP contexts, not the > > more modern v4v6 type. In order to interoperate with old and new > > approach, all three cases (v4, v6 and v4v6) should be supported from > > one code base. > > > It sounds like something that can be subsequently added. Not entirely, at least on the netlink (and any other configuration interface) you will have to reflect this from the very beginning. You have to have an explicit PDP type and cannot rely on the address type to specify the type of PDP context. Whatever interfaces are introduced now will have to remain compatible to any future change. My strategy to avoid any such possible 'road blocks' from being introduced would be to simply add v4v6 and v6 support in one go. The differences are marginal (having both an IPv6 prefix and a v4 address in parallel, rather than mutually exclusive only). > Do you have a reference to the spec? See http://osmocom.org/issues/2418#note-7 which lists Section 11.2.1.3.2 of 3GPP TS 29.061 in combination with RFC3314, RFC7066, RFC6459 and 3GPP TS 23.060 9.2.1 as well as a summary of my understanding of it some months ago. > >> - Configurable networking interfaces so that GTP kernel can be > >> used and tested without needing GSN network emulation (i.e. no > >> user space daemon needed). > > > > We have some pretty decent userspace utilities for configuring the > > GTP interfaces and tunnels in the libgtpnl repository, but if it > > helps people to have another way of configuration, I won't be > > against it. > > > AFAIK those userspace utilities don't support IPv6. Of course not [yet]. libgtpnl and the command line tools have been implemented specifically for the in-kernel GTP driver, and you have to make sure to add related support on both the kernel and the userspace side (libgtpnl). So there's little point in adding features on either side before the other side. There would be no way to test... > Being able to configure GTP like any other encapsulation will > facilitate development of IPv6 and other features. That may very well be the case, but adding "IPv6 support" to kernel GTP in a way that is not in line with the existing userspace libraries and control-plane implementations means that you're developing those features in an artificial environment that doesn't resemble real 3GPP interoperable networks out there. As indicated, I'm not against adding additional interfaces, but we have to make sure that we add IPv6 support (or any new feature support) to at least libgtpnl, and to make sure we test interoperability with existing 3GPP network equipment such as real IPv6 capable phones and SGSNs. > > I'm not sure if this is a useful feature. GTP is used only in > > operator-controlled networks and only on standard ports. It's not > > possible to negotiate any non-standard ports on the signaling plane > > either. > > > Bear in mind that we're not required to do everything the GTP spec > says. Yes, we are, at least as long as it affects interoperability with other implemetations out there. GTP uses well-known port numbers on *both* sides of the tunnel, and you cannot deviate from that. There's no point in having all kinds of feetures in the GTP user plane which are not interoperable with other implementations, and which are completely outside of the information model / architecture of GTP. In the real world, GTP-U is only used in combination with GTP-C. And in GTP-C you can only negotiate the IP address of both sides of GTP-U, and not the port number information. As a result, the port numbers are static on both sides. > My impression is GTP designers probably didn't think in terms of > getting best performance. But we can ;-) I think it's wasted efforts if it's about "random udp ports" as no standards-compliant implementation out there with which you will have to interoperate will be able to support it. GTP is used between home and roaming operator. If you want to introduce changes to how it works, you will have to have control over both sides of the implementation of both the GTP-C and the GTP-u plane, which is very unlikely and rather the exception in the hundreds of operators you interoperate with. Also keep in mind that there often are various "middleboxes" that will suddenly have to reflect your changes. That starts from packet filters at various locations in the operator networks and/or roaming hubs, down to GTP hubs and others. My opinion is: Non-standard GTP ports are not going to happen. > I also brought up open_ggsn. ggsn to sgsn. That's good to hear. For both v4 and v6 PDP contexts? Whcih phones did you use for testing? Particularly given how convolved the address allocation is (see below), I'm surprised it would work. > > For IPv6 (and v4v6) PDP contexts there is quite a bit of extra headache > > related to the way how router solicitation/advertisements are modified > > in the 3GPP world. > > > > The address allocation in v4 is simple: > > * MS/UE requests dynamic or fixed IPv4 address via EUA IE of PDP context > > activation > > * GGSN responds with IPv4 address in EUA of Activate PDP context > > response (and then uses netlink to tell the kernel about that > > IPv4 address) > > > > In v6 or the v6 portion of v4v6 it works differently: > > * MS/UE requests dynamic or fixed IPv4 address in EUA IE of PDP context > > activation > > * GGSN responds with an IPv6 address, but that address is *not* used > > for communication, but simply used as an "interface identifier" to > > build a link-local address. > > * MS then uses router solicitation using that link-local address > > * GGSN responds with router advertisement, allocating a single /64 > > prefix, from which the MS then generates a fully-qualified IPv6 > > source address for communication. > > > > How did you envision this to be done with the v6 support you just added? > > At the very least, the /64 prefix matching would have to be implemented > > so that in fact all addresses within that /64 prefix are matched + > > encapsulated for a given PDP context in the downlink (to phone) > > direction. > > > > [...] > I would hope all the above you're describing is mostly control plane > matters. It is not. The control plane is GTP-C and runs on different UDP ports (at least for GTPv1/v2). The user plane is GTP-U and is what's done in the kernel. And by its very nature, IPv6 router solicitations/advertisements (as well as neighbor solicitations/advertisements) are part of the user plane and thus handled in GTP-U. > At least a good design decouples data palne and control > plane. I know that GTP is a bit convoluted in this regard. The problem is that IPv6 has never been specified properly for point-to-point links. There's no decent PPP specs for IPv6. So the 3GPP folks had to try to be as close as possible to the existing (broadcast) link layer model to facilitate existing IPv6 implemetations to work over 3GPP bearers. That's why they kept whatever possible to re-use in terms of neighbor/router discovery. So the problem is now: Unless you handle GTP-U *entirely* in the kernel (including router + neighbor advertisement/solicitation), you will have a "split GTP-U" plane between kernel and userspace. And in that context the question is who owns the sequence numbers, how will you avoid race conditions, ... - my simple suggestion is thus to keep with the current split and do everything GTP-U related inside the kernel and everything GTP-C related in userspace. I think there has to be a clear plan/architecture on how to implement those bits in terms of the kernel/userspace split, and at least a proof of concept implementation that we can show works with some real phones out there - otherwise there's no point in having IPv6 support that works well with some custom tools. Regards, Harald
On Tue, Sep 19, 2017 at 4:19 PM, Harald Welte <laforge@gnumonks.org> wrote: > Hi Tom, > > On Tue, Sep 19, 2017 at 08:59:28AM -0700, Tom Herbert wrote: >> On Tue, Sep 19, 2017 at 5:43 AM, Harald Welte <laforge@gnumonks.org> >> wrote: >> > On Mon, Sep 18, 2017 at 05:38:50PM -0700, Tom Herbert wrote: >> >> - IPv6 support >> > >> > see my detailed comments in other mails. It's unfortunately only >> > support for the already "deprecated" IPv6-only PDP contexts, not the >> > more modern v4v6 type. In order to interoperate with old and new >> > approach, all three cases (v4, v6 and v4v6) should be supported from >> > one code base. >> > >> It sounds like something that can be subsequently added. > > Not entirely, at least on the netlink (and any other configuration > interface) you will have to reflect this from the very beginning. You > have to have an explicit PDP type and cannot rely on the address type to > specify the type of PDP context. Whatever interfaces are introduced > now will have to remain compatible to any future change. > > My strategy to avoid any such possible 'road blocks' from being > introduced would be to simply add v4v6 and v6 support in one go. The > differences are marginal (having both an IPv6 prefix and a v4 address in > parallel, rather than mutually exclusive only). > >> Do you have a reference to the spec? > > See http://osmocom.org/issues/2418#note-7 which lists Section 11.2.1.3.2 > of 3GPP TS 29.061 in combination with RFC3314, RFC7066, RFC6459 and > 3GPP TS 23.060 9.2.1 as well as a summary of my understanding of it some > months ago. > >> >> - Configurable networking interfaces so that GTP kernel can be >> >> used and tested without needing GSN network emulation (i.e. no >> >> user space daemon needed). >> > >> > We have some pretty decent userspace utilities for configuring the >> > GTP interfaces and tunnels in the libgtpnl repository, but if it >> > helps people to have another way of configuration, I won't be >> > against it. >> > >> AFAIK those userspace utilities don't support IPv6. > > Of course not [yet]. libgtpnl and the command line tools have been > implemented specifically for the in-kernel GTP driver, and you have to > make sure to add related support on both the kernel and the userspace > side (libgtpnl). So there's little point in adding features on either > side before the other side. There would be no way to test... > >> Being able to configure GTP like any other encapsulation will >> facilitate development of IPv6 and other features. > > That may very well be the case, but adding "IPv6 support" to kernel GTP > in a way that is not in line with the existing userspace libraries and > control-plane implementations means that you're developing those > features in an artificial environment that doesn't resemble real 3GPP > interoperable networks out there. > > As indicated, I'm not against adding additional interfaces, but we have > to make sure that we add IPv6 support (or any new feature support) to at > least libgtpnl, and to make sure we test interoperability with existing > 3GPP network equipment such as real IPv6 capable phones and SGSNs. > >> > I'm not sure if this is a useful feature. GTP is used only in >> > operator-controlled networks and only on standard ports. It's not >> > possible to negotiate any non-standard ports on the signaling plane >> > either. >> > >> Bear in mind that we're not required to do everything the GTP spec >> says. > > Yes, we are, at least as long as it affects interoperability with other > implemetations out there. > > GTP uses well-known port numbers on *both* sides of the tunnel, and you > cannot deviate from that. > > There's no point in having all kinds of feetures in the GTP user plane > which are not interoperable with other implementations, and which are > completely outside of the information model / architecture of GTP. > > In the real world, GTP-U is only used in combination with GTP-C. And in > GTP-C you can only negotiate the IP address of both sides of GTP-U, and > not the port number information. As a result, the port numbers are > static on both sides. > >> My impression is GTP designers probably didn't think in terms of >> getting best performance. But we can ;-) > > I think it's wasted efforts if it's about "random udp ports" as no > standards-compliant implementation out there with which you will have to > interoperate will be able to support it. > > GTP is used between home and roaming operator. If you want to introduce > changes to how it works, you will have to have control over both sides > of the implementation of both the GTP-C and the GTP-u plane, which is > very unlikely and rather the exception in the hundreds of operators you > interoperate with. Also keep in mind that there often are various > "middleboxes" that will suddenly have to reflect your changes. That > starts from packet filters at various locations in the operator networks > and/or roaming hubs, down to GTP hubs and others. > > My opinion is: Non-standard GTP ports are not going to happen. > >> I also brought up open_ggsn. ggsn to sgsn. > > That's good to hear. For both v4 and v6 PDP contexts? Whcih phones > did you use for testing? Particularly given how convolved the address > allocation is (see below), I'm surprised it would work. > >> > For IPv6 (and v4v6) PDP contexts there is quite a bit of extra headache >> > related to the way how router solicitation/advertisements are modified >> > in the 3GPP world. >> > >> > The address allocation in v4 is simple: >> > * MS/UE requests dynamic or fixed IPv4 address via EUA IE of PDP context >> > activation >> > * GGSN responds with IPv4 address in EUA of Activate PDP context >> > response (and then uses netlink to tell the kernel about that >> > IPv4 address) >> > >> > In v6 or the v6 portion of v4v6 it works differently: >> > * MS/UE requests dynamic or fixed IPv4 address in EUA IE of PDP context >> > activation >> > * GGSN responds with an IPv6 address, but that address is *not* used >> > for communication, but simply used as an "interface identifier" to >> > build a link-local address. >> > * MS then uses router solicitation using that link-local address >> > * GGSN responds with router advertisement, allocating a single /64 >> > prefix, from which the MS then generates a fully-qualified IPv6 >> > source address for communication. >> > >> > How did you envision this to be done with the v6 support you just added? >> > At the very least, the /64 prefix matching would have to be implemented >> > so that in fact all addresses within that /64 prefix are matched + >> > encapsulated for a given PDP context in the downlink (to phone) >> > direction. >> > >> > [...] >> I would hope all the above you're describing is mostly control plane >> matters. > > It is not. The control plane is GTP-C and runs on different UDP ports > (at least for GTPv1/v2). The user plane is GTP-U and is what's done in > the kernel. And by its very nature, IPv6 router > solicitations/advertisements (as well as neighbor > solicitations/advertisements) are part of the user plane and thus > handled in GTP-U. > >> At least a good design decouples data palne and control >> plane. I know that GTP is a bit convoluted in this regard. > > The problem is that IPv6 has never been specified properly for > point-to-point links. There's no decent PPP specs for IPv6. So the > 3GPP folks had to try to be as close as possible to the existing > (broadcast) link layer model to facilitate existing IPv6 implemetations > to work over 3GPP bearers. That's why they kept whatever possible to > re-use in terms of neighbor/router discovery. > > So the problem is now: Unless you handle GTP-U *entirely* in the kernel > (including router + neighbor advertisement/solicitation), you will have > a "split GTP-U" plane between kernel and userspace. And in that context > the question is who owns the sequence numbers, how will you avoid race > conditions, ... - my simple suggestion is thus to keep with the current > split and do everything GTP-U related inside the kernel and everything > GTP-C related in userspace. > > I think there has to be a clear plan/architecture on how to implement > those bits in terms of the kernel/userspace split, and at least a proof > of concept implementation that we can show works with some real phones > out there - otherwise there's no point in having IPv6 support that works > well with some custom tools. > OTOH, I will argue that the GTP patches should never have been allowed in the kernel in the first place without IPv6 support! ;-) I think the best plan forward is to get the IPv6 data path running that so can demonstrate a functional GTP/IPv6 datapath (my primary purpose here to have something to compare against with ILA). Since "real" configuration path doesn't use the path to set up a standalone interface, I would presume that that will be fleshed when someone has cycles and expertise to work on both sides of the problem. Even if this requires structural changes to how IPv6 is managed in GTP, I doubt that the fundamental TX/RX, GRO/GSO data paths will change much. In other words, please consider this to be a step on an evolutionary path. More work is required to reach the ultimate deployable solution. As for testing on real phones, that is cannot be a requirement for a kernel feature. If you expect Linux community to support this, then we need a way to be develop and test on commodity PC hardware. That is one of the major values of creating a standalone interface configuration-- we can test the datapath just like any other encapsulation supported by the kernel. Tom
Hi Harald, On 20/09/17 01:19, Harald Welte wrote: > Hi Tom, > > On Tue, Sep 19, 2017 at 08:59:28AM -0700, Tom Herbert wrote: >> On Tue, Sep 19, 2017 at 5:43 AM, Harald Welte <laforge@gnumonks.org> >> wrote: >>> On Mon, Sep 18, 2017 at 05:38:50PM -0700, Tom Herbert wrote: >>>> - IPv6 support >>> >>> see my detailed comments in other mails. It's unfortunately only >>> support for the already "deprecated" IPv6-only PDP contexts, not the >>> more modern v4v6 type. In order to interoperate with old and new >>> approach, all three cases (v4, v6 and v4v6) should be supported from >>> one code base. >>> >> It sounds like something that can be subsequently added. > > Not entirely, at least on the netlink (and any other configuration > interface) you will have to reflect this from the very beginning. You > have to have an explicit PDP type and cannot rely on the address type to > specify the type of PDP context. Whatever interfaces are introduced > now will have to remain compatible to any future change. > > My strategy to avoid any such possible 'road blocks' from being > introduced would be to simply add v4v6 and v6 support in one go. The > differences are marginal (having both an IPv6 prefix and a v4 address in > parallel, rather than mutually exclusive only). > >> Do you have a reference to the spec? > > See http://osmocom.org/issues/2418#note-7 which lists Section 11.2.1.3.2 > of 3GPP TS 29.061 in combination with RFC3314, RFC7066, RFC6459 and > 3GPP TS 23.060 9.2.1 as well as a summary of my understanding of it some > months ago. > >>>> - Configurable networking interfaces so that GTP kernel can be >>>> used and tested without needing GSN network emulation (i.e. no >>>> user space daemon needed). >>> >>> We have some pretty decent userspace utilities for configuring the >>> GTP interfaces and tunnels in the libgtpnl repository, but if it >>> helps people to have another way of configuration, I won't be >>> against it. >>> >> AFAIK those userspace utilities don't support IPv6. > > Of course not [yet]. libgtpnl and the command line tools have been > implemented specifically for the in-kernel GTP driver, and you have to > make sure to add related support on both the kernel and the userspace > side (libgtpnl). So there's little point in adding features on either > side before the other side. There would be no way to test... > >> Being able to configure GTP like any other encapsulation will >> facilitate development of IPv6 and other features. > > That may very well be the case, but adding "IPv6 support" to kernel GTP > in a way that is not in line with the existing userspace libraries and > control-plane implementations means that you're developing those > features in an artificial environment that doesn't resemble real 3GPP > interoperable networks out there. > > As indicated, I'm not against adding additional interfaces, but we have > to make sure that we add IPv6 support (or any new feature support) to at > least libgtpnl, and to make sure we test interoperability with existing > 3GPP network equipment such as real IPv6 capable phones and SGSNs. > >>> I'm not sure if this is a useful feature. GTP is used only in >>> operator-controlled networks and only on standard ports. It's not >>> possible to negotiate any non-standard ports on the signaling plane >>> either. >>> >> Bear in mind that we're not required to do everything the GTP spec >> says. > > Yes, we are, at least as long as it affects interoperability with other > implemetations out there. > > GTP uses well-known port numbers on *both* sides of the tunnel, and you > cannot deviate from that. Actually, the well-known port is only mandatory for the receiving side. The sending side can choose any port it wishes as long as it is prepared to receive possible error indication on the well-known port. Of course, it makes the implementation simple to use only one port, but for scalability it might be a good idea to support per PDP context sending ports. Regards Andreas > There's no point in having all kinds of feetures in the GTP user plane > which are not interoperable with other implementations, and which are > completely outside of the information model / architecture of GTP. > > In the real world, GTP-U is only used in combination with GTP-C. And in > GTP-C you can only negotiate the IP address of both sides of GTP-U, and > not the port number information. As a result, the port numbers are > static on both sides. > >> My impression is GTP designers probably didn't think in terms of >> getting best performance. But we can ;-) > > I think it's wasted efforts if it's about "random udp ports" as no > standards-compliant implementation out there with which you will have to > interoperate will be able to support it. > > GTP is used between home and roaming operator. If you want to introduce > changes to how it works, you will have to have control over both sides > of the implementation of both the GTP-C and the GTP-u plane, which is > very unlikely and rather the exception in the hundreds of operators you > interoperate with. Also keep in mind that there often are various > "middleboxes" that will suddenly have to reflect your changes. That > starts from packet filters at various locations in the operator networks > and/or roaming hubs, down to GTP hubs and others. > > My opinion is: Non-standard GTP ports are not going to happen. > >> I also brought up open_ggsn. ggsn to sgsn. > > That's good to hear. For both v4 and v6 PDP contexts? Whcih phones > did you use for testing? Particularly given how convolved the address > allocation is (see below), I'm surprised it would work. > >>> For IPv6 (and v4v6) PDP contexts there is quite a bit of extra headache >>> related to the way how router solicitation/advertisements are modified >>> in the 3GPP world. >>> >>> The address allocation in v4 is simple: >>> * MS/UE requests dynamic or fixed IPv4 address via EUA IE of PDP context >>> activation >>> * GGSN responds with IPv4 address in EUA of Activate PDP context >>> response (and then uses netlink to tell the kernel about that >>> IPv4 address) >>> >>> In v6 or the v6 portion of v4v6 it works differently: >>> * MS/UE requests dynamic or fixed IPv4 address in EUA IE of PDP context >>> activation >>> * GGSN responds with an IPv6 address, but that address is *not* used >>> for communication, but simply used as an "interface identifier" to >>> build a link-local address. >>> * MS then uses router solicitation using that link-local address >>> * GGSN responds with router advertisement, allocating a single /64 >>> prefix, from which the MS then generates a fully-qualified IPv6 >>> source address for communication. >>> >>> How did you envision this to be done with the v6 support you just added? >>> At the very least, the /64 prefix matching would have to be implemented >>> so that in fact all addresses within that /64 prefix are matched + >>> encapsulated for a given PDP context in the downlink (to phone) >>> direction. >>> >>> [...] >> I would hope all the above you're describing is mostly control plane >> matters. > > It is not. The control plane is GTP-C and runs on different UDP ports > (at least for GTPv1/v2). The user plane is GTP-U and is what's done in > the kernel. And by its very nature, IPv6 router > solicitations/advertisements (as well as neighbor > solicitations/advertisements) are part of the user plane and thus > handled in GTP-U. > >> At least a good design decouples data palne and control >> plane. I know that GTP is a bit convoluted in this regard. > > The problem is that IPv6 has never been specified properly for > point-to-point links. There's no decent PPP specs for IPv6. So the > 3GPP folks had to try to be as close as possible to the existing > (broadcast) link layer model to facilitate existing IPv6 implemetations > to work over 3GPP bearers. That's why they kept whatever possible to > re-use in terms of neighbor/router discovery. > > So the problem is now: Unless you handle GTP-U *entirely* in the kernel > (including router + neighbor advertisement/solicitation), you will have > a "split GTP-U" plane between kernel and userspace. And in that context > the question is who owns the sequence numbers, how will you avoid race > conditions, ... - my simple suggestion is thus to keep with the current > split and do everything GTP-U related inside the kernel and everything > GTP-C related in userspace. > > I think there has to be a clear plan/architecture on how to implement > those bits in terms of the kernel/userspace split, and at least a proof > of concept implementation that we can show works with some real phones > out there - otherwise there's no point in having IPv6 support that works > well with some custom tools. > > Regards, > Harald >
Hi Tom, On Tue, Sep 19, 2017 at 04:47:11PM -0700, Tom Herbert wrote: > On Tue, Sep 19, 2017 at 4:19 PM, Harald Welte <laforge@gnumonks.org> wrote: > > > I think there has to be a clear plan/architecture on how to implement > > those bits in terms of the kernel/userspace split, and at least a proof > > of concept implementation that we can show works with some real phones > > out there - otherwise there's no point in having IPv6 support that works > > well with some custom tools. > > > OTOH, I will argue that the GTP patches should never have been allowed > in the kernel in the first place without IPv6 support! ;-) Well, it could be shown that the code works with integration to OpenGGSN (and later ergw, and now OsmoGGSN) and works within a complete 3GPP network. So we were not merging something that we hypothesized it would work once the rest would be implemented, but we could actually show it was working before it got merged. > I think the best plan forward is to get the IPv6 data path running > that so can demonstrate a functional GTP/IPv6 datapath Yes, but a functional "datapath" (= GTP-U) unfortunately includes the way how address allocation/assignment is done. Putting something into the mainline kernel that we know will for sure not work/interop in a real scenario is not a good idea. I think what's worse than not supporting a feature is to implying support for it while actually doing it in an incomplete/incompliant way. So from my point of view, what's needed is * making sure router advertisement/solicitation are covered in some way, either by doing it in the kernel or having a clear strategy how it could be done from userspace while not a) introducing races regarding who owns the sequence numbers * making sure the implementation covers entire /64 prefixes for each PDP context and not single addresses. That is non-negotiable and mandatory by 3GPP specs. > Since "real" configuration path doesn't use the path to set up a > standalone interface, I would presume that that will be fleshed when > someone has cycles and expertise to work on both sides of the problem. The configuration will be different, yes. but we need to ensure that the actual *implementation* of the data path does what it is expected to do, no matter who configures it via which interface. > Even if this requires structural changes to how IPv6 is managed in > GTP, I doubt that the fundamental TX/RX, GRO/GSO data paths will > change much. In other words, please consider this to be a step on an > evolutionary path. More work is required to reach the ultimate > deployable solution. Agreed. But then I'm still against merging something that we know for sure will not be compatible with real-world use case. It should be kept out of mainline until we are sure of that, at the very least theoretically, but even better which we can prove in practise will do what it claims to do. > As for testing on real phones, that is cannot be a requirement for a > kernel feature. GTP is not implemented on phones. GTP is implemented only inside the fixed (land-side) of the cellular network. However, the inner IP data originates from phones, and large parts of what you see on GTP originates from phones in a different format. The inner IP data inside GTP-U originates from phones. And that's where address configuration for IPv6 works. > If you expect Linux community to support this, then we > need a way to be develop and test on commodity PC hardware. I'm not arguing you need to run any of the code on a different architecture such as a phone. I'm arguing for you to run a cellular network including the kernel GTP code, and then use that cellular network from a real phone. This is the only way to know for sure you interoperate. See my other mail related to the Open Soruce based configurations for both 2G and 3G that can be used for this. As a replacement, one can e.g. look at protocol traces of real phones and then simulate the behavior of one or several different phones and implement that as test caess. This is what I did in the GGSN_Test pointed out in my other mail in this thread. And btw, all I've asked is for showing it works with *one* phone model at all. I'm not talking about the various different implementation specifics, such as whether or not the phone will insist on using neighbor solicitation and mandate neighbor advertisement (on a point-to-point link, how absurd!) after the (mandatory) router discovery. > That is one of the major values of creating a standalone interface > configuration-- we can test the datapath just like any other > encapsulation supported by the kernel. Well, you cannot. You might be able to do some benchmarking to compare if an old version of the kernel gtp driver will perform better or worse than some optimizations introduced. But you can *not* have a realistic functional test that will tell you if your implementation is 'valid'. I'm not even talking about being 'complete' here, but simply about being broken or not. Or test whether it will interoperate. Particularly for IPv6 this is impossible, due to the conflated way of involving both GTP-C and GTP-U with router advertisement+solicitation for PDP context activation, as outlined several times in this thread. I wish it was simpler, but I haven't created GTP, sorry :)
This patch set builds upon the initial GTP implementation to make support closer to that enjoyed by other encapsulation protocols. The major items are: - IPv6 support - Configurable networking interfaces so that GTP kernel can be used and tested without needing GSN network emulation (i.e. no user space daemon needed). - GSO,GRO - Control of zero UDP checksums - Port numbers are configurable - Addition of a dst_cache in the GTP structure and other cleanup Additionally, this patch set also includes a couple of general support capabilities: - A facility that allows application specific GSO callbacks - Common functions to get a route fo for an IP tunnel For IPv6 support, the mobile subscriber needs to allow IPv6 addresses, and the remote enpoint can be IPv6. For configurable interfaces, configuration is added to allow an alterate means to configure a GTP and device. This follows the typical UDP encapsulation model of specifying a listener port for receive, and a remote address and port for transmit. GRO was straightfoward to implement following the model of other UDP encapsulations. Providing GSO support had one wrinkle-- the GTP header includes a payload length field that needs to be set per GSO segment. In order to address that in a general way, I create the concept of application specific GSO. To implement application layer GSO I reserved the top four bits of shinfo(skb)->gso_type. The idea is that an application or encapsulation protocol (like GTP in this case) can register a GSO segment callback. The facility returns a gso_type with upper four bits set to a value (index into a table). When the application sets up a packet it includes the code in the gso_type for the skb. At some point (e.g. from UDP segment) the gso_type is checked in the skb and if the application specific GSO is indicated then the callback is called. The registered callbacks include a set of other gso_types so that an application callback can be matched to an appropriate instance. FOr instance, the GTP callback checks for the UDP GSO flags. Zero UDP checksum, port number configuration, and dst_cache are straightforwad. Configuration is performed by iproute2/ip. I will post that in a subsequent patch set. Tested: Configured the matrix of IPv4/IPv6 mobile subscriber, IPv4/IPv6 remote peer, and GTP version 0 and 1 (eight combinations). Observed connectivity and proper GSO/GRO. Also, tested VXLAN for regression. Tom Herbert (14): iptunnel: Add common functions to get a tunnel route vxlan: Call common functions to get tunnel routes gtp: Call common functions to get tunnel routes and add dst_cache gtp: udp recv clean up gtp: Remove special mtu handling gtp: Eliminate pktinfo and add port configuration gtp: Support encapsulation of IPv6 packets gtp: Support encpasulating over IPv6 gtp: Allow configuring GTP interface as standalone gtp: Add support for devnet net: Add a facility to support application defined GSO gtp: Configuration for zero UDP checksum gtp: Support for GRO gtp: GSO support drivers/net/gtp.c | 1300 ++++++++++++++++++++++++++++++++---------- drivers/net/vxlan.c | 84 +-- include/linux/netdevice.h | 31 + include/linux/skbuff.h | 25 + include/net/ip6_tunnel.h | 33 ++ include/net/ip_tunnels.h | 33 ++ include/uapi/linux/gtp.h | 8 + include/uapi/linux/if_link.h | 6 + net/core/dev.c | 47 ++ net/ipv4/ip_tunnel.c | 41 ++ net/ipv4/ip_tunnel_core.c | 6 + net/ipv4/udp_offload.c | 20 +- net/ipv6/ip6_tunnel.c | 43 ++ 13 files changed, 1306 insertions(+), 371 deletions(-)