| Message ID | 2107636851.12713862.1393876035292.JavaMail.zimbra@redhat.com |
|---|---|
| State | RFC, archived |
| Delegated to: | David Miller |
| Headers | show |
Hi Jan, On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote: > I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 > neighbor solicitation packets to KVM guest and as result KVM guest doesn't > respond with neighbor advertisement. Hm, okay, that's not supposed to happen. > The reason I think this packet is related is because when I send same exact > packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. > solicitation packets to KVM guest. Yes, the MLD query is kicking the multicast snooping into gear. If there's never a query, then snooping is basically disabled (compare: "bridge: disable snooping if there is no querier"). > > My current way to reproduce this is: > 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c > guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 > 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP > using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/ > with parameters: > --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029 > -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c > --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 > tcpdump running on guest can see both solicit and advertisement packets > 2. wait ~5 minutes > 3. host B sends Multicast Listener Query packet described above > 4. tcpdump running on guest is no longer seeing any neigh solicit packets Just to clarify, host B is behind eno1 and vnet0 is directly connected to the interface of the guest, no additional bridge or anything else on top of that, right? Would it be possible for you to upload the tcpdumps from host B (or if you can't tcpdump on host B, then capturing on eno1) and the guest somewhere and saying at which time/packet in the dumps it stops working (probably ~10 seconds after the query). Filtering for ICMPv6 should be sufficient. What I'm curious about is, whether the guest receives the MLD query and responds with an MLD report. I suspect that either the bridge doesn't get an MLD report and therefore is shutting down the according port or there's a bug in parsing the MLD report in the bridge code. Thanks for the detailed report so far! Cheers, Linus
On 03/03/2014 04:27 PM, Linus Lüssing wrote: > Hi Jan, > > On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote: >> I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 >> neighbor solicitation packets to KVM guest and as result KVM guest doesn't >> respond with neighbor advertisement. > > Hm, okay, that's not supposed to happen. > >> The reason I think this packet is related is because when I send same exact >> packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. >> solicitation packets to KVM guest. > > Yes, the MLD query is kicking the multicast snooping into gear. If > there's never a query, then snooping is basically disabled > (compare: "bridge: disable snooping if there is no querier"). > >> >> My current way to reproduce this is: >> 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c >> guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 >> 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP >> using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/ >> with parameters: >> --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029 >> -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c >> --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 >> tcpdump running on guest can see both solicit and advertisement packets >> 2. wait ~5 minutes >> 3. host B sends Multicast Listener Query packet described above >> 4. tcpdump running on guest is no longer seeing any neigh solicit packets > > Just to clarify, host B is behind eno1 and vnet0 is directly > connected to the interface of the guest, no additional bridge or > anything else on top of that, right? > > Would it be possible for you to upload the tcpdumps from host B > (or if you can't tcpdump on host B, then capturing on eno1) > and the guest somewhere and saying at which time/packet in the dumps > it stops working (probably ~10 seconds after the query). Filtering > for ICMPv6 should be sufficient. > > What I'm curious about is, whether the guest receives > the MLD query and responds with an MLD report. I suspect that > either the bridge doesn't get an MLD report and therefore is > shutting down the according port or there's a bug in parsing the > MLD report in the bridge code. > I did notice a minor issue in the bridge code. The following code: /* Prevent flooding this packet if there is no listener present */ if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr)) BR_INPUT_SKB_CB(skb)->mrouters_only = 1; if (ip6h->nexthdr != IPPROTO_HOPOPTS || ip6h->payload_len == 0) return 0; will mark most multicast traffic is mrouters_only. The two statement should be probably be reversed. However, that's shouldn't cause the reported problem. -vlad > > Thanks for the detailed report so far! > > Cheers, Linus > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
----- Original Message ----- > From: "Linus Lüssing" <linus.luessing@web.de> > To: "Jan Stancek" <jstancek@redhat.com> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > Sent: Monday, 3 March, 2014 10:27:59 PM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > Hi Jan, > > On Mon, Mar 03, 2014 at 02:47:15PM -0500, Jan Stancek wrote: > > I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 > > neighbor solicitation packets to KVM guest and as result KVM guest doesn't > > respond with neighbor advertisement. > > Hm, okay, that's not supposed to happen. > > > The reason I think this packet is related is because when I send same exact > > packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. > > solicitation packets to KVM guest. > > Yes, the MLD query is kicking the multicast snooping into gear. If > there's never a query, then snooping is basically disabled > (compare: "bridge: disable snooping if there is no querier"). > > > > > My current way to reproduce this is: > > 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / > > 00:21:5a:47:93:1c > > guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 > > 1. host B is sending neigh solicit packets every 5 seconds with KVM guest > > IP > > using ns6 from ipv6toolkit: > > http://www.si6networks.com/tools/ipv6toolkit/ > > with parameters: > > --src-address=2620:52:0:1040:221:5aff:fe47:931c > > --dst-address=ff02::1:ff00:0029 > > -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c > > --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 > > tcpdump running on guest can see both solicit and advertisement packets > > 2. wait ~5 minutes > > 3. host B sends Multicast Listener Query packet described above > > 4. tcpdump running on guest is no longer seeing any neigh solicit packets > > Just to clarify, host B is behind eno1 and vnet0 is directly > connected to the interface of the guest, no additional bridge or > anything else on top of that, right? Yes, host B should be behind eno1 (All hosts are remote to me). There should be only single bridge on host A. Host A has 3 more interfaces but those are all down. # cat /etc/sysconfig/network-scripts/ifcfg-eno1 DEVICE=eno1 ONBOOT=yes BRIDGE=br1 HWADDR=00:23:ae:ed:1a:00 # cat /etc/sysconfig/network-scripts/ifcfg-br1 DEVICE=br1 BOOTPROTO=dhcp ONBOOT=yes TYPE=Bridge DELAY=0 There is also bridge on host B. I assume that doesn't matter but I could set up host B without bridge if needed. > > Would it be possible for you to upload the tcpdumps from host B > (or if you can't tcpdump on host B, then capturing on eno1) > and the guest somewhere and saying at which time/packet in the dumps > it stops working (probably ~10 seconds after the query). Filtering > for ICMPv6 should be sufficient. Here are tcpdumps from hostA, hostB and guest (on hostA): http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces1/ I didn't apply any filter, because that multicast query wasn't showing up for some reason when I tried to filter by icmp6. What I did: 1. started tcpdump on all systems 2. send 3 neigh. solicit from hostB manually with couple seconds in between 3. send multicast listener query from hostB manually 4. send 5 neigh. solicit from hostB manually with couple seconds in between hostA.cap tcpdump -i eno1 -w hostA.cap frame 124, 125 -> OK frame 217, 218 -> OK frame 291, 292 -> OK frame 373 -> Multicast Listener Query frame 484 -> no reply? frame 572 -> no reply? frame 665 -> no reply? hostB.cap tcpdump -i br0 -w hostB.cap frame 106, 108 -> OK frame 214, 216 -> OK frame 300, 302 -> OK frame 396 -> Multicast Listener Query frame 523 -> no reply? frame 623 -> no reply? frame 730 -> no reply? guest.cap tcpdump -i eth0 -w guest.cap frame 89, 90 -> OK frame 181, 182 -> OK frame 254, 255 -> OK frame 334 -> Multicast Listener Query no more neigh. solicit packets > > What I'm curious about is, whether the guest receives > the MLD query and responds with an MLD report. I suspect that > either the bridge doesn't get an MLD report and therefore is > shutting down the according port or there's a bug in parsing the > MLD report in the bridge code. I'm no expert in this area, but shouldn't neigh. solicit packets be forwarded to all ports regardless of any/no MLD reports? Regards, Jan > > > Thanks for the detailed report so far! > > Cheers, Linus > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Mar 03, 2014 at 04:40:40PM -0500, Vlad Yasevich wrote: > I did notice a minor issue in the bridge code. The following > code: > /* Prevent flooding this packet if there is no listener present */ > if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr)) > BR_INPUT_SKB_CB(skb)->mrouters_only = 1; > > if (ip6h->nexthdr != IPPROTO_HOPOPTS || > ip6h->payload_len == 0) > return 0; > > will mark most multicast traffic is mrouters_only. The two > statement should be probably be reversed. However, that's shouldn't > cause the reported problem. Reversing the order of these two if-clauses would reintroduce the issue this commit tried to address, I think: "bridge: prevent flooding IPv6 packets that do not have a listener" Besides, I don't quite see what minor issue you are refering to, would you mind being a little more verbose? Cheers, Linus PS: mrouters_only has a kind of confusing naming... for MLD/IGMP packets it means sending to multicast routers only, there the name fits. But for non-MLD/IGMP packets it means something else since "bridge: Only flood unregistered groups to routers" (and I went along with it with "bridge: prevent flooding IPv6 packets that do not have a listener"), there it means dropping the skb if there is no router or matching listener.
Hi Jan, On Mon, Mar 03, 2014 at 05:45:49PM -0500, Jan Stancek wrote: > There is also bridge on host B. I assume that doesn't matter > but I could set up host B without bridge if needed. It can matter, but in this case it doesn't :). > > What I'm curious about is, whether the guest receives > > the MLD query and responds with an MLD report. I suspect that > > either the bridge doesn't get an MLD report and therefore is > > shutting down the according port or there's a bug in parsing the > > MLD report in the bridge code. > > I'm no expert in this area, but shouldn't neigh. solicit packets > be forwarded to all ports regardless of any/no MLD reports? That's the beauty of IPv6 Neighbor Discovery using these neat solicited-node multicast addresses :). With IPv4 and ARP requests there's no other way than flooding. But for IPv6 we know in advance behind which bridge port someone interested in the neighbor solicitation message might be (assuming MLD is working, properly), allowing us to save bandwidth. In this case, MLD is not working properly, the main issue is the following: Host B sends broken MLD queries, the source address should be an IPv6 link-local one, not "100:0:600:0:78fb:100::". MLDv2 mandates this (see RFC3810, section 5.1.14.: "Source Addresses for Queries"). Though I couldn't find that requirement for MLDv1, Linux ignores MLDv1 queries with a non-link-local source address, too (see net/ipv6/mcast.c, igmp6_event_query() ). So Linux never sends an MLD report in reply to these broken queries. The second "minor" but in this case fatal issue is, that the bridge code doesn't have this link-local-src check, therefore kicking the snooping into gear even though it shouldn't because we don't have a _working_ querier. I'm going to make a patch for the bridge code adding this sanity check. For the broken query, ok, it's your manually crafted query. But did you see a query with such a bogus source address "in the wild", too? (I'm curious how urgent this sanity check is) Cheers, Linus
----- Original Message ----- > From: "Linus Lüssing" <linus.luessing@web.de> > To: "Jan Stancek" <jstancek@redhat.com> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > Sent: Tuesday, 4 March, 2014 1:00:41 AM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > Hi Jan, > > On Mon, Mar 03, 2014 at 05:45:49PM -0500, Jan Stancek wrote: > > There is also bridge on host B. I assume that doesn't matter > > but I could set up host B without bridge if needed. > > It can matter, but in this case it doesn't :). > > > > What I'm curious about is, whether the guest receives > > > the MLD query and responds with an MLD report. I suspect that > > > either the bridge doesn't get an MLD report and therefore is > > > shutting down the according port or there's a bug in parsing the > > > MLD report in the bridge code. > > > > I'm no expert in this area, but shouldn't neigh. solicit packets > > be forwarded to all ports regardless of any/no MLD reports? > > That's the beauty of IPv6 Neighbor Discovery using these neat > solicited-node multicast addresses :). With IPv4 and ARP > requests there's no other way than flooding. But for IPv6 we know > in advance behind which bridge port someone interested in the > neighbor solicitation message might be (assuming MLD is working, > properly), allowing us to save bandwidth. > > In this case, MLD is not working properly, the main issue is the > following: > > Host B sends broken MLD queries, the source address should be an > IPv6 link-local one, not "100:0:600:0:78fb:100::". MLDv2 mandates > this (see RFC3810, section 5.1.14.: "Source Addresses for > Queries"). > > Though I couldn't find that requirement for MLDv1, Linux ignores > MLDv1 queries with a non-link-local source address, too (see > net/ipv6/mcast.c, igmp6_event_query() ). So Linux never sends an > MLD report in reply to these broken queries. > > > The second "minor" but in this case fatal issue is, that the > bridge code doesn't have this link-local-src check, therefore > kicking the snooping into gear even though it shouldn't because we > don't have a _working_ querier. > > I'm going to make a patch for the bridge code adding this sanity > check. > > > For the broken query, ok, it's your manually crafted query. But > did you see a query with such a bogus source address "in the > wild", too? (I'm curious how urgent this sanity check is) It's real packet I managed to capture during one such occurrence. I'm sending it with small C program over raw socket, but it's byte by byte exact copy of what I captured with tcpdump previously. I'm not sure how that packet came to existence. Based on IPv6 address it came from host B, but all host B was doing at the time was running RHEL6 with couple qemu-kvm instances. KVM guests were set up to use bridge, so I'm assuming if any of them crafted this packet, source IPv6 address would be different. Regards, Jan > > Cheers, Linus > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Jan, On Tue, Mar 04, 2014 at 03:02:36AM -0500, Jan Stancek wrote: > > For the broken query, ok, it's your manually crafted query. But > > did you see a query with such a bogus source address "in the > > wild", too? (I'm curious how urgent this sanity check is) > > It's real packet I managed to capture during one such occurrence. > I'm sending it with small C program over raw socket, but it's byte > by byte exact copy of what I captured with tcpdump previously. > > I'm not sure how that packet came to existence. Based on IPv6 address > it came from host B, but all host B was doing at the time > was running RHEL6 with couple qemu-kvm instances. KVM guests were > set up to use bridge, so I'm assuming if any of them crafted > this packet, source IPv6 address would be different. > Ah, okay. Can you check whether it maybe came from the querier code in the Linux bridge on host B? Is "cat /sys/class/net/br0/bridge/multicast_querier" 1? Can you isolate host B and disable any multicast router daemon on it? Then check again, if you still see these queries. What kernel version is running on host B? Cheers, Linus
----- Original Message ----- > From: "Linus Lüssing" <linus.luessing@web.de> > To: "Jan Stancek" <jstancek@redhat.com> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > Sent: Tuesday, 4 March, 2014 11:52:54 AM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > Hi Jan, > > On Tue, Mar 04, 2014 at 03:02:36AM -0500, Jan Stancek wrote: > > > For the broken query, ok, it's your manually crafted query. But > > > did you see a query with such a bogus source address "in the > > > wild", too? (I'm curious how urgent this sanity check is) > > > > It's real packet I managed to capture during one such occurrence. > > I'm sending it with small C program over raw socket, but it's byte > > by byte exact copy of what I captured with tcpdump previously. > > > > I'm not sure how that packet came to existence. Based on IPv6 address > > it came from host B, but all host B was doing at the time > > was running RHEL6 with couple qemu-kvm instances. KVM guests were > > set up to use bridge, so I'm assuming if any of them crafted > > this packet, source IPv6 address would be different. > > > > Ah, okay. Can you check whether it maybe came from the querier > code in the Linux bridge on host B? Is > "cat /sys/class/net/br0/bridge/multicast_querier" 1? # cat /sys/class/net/br0/bridge/multicast_querier cat: /sys/class/net/br0/bridge/multicast_querier: No such file or directory > Can you isolate host B and disable any multicast router daemon on it? Then > check again, if you still see these queries. Besides those cases where I sent it by myself, I haven't seen host B send that query for couple days now. > What kernel version is running on host B? 2.6.32-279.42.1.el6.x86_64 It's a RHEL6.3.z kernel. > Where does Linux use :: for queries? I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I see packets like these on my network every ~125 seconds: No. Time Source Destination Protocol Length Info 22675 1334.751135 :: ff02::1 ICMPv6 86 Multicast Listener Query Internet Control Message Protocol v6 Type: Multicast Listener Query (130) Code: 0 Checksum: 0x7ac1 [correct] Maximum Response Delay [ms]: 1000 Reserved: 0000 Multicast Address: :: (::) Regards, Jan -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, Mar 04, 2014 at 06:06:29AM -0500, Jan Stancek wrote: > > > ----- Original Message ----- > > From: "Linus Lüssing" <linus.luessing@web.de> > > To: "Jan Stancek" <jstancek@redhat.com> > > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > > Sent: Tuesday, 4 March, 2014 11:52:54 AM > > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > > > Hi Jan, > > > > On Tue, Mar 04, 2014 at 03:02:36AM -0500, Jan Stancek wrote: > > > > For the broken query, ok, it's your manually crafted query. But > > > > did you see a query with such a bogus source address "in the > > > > wild", too? (I'm curious how urgent this sanity check is) > > > > > > It's real packet I managed to capture during one such occurrence. > > > I'm sending it with small C program over raw socket, but it's byte > > > by byte exact copy of what I captured with tcpdump previously. > > > > > > I'm not sure how that packet came to existence. Based on IPv6 address > > > it came from host B, but all host B was doing at the time > > > was running RHEL6 with couple qemu-kvm instances. KVM guests were > > > set up to use bridge, so I'm assuming if any of them crafted > > > this packet, source IPv6 address would be different. > > > > > > > Ah, okay. Can you check whether it maybe came from the querier > > code in the Linux bridge on host B? Is > > "cat /sys/class/net/br0/bridge/multicast_querier" 1? > > # cat /sys/class/net/br0/bridge/multicast_querier > cat: /sys/class/net/br0/bridge/multicast_querier: No such file or directory Ok, that's not present on such old kernels, there it is actually enabled by default. > > > Can you isolate host B and disable any multicast router daemon on it? Then > > check again, if you still see these queries. > > Besides those cases where I sent it by myself, I haven't seen host B send > that query for couple days now. > > > What kernel version is running on host B? > > 2.6.32-279.42.1.el6.x86_64 > It's a RHEL6.3.z kernel. > > > Where does Linux use :: for queries? > > I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I see > packets like these on my network every ~125 seconds: > > No. Time Source Destination Protocol Length Info > 22675 1334.751135 :: ff02::1 ICMPv6 86 Multicast Listener Query It's probably the bridge on this ancient kernel, you might want to backport the following patch: "bridge: Use IPv6 link-local address for multicast listener queries" (fe29ec41aaa51902aebd63658dfb04fe6fea8be5) And it's follow-up fixes: "bridge: Fix possibly wrong MLD queries' ethernet source address" (a7bff75b087e7a355838a32efe61707cfa73c194) "bridge: check return value of ipv6_dev_get_saddr()" (d1d81d4c3dd886d5fa25a2c4fa1e39cb89613712) If these patches on host B xor the sanity check I just submitted applied on your host A / VM host fix your issue, then they might be worth considering for the stable queue. Cheers, Linus
----- Original Message ----- > From: "Linus Lüssing" <linus.luessing@web.de> > To: "Jan Stancek" <jstancek@redhat.com> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > Sent: Tuesday, 4 March, 2014 10:37:57 PM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I > > see > > packets like these on my network every ~125 seconds: > > > > No. Time Source Destination Protocol > > Length Info > > 22675 1334.751135 :: ff02::1 ICMPv6 86 > > Multicast Listener Query > > It's probably the bridge on this ancient kernel, you might want to > backport the following patch: Hi, The example above was from box, which turned out not be running Linux. > > If these patches on host B xor the sanity check I just submitted applied > on your host A / VM host fix your issue, then they might be worth > considering for the stable queue. Is it necessary to apply same patch also on any (older) kernel running in VM? I applied your sanity check patch [1] on host A only and I wasn't able to reproduce this issue any longer when I sent this malformed packet from host B: Ethernet II, Src: Hewlett-_47:93:1c (00:21:5a:47:93:1c), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) Internet Protocol Version 6, Src: 100:0:600:0:78fb:100:: (100:0:600:0:78fb:100::), Dst: ff02::1 (ff02::1) Internet Control Message Protocol v6 I hand-crafted one new packet from malformed one used in previous tests. I modified source address from :: to host B link-scope address and changed dst address from ff02::1 to ff02::1:ffaa:aaaa Ethernet II, Src: Hewlett-_47:93:1c (00:21:5a:47:93:1c), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) Internet Protocol Version 6, Src: fe80::221:5aff:fe47:931c (fe80::221:5aff:fe47:931c), Dst: ff02::1:ffaa:aaaa (ff02::1:ffaa:aaaa) Internet Control Message Protocol v6 Type: Multicast Listener Query (130) Code: 0 Checksum: 0xe365 [correct] Maximum Response Delay [ms]: 1000 Reserved: 0000 Multicast Address: :: (::) When I sent it from host B, guest on host A hit same issue - it stopped seeing neighbor solicitation packets. host A was running 3.14.0-rc5 kernel with patch [1] applied. Then I updated kernel on guest to 3.14.0-rc5 + patch [1], but result was the same. Here is trace from guest: http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/ host B was sending neigh solicit every 5 seconds: frame 134, 135 -> OK frame 216, 217 -> OK frame 329, 330 -> OK frame 432, 433 -> OK frame 459 -> new hand-crafted packet after 35 second mark, guest doesn't see any longer neigh solicit packets Regards, Jan [1] bridge: multicast: add sanity check for query source addresses -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Mar 05, 2014 at 07:10:07AM -0500, Jan Stancek wrote: > > > ----- Original Message ----- > > From: "Linus Lüssing" <linus.luessing@web.de> > > To: "Jan Stancek" <jstancek@redhat.com> > > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > > Sent: Tuesday, 4 March, 2014 10:37:57 PM > > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > > I'm not sure if it's Linux (I'm trying to locate that system by MAC), but I > > > see > > > packets like these on my network every ~125 seconds: > > > > > > No. Time Source Destination Protocol > > > Length Info > > > 22675 1334.751135 :: ff02::1 ICMPv6 86 > > > Multicast Listener Query > > > > It's probably the bridge on this ancient kernel, you might want to > > backport the following patch: > > Hi, > > The example above was from box, which turned out not be running Linux. Okay, yeah, there's somehow the misconception flying around, that a snooping switch could use "::" as a source address. Even the informational RFC4541 writes about "proxy IGMP queries" disregarding the source address requirements for MLDv1/2, feeding this misconception. I just skimmed RFC4605 ("IGMP/MLD Proxying") briefly, but couldn't find any note for "0.0.0.0" or "::" there. But I think we could see with your practical example, that "::" MLD query source addresses should be considered broken. > > > > > If these patches on host B xor the sanity check I just submitted applied > > on your host A / VM host fix your issue, then they might be worth > > considering for the stable queue. > > Is it necessary to apply same patch also on any (older) kernel running in VM? The source address patches are only required on boxes where the internal MLD querier code of the bridge is used. Afaik tell the source address patch is present on all kernels since v2.6.38 and all kernels listed on the kernel.org frontpage, except 2.6.34.15 and 2.6.32.61. For the sanity check patch, it should probably be added to any kernel having a bridge (though in your particular scenario and what we are currently debugging at, having it on host A should be sufficient). > I hand-crafted one new packet from malformed one used in previous tests. > I modified source address from :: to host B link-scope address and changed > dst address from ff02::1 to ff02::1:ffaa:aaaa Okay, again according to your capture the guest is receiving the MLD query on its interface but does not react with an MLD report. Two things I'd like to know: Is using the link-scope address as a source and "ff02::1" as the destination address for the MLD query work for you? Is using the link-scope address as a source and "ff02::1:ff00:29" as the destination address for the MLD query "work" for you (do we see an MLD report from the guest and keep on seeing neighbor solicitations from host B then?). For the latter, I don't see anything in particular filtering these for a general MLD query wrong destination address in the IPv6 code from igmp6_event_query() on. But I suspect that the query doesn't even get that far on the kernel of the guest, as it is not listening on ff02::1:ffaa:aaaa. Therefore the test with "ff02::1:ff00:29", an address the guest is listening on, would be interesting. If that works, then I'm going to make a patch ignore General MLD Queries without ff02::1 as their destination address, too. Hm, looking at more checks in igmp6_event_query(), I'm currently wondering whether we should only enable the snooping behaviour in the bridge when receiving a General MLD Query, so one with "::" in the multicast field of the MLD message, instead of activating it upon a Multicast-Address-Specific Query, too. That'd seem more sane to me, I'm going to make a patch for that tomorrow. Cheers, Linus -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
----- Original Message ----- > From: "Linus Lüssing" <linus.luessing@web.de> > To: "Jan Stancek" <jstancek@redhat.com> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > Sent: Wednesday, 5 March, 2014 3:27:07 PM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest <snip> > > > I hand-crafted one new packet from malformed one used in previous tests. > > I modified source address from :: to host B link-scope address and changed > > dst address from ff02::1 to ff02::1:ffaa:aaaa > > Okay, again according to your capture the guest is receiving the > MLD query on its interface but does not react with an MLD report. > > Two things I'd like to know: > > Is using the link-scope address as a source and "ff02::1" as the > destination address for the MLD query work for you? Yes, I could not trigger it with such query: http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1.cap frame 795 -> query frame 1040 -> MLD report from guest ~20 seconds later frame 1507, 1508 -> neigh solicit/advert frame 1580, 1581 -> neigh solicit/advert > > Is using the link-scope address as a source and "ff02::1:ff00:29" > as the destination address for the MLD query "work" for you (do > we see an MLD report from the guest and keep on seeing neighbor > solicitations from host B then?). Yes, this also worked (though I received 2 reports): http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1_ff0029.cap frame 446 -> query frame 448 -> MLD report from guest frame 465 -> MLD report from guest frame 689, 690 -> neigh solicit/advert frame 760, 761 -> neigh solicit/advert ... Both host and guest were running 3.14.0-rc5 with your sanity check patch. Regards, Jan > > For the latter, I don't see anything in particular filtering these > for a general MLD query wrong destination address in the IPv6 > code from igmp6_event_query() on. But I suspect that the query > doesn't even get that far on the kernel of the guest, as it is not > listening on ff02::1:ffaa:aaaa. Therefore the test with > "ff02::1:ff00:29", an address the guest is listening on, would be > interesting. > > If that works, then I'm going to make a patch ignore General MLD > Queries without ff02::1 as their destination address, too. > > > Hm, looking at more checks in igmp6_event_query(), I'm currently > wondering whether we should only enable the snooping behaviour in > the bridge when receiving a General MLD Query, so one with "::" in > the multicast field of the MLD message, instead of activating it > upon a Multicast-Address-Specific Query, too. That'd seem more > sane to me, I'm going to make a patch for that tomorrow. > > Cheers, Linus > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Jan, Hope your bridge-snooping related issues are resolved with the two new patches present in net. If not just let me/us know. Thanks again for the detailed and helpful reporting! Cheers, Linus On Wed, Mar 05, 2014 at 09:57:52AM -0500, Jan Stancek wrote: > > > > > ----- Original Message ----- > > From: "Linus Lüssing" <linus.luessing@web.de> > > To: "Jan Stancek" <jstancek@redhat.com> > > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > > Sent: Wednesday, 5 March, 2014 3:27:07 PM > > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > <snip> > > > > > > I hand-crafted one new packet from malformed one used in previous tests. > > > I modified source address from :: to host B link-scope address and changed > > > dst address from ff02::1 to ff02::1:ffaa:aaaa > > > > Okay, again according to your capture the guest is receiving the > > MLD query on its interface but does not react with an MLD report. > > > > Two things I'd like to know: > > > > Is using the link-scope address as a source and "ff02::1" as the > > destination address for the MLD query work for you? > > Yes, I could not trigger it with such query: > http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1.cap > frame 795 -> query > frame 1040 -> MLD report from guest > ~20 seconds later > frame 1507, 1508 -> neigh solicit/advert > frame 1580, 1581 -> neigh solicit/advert > > > > > Is using the link-scope address as a source and "ff02::1:ff00:29" > > as the destination address for the MLD query "work" for you (do > > we see an MLD report from the guest and keep on seeing neighbor > > solicitations from host B then?). > > Yes, this also worked (though I received 2 reports): > http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1_ff0029.cap > frame 446 -> query > frame 448 -> MLD report from guest > frame 465 -> MLD report from guest > frame 689, 690 -> neigh solicit/advert > frame 760, 761 -> neigh solicit/advert > ... > > Both host and guest were running 3.14.0-rc5 with your sanity check patch. > > Regards, > Jan > > > > > For the latter, I don't see anything in particular filtering these > > for a general MLD query wrong destination address in the IPv6 > > code from igmp6_event_query() on. But I suspect that the query > > doesn't even get that far on the kernel of the guest, as it is not > > listening on ff02::1:ffaa:aaaa. Therefore the test with > > "ff02::1:ff00:29", an address the guest is listening on, would be > > interesting. > > > > If that works, then I'm going to make a patch ignore General MLD > > Queries without ff02::1 as their destination address, too. > > > > > > Hm, looking at more checks in igmp6_event_query(), I'm currently > > wondering whether we should only enable the snooping behaviour in > > the bridge when receiving a General MLD Query, so one with "::" in > > the multicast field of the MLD message, instead of activating it > > upon a Multicast-Address-Specific Query, too. That'd seem more > > sane to me, I'm going to make a patch for that tomorrow. > > > > Cheers, Linus > > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
----- Original Message ----- > From: "Linus Lüssing" <linus.luessing@web.de> > To: "Jan Stancek" <jstancek@redhat.com> > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, bridge@lists.linux-foundation.org > Sent: Wednesday, 12 March, 2014 5:37:40 AM > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM guest > > Hi Jan, > > Hope your bridge-snooping related issues are resolved with the two > new patches present in net. If not just let me/us know. Thanks > again for the detailed and helpful reporting! Hi, looks good, I wasn't able to reproduce this issue after I applied patches you posted recently. Regards, Jan > > Cheers, Linus > > > On Wed, Mar 05, 2014 at 09:57:52AM -0500, Jan Stancek wrote: > > > > > > > > > > ----- Original Message ----- > > > From: "Linus Lüssing" <linus.luessing@web.de> > > > To: "Jan Stancek" <jstancek@redhat.com> > > > Cc: netdev@vger.kernel.org, "Florian Westphal" <fwestpha@redhat.com>, > > > bridge@lists.linux-foundation.org > > > Sent: Wednesday, 5 March, 2014 3:27:07 PM > > > Subject: Re: bridge is not forwaring ICMP6 neighbor solicitation to KVM > > > guest > > > > <snip> > > > > > > > > > I hand-crafted one new packet from malformed one used in previous > > > > tests. > > > > I modified source address from :: to host B link-scope address and > > > > changed > > > > dst address from ff02::1 to ff02::1:ffaa:aaaa > > > > > > Okay, again according to your capture the guest is receiving the > > > MLD query on its interface but does not react with an MLD report. > > > > > > Two things I'd like to know: > > > > > > Is using the link-scope address as a source and "ff02::1" as the > > > destination address for the MLD query work for you? > > > > Yes, I could not trigger it with such query: > > http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1.cap > > frame 795 -> query > > frame 1040 -> MLD report from guest > > ~20 seconds later > > frame 1507, 1508 -> neigh solicit/advert > > frame 1580, 1581 -> neigh solicit/advert > > > > > > > > Is using the link-scope address as a source and "ff02::1:ff00:29" > > > as the destination address for the MLD query "work" for you (do > > > we see an MLD report from the guest and keep on seeing neighbor > > > solicitations from host B then?). > > > > Yes, this also worked (though I received 2 reports): > > http://jan.stancek.eu/tmp/neigh_solicit_and_bridge_traces2/guest_mld_query_ff02_1_ff0029.cap > > frame 446 -> query > > frame 448 -> MLD report from guest > > frame 465 -> MLD report from guest > > frame 689, 690 -> neigh solicit/advert > > frame 760, 761 -> neigh solicit/advert > > ... > > > > Both host and guest were running 3.14.0-rc5 with your sanity check patch. > > > > Regards, > > Jan > > > > > > > > For the latter, I don't see anything in particular filtering these > > > for a general MLD query wrong destination address in the IPv6 > > > code from igmp6_event_query() on. But I suspect that the query > > > doesn't even get that far on the kernel of the guest, as it is not > > > listening on ff02::1:ffaa:aaaa. Therefore the test with > > > "ff02::1:ff00:29", an address the guest is listening on, would be > > > interesting. > > > > > > If that works, then I'm going to make a patch ignore General MLD > > > Queries without ff02::1 as their destination address, too. > > > > > > > > > Hm, looking at more checks in igmp6_event_query(), I'm currently > > > wondering whether we should only enable the snooping behaviour in > > > the bridge when receiving a General MLD Query, so one with "::" in > > > the multicast field of the MLD message, instead of activating it > > > upon a Multicast-Address-Specific Query, too. That'd seem more > > > sane to me, I'm going to make a patch for that tomorrow. > > > > > > Cheers, Linus > > > > -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c index ef66365..8ccc0bf 100644 --- a/net/bridge/br_multicast.c +++ b/net/bridge/br_multicast.c @@ -1562,8 +1562,8 @@ static int br_multicast_ipv6_rcv(struct net_bridge *br, return 0; /* Prevent flooding this packet if there is no listener present */ - if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr)) - BR_INPUT_SKB_CB(skb)->mrouters_only = 1; +/* if (!ipv6_addr_is_ll_all_nodes(&ip6h->daddr)) + BR_INPUT_SKB_CB(skb)->mrouters_only = 1;*/ if (ip6h->nexthdr != IPPROTO_HOPOPTS || ip6h->payload_len == 0)
Hi, I have a host A running 3.14.0-rc5 with one KVM guest, where network is set up to use bridge: # brctl show bridge name bridge id STP enabled interfaces br1 8000.0023aeed1a00 no eno1 vnet0 virbr0 8000.000000000000 yes I'm seeing an issue where bridge (sometimes) stops forwarding ICMP6 neighbor solicitation packets to KVM guest and as result KVM guest doesn't respond with neighbor advertisement. I was trying to narrow down conditions when this happens and eventually came to following packet sent by other host B on same network: Ethernet II, Src: Hewlett-_47:93:1c (00:21:5a:47:93:1c), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) Destination: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) Address: IPv6mcast_00:00:00:01 (33:33:00:00:00:01) .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default) .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast) Source: Hewlett-_47:93:1c (00:21:5a:47:93:1c) Address: Hewlett-_47:93:1c (00:21:5a:47:93:1c) .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv6 (0x86dd) Internet Protocol Version 6, Src: 100:0:600:0:78fb:100:: (100:0:600:0:78fb:100::), Dst: ff02::1 (ff02::1) 0110 .... = Version: 6 [0110 .... = This field makes the filter "ip.version == 6" possible: 6] .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000 .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000) .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set .... .... ...0 .... .... .... .... .... = ECN-CE: Not set .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000 Payload length: 32 Next header: IPv6 hop-by-hop option (0) Hop limit: 1 Source: 100:0:600:0:78fb:100:: (100:0:600:0:78fb:100::) Destination: ff02::1 (ff02::1) Hop-by-Hop Option Next header: ICMPv6 (58) Length: 0 (8 bytes) IPv6 Option (Router Alert) Type: Router Alert (5) Length: 2 Router Alert: MLD (0) IPv6 Option (Pad1) Type: Pad1 (0) Pad1 IPv6 Option (Pad1) Type: Pad1 (0) Pad1 Internet Control Message Protocol v6 Type: Multicast Listener Query (130) Code: 0 Checksum: 0xf9c5 [correct] Maximum Response Delay [ms]: 1000 Reserved: 0000 Multicast Address: :: (::) The reason I think this packet is related is because when I send same exact packet I'm often hitting same issue - bridge stops forwarding ICMP6 neigh. solicitation packets to KVM guest. My current way to reproduce this is: 0. host B IP / MAC is: 2620:52:0:1040:221:5aff:fe47:931c / 00:21:5a:47:93:1c guest IP / MAC is: 2620:52:0:1040:5056:ff:fe00:29 / 52:56:00:00:00:29 1. host B is sending neigh solicit packets every 5 seconds with KVM guest IP using ns6 from ipv6toolkit: http://www.si6networks.com/tools/ipv6toolkit/ with parameters: --src-address=2620:52:0:1040:221:5aff:fe47:931c --dst-address=ff02::1:ff00:0029 -t 2620:52:0:1040:5056:ff:fe00:29 --link-src-address=00:21:5a:47:93:1c --source-lla-opt=00:21:5a:47:93:1c --link-dst-address=33:33:ff:00:00:29 tcpdump running on guest can see both solicit and advertisement packets 2. wait ~5 minutes 3. host B sends Multicast Listener Query packet described above 4. tcpdump running on guest is no longer seeing any neigh solicit packets I also found out that issue goes away if I do any of following: 1. disable CONFIG_BRIDGE_IGMP_SNOOPING=y 2. echo 0 > /sys/devices/virtual/net/br1/bridge/multicast_snooping 3. make following change: Change above makes me suspect following patch: commit 3c3769e63301fd92fcaf51870c371583dd0282ce Author: Linus Lüssing <linus.luessing@web.de> Date: Wed Sep 4 02:13:39 2013 +0200 bridge: apply multicast snooping to IPv6 link-local, too Please let me know if I can assist in any way to help resolve this. Regards, Jan -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html