| Message ID | 1389519069-1619-3-git-send-email-w@1wt.eu |
|---|---|
| State | Changes Requested, archived |
| Delegated to: | David Miller |
| Headers | show |
On Sun, 2014-01-12 at 10:31 +0100, Willy Tarreau wrote: > Stats writers are mvneta_rx() and mvneta_tx(). They don't lock anything > when they update the stats, and as a result, it randomly happens that > the stats freeze on SMP if two updates happen during stats retrieval. Your patch is OK, but I dont understand how this freeze can happen. TX and RX uses a separate syncp, and TX is protected by a lock, RX is protected by NAPI bit. Stats retrieval uses the appropriate BH disable before the fetches... > This is very easily reproducible by starting two HTTP servers and binding > each of them to a different CPU, then consulting /proc/net/dev in loops > during transfers, the interface should immediately lock up. This issue > also randomly happens upon link state changes during transfers, because > the stats are collected in this situation, but it takes more attempts to > reproduce it. -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Eric! On Sun, Jan 12, 2014 at 10:07:36AM -0800, Eric Dumazet wrote: > On Sun, 2014-01-12 at 10:31 +0100, Willy Tarreau wrote: > > Stats writers are mvneta_rx() and mvneta_tx(). They don't lock anything > > when they update the stats, and as a result, it randomly happens that > > the stats freeze on SMP if two updates happen during stats retrieval. > > Your patch is OK, but I dont understand how this freeze can happen. > > TX and RX uses a separate syncp, and TX is protected by a lock, RX > is protected by NAPI bit. But we can have multiple tx in parallel, one per queue. And it's only when I explicitly bind two servers to two distinct CPU cores that I can trigger the issue, which seems to confirm that this is the cause of the issue. > Stats retrieval uses the appropriate BH disable before the fetches... From the numerous printks I have added inside the syncp blocks, it appears that the stats themselves are not responsible for the issue, but the concurrent Tx are. I ended up several times stuck if I had two Tx on different CPUs right before a stats retrieval. From the info I found on the syncp docs, the caller is responsible for locking and I don't see where there's any lock here since the syncp are global and not even per tx queue. But this stuff is very new to me, I can have missed something. That said, I'm quite certain that the lock happened within the syncp blocks and only in this case! At least my reading of the relevant includes seemed to confirm to me that this hypothesis was valid :-/ Thanks, Willy -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, 2014-01-12 at 23:09 +0100, Willy Tarreau wrote: > But we can have multiple tx in parallel, one per queue. And it's only > when I explicitly bind two servers to two distinct CPU cores that I > can trigger the issue, which seems to confirm that this is the cause > of the issue. So this driver has multiqueue ? Definitely it should have one syncp per queue. Or per cpu stats, as your patch did. Thanks ! -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, 2014-01-12 at 10:31 +0100, Willy Tarreau wrote: > This patch implements this. It merges both rx_stats and tx_stats into > a single "stats" member with a single syncp. Both mvneta_rx() and > mvneta_rx() now only update the a single CPU's counters. Reviewed-by: Eric Dumazet <edumazet@google.com> -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Jan 12, 2014 at 04:45:03PM -0800, Eric Dumazet wrote: > On Sun, 2014-01-12 at 23:09 +0100, Willy Tarreau wrote: > > > But we can have multiple tx in parallel, one per queue. And it's only > > when I explicitly bind two servers to two distinct CPU cores that I > > can trigger the issue, which seems to confirm that this is the cause > > of the issue. > > So this driver has multiqueue ? Yes, it defaults to 8 queues in each direction. > Definitely it should have one syncp per queue. > > Or per cpu stats, as your patch did. OK thank you for your review and explanation then, I'm reassured :-) Thanks, Willy -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c index baa85af..40d3e8b 100644 --- a/drivers/net/ethernet/marvell/mvneta.c +++ b/drivers/net/ethernet/marvell/mvneta.c @@ -221,10 +221,12 @@ #define MVNETA_RX_BUF_SIZE(pkt_size) ((pkt_size) + NET_SKB_PAD) -struct mvneta_stats { +struct mvneta_pcpu_stats { struct u64_stats_sync syncp; - u64 packets; - u64 bytes; + u64 rx_packets; + u64 rx_bytes; + u64 tx_packets; + u64 tx_bytes; }; struct mvneta_port { @@ -250,8 +252,7 @@ struct mvneta_port { u8 mcast_count[256]; u16 tx_ring_size; u16 rx_ring_size; - struct mvneta_stats tx_stats; - struct mvneta_stats rx_stats; + struct mvneta_pcpu_stats *stats; struct mii_bus *mii_bus; struct phy_device *phy_dev; @@ -461,21 +462,29 @@ struct rtnl_link_stats64 *mvneta_get_stats64(struct net_device *dev, { struct mvneta_port *pp = netdev_priv(dev); unsigned int start; + int cpu; - memset(stats, 0, sizeof(struct rtnl_link_stats64)); - - do { - start = u64_stats_fetch_begin_bh(&pp->rx_stats.syncp); - stats->rx_packets = pp->rx_stats.packets; - stats->rx_bytes = pp->rx_stats.bytes; - } while (u64_stats_fetch_retry_bh(&pp->rx_stats.syncp, start)); + for_each_possible_cpu(cpu) { + struct mvneta_pcpu_stats *cpu_stats; + u64 rx_packets; + u64 rx_bytes; + u64 tx_packets; + u64 tx_bytes; + cpu_stats = per_cpu_ptr(pp->stats, cpu); + do { + start = u64_stats_fetch_begin_bh(&cpu_stats->syncp); + rx_packets = cpu_stats->rx_packets; + rx_bytes = cpu_stats->rx_bytes; + tx_packets = cpu_stats->tx_packets; + tx_bytes = cpu_stats->tx_bytes; + } while (u64_stats_fetch_retry_bh(&cpu_stats->syncp, start)); - do { - start = u64_stats_fetch_begin_bh(&pp->tx_stats.syncp); - stats->tx_packets = pp->tx_stats.packets; - stats->tx_bytes = pp->tx_stats.bytes; - } while (u64_stats_fetch_retry_bh(&pp->tx_stats.syncp, start)); + stats->rx_packets += rx_packets; + stats->rx_bytes += rx_bytes; + stats->tx_packets += tx_packets; + stats->tx_bytes += tx_bytes; + } stats->rx_errors = dev->stats.rx_errors; stats->rx_dropped = dev->stats.rx_dropped; @@ -1453,10 +1462,12 @@ static int mvneta_rx(struct mvneta_port *pp, int rx_todo, } if (rcvd_pkts) { - u64_stats_update_begin(&pp->rx_stats.syncp); - pp->rx_stats.packets += rcvd_pkts; - pp->rx_stats.bytes += rcvd_bytes; - u64_stats_update_end(&pp->rx_stats.syncp); + struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats); + + u64_stats_update_begin(&stats->syncp); + stats->rx_packets += rcvd_pkts; + stats->rx_bytes += rcvd_bytes; + u64_stats_update_end(&stats->syncp); } /* Update rxq management counters */ @@ -1589,11 +1600,12 @@ static int mvneta_tx(struct sk_buff *skb, struct net_device *dev) out: if (frags > 0) { - u64_stats_update_begin(&pp->tx_stats.syncp); - pp->tx_stats.packets++; - pp->tx_stats.bytes += skb->len; - u64_stats_update_end(&pp->tx_stats.syncp); + struct mvneta_pcpu_stats *stats = this_cpu_ptr(pp->stats); + u64_stats_update_begin(&stats->syncp); + stats->tx_packets++; + stats->tx_bytes += skb->len; + u64_stats_update_end(&stats->syncp); } else { dev->stats.tx_dropped++; dev_kfree_skb_any(skb); @@ -2758,6 +2770,7 @@ static int mvneta_probe(struct platform_device *pdev) const char *mac_from; int phy_mode; int err; + int cpu; /* Our multiqueue support is not complete, so for now, only * allow the usage of the first RX queue @@ -2799,9 +2812,6 @@ static int mvneta_probe(struct platform_device *pdev) pp = netdev_priv(dev); - u64_stats_init(&pp->tx_stats.syncp); - u64_stats_init(&pp->rx_stats.syncp); - pp->weight = MVNETA_RX_POLL_WEIGHT; pp->phy_node = phy_node; pp->phy_interface = phy_mode; @@ -2820,6 +2830,19 @@ static int mvneta_probe(struct platform_device *pdev) goto err_clk; } + /* Alloc per-cpu stats */ + pp->stats = alloc_percpu(struct mvneta_pcpu_stats); + if (!pp->stats) { + err = -ENOMEM; + goto err_unmap; + } + + for_each_possible_cpu(cpu) { + struct mvneta_pcpu_stats *stats; + stats = per_cpu_ptr(pp->stats, cpu); + u64_stats_init(&stats->syncp); + } + dt_mac_addr = of_get_mac_address(dn); if (dt_mac_addr) { mac_from = "device tree"; @@ -2849,7 +2872,7 @@ static int mvneta_probe(struct platform_device *pdev) err = mvneta_init(pp, phy_addr); if (err < 0) { dev_err(&pdev->dev, "can't init eth hal\n"); - goto err_unmap; + goto err_free_stats; } mvneta_port_power_up(pp, phy_mode); @@ -2879,6 +2902,8 @@ static int mvneta_probe(struct platform_device *pdev) err_deinit: mvneta_deinit(pp); +err_free_stats: + free_percpu(pp->stats); err_unmap: iounmap(pp->base); err_clk: @@ -2899,6 +2924,7 @@ static int mvneta_remove(struct platform_device *pdev) unregister_netdev(dev); mvneta_deinit(pp); clk_disable_unprepare(pp->clk); + free_percpu(pp->stats); iounmap(pp->base); irq_dispose_mapping(dev->irq); free_netdev(dev);
Stats writers are mvneta_rx() and mvneta_tx(). They don't lock anything when they update the stats, and as a result, it randomly happens that the stats freeze on SMP if two updates happen during stats retrieval. This is very easily reproducible by starting two HTTP servers and binding each of them to a different CPU, then consulting /proc/net/dev in loops during transfers, the interface should immediately lock up. This issue also randomly happens upon link state changes during transfers, because the stats are collected in this situation, but it takes more attempts to reproduce it. The comments in netdevice.h suggest using per_cpu stats instead to get rid of this issue. This patch implements this. It merges both rx_stats and tx_stats into a single "stats" member with a single syncp. Both mvneta_rx() and mvneta_rx() now only update the a single CPU's counters. In turn, mvneta_get_stats64() does the summing by iterating over all CPUs to get their respective stats. With this change, stats are still correct and no more lockup is encountered. Note that this bug was present since the first import of the mvneta driver. It might make sense to backport it to some stable trees. If so, it depends on "d33dc73 net: mvneta: increase the 64-bit rx/tx stats out of the hot path". Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> Cc: Gregory CLEMENT <gregory.clement@free-electrons.com> Signed-off-by: Willy Tarreau <w@1wt.eu> --- drivers/net/ethernet/marvell/mvneta.c | 84 +++++++++++++++++++++++------------ 1 file changed, 55 insertions(+), 29 deletions(-)