From patchwork Tue Jun 14 14:34:13 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Koen Vandeputte X-Patchwork-Id: 635315 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2001:1868:205::9]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3rTXK53cN2z9t0q for ; Wed, 15 Jun 2016 00:36:41 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=ncentric.com header.i=@ncentric.com header.b=a1wcw25o; dkim-atps=neutral Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.80.1 #2 (Red Hat Linux)) id 1bCpQP-0007Bj-C2; Tue, 14 Jun 2016 14:34:41 +0000 Received: from mail-wm0-x231.google.com ([2a00:1450:400c:c09::231]) by bombadil.infradead.org with esmtps (Exim 4.80.1 #2 (Red Hat Linux)) id 1bCpQL-00078G-GM for lede-dev@lists.infradead.org; Tue, 14 Jun 2016 14:34:39 +0000 Received: by mail-wm0-x231.google.com with SMTP id m124so126049458wme.1 for ; Tue, 14 Jun 2016 07:34:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ncentric.com; s=google; h=subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding; bh=H00jBcJuMuw0MeKtr6ZcBaZh70ihfkD52FcA0GLTiIw=; b=a1wcw25o8yFXblGyQS35Lbviuz3fl3G/BJR6ZMakbHnaz/NMUALMJjjltnlAImPD4o WTNVa81A4pK0XkydNFG4h09im36a7vqeIRzuYPqYt05NtlUCQ1MtLhOGoEhD6yo62avs xVDZgBFqh9OTZDq2P4doNJ6+y+hX/I1AfmmNc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=H00jBcJuMuw0MeKtr6ZcBaZh70ihfkD52FcA0GLTiIw=; b=I255haLny0VyCXYHuN+gDlFzDxwAQ5wMKmpHJuGepebARiDtUH19RfPQcs+XOJZuqh ASgMBYDHGUospnKJtwxjXTlwhnKoXir0IXXHF6k0BqpPRgQgvTsqjRSsCnn0N4NCDWpN zVH+hoxTUCU/oRlSPJKYt6Xm+tcQajWJQ2lvcdatBI2dFNIKQ7zsrqwsYyBYMrXu6m2G 67TGwmCmh2iooFeV6Ktmic5SRA2KNOsmCd0phUEJoq6q3tRRqSeBmVUEHCfres2ZmhWi HKm1NuZhLMcsJXc7W7O4JpyE7nU+2E8bHxbFvBxL64gHAvRK8nSVeuWaFRaP2+espP60 R8bQ== X-Gm-Message-State: ALyK8tLyN3AwfdsFNC1PGmSpdXA3QZUcNs1BGYjSVLdIaT0bjDy6yuXEN59zRXIRBshF2g== X-Received: by 10.194.201.70 with SMTP id jy6mr7360889wjc.150.1465914854928; Tue, 14 Jun 2016 07:34:14 -0700 (PDT) Received: from [192.168.3.149] (d515300d8.static.telenet.be. [81.83.0.216]) by smtp.gmail.com with ESMTPSA id n66sm4461701wmn.7.2016.06.14.07.34.14 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 14 Jun 2016 07:34:14 -0700 (PDT) To: Tim Harvey References: <575EDB77.4040400@ncentric.com> From: Koen Vandeputte Message-ID: <576015E5.7060106@ncentric.com> Date: Tue, 14 Jun 2016 16:34:13 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.8.0 MIME-Version: 1.0 In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20160614_073438_049537_104B2AD9 X-CRM114-Status: GOOD ( 14.59 ) X-Spam-Score: -2.7 (--) X-Spam-Report: SpamAssassin version 3.4.0 on bombadil.infradead.org summary: Content analysis details: (-2.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [2a00:1450:400c:c09:0:0:0:231 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid Subject: Re: [LEDE-DEV] Bug - spinlock loop in cns3xxx_eth.c X-BeenThere: lede-dev@lists.infradead.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: lede-dev@lists.infradead.org, Felix Fietkau Sender: "Lede-dev" Errors-To: lede-dev-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Hi Tim, I fully understand the reason for the check. However, I think the bug has nothing to do with the rotting packet issue. The bug is actually: - eth_schedule_poll() gets called as it's detected that the ring is full after IRQ_enable. (cown flag is set in the previous slot of the ring) --> During this extra loop, _0_ packets are received/processed! The ring still seems to be full, and another reschedule takes place .. and so on .. Iv'e added 3 debug counters in the code: (see below after the patch) - Loop --> Top of eth_poll() - IRQ --> Top of eth_rx_irq() - Resched --> Inside the reschedule check Patch: spin_lock_bh(&tx_lock); When the bug triggers, you can see below that consecutive loops in eth_poll(), triggered by a reschedule, doesn't process any RX packets at all .. (As the ReSched counter increases linear with the loop counter and ReSched can only be incremented when "!received") Also Checking /proc/interrupts shows that gig_switch IRQ counter goes berserk (+10k IRQ/s) [ 477.440000] Loop: 835307 irq: 300583 ReSched: 1 [ 477.440000] Loop: 835308 irq: 300583 ReSched: 2 [ 477.440000] Loop: 835310 irq: 300583 ReSched: 3 [ 477.440000] Loop: 835311 irq: 300583 ReSched: 4 [ 477.440000] Loop: 835312 irq: 300583 ReSched: 5 [ 477.440000] Loop: 835313 irq: 300583 ReSched: 6 [ 477.440000] Loop: 835314 irq: 300583 ReSched: 7 [ 477.440000] Loop: 835315 irq: 300583 ReSched: 8 [ 477.440000] Loop: 835316 irq: 300583 ReSched: 9 [ 477.440000] Loop: 835317 irq: 300583 ReSched: 10 [ 477.440000] Loop: 835318 irq: 300583 ReSched: 11 [ 477.440000] Loop: 835319 irq: 300583 ReSched: 12 [ 477.440000] Loop: 835320 irq: 300583 ReSched: 13 [ 477.440000] Loop: 835321 irq: 300583 ReSched: 14 [ 477.440000] Loop: 835322 irq: 300583 ReSched: 15 [ 477.440000] Loop: 835323 irq: 300583 ReSched: 16 [ 477.440000] Loop: 835324 irq: 300583 ReSched: 17 [ 477.440000] Loop: 835325 irq: 300583 ReSched: 18 [ 477.440000] Loop: 835326 irq: 300583 ReSched: 19 [ 477.440000] Loop: 835327 irq: 300583 ReSched: 20 [ 477.440000] Loop: 835328 irq: 300583 ReSched: 21 [ 477.440000] Loop: 835329 irq: 300583 ReSched: 22 [ 477.440000] Loop: 835330 irq: 300583 ReSched: 23 [ 477.440000] Loop: 835331 irq: 300583 ReSched: 24 [ 477.440000] Loop: 835332 irq: 300583 ReSched: 25 [ 477.440000] Loop: 835333 irq: 300583 ReSched: 26 [ 477.440000] Loop: 835334 irq: 300583 ReSched: 27 [ 477.440000] Loop: 835335 irq: 300583 ReSched: 28 [ 477.440000] Loop: 835336 irq: 300583 ReSched: 29 [ 477.440000] Loop: 835337 irq: 300583 ReSched: 30 [ 477.440000] Loop: 835338 irq: 300583 ReSched: 31 [ 477.440000] Loop: 835339 irq: 300583 ReSched: 32 [ 477.440000] Loop: 835340 irq: 300583 ReSched: 33 [ 477.440000] Loop: 835341 irq: 300583 ReSched: 34 [ 477.440000] Loop: 835342 irq: 300583 ReSched: 35 [ 477.440000] Loop: 835343 irq: 300583 ReSched: 36 [ 477.440000] Loop: 835344 irq: 300583 ReSched: 37 [ 477.440000] Loop: 835345 irq: 300583 ReSched: 38 [ 477.440000] Loop: 835346 irq: 300583 ReSched: 39 [ 477.440000] Loop: 835347 irq: 300583 ReSched: 40 [ 477.440000] Loop: 835348 irq: 300583 ReSched: 41 [ 477.440000] Loop: 835349 irq: 300583 ReSched: 42 [ 477.440000] Loop: 835350 irq: 300583 ReSched: 43 [ 477.440000] Loop: 835351 irq: 300583 ReSched: 44 [ 477.440000] Loop: 835352 irq: 300583 ReSched: 45 [ 477.440000] Loop: 835353 irq: 300583 ReSched: 46 [ 477.440000] Loop: 835354 irq: 300583 ReSched: 47 [ 477.440000] Loop: 835355 irq: 300583 ReSched: 48 [ 477.440000] Loop: 835356 irq: 300583 ReSched: 49 [ 477.440000] Loop: 835357 irq: 300583 ReSched: 50 [ 477.440000] Loop: 835358 irq: 300583 ReSched: 51 [ 477.440000] Loop: 835359 irq: 300583 ReSched: 52 [ 477.440000] Loop: 835360 irq: 300583 ReSched: 53 [ 477.440000] Loop: 835361 irq: 300583 ReSched: 54 [ 477.440000] Loop: 835362 irq: 300583 ReSched: 55 [ 477.440000] Loop: 835363 irq: 300583 ReSched: 56 [ 477.440000] Loop: 835364 irq: 300583 ReSched: 57 [ 477.440000] Loop: 835365 irq: 300583 ReSched: 58 [ 477.440000] Loop: 835366 irq: 300583 ReSched: 59 [ 477.440000] Loop: 835367 irq: 300583 ReSched: 60 [ 477.440000] Loop: 835368 irq: 300583 ReSched: 61 [ 477.440000] Loop: 835369 irq: 300583 ReSched: 62 [ 477.440000] Loop: 835370 irq: 300583 ReSched: 63 [ 477.440000] Loop: 835371 irq: 300583 ReSched: 64 [ 477.440000] Loop: 835372 irq: 300583 ReSched: 65 [ 477.440000] Loop: 835373 irq: 300583 ReSched: 66 [ 477.440000] Loop: 835374 irq: 300583 ReSched: 67 [ 477.440000] Loop: 835375 irq: 300583 ReSched: 68 [ 477.440000] Loop: 835376 irq: 300583 ReSched: 69 [ 477.440000] Loop: 835377 irq: 300583 ReSched: 70 [ 477.440000] Loop: 835378 irq: 300583 ReSched: 71 [ 477.440000] Loop: 835379 irq: 300583 ReSched: 72 [ 477.440000] Loop: 835380 irq: 300583 ReSched: 73 [ 477.440000] Loop: 835381 irq: 300583 ReSched: 74 [ 477.440000] Loop: 835382 irq: 300583 ReSched: 75 [ 477.440000] Loop: 835383 irq: 300583 ReSched: 76 [ 477.440000] Loop: 835384 irq: 300583 ReSched: 77 [ 477.440000] Loop: 835385 irq: 300583 ReSched: 78 [ 477.440000] Loop: 835386 irq: 300583 ReSched: 79 [ 477.440000] Loop: 835387 irq: 300583 ReSched: 80 [ 477.440000] Loop: 835388 irq: 300583 ReSched: 81 [ 477.440000] Loop: 835389 irq: 300583 ReSched: 82 [ 477.440000] Loop: 835390 irq: 300583 ReSched: 83 [ 477.440000] Loop: 835391 irq: 300583 ReSched: 84 [ 477.440000] Loop: 835392 irq: 300583 ReSched: 85 [ 477.440000] Loop: 835393 irq: 300583 ReSched: 86 [ 477.440000] Loop: 835394 irq: 300583 ReSched: 87 [ 477.440000] Loop: 835396 irq: 300583 ReSched: 88 [ 477.440000] Loop: 835397 irq: 300583 ReSched: 89 [ 477.440000] Loop: 835398 irq: 300583 ReSched: 90 [ 477.440000] Loop: 835399 irq: 300583 ReSched: 91 [ 477.440000] Loop: 835400 irq: 300583 ReSched: 92 [ 477.440000] Loop: 835403 irq: 300583 ReSched: 93 [ 477.440000] Loop: 835406 irq: 300583 ReSched: 94 [ 477.440000] Loop: 835407 irq: 300583 ReSched: 95 [ 477.440000] Loop: 835408 irq: 300583 ReSched: 96 [ 477.440000] Loop: 835409 irq: 300583 ReSched: 97 [ 477.440000] Loop: 835410 irq: 300583 ReSched: 98 [ 477.440000] Loop: 835411 irq: 300583 ReSched: 99 [ 477.440000] Loop: 835412 irq: 300583 ReSched: 100 [ 477.440000] Loop: 835413 irq: 300583 ReSched: 101 [ 477.440000] Loop: 835414 irq: 300583 ReSched: 102 [ 477.440000] Loop: 835415 irq: 300583 ReSched: 103 [ 477.440000] Loop: 835416 irq: 300583 ReSched: 104 [ 477.440000] Loop: 835417 irq: 300583 ReSched: 105 [ 477.440000] Loop: 835418 irq: 300583 ReSched: 106 [ 477.440000] Loop: 835419 irq: 300583 ReSched: 107 [ 477.440000] Loop: 835420 irq: 300583 ReSched: 108 [ 477.440000] Loop: 835421 irq: 300583 ReSched: 109 [ 477.440000] Loop: 835422 irq: 300583 ReSched: 110 [ 477.440000] Loop: 835423 irq: 300583 ReSched: 111 [ 477.440000] Loop: 835424 irq: 300583 ReSched: 112 [ 477.440000] Loop: 835425 irq: 300583 ReSched: 113 [ 477.440000] Loop: 835426 irq: 300583 ReSched: 114 [ 477.440000] Loop: 835427 irq: 300583 ReSched: 115 [ 477.440000] Loop: 835428 irq: 300583 ReSched: 116 [ 477.440000] Loop: 835429 irq: 300583 ReSched: 117 [ 477.440000] Loop: 835430 irq: 300583 ReSched: 118 [ 477.440000] Loop: 835431 irq: 300583 ReSched: 119 [ 477.440000] Loop: 835432 irq: 300583 ReSched: 120 [ 477.440000] Loop: 835433 irq: 300583 ReSched: 121 [ 477.440000] Loop: 835434 irq: 300583 ReSched: 122 [ 477.440000] Loop: 835435 irq: 300583 ReSched: 123 [ 477.440000] Loop: 835436 irq: 300583 ReSched: 124 [ 477.440000] Loop: 835437 irq: 300583 ReSched: 125 [ 477.440000] Loop: 835438 irq: 300583 ReSched: 126 [ 477.440000] Loop: 835439 irq: 300583 ReSched: 127 [ 477.440000] Loop: 835440 irq: 300583 ReSched: 128 [ 477.440000] Loop: 835441 irq: 300583 ReSched: 129 [ 477.440000] Loop: 835442 irq: 300583 ReSched: 130 [ 477.440000] Loop: 835443 irq: 300583 ReSched: 131 [ 477.440000] Loop: 835444 irq: 300583 ReSched: 132 [ 477.440000] Loop: 835445 irq: 300583 ReSched: 133 [ 477.440000] Loop: 835446 irq: 300583 ReSched: 134 [ 477.440000] Loop: 835447 irq: 300583 ReSched: 135 [ 477.440000] Loop: 835448 irq: 300583 ReSched: 136 [ 477.440000] Loop: 835449 irq: 300583 ReSched: 137 [ 477.440000] Loop: 835450 irq: 300583 ReSched: 138 [ 477.450000] Loop: 835451 irq: 300583 ReSched: 139 [ 477.450000] Loop: 835452 irq: 300583 ReSched: 140 [ 477.450000] Loop: 835453 irq: 300583 ReSched: 141 [ 477.450000] Loop: 835454 irq: 300583 ReSched: 142 [ 477.450000] Loop: 835455 irq: 300583 ReSched: 143 [ 477.450000] Loop: 835456 irq: 300583 ReSched: 144 [ 477.450000] Loop: 835457 irq: 300583 ReSched: 145 [ 477.450000] Loop: 835458 irq: 300583 ReSched: 146 [ 477.450000] Loop: 835459 irq: 300583 ReSched: 147 [ 477.450000] Loop: 835460 irq: 300583 ReSched: 148 [ 477.450000] Loop: 835461 irq: 300583 ReSched: 149 [ 477.450000] Loop: 835462 irq: 300583 ReSched: 150 [ 477.450000] Loop: 835463 irq: 300583 ReSched: 151 [ 477.450000] Loop: 835464 irq: 300583 ReSched: 152 [ 477.450000] Loop: 835465 irq: 300583 ReSched: 153 [ 477.450000] Loop: 835466 irq: 300583 ReSched: 154 [ 477.450000] Loop: 835467 irq: 300583 ReSched: 155 [ 477.450000] Loop: 835468 irq: 300583 ReSched: 156 [ 477.450000] Loop: 835469 irq: 300583 ReSched: 157 [ 477.450000] Loop: 835470 irq: 300583 ReSched: 158 [ 477.450000] Loop: 835472 irq: 300583 ReSched: 159 [ 477.450000] Loop: 835473 irq: 300583 ReSched: 160 [ 477.450000] Loop: 835474 irq: 300583 ReSched: 161 [ 477.450000] Loop: 835475 irq: 300583 ReSched: 162 [ 477.450000] Loop: 835476 irq: 300583 ReSched: 163 [ 477.450000] Loop: 835477 irq: 300583 ReSched: 164 [ 477.450000] Loop: 835478 irq: 300583 ReSched: 165 [ 477.450000] Loop: 835479 irq: 300583 ReSched: 166 [ 477.450000] Loop: 835480 irq: 300583 ReSched: 167 [ 477.450000] Loop: 835481 irq: 300583 ReSched: 168 [ 477.450000] Loop: 835482 irq: 300583 ReSched: 169 [ 477.450000] Loop: 835483 irq: 300583 ReSched: 170 [ 477.450000] Loop: 835484 irq: 300583 ReSched: 171 [ 477.450000] Loop: 835485 irq: 300583 ReSched: 172 [ 477.450000] Loop: 835486 irq: 300583 ReSched: 173 [ 477.450000] Loop: 835487 irq: 300583 ReSched: 174 [ 477.450000] Loop: 835488 irq: 300583 ReSched: 175 [ 477.450000] Loop: 835489 irq: 300583 ReSched: 176 [ 477.450000] Loop: 835490 irq: 300583 ReSched: 177 [ 477.450000] Loop: 835491 irq: 300583 ReSched: 178 [ 477.450000] Loop: 835492 irq: 300583 ReSched: 179 [ 477.450000] Loop: 835493 irq: 300583 ReSched: 180 [ 477.450000] Loop: 835494 irq: 300583 ReSched: 181 [ 477.450000] Loop: 835495 irq: 300583 ReSched: 182 [ 477.450000] Loop: 835496 irq: 300583 ReSched: 183 [ 477.450000] Loop: 835497 irq: 300583 ReSched: 184 [ 477.450000] Loop: 835498 irq: 300583 ReSched: 185 [ 477.450000] Loop: 835499 irq: 300583 ReSched: 186 [ 477.450000] Loop: 835500 irq: 300583 ReSched: 187 [ 477.450000] Loop: 835501 irq: 300583 ReSched: 188 [ 477.450000] Loop: 835502 irq: 300583 ReSched: 189 [ 477.450000] Loop: 835503 irq: 300583 ReSched: 190 [ 477.450000] Loop: 835504 irq: 300583 ReSched: 191 [ 477.450000] Loop: 835505 irq: 300583 ReSched: 192 [ 477.450000] Loop: 835506 irq: 300583 ReSched: 193 [ 477.450000] Loop: 835507 irq: 300583 ReSched: 194 [ 477.450000] Loop: 835508 irq: 300583 ReSched: 195 [ 477.450000] Loop: 835509 irq: 300583 ReSched: 196 [ 477.450000] Loop: 835510 irq: 300583 ReSched: 197 [ 477.450000] Loop: 835511 irq: 300583 ReSched: 198 [ 477.450000] Loop: 835512 irq: 300583 ReSched: 199 [ 477.450000] Loop: 835513 irq: 300583 ReSched: 200 [ 477.450000] Loop: 835514 irq: 300583 ReSched: 201 [ 477.450000] Loop: 835515 irq: 300583 ReSched: 202 [ 477.450000] Loop: 835516 irq: 300583 ReSched: 203 [ 477.450000] Loop: 835517 irq: 300583 ReSched: 204 [ 477.450000] Loop: 835518 irq: 300583 ReSched: 205 [ 477.450000] Loop: 835519 irq: 300583 ReSched: 206 [ 477.450000] Loop: 835520 irq: 300583 ReSched: 207 [ 477.450000] Loop: 835521 irq: 300583 ReSched: 208 [ 477.450000] Loop: 835522 irq: 300583 ReSched: 209 [ 477.450000] Loop: 835523 irq: 300583 ReSched: 210 [ 477.450000] Loop: 835524 irq: 300583 ReSched: 211 [ 477.450000] Loop: 835525 irq: 300583 ReSched: 212 [ 477.450000] Loop: 835526 irq: 300583 ReSched: 213 [ 477.450000] Loop: 835527 irq: 300583 ReSched: 214 [ 477.450000] Loop: 835528 irq: 300583 ReSched: 215 [ 477.450000] Loop: 835529 irq: 300583 ReSched: 216 [ 477.450000] Loop: 835530 irq: 300583 ReSched: 217 [ 478.450000] Loop: 835534 irq: 300584 ReSched: 218 [ 478.460000] Loop: 835536 irq: 300585 ReSched: 219 [ 478.470000] Loop: 835538 irq: 300586 ReSched: 220 [ 478.470000] Loop: 835539 irq: 300586 ReSched: 221 [ 478.500000] Loop: 835983 irq: 300586 ReSched: 622 ... [ 478.510000] Loop: 836072 irq: 300586 ReSched: 697 [ 478.510000] Loop: 836073 irq: 300586 ReSched: 698 [ 478.510000] Loop: 836074 irq: 300586 ReSched: 699 [ 478.510000] Loop: 836075 irq: 300586 ReSched: 700 [ 478.510000] Loop: 836076 irq: 300586 ReSched: 701 [ 478.510000] Loop: 836077 irq: 300586 ReSched: 702 [ 478.510000] Loop: 836078 irq: 300586 ReSched: 703 [ 478.510000] Loop: 836079 irq: 300586 ReSched: 704 [ 478.510000] Loop: 836080 irq: 300586 ReSched: 705 [ 478.510000] Loop: 836081 irq: 300586 ReSched: 706 [ 478.510000] Loop: 836082 irq: 300586 ReSched: 707 [ 478.510000] Loop: 836083 irq: 300586 ReSched: 708 [ 478.510000] Loop: 836084 irq: 300586 ReSched: 709 [ 478.510000] Loop: 836085 irq: 300586 ReSched: 710 [ 478.510000] Loop: 836086 irq: 300586 ReSched: 711 [ 478.510000] Loop: 836087 irq: 300586 ReSched: 712 [ 478.510000] Loop: 836088 irq: 300586 ReSched: 713 [ 478.510000] Loop: 836089 irq: 300586 ReSched: 714 [ 478.510000] Loop: 836090 irq: 300586 ReSched: 715 [ 478.510000] Loop: 836091 irq: 300586 ReSched: 716 [ 478.510000] Loop: 836092 irq: 300586 ReSched: 717 [ 478.510000] Loop: 836093 irq: 300586 ReSched: 718 [ 478.510000] Loop: 836094 irq: 300586 ReSched: 719 [ 478.510000] Loop: 836095 irq: 300586 ReSched: 720 [ 478.510000] Loop: 836096 irq: 300586 ReSched: 721 [ 478.510000] Loop: 836097 irq: 300586 ReSched: 722 [ 478.510000] Loop: 836098 irq: 300586 ReSched: 723 [ 478.510000] Loop: 836099 irq: 300586 ReSched: 724 [ 478.510000] Loop: 836100 irq: 300586 ReSched: 725 [ 478.510000] Loop: 836101 irq: 300586 ReSched: 726 [ 478.510000] Loop: 836102 irq: 300586 ReSched: 727 [ 478.510000] Loop: 836103 irq: 300586 ReSched: 728 [ 478.510000] Loop: 836104 irq: 300586 ReSched: 729 [ 478.510000] Loop: 836105 irq: 300586 ReSched: 730 [ 478.510000] Loop: 836106 irq: 300586 ReSched: 731 [ 478.510000] Loop: 836107 irq: 300586 ReSched: 732 [ 478.510000] Loop: 836108 irq: 300586 ReSched: 733 [ 478.510000] Loop: 836109 irq: 300586 ReSched: 734 [ 478.510000] Loop: 836110 irq: 300586 ReSched: 735 [ 478.510000] Loop: 836111 irq: 300586 ReSched: 736 [ 478.510000] Loop: 836112 irq: 300586 ReSched: 737 [ 478.510000] Loop: 836113 irq: 300586 ReSched: 738 [ 478.510000] Loop: 836114 irq: 300586 ReSched: 739 [ 478.510000] Loop: 836115 irq: 300586 ReSched: 740 [ 478.510000] Loop: 836116 irq: 300586 ReSched: 741 [ 478.510000] Loop: 836117 irq: 300586 ReSched: 742 [ 478.510000] Loop: 836118 irq: 300586 ReSched: 743 [ 478.510000] Loop: 836119 irq: 300586 ReSched: 744 [ 478.510000] Loop: 836120 irq: 300586 ReSched: 745 [ 478.510000] Loop: 836121 irq: 300586 ReSched: 746 [ 478.510000] Loop: 836122 irq: 300586 ReSched: 747 [ 478.510000] Loop: 836123 irq: 300586 ReSched: 748 [ 478.510000] Loop: 836124 irq: 300586 ReSched: 749 [ 478.510000] Loop: 836125 irq: 300586 ReSched: 750 [ 478.510000] Loop: 836126 irq: 300586 ReSched: 751 [ 478.510000] Loop: 836127 irq: 300586 ReSched: 752 [ 478.510000] Loop: 836128 irq: 300586 ReSched: 753 [ 478.510000] Loop: 836129 irq: 300586 ReSched: 754 [ 478.510000] Loop: 836130 irq: 300586 ReSched: 755 [ 478.510000] Loop: 836131 irq: 300586 ReSched: 756 [ 478.510000] Loop: 836132 irq: 300586 ReSched: 757 [ 478.510000] Loop: 836133 irq: 300586 ReSched: 758 [ 478.510000] Loop: 836134 irq: 300586 ReSched: 759 Koen On 2016-06-13 20:17, Tim Harvey wrote: > On Mon, Jun 13, 2016 at 9:12 AM, Koen Vandeputte > wrote: >> Hi All, >> >> There seems to be a bug in the function eth_poll() in this driver >> >> When the RX ring gets full once, the re-schedule is called forever, even >> when the ring is empty afterwards. >> >> >> if (!received) { >> napi_complete(napi); >> enable_irq(sw->rx_irq); >> budget = 0; >> >> /* if rx descriptors are full schedule another poll */ >> if (rx_ring->desc[(i-1) & (RX_DESCS-1)].cown) >> { >> eth_schedule_poll(sw); <---- Gets called on each function >> entry >> } >> } >> >> >> This causes SoftIRQ to fully load a core forever. >> >> >> I didn't fix it yet, but should I be the first, i'll supply a patch .. >> > Koen, > > We have seen this before, but admittedly don't understand why we enter > into the same condition on each subsequent call to eth_poll(). The > check is to catch the condition described as irq rot [1] and is to > catch the case where after processing the full budget, we are > immediately full again (a situation which is easily re-producible with > a flood-ping). If this occurs we will no longer get an rx interrupt > (because the descriptors are full) and our napi function will never > get called again (unless transmitting packets). > > What is your proposed patch? > > Tim > > 1 .http://www.linuxfoundation.org/collaborate/workgroups/networking/napi#IRQ_race_a.k.a_rotting_packet --- a/drivers/net/ethernet/cavium/cns3xxx_eth.c +++ b/drivers/net/ethernet/cavium/cns3xxx_eth.c @@ -308,6 +308,8 @@ static int ports_open; static struct port *switch_port_tab[4]; struct net_device *napi_dev; +static int CntIRQ; + static int cns3xxx_mdio_cmd(struct mii_bus *bus, int phy_id, int location, int write, u16 cmd) { @@ -437,7 +439,10 @@ static void cns3xxx_adjust_link(struct n static void eth_schedule_poll(struct sw *sw) { if (unlikely(!napi_schedule_prep(&sw->napi))) + { + //printk(KERN_ERR "sched err\n"); return; + } disable_irq_nosync(sw->rx_irq); __napi_schedule(&sw->napi); @@ -447,6 +452,7 @@ irqreturn_t eth_rx_irq(int irq, void *pd { struct net_device *dev = pdev; struct sw *sw = netdev_priv(dev); + CntIRQ++; eth_schedule_poll(sw); return (IRQ_HANDLED); } @@ -617,6 +623,7 @@ static void eth_complete_tx(struct sw *s static int eth_poll(struct napi_struct *napi, int budget) { + static int Loop = 0; struct sw *sw = container_of(napi, struct sw, napi); struct _rx_ring *rx_ring = &sw->rx_ring; int received = 0; @@ -712,6 +719,9 @@ static int eth_poll(struct napi_struct * } rx_ring->cur_index = i; + + Loop++; + if (!received) { napi_complete(napi); enable_irq(sw->rx_irq); @@ -719,7 +729,12 @@ static int eth_poll(struct napi_struct * /* if rx descriptors are full schedule another poll */ if (rx_ring->desc[(i-1) & (RX_DESCS-1)].cown) + { + static int Sched = 0; + + printk(KERN_ERR "Loop: %d irq: %d ReSched: %d\n", Loop, CntIRQ, ++Sched); eth_schedule_poll(sw); + } }