From patchwork Thu May 5 14:56:02 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eric Dumazet X-Patchwork-Id: 94282 Return-Path: X-Original-To: incoming-imx@patchwork.ozlabs.org Delivered-To: patchwork-incoming-imx@bilbo.ozlabs.org Received: from bombadil.infradead.org (bombadil.infradead.org [18.85.46.34]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id B9E5BB6FDE for ; Fri, 6 May 2011 00:56:16 +1000 (EST) Received: from canuck.infradead.org ([2001:4978:20e::1]) by bombadil.infradead.org with esmtps (Exim 4.72 #1 (Red Hat Linux)) id 1QHzyP-0003Dw-J8; Thu, 05 May 2011 14:56:13 +0000 Received: from localhost ([127.0.0.1] helo=canuck.infradead.org) by canuck.infradead.org with esmtp (Exim 4.72 #1 (Red Hat Linux)) id 1QHzyO-0006p2-3Y; Thu, 05 May 2011 14:56:12 +0000 Received: from mail-wy0-f177.google.com ([74.125.82.177]) by canuck.infradead.org with esmtps (Exim 4.72 #1 (Red Hat Linux)) id 1QHzyJ-0006oj-A6 for linux-arm-kernel@lists.infradead.org; Thu, 05 May 2011 14:56:08 +0000 Received: by wyb28 with SMTP id 28so2144228wyb.36 for ; Thu, 05 May 2011 07:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:from:to:cc:in-reply-to:references :content-type:date:message-id:mime-version:x-mailer :content-transfer-encoding; bh=cuqeFj0EEqnuKkLyQRJPBSgDIt3spzgO2lkX/iYj/VM=; b=Xo5RvcAiPboIdi475LQFNQ/EUNSd0QsOkUnrM3RiMI35GL9cRblVCwNhPkHq1exYmN H66JLF+rxJWhQCWHj6p5ZPHyy5h/lfalV/Z6r8ProqRPpE+JGz4WO3DAFOEm9ubKZwdW fRjVIvK3HzOKkzRdD+me7JWuly+lw/i+O0PX0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:from:to:cc:in-reply-to:references:content-type:date :message-id:mime-version:x-mailer:content-transfer-encoding; b=vNDZ/3deUx3010ACg26+soVVVu+HIw389lMzYK0lPxlywxc+ORLX8pO4uFtl8fOsXp rh2lDMZ6cOUh2iN7JU+R3m8VGts+SU2OiHCP+BlOGyx9xSN5Bu6TWcpvrJdHxCV0bEE4 Dv8otI5efpqvDleZgvNNoeqVjL553LZiep15E= Received: by 10.216.82.16 with SMTP id n16mr2580473wee.13.1304607365417; Thu, 05 May 2011 07:56:05 -0700 (PDT) Received: from [10.150.51.218] (gw0.net.jmsp.net [212.23.165.14]) by mx.google.com with ESMTPS id c43sm1133664weo.42.2011.05.05.07.56.03 (version=SSLv3 cipher=OTHER); Thu, 05 May 2011 07:56:03 -0700 (PDT) Subject: Re: ARM, AF_PACKET: caching problems on Marvell Kirkwood From: Eric Dumazet To: Phil Sutter In-Reply-To: <20110505141107.GC30443@orbit.nwl.cc> References: <20110408130643.GA8730@orbit.nwl.cc> <20110505141107.GC30443@orbit.nwl.cc> Date: Thu, 05 May 2011 16:56:02 +0200 Message-ID: <1304607362.3032.84.camel@edumazet-laptop> Mime-Version: 1.0 X-Mailer: Evolution 2.30.3 X-CRM114-Version: 20090807-BlameThorstenAndJenny ( TRE 0.7.6 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20110505_105607_646327_2774D1FB X-CRM114-Status: GOOD ( 38.05 ) X-Spam-Score: -0.8 (/) X-Spam-Report: SpamAssassin version 3.3.1 on canuck.infradead.org summary: Content analysis details: (-0.8 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low trust [74.125.82.177 listed in list.dnswl.org] 0.0 FREEMAIL_FROM Sender email is freemail (eric.dumazet[at]gmail.com) 0.0 T_FRT_STOCK2 BODY: ReplaceTags: Stock (2) -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.0 RFC_ABUSE_POST Both abuse and postmaster missing on sender domain Cc: Nicolas Pitre , netdev@vger.kernel.org, ne@erfurth.eu, Johann Baudy , Lennert Buytenhek , linux-arm-kernel@lists.infradead.org X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.12 Precedence: list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: linux-arm-kernel-bounces@lists.infradead.org Errors-To: linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org List-Id: linux-imx-kernel.lists.patchwork.ozlabs.org Le jeudi 05 mai 2011 à 16:11 +0200, Phil Sutter a écrit : > Hi, > > Hasn't anyone experienced this bug but me? Can anyone reproduce the > described behaviour on his Kirkwood-based (or even generic ARM) machine? > I am still not sure if this is a problem of just my CPU or common > amongst Kirkwood/VIPT/ARM machines. > > My workaround looks like this: > | diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c > | index b5362e9..0672f50 100644 > | --- a/net/packet/af_packet.c > | +++ b/net/packet/af_packet.c > | @@ -1298,10 +1298,13 @@ static int packet_sendmsg(struct kiocb *iocb, struct socket *sock, > | { > | struct sock *sk = sock->sk; > | struct packet_sock *po = pkt_sk(sk); > | - if (po->tx_ring.pg_vec) > | - return tpacket_snd(po, msg); > | - else > | - return packet_snd(sock, msg, len); > | + int rc; > | + > | + flush_cache_all(); > | + rc = po->tx_ring.pg_vec ? tpacket_snd(po, msg) : > | + packet_snd(sock, msg, len); > | + flush_cache_all(); > | + return rc; > | } > | > | /* > > Greetings, Phil > > (Full-quoting here because I've added the TX ring author and the Kirkwood > maintainers to Cc.) > > On Fri, Apr 08, 2011 at 03:06:43PM +0200, Phil Sutter wrote: > > Dear lists, > > > > I am experiencing severe caching issues using the TX_RING feature of > > AF_PACKET on a Kirkwood-based system (i.e., OpenRD). This may likely be > > a bug of the CPU/SoC itself, at least it reacts a bit picky when using > > the preload data instruction (pld) in rather useless cases (but that's a > > different story). > > > > There is simple testing code at the end of this email, effectively just > > preparing a packet in the TX_RING and triggering it's delivery once per > > second. The experienced symptom is that sporadically nothing goes out in > > one iteration, and two packets in the following one. > > > > It looks like the kernel doesn't get the changed value of tp_status in > > time, although userspace sees the correct value. Note that moving the > > sleep(1) from the end of the loop to just before calling sendto() fixes > > the problem. > > > > Another (more useful) workaround is to call flush_cache_all() at the > > beginning of packet_sendmsg() in net/packet/af_packet.c. I was not able > > to fix this with some more specific flushing at that place. Anyway, the > > call to flush_dcache_page() from __packet_get_status() in the same > > source file is meant to do the trick I guess. But somehow doesn't. > > > > Feedback regardles of which kind is highly appreciated, of course! > > > > Greetings, Phil > > > > ------------------[start of packet_mmap_test.c]-------------------- > > #include > > #include > > #include > > #include > > #include > > #include > > #include > > #include > > #include > > #include > > #include > > > > #define PERROR_EXIT(rc, mesg) { \ > > perror(mesg); \ > > return rc; \ > > } > > > > int main(void) > > { > > uint32_t size; > > struct sockaddr_ll sa; > > struct ifreq ifr; > > int index; > > int tmp; > > int fd; > > struct tpacket_req packet_req; > > struct tpacket2_hdr * ps_header_start, *ps_header; > > > > if ((fd = socket(AF_PACKET, SOCK_DGRAM, htons(ETH_P_ALL))) < 0) > > PERROR_EXIT(EXIT_FAILURE, "socket"); > > > > /* retrieve eth0's interface index number */ > > strncpy (ifr.ifr_name, "eth0", sizeof(ifr.ifr_name)); > > if (ioctl(fd, SIOCGIFINDEX, &ifr) < 0) > > PERROR_EXIT(EXIT_FAILURE, "ioctl(SIOCGIFINDEX)"); > > > > /* set sockaddr info */ > > memset(&sa, 0, sizeof(sa)); > > sa.sll_family = AF_PACKET; > > sa.sll_protocol = ETH_P_ALL; > > sa.sll_ifindex = ifr.ifr_ifindex; > > > > /* bind port */ > > if (bind(fd, (struct sockaddr *)&sa, sizeof(sa)) < 0) > > PERROR_EXIT(EXIT_FAILURE, "bind()"); > > > > tmp = TPACKET_V2; > > if (setsockopt(fd, SOL_PACKET, PACKET_VERSION, &tmp, sizeof(tmp)) < 0) > > PERROR_EXIT(EXIT_FAILURE, "setsockopt(PACKET_VERSION)"); > > > > /* set packet loss option */ > > tmp = 1; > > if (setsockopt(fd, SOL_PACKET, PACKET_LOSS, &tmp, sizeof(tmp)) < 0) > > PERROR_EXIT(EXIT_FAILURE, "setsockopt(PACKET_LOSS)"); > > > > /* prepare Tx ring request */ > > packet_req.tp_block_size = 1024 * 8; > > packet_req.tp_frame_size = 1024 * 8; > > packet_req.tp_block_nr = 1024; > > packet_req.tp_frame_nr = 1024; > > > > /* send TX ring request */ > > if (setsockopt(fd, SOL_PACKET, PACKET_TX_RING, > > &packet_req, sizeof(packet_req)) < 0) > > PERROR_EXIT(EXIT_FAILURE, "setsockopt: PACKET_TX_RING"); > > > > /* calculate memory to mmap in the kernel */ > > size = packet_req.tp_block_size * packet_req.tp_block_nr; > > > > /* mmap Tx ring buffers memory */ > > ps_header_start = mmap(0, size, > > PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); > > if (ps_header_start < 0) > > PERROR_EXIT(EXIT_FAILURE, "mmap"); > > > > /* fill peer sockaddr for SOCK_DGRAM */ > > sa.sll_family = AF_PACKET; > > sa.sll_protocol = htons(ETH_P_IP); > > sa.sll_ifindex = ifr.ifr_ifindex; > > sa.sll_halen = ETH_ALEN; > > memset(&sa.sll_addr, 0xff, ETH_ALEN); > > > > ps_header = ps_header_start; > > while (1) { > > int sendlen, j; > > > > char *data = (void*)ps_header + TPACKET_HDRLEN > > - sizeof(struct sockaddr_ll); > > > > switch((volatile uint32_t)ps_header->tp_status) > > { > > case TP_STATUS_AVAILABLE: > > memset(data, 0x23, 150); > > break; > > > > case TP_STATUS_WRONG_FORMAT: > > printf("An error has occured during transfer\n"); > > exit(EXIT_FAILURE); > > break; > > > > default: > > printf("Buffer is not available, aborting\n"); > > exit(1); > > break; > > } > > ps_header->tp_len = 150; > > ps_header->tp_status = TP_STATUS_SEND_REQUEST; > > > > sendlen = sendto(fd, NULL, 0, 0, > > (struct sockaddr *)&sa, sizeof(sa)); > > if (sendlen < 0) > > perror("sendto"); > > else if (sendlen == 0) > > printf("sendto(): nothing sent!\n"); > > else > > printf("sendto(): sent %d bytes out\n", sendlen); > > > > #define ST_IS(x) ((volatile uint32_t)ps_header->tp_status == x) > > printf("tp_status after sending: %s\n", > > ST_IS(TP_STATUS_AVAILABLE) ? "AVAILABLE" : > > ST_IS(TP_STATUS_SEND_REQUEST) ? "SEND_REQUEST" : > > ST_IS(TP_STATUS_WRONG_FORMAT) ? "WRONG_FORMAT" : > > "unknown"); > > #undef ST_IS > > > > ps_header = (void *)ps_header + packet_req.tp_frame_size; > > if (ps_header >= ps_header_start + size) > > ps_header = ps_header_start; > > > > sleep(1); > > } > > return 0; > > } > > --------------------[end of packet_mmap_test.c]-------------------- Hi Phil I assume you use latest linux-2.6 or net-next-2.6 ? Could you try to force vmalloc() use ? diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index b5362e9..0b5a89c 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -2383,7 +2383,7 @@ static inline char *alloc_one_pg_vec_page(unsigned long order) gfp_t gfp_flags = GFP_KERNEL | __GFP_COMP | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY; - buffer = (char *) __get_free_pages(gfp_flags, order); + buffer = NULL; if (buffer) return buffer;