[{"id":1766349,"web_url":"http://patchwork.ozlabs.org/comment/1766349/","msgid":"<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>","list_archive_url":null,"date":"2017-09-11T14:49:17","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":2404,"url":"http://patchwork.ozlabs.org/api/people/2404/","name":"Eric Dumazet","email":"eric.dumazet@gmail.com"},"content":"On Mon, 2017-09-11 at 14:27 +0800, liujian56@huawei.com wrote:\n> From: liujian \n> \n> After the tcp socket go to ESTABLISHED stat, change IP address (server\n> side),\n> then the tcp socket will go tcp_probe_timer process.\n> \n> [root@localhost net]# netstat -toe\n> Active Internet connections (w/o servers)\n> Proto Recv-Q Send-Q Local Address Foreign Address State User Inode Timer\n> tcp 0 1104 9.81.254:personal-agent 9.84.201.213:23597 ESTABLISHED root 12819 probe (4.36/0/7)\n> [root@localhost net]# cat /proc/net/tcp\n> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode \n> 3: B1FE5109:15B3 D5C95409:5C2D 01 00000495:00000000 04:0000005E 00000000 0 7 12819 2 ffff95cdcf45a000 20 4 1 10 -1 \n> \n> In my test case, tcp_write_queue_head(sk) and tcp_send_head(sk) is same\n> SKB.\n> And ((s32)(tcp_time_stamp(tp) - start_ts) >\n> jiffies_to_msecs(icsk->icsk_user_timeout))\n> always is false.\n\nInteresting.\n\n> Here use keepalive_time_elapsed(tp) to do the compare as\n> tcp_keepalive_timer do.\n\n\nBut zero window probe and TCP_USER_TIMEOUT can be used without\nkeepalives...\n\n\nA packetdrill test would help, I will write one.\n\nThanks.","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"UtGOWCCi\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xrW6J2jPJz9s4q\n\tfor ;\n\tTue, 12 Sep 2017 00:49:28 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751049AbdIKOtW (ORCPT );\n\tMon, 11 Sep 2017 10:49:22 -0400","from mail-pf0-f182.google.com ([209.85.192.182]:33183 \"EHLO\n\tmail-pf0-f182.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750969AbdIKOtV (ORCPT\n\t); Mon, 11 Sep 2017 10:49:21 -0400","by mail-pf0-f182.google.com with SMTP id y29so13526181pff.0\n\tfor ; Mon, 11 Sep 2017 07:49:21 -0700 (PDT)","from [10.1.104.73] ([207.198.105.19])\n\tby smtp.googlemail.com with ESMTPSA id\n\ts68sm15770816pfd.72.2017.09.11.07.49.18\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tMon, 11 Sep 2017 07:49:20 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=message-id:subject:from:to:cc:date:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=PXYwD+gRp/+u1d+Aa8CrGV8u/nBd9hmWH9Z3trwSXNE=;\n\tb=UtGOWCCi32y6MGbDuawqzmyKjdlGeQ0lD1q/qQAiKeRyo8M8mXiGX2PyKaA0UuiQNi\n\tR3mcw51KwGfvhylksBQtqKHQ3+ksuiehHRtq+S+vk34rMqqKBqtMoBjOizk9g1+wXguA\n\tD+UBqlgVSca12uA4+mlx77OoJ3HeM1ShrROTeDT6z/AkwEKzEhmZSeB60Y8zSjl71YQr\n\tEyHWuYutT0x4jKbZ9A5V+umpGGuJUBe06UHAgYOFtxEenw4TmOsxCAhIh6zSbCTs4K2N\n\tLcGnNHLhC/eNkeEp2AO7qP8sIVDWUBQmVKQftmCic8exrmXWFNxDcD8wz79iKBe9QpnJ\n\t2XwA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=PXYwD+gRp/+u1d+Aa8CrGV8u/nBd9hmWH9Z3trwSXNE=;\n\tb=A6Y+aZx5NIPKqPobrV/ozSQsHV6Ys1m7ECq3WGKOzT5r5L66AwDnLI5R8zENZ/4TzO\n\tCQ/w4ifz3qCkROnelK7JpYAIyNvjAgY36ouJgkzp5Yfmq/6qmFJc5Y7WjPmoHd7DjQLy\n\tPviPvKikyiSxg0TNty2WJQmhx+DIyFVTuJ+QURYl/S8GQVMP4nysEsMKKPxwjLv2JpMS\n\tKqsCY2wf09UtftAgNSczz46ig8tvoI4LT9UQIUvRmhX6nxnuXFKVYa1RNUIN+sndOAZC\n\tAt4bODYML4S/cI/fdN5dXcnSHekgocp/ueAU/RXu9Sbqqy+zlqwG4AucZi30ooMlyZzD\n\tzsSg==","X-Gm-Message-State":"AHPjjUi24ngOEgMz+6XIU+SJ6P6oUcUJ0HWJO5wTu6AcIXmWx7UQqhNw\n\tQKbLNDak2zx0QQ==","X-Google-Smtp-Source":"ADKCNb4GFrGxWa8m4z7B+i7Brrcg3Bl5xZTvAwoif4JHAVdU6PTml919u7rJZbKvg8ngbmuUo9Z9/Q==","X-Received":"by 10.99.185.86 with SMTP id v22mr12750467pgo.236.1505141361303; \n\tMon, 11 Sep 2017 07:49:21 -0700 (PDT)","Message-ID":"<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>","Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","From":"Eric Dumazet ","To":"liujian56@huawei.com","Cc":"davem@davemloft.net, kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.org,\n\tedumazet@google.com, ycheng@google.com, hkchu@google.com,\n\tnetdev@vger.kernel.org, weiyongjun1@huawei.com","Date":"Mon, 11 Sep 2017 07:49:17 -0700","In-Reply-To":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>","Content-Type":"text/plain; charset=\"UTF-8\"","X-Mailer":"Evolution 3.10.4-0ubuntu2 ","Mime-Version":"1.0","Content-Transfer-Encoding":"7bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1766356,"web_url":"http://patchwork.ozlabs.org/comment/1766356/","msgid":"<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>","list_archive_url":null,"date":"2017-09-11T15:13:05","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":2404,"url":"http://patchwork.ozlabs.org/api/people/2404/","name":"Eric Dumazet","email":"eric.dumazet@gmail.com"},"content":"On Mon, 2017-09-11 at 07:49 -0700, Eric Dumazet wrote:\n> On Mon, 2017-09-11 at 14:27 +0800, liujian56@huawei.com wrote:\n> > From: liujian \n> > \n> > After the tcp socket go to ESTABLISHED stat, change IP address (server\n> > side),\n> > then the tcp socket will go tcp_probe_timer process.\n> > \n> > [root@localhost net]# netstat -toe\n> > Active Internet connections (w/o servers)\n> > Proto Recv-Q Send-Q Local Address Foreign Address State User Inode Timer\n> > tcp 0 1104 9.81.254:personal-agent 9.84.201.213:23597 ESTABLISHED root 12819 probe (4.36/0/7)\n> > [root@localhost net]# cat /proc/net/tcp\n> > sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode \n> > 3: B1FE5109:15B3 D5C95409:5C2D 01 00000495:00000000 04:0000005E 00000000 0 7 12819 2 ffff95cdcf45a000 20 4 1 10 -1 \n> > \n> > In my test case, tcp_write_queue_head(sk) and tcp_send_head(sk) is same\n> > SKB.\n> > And ((s32)(tcp_time_stamp(tp) - start_ts) >\n> > jiffies_to_msecs(icsk->icsk_user_timeout))\n> > always is false.\n> \n> Interesting.\n> \n> > Here use keepalive_time_elapsed(tp) to do the compare as\n> > tcp_keepalive_timer do.\n> \n> \n> But zero window probe and TCP_USER_TIMEOUT can be used without\n> keepalives...\n> \n> \n> A packetdrill test would help, I will write one.\n\nSo existing code seems to work :\n\n# cat window-probe-without-data-user-timeout.pkt\n\n 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n +0 bind(3, ..., ...) = 0\n +0 listen(3, 1) = 0\n\n +0 < S 0:0(0) win 0 \n +0 > S. 0:0(0) ack 1 \n\n// Client advertises a zero receive window, so we can't send.\n +.1 < . 1:1(0) ack 1 win 0\n +0 accept(3, ..., ...) = 4\n\n +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n +0 write(4, ..., 2920) = 2920\n\n// Window probes are scheduled just like RTOs.\n +.3~+.31 > . 0:0(0) ack 1\n +.6~+.62 > . 0:0(0) ack 1\n +1.2~+1.24 > . 0:0(0) ack 1\n +2.4~+2.48 > . 0:0(0) ack 1\n\n# ./packetdrill window-probe-without-data-user_timeout.pkt\n08:10:39.306137 IP 192.0.2.1.58149 > 192.168.79.31.8080: Flags [S], seq 0, win 0, options [mss 1460,sackOK,nop,nop,nop,wscale 7], length 0\n08:10:39.306166 IP 192.168.79.31.8080 > 192.0.2.1.58149: Flags [S.], seq 3982794529, ack 1, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0\n08:10:39.406296 IP 192.0.2.1.58149 > 192.168.79.31.8080: Flags [.], ack 1, win 0, length 0\n08:10:39.716004 IP 192.168.79.31.8080 > 192.0.2.1.58149: Flags [.], ack 1, win 115, length 0\n08:10:40.327133 IP 192.168.79.31.8080 > 192.0.2.1.58149: Flags [.], ack 1, win 115, length 0\n08:10:41.540243 IP 192.168.79.31.8080 > 192.0.2.1.58149: Flags [.], ack 1, win 115, length 0\n\nYou can see we got only 3 probes, not 4.","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"KkW38Pgg\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xrWdh4fz5z9s7C\n\tfor ;\n\tTue, 12 Sep 2017 01:13:12 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751028AbdIKPNJ (ORCPT );\n\tMon, 11 Sep 2017 11:13:09 -0400","from mail-pg0-f66.google.com ([74.125.83.66]:35931 \"EHLO\n\tmail-pg0-f66.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750782AbdIKPNI (ORCPT\n\t); Mon, 11 Sep 2017 11:13:08 -0400","by mail-pg0-f66.google.com with SMTP id d8so4766804pgt.3\n\tfor ; Mon, 11 Sep 2017 08:13:08 -0700 (PDT)","from [10.1.104.73] ([207.198.105.19])\n\tby smtp.googlemail.com with ESMTPSA id\n\tl12sm13514842pgr.10.2017.09.11.08.13.06\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tMon, 11 Sep 2017 08:13:07 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=message-id:subject:from:to:cc:date:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=NBXyPQEGn+zk9JBFaVEv8pcDjjsi2MlqffvfpldzymQ=;\n\tb=KkW38PggytEfJmlgykFxK63JZIR5LyL/mvGZAnL0s2WaK77A6IqF86grK4yYWIDQps\n\tAmqfTZeQNIdauPSyAnO89Kd1lalKd8DtStvgcyoW47E+4uP1M4PzstCiiax+4aYlNSqy\n\tDjxlOGzKC/FwWuJc0UjUvHAxg+p7GzXHSSisf5zn51djmRM6KVOsC++pbSmqqIiNixOe\n\tSiHOJuj7pi/1s8eJqs9tFeBB/+pY/JbqDjVgqn/uVNkvDqWVtDcXv+h/+85pw3s2NIxA\n\tM18iKUXn4+LzeEV6/bx3T19cj+78e8aor3pW+gq2cQmvKRxouWpHpJAid9fqU9EUCq9d\n\t+JPA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=NBXyPQEGn+zk9JBFaVEv8pcDjjsi2MlqffvfpldzymQ=;\n\tb=XlWERSsMuWzh2tUJxivqBNon6N1Yl4vUV6l7miWR1ZoMd7o7oh/MCIPFBH7/y3NkmB\n\t0TF1KHIUSxww3za+T8i1N/8FsGIH4L6F16ruZbNnEoNsFcoQYJhvOdeuZBarbF2KNq0l\n\tguAqfaHkQRtcC42RXQcjr0O6x02018URaibNhqISqFtNPKJvQ/bvEL8rT+YaEaG9mzwo\n\t+45txoXBDJc/uiH/JF6R2Lq5xuYUxaUc5Ik/QIDSc+iNSosTy8rg7K5VBVOAy5ONiNmz\n\tMaF5fpFU6ZqqsILs2o+p5BrngCZe2l94QJEFKwev3fnmIl4YXAmeaytQFxl6+0Rpa8+4\n\tvJJg==","X-Gm-Message-State":"AHPjjUij3/QRcKuYZ2Bnc2t3LfnKur6PttUAU+H0c2DaMdKGR+rF2EKO\n\tIfG1uVnkOq79xA==","X-Google-Smtp-Source":"ADKCNb6f1dLK+xZ8Gr+ZKD8FF7wYqdEFzxHc2dme1/1iFrmHhPVENi8t8bTegiqaIw8hNx8h8OUeRg==","X-Received":"by 10.84.185.106 with SMTP id e39mr5438247plg.333.1505142788226; \n\tMon, 11 Sep 2017 08:13:08 -0700 (PDT)","Message-ID":"<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>","Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","From":"Eric Dumazet ","To":"liujian56@huawei.com","Cc":"davem@davemloft.net, kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.org,\n\tedumazet@google.com, ycheng@google.com, hkchu@google.com,\n\tnetdev@vger.kernel.org, weiyongjun1@huawei.com","Date":"Mon, 11 Sep 2017 08:13:05 -0700","In-Reply-To":"<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>","Content-Type":"text/plain; charset=\"UTF-8\"","X-Mailer":"Evolution 3.10.4-0ubuntu2 ","Mime-Version":"1.0","Content-Transfer-Encoding":"7bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1766359,"web_url":"http://patchwork.ozlabs.org/comment/1766359/","msgid":"<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>","list_archive_url":null,"date":"2017-09-11T15:22:28","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":2404,"url":"http://patchwork.ozlabs.org/api/people/2404/","name":"Eric Dumazet","email":"eric.dumazet@gmail.com"},"content":"On Mon, 2017-09-11 at 08:13 -0700, Eric Dumazet wrote:\n\n> You can see we got only 3 probes, not 4.\n\nHere is complete packetdrill test showing that code behaves as expected.\n\n 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n +0 bind(3, ..., ...) = 0\n +0 listen(3, 1) = 0\n\n +0 < S 0:0(0) win 0 \n +0 > S. 0:0(0) ack 1 \n\n// Client advertises a zero receive window, so we can't send.\n +.1 < . 1:1(0) ack 1 win 0\n +0 accept(3, ..., ...) = 4\n\n +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n +0 write(4, ..., 2920) = 2920\n\n// Window probes are scheduled just like RTOs.\n +.3~+.31 > . 0:0(0) ack 1\n +.6~+.62 > . 0:0(0) ack 1\n +1.2~+1.24 > . 0:0(0) ack 1\n\n// Peer opens its window too late !\n +3 < . 1:1(0) ack 1 win 1000\n +0 > R 1:1(0)","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"JAORGCLU\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xrWrW5QxCz9s7B\n\tfor ;\n\tTue, 12 Sep 2017 01:22:35 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1750975AbdIKPWd (ORCPT );\n\tMon, 11 Sep 2017 11:22:33 -0400","from mail-pf0-f182.google.com ([209.85.192.182]:34294 \"EHLO\n\tmail-pf0-f182.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750810AbdIKPWb (ORCPT\n\t); Mon, 11 Sep 2017 11:22:31 -0400","by mail-pf0-f182.google.com with SMTP id e1so14439934pfk.1\n\tfor ; Mon, 11 Sep 2017 08:22:31 -0700 (PDT)","from ?IPv6:2620:15c:2c1:100:5588:2121:6039:1ef6?\n\t([2620:15c:2c1:100:5588:2121:6039:1ef6])\n\tby smtp.googlemail.com with ESMTPSA id\n\tu12sm9683695pfg.146.2017.09.11.08.22.29\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tMon, 11 Sep 2017 08:22:30 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=message-id:subject:from:to:cc:date:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=FEg5iiYCckVncHjL4u00o33gfJepLK9euxlcXKa1WCw=;\n\tb=JAORGCLUWQvZqDwf52x/hgr0gzO7fqKlU6gPmFj5K3dicZFbNDWjvxrdEbB6PNU7fz\n\tRTLp0MXKB0vBvmZsOMchqDZt7uLHjbLi0hl5BQX6wtwc+duu4+buUac+tKJHsPfB1qqT\n\tybKnpIEqLvF41b/DTbp3ifTkfhB08SUKcuhSIsN6cKN8TLvYoqAHDXdUWNG9f76mA7xf\n\tLssxKzQy4ZKNEW+qrd0rv3IU8C14iC8SGWm7GBwKAVj39T+HlK7z4sfP2k0J1yONCjgj\n\tsZ5fgf8OE14OM+HckM7PJ818ShGPeu7RUlsOtTu50MmPBWLfthA/KO4RT1Qi3MvuAf2l\n\t0dPA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=FEg5iiYCckVncHjL4u00o33gfJepLK9euxlcXKa1WCw=;\n\tb=B9QTURSwWV95kWudUMLD8JU5WiJzA3n4D3EW/Dh1wJIPjt2poqXJOiJ4tL+p727p9/\n\tFGosAQ6IKmX/AEf1gYD22RmHt6hjHePuYZcfnlaGosdKbRzdCkIbnRoaENVnjDZAoj/h\n\taKXWmO/i/nE1ek6rMSD7n6VThxiIXAcHs1bMrtrXk3zgOuHs9BJnqz37rRdyL8d+HlPf\n\t/PCRmydZg+Qxws1NPtlY4CqEiYKq+70kbc9FR19L7JLfC40gMCS/nPaDY3WT6qxmAtyQ\n\t2qzM9+HSlnjYj0eCQNVEcX3BJKCC/abzrLlwjoVnCcuhUbezFfENV2bsgz7Fk/Fk4DTf\n\t3J+g==","X-Gm-Message-State":"AHPjjUgWAsVqgIL10sIBT32oAbO/ynyf5OZUTeuzXddMcAdYzrNo3poU\n\tGcnnywfHop7mug==","X-Google-Smtp-Source":"ADKCNb7rOXFKHO9gAuyK095/AiadUzejNRGsgRl6emqGK5NxbYBxw3JmBz5/G53hzoeTaHCkBGLGTg==","X-Received":"by 10.84.235.5 with SMTP id o5mr13800982plk.4.1505143351141;\n\tMon, 11 Sep 2017 08:22:31 -0700 (PDT)","Message-ID":"<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>","Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","From":"Eric Dumazet ","To":"liujian56@huawei.com","Cc":"davem@davemloft.net, kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.org,\n\tedumazet@google.com, ycheng@google.com, hkchu@google.com,\n\tnetdev@vger.kernel.org, weiyongjun1@huawei.com","Date":"Mon, 11 Sep 2017 08:22:28 -0700","In-Reply-To":"<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>","Content-Type":"text/plain; charset=\"UTF-8\"","X-Mailer":"Evolution 3.10.4-0ubuntu2 ","Mime-Version":"1.0","Content-Transfer-Encoding":"7bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1766698,"web_url":"http://patchwork.ozlabs.org/comment/1766698/","msgid":"","list_archive_url":null,"date":"2017-09-12T06:08:48","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":71967,"url":"http://patchwork.ozlabs.org/api/people/71967/","name":"liujian \\(CE\\)","email":"liujian56@huawei.com"},"content":"Hi,\n\nIn the scenario, tcp server side IP changed, and at that memont,\nuserspace application still send data continuously;\ntcp_send_head(sk)'s timestamp always be refreshed.\n\nHere is the packetdrill script:\n\n 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n +0 bind(3, ..., ...) = 0\n +0 listen(3, 1) = 0\n\n +0 < S 0:0(0) win 0 \n +0 > S. 0:0(0) ack 1 \n\n +.1 < . 1:1(0) ack 1 win 65530\n +0 accept(3, ..., ...) = 4\n\n +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n +0 write(4, ..., 24) = 24\n +0 > P. 1:25(24) ack 1 win 229\n +.1 < . 1:1(0) ack 25 win 65530\n\n//change the ipaddress\n +1 `ifconfig tun0 192.168.0.10/16`\n\n +1 write(4, ..., 24) = 24\n +1 write(4, ..., 24) = 24\n +1 write(4, ..., 24) = 24\n +1 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n\n +0 `ifconfig tun0 192.168.0.1/16`\n +0 < . 1:1(0) ack 1 win 1000\n +0 write(4, ..., 24) = -1\n\n\n[root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\ntest.pkt:50: runtime error in write call: Expected result -1 but got 24 with errno 2 (No such file or directory)\n\nreal\t1m11.364s\nuser\t0m0.028s\nsys\t0m0.106s\n\n[root@localhost ~]# netstat -toen\nActive Internet connections (w/o servers)\nProto Recv-Q Send-Q Local Address Foreign Address State User Inode Timer\ntcp 0 504 192.168.0.1:8080 192.0.2.1:33993 ESTABLISHED 0 45453 probe (22.38/0/7)\n\nsince the script didn't wait for enough time, here only got 7 probes.\n\n在 2017/9/11 23:22, Eric Dumazet 写道:\n> On Mon, 2017-09-11 at 08:13 -0700, Eric Dumazet wrote:\n> \n>> You can see we got only 3 probes, not 4.\n> \n> Here is complete packetdrill test showing that code behaves as expected.\n> \n> 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n> +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n> +0 bind(3, ..., ...) = 0\n> +0 listen(3, 1) = 0\n> \n> +0 < S 0:0(0) win 0 \n> +0 > S. 0:0(0) ack 1 \n> \n> // Client advertises a zero receive window, so we can't send.\n> +.1 < . 1:1(0) ack 1 win 0\n> +0 accept(3, ..., ...) = 4\n> \n> +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n> +0 write(4, ..., 2920) = 2920\n> \n> // Window probes are scheduled just like RTOs.\n> +.3~+.31 > . 0:0(0) ack 1\n> +.6~+.62 > . 0:0(0) ack 1\n> +1.2~+1.24 > . 0:0(0) ack 1\n> \n> // Peer opens its window too late !\n> +3 < . 1:1(0) ack 1 win 1000\n> +0 > R 1:1(0)\n> \n> \n> \n> .\n>","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xrvWh1KQlz9s76\n\tfor ;\n\tTue, 12 Sep 2017 16:09:20 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751244AbdILGJR (ORCPT );\n\tTue, 12 Sep 2017 02:09:17 -0400","from szxga04-in.huawei.com ([45.249.212.190]:6451 \"EHLO\n\tszxga04-in.huawei.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750911AbdILGJQ (ORCPT\n\t); Tue, 12 Sep 2017 02:09:16 -0400","from 172.30.72.60 (EHLO DGGEMS401-HUB.china.huawei.com)\n\t([172.30.72.60])\n\tby dggrg04-dlp.huawei.com (MOS 4.4.6-GA FastPath queued)\n\twith ESMTP id DHB20973; Tue, 12 Sep 2017 14:09:07 +0800 (CST)","from [127.0.0.1] (10.177.97.126) by DGGEMS401-HUB.china.huawei.com\n\t(10.3.19.201) with Microsoft SMTP Server id 14.3.301.0;\n\tTue, 12 Sep 2017 14:08:59 +0800"],"Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","To":"Eric Dumazet ","CC":", ,\n\t, ,\n\t, , ,\n\t,\n\twangkefeng 00227729 ","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>","From":"liujian ","Message-ID":"","Date":"Tue, 12 Sep 2017 14:08:48 +0800","User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"8bit","X-Originating-IP":"[10.177.97.126]","X-CFilter-Loop":"Reflected","X-Mirapoint-Virus-RAPID-Raw":"score=unknown(0),\n\trefid=str=0001.0A020201.59B77A04.00A0, ss=1, re=0.000, recu=0.000,\n\treip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0,\n\tso=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32","X-Mirapoint-Loop-Id":"81853c1dfab888c415e3275a5fb5bddb","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1767153,"web_url":"http://patchwork.ozlabs.org/comment/1767153/","msgid":"<1505228700.15310.138.camel@edumazet-glaptop3.roam.corp.google.com>","list_archive_url":null,"date":"2017-09-12T15:05:00","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":2404,"url":"http://patchwork.ozlabs.org/api/people/2404/","name":"Eric Dumazet","email":"eric.dumazet@gmail.com"},"content":"On Tue, 2017-09-12 at 14:08 +0800, liujian wrote:\n> Hi,\n> \n> In the scenario, tcp server side IP changed, and at that memont,\n> userspace application still send data continuously;\n> tcp_send_head(sk)'s timestamp always be refreshed.\n> \n> Here is the packetdrill script:\n> \n> 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n> +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n> +0 bind(3, ..., ...) = 0\n> +0 listen(3, 1) = 0\n> \n> +0 < S 0:0(0) win 0 \n> +0 > S. 0:0(0) ack 1 \n> \n> +.1 < . 1:1(0) ack 1 win 65530\n> +0 accept(3, ..., ...) = 4\n> \n> +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n> +0 write(4, ..., 24) = 24\n> +0 > P. 1:25(24) ack 1 win 229\n> +.1 < . 1:1(0) ack 25 win 65530\n> \n> //change the ipaddress\n> +1 `ifconfig tun0 192.168.0.10/16`\n> \n> +1 write(4, ..., 24) = 24\n> +1 write(4, ..., 24) = 24\n> +1 write(4, ..., 24) = 24\n> +1 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> +3 write(4, ..., 24) = 24\n> \n> +0 `ifconfig tun0 192.168.0.1/16`\n> +0 < . 1:1(0) ack 1 win 1000\n> +0 write(4, ..., 24) = -1\n> \n> \n\nThis has nothing to do with the code patch you have changed.\n\nHow have you tested your patch exactly ?\n\n\n> [root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\n> test.pkt:50: runtime error in write call: Expected result -1 but got 24 with errno 2 (No such file or directory)\n> \n> real\t1m11.364s\n> user\t0m0.028s\n> sys\t0m0.106s\n> \n> [root@localhost ~]# netstat -toen\n> Active Internet connections (w/o servers)\n> Proto Recv-Q Send-Q Local Address Foreign Address State User Inode Timer\n> tcp 0 504 192.168.0.1:8080 192.0.2.1:33993 ESTABLISHED 0 45453 probe (22.38/0/7)\n> \n> since the script didn't wait for enough time, here only got 7 probes.\n> \n> 在 2017/9/11 23:22, Eric Dumazet 写道:\n> > On Mon, 2017-09-11 at 08:13 -0700, Eric Dumazet wrote:\n> > \n> >> You can see we got only 3 probes, not 4.\n> > \n> > Here is complete packetdrill test showing that code behaves as expected.\n> > \n> > 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n> > +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n> > +0 bind(3, ..., ...) = 0\n> > +0 listen(3, 1) = 0\n> > \n> > +0 < S 0:0(0) win 0 \n> > +0 > S. 0:0(0) ack 1 \n> > \n> > // Client advertises a zero receive window, so we can't send.\n> > +.1 < . 1:1(0) ack 1 win 0\n> > +0 accept(3, ..., ...) = 4\n> > \n> > +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n> > +0 write(4, ..., 2920) = 2920\n> > \n> > // Window probes are scheduled just like RTOs.\n> > +.3~+.31 > . 0:0(0) ack 1\n> > +.6~+.62 > . 0:0(0) ack 1\n> > +1.2~+1.24 > . 0:0(0) ack 1\n> > \n> > // Peer opens its window too late !\n> > +3 < . 1:1(0) ack 1 win 1000\n> > +0 > R 1:1(0)\n> > \n> > \n> > \n> > .\n> > \n>","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"fywyNe/M\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xs7Pz2q8mz9s5L\n\tfor ;\n\tWed, 13 Sep 2017 01:05:11 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751439AbdILPFH (ORCPT );\n\tTue, 12 Sep 2017 11:05:07 -0400","from mail-pg0-f52.google.com ([74.125.83.52]:37076 \"EHLO\n\tmail-pg0-f52.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751404AbdILPFG (ORCPT\n\t); Tue, 12 Sep 2017 11:05:06 -0400","by mail-pg0-f52.google.com with SMTP id d8so21763444pgt.4\n\tfor ; Tue, 12 Sep 2017 08:05:06 -0700 (PDT)","from [10.1.104.73] ([207.198.105.19])\n\tby smtp.googlemail.com with ESMTPSA id\n\te76sm2502658pfd.149.2017.09.12.08.05.03\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tTue, 12 Sep 2017 08:05:04 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=message-id:subject:from:to:cc:date:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=V35qyIUepV3kWFrbqI0X80/QXK/OxIAyHv8MaVOEgig=;\n\tb=fywyNe/M3+DEwRe5Lmeyippcta5okJ63NDmkjw70ZWaOZGSTQYEmpn3kq+/x6RKtwU\n\tOjgX93y2mQtOgsGPvFtnngslwwI/B7VYCXQQaCj+62s8AYlPUNvTg5hby4eBzh+va2ob\n\tw6wTksTXw1NETtJJ6ksf/O27Dd0fuQwepB5Se+mA3WSD0sMly5Y7ixy6akpeZ7Hn/UlY\n\tmBe+a2LMZyVsYe0dpXC1HX839vJGBHdpWYJw0K8K09Tug7O3Hih2a26usdn/35OhhM8v\n\tOv4yrYKcSZoX1Z9Ch9jjhdm4S7AFDAMg3jrIt8AKIhkrst+2O+OPjtVuKd0F6/tTUSFM\n\tstnA==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=V35qyIUepV3kWFrbqI0X80/QXK/OxIAyHv8MaVOEgig=;\n\tb=itMs8MoFh4fmImUs81zyFZ0oNS8aB7oIMCrILXGVpl0XZ78tAkrx98B6har46RSZ7q\n\tF9B6dAxnjP0m9ph130w26dpvErRiFsVuag1R+7gvFIhluOQM1riOyCIW37OUHm+khOwV\n\t1wSijVVfiQKGEjMJueed7OGNsmNTUax3aW7+hiXU6WECkn9vZnquQlUwC4BS6DU226Ni\n\tJbKtX0rXeg1YycLR5mNX34MWNwzhrpBiZgJFyioo4a9K2YT/gj+KtVNM3ACkvMiMD108\n\tpQggIUBTI14ACZDRQSxrvlkw9GeRzpTVeMCXP/wN8oVIqgWn0e0l/ihFNQARhdnBdftn\n\tKsDQ==","X-Gm-Message-State":"AHPjjUjyLqRFi+ZSIQB3LW4howuPtSk6Dc4M+Trg4jwSyoY7gXjwct6D\n\tUemIKQO5cBSK8r6w","X-Google-Smtp-Source":"ADKCNb6z8d4gmVBsqP7PONQtQ7dorkfnqjYfExRW8dx3Zy0uhV5UYrdCmY5LZDYtfWp2GXc4dFN8GQ==","X-Received":"by 10.99.131.193 with SMTP id\n\th184mr14781701pge.288.1505228706071; \n\tTue, 12 Sep 2017 08:05:06 -0700 (PDT)","Message-ID":"<1505228700.15310.138.camel@edumazet-glaptop3.roam.corp.google.com>","Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","From":"Eric Dumazet ","To":"liujian ","Cc":"davem@davemloft.net, kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.org,\n\tedumazet@google.com, ycheng@google.com, hkchu@google.com,\n\tnetdev@vger.kernel.org, weiyongjun1@huawei.com,\n\twangkefeng 00227729 ","Date":"Tue, 12 Sep 2017 08:05:00 -0700","In-Reply-To":"","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>\n\t","Content-Type":"text/plain; charset=\"UTF-8\"","X-Mailer":"Evolution 3.10.4-0ubuntu2 ","Mime-Version":"1.0","Content-Transfer-Encoding":"8bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1767168,"web_url":"http://patchwork.ozlabs.org/comment/1767168/","msgid":"<1505230686.15310.141.camel@edumazet-glaptop3.roam.corp.google.com>","list_archive_url":null,"date":"2017-09-12T15:38:06","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":2404,"url":"http://patchwork.ozlabs.org/api/people/2404/","name":"Eric Dumazet","email":"eric.dumazet@gmail.com"},"content":"On Tue, 2017-09-12 at 08:05 -0700, Eric Dumazet wrote:\n> On Tue, 2017-09-12 at 14:08 +0800, liujian wrote:\n> > Hi,\n> > \n> > In the scenario, tcp server side IP changed, and at that memont,\n> > userspace application still send data continuously;\n> > tcp_send_head(sk)'s timestamp always be refreshed.\n> > \n> > Here is the packetdrill script:\n> > \n> > 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n> > +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n> > +0 bind(3, ..., ...) = 0\n> > +0 listen(3, 1) = 0\n> > \n> > +0 < S 0:0(0) win 0 \n> > +0 > S. 0:0(0) ack 1 \n> > \n> > +.1 < . 1:1(0) ack 1 win 65530\n> > +0 accept(3, ..., ...) = 4\n> > \n> > +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n> > +0 write(4, ..., 24) = 24\n> > +0 > P. 1:25(24) ack 1 win 229\n> > +.1 < . 1:1(0) ack 25 win 65530\n> > \n> > //change the ipaddress\n> > +1 `ifconfig tun0 192.168.0.10/16`\n> > \n> > +1 write(4, ..., 24) = 24\n> > +1 write(4, ..., 24) = 24\n> > +1 write(4, ..., 24) = 24\n> > +1 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > +3 write(4, ..., 24) = 24\n> > \n> > +0 `ifconfig tun0 192.168.0.1/16`\n> > +0 < . 1:1(0) ack 1 win 1000\n> > +0 write(4, ..., 24) = -1\n> > \n> > \n> \n> This has nothing to do with the code patch you have changed.\n> \n> How have you tested your patch exactly ?\n> \n\n\nlpaa23:~# ss -toenmi src :8080\nState Recv-Q Send-Q Local Address:Port Peer\nAddress:Port \nESTAB 0 144 192.168.134.161:8080\n192.0.2.1:51165 timer:(persist,8.262ms,5) ino:1\n82083 sk:3 <->\n\t skmem:(r0,rb359040,t0,tb46080,f1792,w2304,o0,bl0,d0) sack cubic\nwscale:7,8 rto:301 backoff:5 rtt:100.127/37.576 \nmss:1460 rcvmss:536 advmss:1460 cwnd:10 bytes_acked:24 segs_out:12\nsegs_in:3 data_segs_out:12 send 1.2Mbps lastsnd:1370 l\nastrcv:13348 lastack:13248 pacing_rate 2.3Mbps delivery_rate 116.7Kbps\napp_limited busy:11346ms rcv_space:29200 notsent:1\n44 minrtt:100.043\n\nThis is the typical RTO timer, not zero window probe.","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"ceHgy4OT\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xs8844YKPz9ryr\n\tfor ;\n\tWed, 13 Sep 2017 01:38:12 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751443AbdILPiJ (ORCPT );\n\tTue, 12 Sep 2017 11:38:09 -0400","from mail-pg0-f66.google.com ([74.125.83.66]:37525 \"EHLO\n\tmail-pg0-f66.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751404AbdILPiI (ORCPT\n\t); Tue, 12 Sep 2017 11:38:08 -0400","by mail-pg0-f66.google.com with SMTP id v5so6326756pgn.4\n\tfor ; Tue, 12 Sep 2017 08:38:08 -0700 (PDT)","from ?IPv6:2620:15c:2c1:100:e4ac:feb:7de5:11b8?\n\t([2620:15c:2c1:100:e4ac:feb:7de5:11b8])\n\tby smtp.googlemail.com with ESMTPSA id\n\ty63sm20244031pgd.20.2017.09.12.08.38.06\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tTue, 12 Sep 2017 08:38:07 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=message-id:subject:from:to:cc:date:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=bbHcTRs3QV83hJfrwrYiql1Usy8uy0kAdGXH+AOPZQY=;\n\tb=ceHgy4OTk2GRf7B88v4P17J37+jQ/kP/IvlRDzZlRBq7vEolj8NehrX7VV55xS50+D\n\tmndzrkc2Nr1B+EyycsgSiAC80xOQkahw7j+tWGbS0c813QNlgD6HKr9RT12hFYqA4L6A\n\tf+RDKAgehLOFx6ECsNUMLTbYwgNBlNxMCi6UAn0vBC94ftUxljRf+l7RF9cjr4WPyriN\n\tUgTgJP4I9PNlU4VMVxcmedwj4WQwEzGe43mBQxP9GdPHg60Jj9OSB+rT+FIOxCjgcFJo\n\tr2ZHbrug62G/AptADjqfo69bE2yZyVVBoFIkup0nauMO7TJ796YnVMoJJ5x8ysUEmFfy\n\tmPzw==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=bbHcTRs3QV83hJfrwrYiql1Usy8uy0kAdGXH+AOPZQY=;\n\tb=ZSt9JRVB/RvxvQnH539eaQulxQ5S9SYGPWucc79+KyznZ4nBws31/MeZ83Boy6dSCU\n\tM9LdHNt6DDWmK2uw9TN8ECM9DXYZWHs+QuGZ+yzy7+riL9V/qil4U7zk7DekBUOLEGOb\n\tHWdwnNPjEtv/t68DcnVVElmGu83vxe5nME9zSlBJ56bk/GV58yZzR3SPOJAYdlEPQpmb\n\tAtS22wM6wuBS2G+B7+AALHJIZl5WDioPF4USYB3YM+K97C30wdYHKM5VJasMAc3HKT39\n\tyuvSuZq+RB6RlikLIh2PaQB9db0djOfCr/Wh2cOZKDq9wV8TNo6z1J5R+D6ce/iJ9m3y\n\tCq2Q==","X-Gm-Message-State":"AHPjjUjLUzGSHw5oV7jYpNwfHXHyM7kjfkUl6AIb77EnUX1hxOvO5tSh\n\tsZWNrnuS2nVmGA==","X-Google-Smtp-Source":"ADKCNb4sDjJ3iKA74JP+nD0SHxNYEL8CxgKWzPBz/QAiKgcMYNZNnUBDja28Ti5uOSvvcSm1OowUuA==","X-Received":"by 10.84.238.206 with SMTP id l14mr12456353pln.156.1505230688379;\n\tTue, 12 Sep 2017 08:38:08 -0700 (PDT)","Message-ID":"<1505230686.15310.141.camel@edumazet-glaptop3.roam.corp.google.com>","Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","From":"Eric Dumazet ","To":"liujian ","Cc":"davem@davemloft.net, kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.org,\n\tedumazet@google.com, ycheng@google.com, hkchu@google.com,\n\tnetdev@vger.kernel.org, weiyongjun1@huawei.com,\n\twangkefeng 00227729 ","Date":"Tue, 12 Sep 2017 08:38:06 -0700","In-Reply-To":"<1505228700.15310.138.camel@edumazet-glaptop3.roam.corp.google.com>","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>\n\t\n\t<1505228700.15310.138.camel@edumazet-glaptop3.roam.corp.google.com>","Content-Type":"text/plain; charset=\"UTF-8\"","X-Mailer":"Evolution 3.10.4-0ubuntu2 ","Mime-Version":"1.0","Content-Transfer-Encoding":"7bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1767599,"web_url":"http://patchwork.ozlabs.org/comment/1767599/","msgid":"","list_archive_url":null,"date":"2017-09-13T06:56:06","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":71967,"url":"http://patchwork.ozlabs.org/api/people/71967/","name":"liujian \\(CE\\)","email":"liujian56@huawei.com"},"content":"在 2017/9/12 23:38, Eric Dumazet 写道:\n> On Tue, 2017-09-12 at 08:05 -0700, Eric Dumazet wrote:\n>> On Tue, 2017-09-12 at 14:08 +0800, liujian wrote:\n>>> Hi,\n>>>\n>>> In the scenario, tcp server side IP changed, and at that memont,\n>>> userspace application still send data continuously;\n>>> tcp_send_head(sk)'s timestamp always be refreshed.\n>>>\n>>> Here is the packetdrill script:\n>>>\n>>> 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n>>> +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n>>> +0 bind(3, ..., ...) = 0\n>>> +0 listen(3, 1) = 0\n>>>\n>>> +0 < S 0:0(0) win 0 \n>>> +0 > S. 0:0(0) ack 1 \n>>>\n>>> +.1 < . 1:1(0) ack 1 win 65530\n>>> +0 accept(3, ..., ...) = 4\n>>>\n>>> +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n>>> +0 write(4, ..., 24) = 24\n>>> +0 > P. 1:25(24) ack 1 win 229\n>>> +.1 < . 1:1(0) ack 25 win 65530\n>>>\n>>> //change the ipaddress\n>>> +1 `ifconfig tun0 192.168.0.10/16`\n>>>\n>>> +1 write(4, ..., 24) = 24\n>>> +1 write(4, ..., 24) = 24\n>>> +1 write(4, ..., 24) = 24\n>>> +1 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>> +3 write(4, ..., 24) = 24\n>>>\n>>> +0 `ifconfig tun0 192.168.0.1/16`\n>>> +0 < . 1:1(0) ack 1 win 1000\n>>> +0 write(4, ..., 24) = -1\n>>>\n>>>\n>>\n>> This has nothing to do with the code patch you have changed.\n>>\n>> How have you tested your patch exactly ?\n>>\nI tested the patch, it can work.\n\n[root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\ntest.pkt:24: runtime error in write call: Expected result 24 but got -1 with errno 110 (Connection timed out)\n\nreal\t0m5.356s\nuser\t0m0.026s\nsys\t0m0.104s\n\n[root@localhost ~]# ss -toenmi src :8080\nState Recv-Q Send-Q Local Address:Port Peer Address:Port\nESTAB 0 48 192.168.0.1:8080 192.0.2.1:39559 timer:(persist,186ms,2) ino:37178 sk:6 <->\n\t skmem:(r0,rb369280,t0,tb87040,f1792,w2304,o0,bl0) sack cubic wscale:7,7 rto:301 backoff:2 rtt:100.253/37.643 mss:1460 cwnd:10 bytes_acked:24 segs_out:5 segs_in:3 send 1.2Mbps lastsnd:976 lastrcv:4082 lastack:3982 pacing_rate 2.3Mbps rcv_space:29200\n\n\nif change the TCP_USER_TIMEOUT to 30s, test result as below:\n\n[root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\ntest.pkt:37: runtime error in write call: Expected result 24 but got -1 with errno 110 (Connection timed out)\n\nreal\t0m44.362s\nuser\t0m0.018s\nsys\t0m0.110s\n\nESTAB 0 360 192.168.0.1:8080 192.0.2.1:47577 timer:(persist,516ms,6) ino:18806 sk:5 <->\n\t skmem:(r0,rb369280,t0,tb87040,f1792,w2304,o0,bl0) sack cubic wscale:7,7 rto:301 backoff:6 rtt:100.228/37.623 mss:10 bytes_acked:24 segs_out:22 segs_in:3 send 1.2Mbps lastsnd:2343 lastrcv:40450 lastack:40350 pacing_rate 2.3Mbps rcv_sp\n\n> \n> lpaa23:~# ss -toenmi src :8080\n> State Recv-Q Send-Q Local Address:Port Peer\n> Address:Port \n> ESTAB 0 144 192.168.134.161:8080\n> 192.0.2.1:51165 timer:(persist,8.262ms,5) ino:1\n> 82083 sk:3 <->\n> \t skmem:(r0,rb359040,t0,tb46080,f1792,w2304,o0,bl0,d0) sack cubic\n> wscale:7,8 rto:301 backoff:5 rtt:100.127/37.576 \n> mss:1460 rcvmss:536 advmss:1460 cwnd:10 bytes_acked:24 segs_out:12\n> segs_in:3 data_segs_out:12 send 1.2Mbps lastsnd:1370 l\n> astrcv:13348 lastack:13248 pacing_rate 2.3Mbps delivery_rate 116.7Kbps\n> app_limited busy:11346ms rcv_space:29200 notsent:1\n> 44 minrtt:100.043userspace application still send data continuously\n> \n> This is the typical RTO timer, not zero window probe.\n> \nwith the script, it is not zero window; but the code enter tcp_probe_timer.\n->tcp_sendmsg\n-->tcp_push\n---->__tcp_push_pending_frames\n------>tcp_write_xmit\n-------->tcp_transmit_skb (at this function, tcp_send_head(sk)'s timestamp was be refreshed)\n---------->ip_queue_xmit (can not find route, return -EHOSTUNREACH)\n-------->tcp_transmit_skb (send fail, do not advance send head)\n------->tcp_check_probe_timer (reset ICSK_TIME_PROBE0 timer)\n\nHere userspace application still send data continuously,\nit triggered ICSK_TIME_PROBE0 timer and tcp_send_head(sk)'s timestamp refresh.","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xsXX949yTz9sNV\n\tfor ;\n\tWed, 13 Sep 2017 16:56:57 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751388AbdIMG4z (ORCPT );\n\tWed, 13 Sep 2017 02:56:55 -0400","from szxga05-in.huawei.com ([45.249.212.191]:6042 \"EHLO\n\tszxga05-in.huawei.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1750999AbdIMG4y (ORCPT\n\t); Wed, 13 Sep 2017 02:56:54 -0400","from 172.30.72.58 (EHLO DGGEMS401-HUB.china.huawei.com)\n\t([172.30.72.58])\n\tby dggrg05-dlp.huawei.com (MOS 4.4.6-GA FastPath queued)\n\twith ESMTP id DHF09888; Wed, 13 Sep 2017 14:56:37 +0800 (CST)","from [127.0.0.1] (10.177.97.126) by DGGEMS401-HUB.china.huawei.com\n\t(10.3.19.201) with Microsoft SMTP Server id 14.3.301.0;\n\tWed, 13 Sep 2017 14:56:27 +0800"],"Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","To":"Eric Dumazet ","CC":", ,\n\t, ,\n\t, , ,\n\t,\n\twangkefeng 00227729 ","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>\n\t\n\t<1505228700.15310.138.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505230686.15310.141.camel@edumazet-glaptop3.roam.corp.google.com>","From":"liujian ","Message-ID":"","Date":"Wed, 13 Sep 2017 14:56:06 +0800","User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"<1505230686.15310.141.camel@edumazet-glaptop3.roam.corp.google.com>","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"8bit","X-Originating-IP":"[10.177.97.126]","X-CFilter-Loop":"Reflected","X-Mirapoint-Virus-RAPID-Raw":"score=unknown(0),\n\trefid=str=0001.0A090205.59B8D6A7.0006, ss=1, re=0.000, recu=0.000,\n\treip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0,\n\tso=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32","X-Mirapoint-Loop-Id":"4869d8a2ae786b7253b666d744f3b1de","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1767614,"web_url":"http://patchwork.ozlabs.org/comment/1767614/","msgid":"<3c750ac3-8739-89ed-ce41-d4cde755171e@huawei.com>","list_archive_url":null,"date":"2017-09-13T07:15:24","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":71967,"url":"http://patchwork.ozlabs.org/api/people/71967/","name":"liujian \\(CE\\)","email":"liujian56@huawei.com"},"content":"在 2017/9/13 14:56, liujian 写道:\n> \n> \n> 在 2017/9/12 23:38, Eric Dumazet 写道:\n>> On Tue, 2017-09-12 at 08:05 -0700, Eric Dumazet wrote:\n>>> On Tue, 2017-09-12 at 14:08 +0800, liujian wrote:\n>>>> Hi,\n>>>>\n>>>> In the scenario, tcp server side IP changed, and at that memont,\n>>>> userspace application still send data continuously;\n>>>> tcp_send_head(sk)'s timestamp always be refreshed.\n>>>>\n>>>> Here is the packetdrill script:\n>>>>\n>>>> 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n>>>> +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n>>>> +0 bind(3, ..., ...) = 0\n>>>> +0 listen(3, 1) = 0\n>>>>\n>>>> +0 < S 0:0(0) win 0 \n>>>> +0 > S. 0:0(0) ack 1 \n>>>>\n>>>> +.1 < . 1:1(0) ack 1 win 65530\n>>>> +0 accept(3, ..., ...) = 4\n>>>>\n>>>> +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n>>>> +0 write(4, ..., 24) = 24\n>>>> +0 > P. 1:25(24) ack 1 win 229\n>>>> +.1 < . 1:1(0) ack 25 win 65530\n>>>>\n>>>> //change the ipaddress\n>>>> +1 `ifconfig tun0 192.168.0.10/16`\n>>>>\n>>>> +1 write(4, ..., 24) = 24\n>>>> +1 write(4, ..., 24) = 24\n>>>> +1 write(4, ..., 24) = 24\n>>>> +1 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>> +3 write(4, ..., 24) = 24\n>>>>\n>>>> +0 `ifconfig tun0 192.168.0.1/16`\n>>>> +0 < . 1:1(0) ack 1 win 1000\n>>>> +0 write(4, ..., 24) = -1\n>>>>\n>>>>\n>>>\n>>> This has nothing to do with the code patch you have changed.\n>>>\n>>> How have you tested your patch exactly ?\n>>>\n> I tested the patch, it can work.\n> \n> [root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\n> test.pkt:24: runtime error in write call: Expected result 24 but got -1 with errno 110 (Connection timed out)\n> \n> real\t0m5.356s\n> user\t0m0.026s\n> sys\t0m0.104s\n> \n> [root@localhost ~]# ss -toenmi src :8080\n> State Recv-Q Send-Q Local Address:Port Peer Address:Port\n> ESTAB 0 48 192.168.0.1:8080 192.0.2.1:39559 timer:(persist,186ms,2) ino:37178 sk:6 <->\n> \t skmem:(r0,rb369280,t0,tb87040,f1792,w2304,o0,bl0) sack cubic wscale:7,7 rto:301 backoff:2 rtt:100.253/37.643 mss:1460 cwnd:10 bytes_acked:24 segs_out:5 segs_in:3 send 1.2Mbps lastsnd:976 lastrcv:4082 lastack:3982 pacing_rate 2.3Mbps rcv_space:29200\n> \n> \n> if change the TCP_USER_TIMEOUT to 30s, test result as below:\n> \n> [root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\n> test.pkt:37: runtime error in write call: Expected result 24 but got -1 with errno 110 (Connection timed out)\n> \n> real\t0m44.362s\n> user\t0m0.018s\n> sys\t0m0.110s\n> \n> ESTAB 0 360 192.168.0.1:8080 192.0.2.1:47577 timer:(persist,516ms,6) ino:18806 sk:5 <->\n> \t skmem:(r0,rb369280,t0,tb87040,f1792,w2304,o0,bl0) sack cubic wscale:7,7 rto:301 backoff:6 rtt:100.228/37.623 mss:10 bytes_acked:24 segs_out:22 segs_in:3 send 1.2Mbps lastsnd:2343 lastrcv:40450 lastack:40350 pacing_rate 2.3Mbps rcv_sp\n> \n>>\n>> lpaa23:~# ss -toenmi src :8080\n>> State Recv-Q Send-Q Local Address:Port Peer\n>> Address:Port \n>> ESTAB 0 144 192.168.134.161:8080\n>> 192.0.2.1:51165 timer:(persist,8.262ms,5) ino:1\n>> 82083 sk:3 <->\n>> \t skmem:(r0,rb359040,t0,tb46080,f1792,w2304,o0,bl0,d0) sack cubic\n>> wscale:7,8 rto:301 backoff:5 rtt:100.127/37.576 \n>> mss:1460 rcvmss:536 advmss:1460 cwnd:10 bytes_acked:24 segs_out:12\n>> segs_in:3 data_segs_out:12 send 1.2Mbps lastsnd:1370 l\n>> astrcv:13348 lastack:13248 pacing_rate 2.3Mbps delivery_rate 116.7Kbps\n>> app_limited busy:11346ms rcv_space:29200 notsent:1\n>> 44 minrtt:100.043userspace application still send data continuously\n>>\n>> This is the typical RTO timer, not zero window probe.\n>>\n> with the script, it is not zero window; but the code enter tcp_probe_timer.\n> ->tcp_sendmsg\n> -->tcp_push\n> ---->__tcp_push_pending_frames\n> ------>tcp_write_xmit\n> -------->tcp_transmit_skb (at this function, tcp_send_head(sk)'s timestamp was be refreshed)\n> ---------->ip_queue_xmit (can not find route, return -EHOSTUNREACH)\n> -------->tcp_transmit_skb (send fail, do not advance send head)\n> ------->tcp_check_probe_timer (reset ICSK_TIME_PROBE0 timer)\nsorry, change to\n->tcp_sendmsg\n-->tcp_push\n---->__tcp_push_pending_frames\n------>tcp_write_xmit\n-------->tcp_transmit_skb (at this function, tcp_send_head(sk)'s timestamp was be refreshed)\n---------->ip_queue_xmit (can not find route, return -EHOSTUNREACH)\n------>tcp_write_xmit (send fail, do not advance send head)\n------>tcp_check_probe_timer (reset ICSK_TIME_PROBE0 timer)\n\n> Here userspace application still send data continuously,\n> it triggered ICSK_TIME_PROBE0 timer and tcp_send_head(sk)'s timestamp refresh.\n> \n>","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":"ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xsXyS25gfz9sNw\n\tfor ;\n\tWed, 13 Sep 2017 17:16:16 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751497AbdIMHQM (ORCPT );\n\tWed, 13 Sep 2017 03:16:12 -0400","from szxga04-in.huawei.com ([45.249.212.190]:6462 \"EHLO\n\tszxga04-in.huawei.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751392AbdIMHQK (ORCPT\n\t); Wed, 13 Sep 2017 03:16:10 -0400","from 172.30.72.59 (EHLO DGGEMS412-HUB.china.huawei.com)\n\t([172.30.72.59])\n\tby dggrg04-dlp.huawei.com (MOS 4.4.6-GA FastPath queued)\n\twith ESMTP id DHD12820; Wed, 13 Sep 2017 15:15:49 +0800 (CST)","from [127.0.0.1] (10.177.97.126) by DGGEMS412-HUB.china.huawei.com\n\t(10.3.19.212) with Microsoft SMTP Server id 14.3.301.0;\n\tWed, 13 Sep 2017 15:15:40 +0800"],"Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","From":"liujian ","To":"Eric Dumazet ","CC":", ,\n\t, ,\n\t, , ,\n\t,\n\twangkefeng 00227729 ","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>\n\t\n\t<1505228700.15310.138.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505230686.15310.141.camel@edumazet-glaptop3.roam.corp.google.com>\n\t","Message-ID":"<3c750ac3-8739-89ed-ce41-d4cde755171e@huawei.com>","Date":"Wed, 13 Sep 2017 15:15:24 +0800","User-Agent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101\n\tThunderbird/52.3.0","MIME-Version":"1.0","In-Reply-To":"","Content-Type":"text/plain; charset=\"utf-8\"","Content-Transfer-Encoding":"8bit","X-Originating-IP":"[10.177.97.126]","X-CFilter-Loop":"Reflected","X-Mirapoint-Virus-RAPID-Raw":"score=unknown(0),\n\trefid=str=0001.0A020206.59B8DB25.00E5, ss=1, re=0.000, recu=0.000,\n\treip=0.000, cl=1, cld=1, fgs=0, ip=0.0.0.0,\n\tso=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32","X-Mirapoint-Loop-Id":"81853c1dfab888c415e3275a5fb5bddb","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}},{"id":1767937,"web_url":"http://patchwork.ozlabs.org/comment/1767937/","msgid":"<1505313647.15310.165.camel@edumazet-glaptop3.roam.corp.google.com>","list_archive_url":null,"date":"2017-09-13T14:40:47","subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","submitter":{"id":2404,"url":"http://patchwork.ozlabs.org/api/people/2404/","name":"Eric Dumazet","email":"eric.dumazet@gmail.com"},"content":"On Wed, 2017-09-13 at 15:15 +0800, liujian wrote:\n> \n> 在 2017/9/13 14:56, liujian 写道:\n> > \n> > \n> > 在 2017/9/12 23:38, Eric Dumazet 写道:\n> >> On Tue, 2017-09-12 at 08:05 -0700, Eric Dumazet wrote:\n> >>> On Tue, 2017-09-12 at 14:08 +0800, liujian wrote:\n> >>>> Hi,\n> >>>>\n> >>>> In the scenario, tcp server side IP changed, and at that memont,\n> >>>> userspace application still send data continuously;\n> >>>> tcp_send_head(sk)'s timestamp always be refreshed.\n> >>>>\n> >>>> Here is the packetdrill script:\n> >>>>\n> >>>> 0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n> >>>> +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n> >>>> +0 bind(3, ..., ...) = 0\n> >>>> +0 listen(3, 1) = 0\n> >>>>\n> >>>> +0 < S 0:0(0) win 0 \n> >>>> +0 > S. 0:0(0) ack 1 \n> >>>>\n> >>>> +.1 < . 1:1(0) ack 1 win 65530\n> >>>> +0 accept(3, ..., ...) = 4\n> >>>>\n> >>>> +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n> >>>> +0 write(4, ..., 24) = 24\n> >>>> +0 > P. 1:25(24) ack 1 win 229\n> >>>> +.1 < . 1:1(0) ack 25 win 65530\n> >>>>\n> >>>> //change the ipaddress\n> >>>> +1 `ifconfig tun0 192.168.0.10/16`\n> >>>>\n> >>>> +1 write(4, ..., 24) = 24\n> >>>> +1 write(4, ..., 24) = 24\n> >>>> +1 write(4, ..., 24) = 24\n> >>>> +1 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>> +3 write(4, ..., 24) = 24\n> >>>>\n> >>>> +0 `ifconfig tun0 192.168.0.1/16`\n> >>>> +0 < . 1:1(0) ack 1 win 1000\n> >>>> +0 write(4, ..., 24) = -1\n> >>>>\n> >>>>\n> >>>\n> >>> This has nothing to do with the code patch you have changed.\n> >>>\n> >>> How have you tested your patch exactly ?\n> >>>\n> > I tested the patch, it can work.\n> > \n> > [root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\n> > test.pkt:24: runtime error in write call: Expected result 24 but got -1 with errno 110 (Connection timed out)\n> > \n> > real\t0m5.356s\n> > user\t0m0.026s\n> > sys\t0m0.104s\n> > \n> > [root@localhost ~]# ss -toenmi src :8080\n> > State Recv-Q Send-Q Local Address:Port Peer Address:Port\n> > ESTAB 0 48 192.168.0.1:8080 192.0.2.1:39559 timer:(persist,186ms,2) ino:37178 sk:6 <->\n> > \t skmem:(r0,rb369280,t0,tb87040,f1792,w2304,o0,bl0) sack cubic wscale:7,7 rto:301 backoff:2 rtt:100.253/37.643 mss:1460 cwnd:10 bytes_acked:24 segs_out:5 segs_in:3 send 1.2Mbps lastsnd:976 lastrcv:4082 lastack:3982 pacing_rate 2.3Mbps rcv_space:29200\n> > \n> > \n> > if change the TCP_USER_TIMEOUT to 30s, test result as below:\n> > \n> > [root@localhost ~]# time ./gtests/net/packetdrill/packetdrill test.pkt\n> > test.pkt:37: runtime error in write call: Expected result 24 but got -1 with errno 110 (Connection timed out)\n> > \n> > real\t0m44.362s\n> > user\t0m0.018s\n> > sys\t0m0.110s\n> > \n> > ESTAB 0 360 192.168.0.1:8080 192.0.2.1:47577 timer:(persist,516ms,6) ino:18806 sk:5 <->\n> > \t skmem:(r0,rb369280,t0,tb87040,f1792,w2304,o0,bl0) sack cubic wscale:7,7 rto:301 backoff:6 rtt:100.228/37.623 mss:10 bytes_acked:24 segs_out:22 segs_in:3 send 1.2Mbps lastsnd:2343 lastrcv:40450 lastack:40350 pacing_rate 2.3Mbps rcv_sp\n> > \n> >>\n> >> lpaa23:~# ss -toenmi src :8080\n> >> State Recv-Q Send-Q Local Address:Port Peer\n> >> Address:Port \n> >> ESTAB 0 144 192.168.134.161:8080\n> >> 192.0.2.1:51165 timer:(persist,8.262ms,5) ino:1\n> >> 82083 sk:3 <->\n> >> \t skmem:(r0,rb359040,t0,tb46080,f1792,w2304,o0,bl0,d0) sack cubic\n> >> wscale:7,8 rto:301 backoff:5 rtt:100.127/37.576 \n> >> mss:1460 rcvmss:536 advmss:1460 cwnd:10 bytes_acked:24 segs_out:12\n> >> segs_in:3 data_segs_out:12 send 1.2Mbps lastsnd:1370 l\n> >> astrcv:13348 lastack:13248 pacing_rate 2.3Mbps delivery_rate 116.7Kbps\n> >> app_limited busy:11346ms rcv_space:29200 notsent:1\n> >> 44 minrtt:100.043userspace application still send data continuously\n> >>\n> >> This is the typical RTO timer, not zero window probe.\n> >>\n> > with the script, it is not zero window; but the code enter tcp_probe_timer.\n> > ->tcp_sendmsg\n> > -->tcp_push\n> > ---->__tcp_push_pending_frames\n> > ------>tcp_write_xmit\n> > -------->tcp_transmit_skb (at this function, tcp_send_head(sk)'s timestamp was be refreshed)\n> > ---------->ip_queue_xmit (can not find route, return -EHOSTUNREACH)\n> > -------->tcp_transmit_skb (send fail, do not advance send head)\n> > ------->tcp_check_probe_timer (reset ICSK_TIME_PROBE0 timer)\n> sorry, change to\n> ->tcp_sendmsg\n> -->tcp_push\n> ---->__tcp_push_pending_frames\n> ------>tcp_write_xmit\n> -------->tcp_transmit_skb (at this function, tcp_send_head(sk)'s timestamp was be refreshed)\n> ---------->ip_queue_xmit (can not find route, return -EHOSTUNREACH)\n> ------>tcp_write_xmit (send fail, do not advance send head)\n> ------>tcp_check_probe_timer (reset ICSK_TIME_PROBE0 timer)\n\n\nOK but :\n\n1) keepalive_time_elapsed() returns jiffies, and you compare it to a ms\nvalue. It will break on kernels with HZ=100\n\n\n2) keepalive_time_elapsed() is reset every time a packet is received.\n\nWith following packetdrill your patch (even if we fix 1)) breaks again\n\n\n# ./packetdrill --local_ip 192.168.102.64 user_timeout.pkt\n# cat user_timeout.pkt \n\n0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3\n +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0\n +0 bind(3, ..., ...) = 0\n +0 listen(3, 1) = 0\n\n +0 `ifconfig tun0 192.168.102.64/16; ip ro add 192.0.2.1 dev tun0`\n\n +0 < S 0:0(0) win 0 \n +0 > S. 0:0(0) ack 1 \n\n +.1 < . 1:1(0) ack 1 win 65530\n +0 accept(3, ..., ...) = 4\n\n +0 setsockopt(4, SOL_TCP, TCP_USER_TIMEOUT, [3000], 4) = 0\n +0 write(4, ..., 24) = 24\n +0 > P. 1:25(24) ack 1 win 115\n +.1 < . 1:1(0) ack 25 win 65530\n\n//change the ipaddress\n +1 `ifconfig tun0 192.168.0.10/16`\n\n +1 write(4, ..., 24) = 24\n +1 write(4, ..., 24) = 24\n +1 write(4, ..., 24) = 24\n +1 write(4, ..., 24) = 24\n\n +0 `ifconfig tun0 192.168.102.64/16`\n +0 < . 1:2(1) ack 25 win 65530\n +0 `ifconfig tun0 192.168.0.10/16`\n\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n +3 write(4, ..., 24) = 24\n\n +0 `ifconfig tun0 192.168.0.1/16`\n +0 < . 1:1(0) ack 1 win 1000\n +0 write(4, ..., 24) = -1\n\n\nI would instead try to keep a meaningful value in\ntcp_send_head(sk)->skb_mstamp if the transmit failed.\n\nOr more exactly change skb->skb_mstamp only on (successfully)\ntransmitted skb.\n\nI will provide a patch ( fixing net/ipv4/tcp_output.c )","headers":{"Return-Path":"","X-Original-To":"patchwork-incoming@ozlabs.org","Delivered-To":"patchwork-incoming@ozlabs.org","Authentication-Results":["ozlabs.org;\n\tspf=none (mailfrom) smtp.mailfrom=vger.kernel.org\n\t(client-ip=209.132.180.67; helo=vger.kernel.org;\n\tenvelope-from=netdev-owner@vger.kernel.org;\n\treceiver=)","ozlabs.org; dkim=pass (2048-bit key;\n\tunprotected) header.d=gmail.com header.i=@gmail.com\n\theader.b=\"F4fXU8Q0\"; dkim-atps=neutral"],"Received":["from vger.kernel.org (vger.kernel.org [209.132.180.67])\n\tby ozlabs.org (Postfix) with ESMTP id 3xskqq4hJbz9sNr\n\tfor ;\n\tThu, 14 Sep 2017 00:41:11 +1000 (AEST)","(majordomo@vger.kernel.org) by vger.kernel.org via listexpand\n\tid S1751953AbdIMOlJ (ORCPT );\n\tWed, 13 Sep 2017 10:41:09 -0400","from mail-pf0-f195.google.com ([209.85.192.195]:33712 \"EHLO\n\tmail-pf0-f195.google.com\" rhost-flags-OK-OK-OK-OK) by vger.kernel.org\n\twith ESMTP id S1751081AbdIMOlH (ORCPT\n\t); Wed, 13 Sep 2017 10:41:07 -0400","by mail-pf0-f195.google.com with SMTP id h4so176546pfk.0\n\tfor ; Wed, 13 Sep 2017 07:41:07 -0700 (PDT)","from [10.1.104.73] ([207.198.105.19])\n\tby smtp.googlemail.com with ESMTPSA id\n\tk27sm20949259pfk.20.2017.09.13.07.40.57\n\t(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);\n\tWed, 13 Sep 2017 07:41:06 -0700 (PDT)"],"DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=gmail.com; s=20161025;\n\th=message-id:subject:from:to:cc:date:in-reply-to:references\n\t:mime-version:content-transfer-encoding;\n\tbh=JgJaZK9HC884trYHaYcU38pDkUAkJtabNy6IAKOoK+s=;\n\tb=F4fXU8Q0ZWq1pDexVy7LAEcD4y5W2vCvUnRfSUB6NpQ939QFmAzMMKkMng8Z+lsOOF\n\tRgcRTtoMzix/Mg0NShZyb46L+4UH1Lo+jTIaxZ9rlvw1TzTLDsN7iYSjRALsYBvPkOqR\n\ttPtEqoGreHMlDfq8J4LTCzz7wU1x2E/4daffHVHqPgp73K/RdPEDwmGXgbRfqxVGbEUg\n\tWYzq+7K3tfMUWHXIcLgmIbY8Hou2HabvaVC9MIPyzsy2gUpgngr04j37RZVwBXqS4SAs\n\tPBoGeRCb5VEw98WYCMiYV9OTeYjeaCQ/d5uPD2UFIFoEsglc5zhkBTuQDrE1241tpGGA\n\t1HdQ==","X-Google-DKIM-Signature":"v=1; a=rsa-sha256; c=relaxed/relaxed;\n\td=1e100.net; s=20161025;\n\th=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to\n\t:references:mime-version:content-transfer-encoding;\n\tbh=JgJaZK9HC884trYHaYcU38pDkUAkJtabNy6IAKOoK+s=;\n\tb=YokemKWyQIn94Kr6GQ94FGXZVNEhF+qT3QxEVmMTKi3PYl67yfs/tp76liJ06mtJmz\n\tniACk/ac9usGVC3cdF60AIGTgaDMQaXyIwqwTXu8qCYilojs784KzBkKRsKT3W0RewPo\n\tW0jkIKvvHj0SGrKVQyYLDNJoEECxPuOcEpimHnrqsjQv8aJT8B7FBnrI/qbmb9hgP0YQ\n\tRPTu3OX/8M3/rLcNUHS9gxWSfwuRhYSP/3pXUhjSd71s5BKhflkrNdYnT/kFqrqCy9eM\n\tNPMxaGXfzxH/Ilsbr9Ws9r+e/eSJIFgV+fM1bIS6sru5HKm3K86YjtZwRJMtlnQg0a9u\n\tL+Zw==","X-Gm-Message-State":"AHPjjUg+BYFYLRAbmRBU23Cd63IJpMtjHZNNAFAgGhnuu83iUXYriN+Q\n\tUZTjnOp1C2OWYg==","X-Google-Smtp-Source":"ADKCNb6OIvJvx8jICZcIMD+JE7cZ4yLHE2kMoFoH4kev8L0U2aBq8rImuuGJfsIwvHek6niBWDy28A==","X-Received":"by 10.84.209.204 with SMTP id y70mr20634340plh.183.1505313667353;\n\tWed, 13 Sep 2017 07:41:07 -0700 (PDT)","Message-ID":"<1505313647.15310.165.camel@edumazet-glaptop3.roam.corp.google.com>","Subject":"Re: [PATCH] tcp: TCP_USER_TIMEOUT can not work in tcp_probe_timer()","From":"Eric Dumazet ","To":"liujian ","Cc":"davem@davemloft.net, kuznet@ms2.inr.ac.ru, yoshfuji@linux-ipv6.org,\n\tedumazet@google.com, ycheng@google.com, hkchu@google.com,\n\tnetdev@vger.kernel.org, weiyongjun1@huawei.com,\n\twangkefeng 00227729 ","Date":"Wed, 13 Sep 2017 07:40:47 -0700","In-Reply-To":"<3c750ac3-8739-89ed-ce41-d4cde755171e@huawei.com>","References":"<1505111262-12620-1-git-send-email-liujian56@huawei.com>\n\t<1505141357.15310.115.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505142785.15310.117.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505143348.15310.118.camel@edumazet-glaptop3.roam.corp.google.com>\n\t\n\t<1505228700.15310.138.camel@edumazet-glaptop3.roam.corp.google.com>\n\t<1505230686.15310.141.camel@edumazet-glaptop3.roam.corp.google.com>\n\t\n\t<3c750ac3-8739-89ed-ce41-d4cde755171e@huawei.com>","Content-Type":"text/plain; charset=\"UTF-8\"","X-Mailer":"Evolution 3.10.4-0ubuntu2 ","Mime-Version":"1.0","Content-Transfer-Encoding":"8bit","Sender":"netdev-owner@vger.kernel.org","Precedence":"bulk","List-ID":"","X-Mailing-List":"netdev@vger.kernel.org"}}]