From patchwork Mon Sep 18 07:41:15 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Cochran X-Patchwork-Id: 814802 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3xwdLr6wvDz9s3w for ; Mon, 18 Sep 2017 17:44:36 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752706AbdIRHoY (ORCPT ); Mon, 18 Sep 2017 03:44:24 -0400 Received: from Galois.linutronix.de ([146.0.238.70]:49795 "EHLO Galois.linutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752519AbdIRHnZ (ORCPT ); Mon, 18 Sep 2017 03:43:25 -0400 Received: from rcochran by Galois.linutronix.de with local (Exim 4.80) (envelope-from ) id 1dtqgD-0006bH-3S; Mon, 18 Sep 2017 09:41:21 +0200 From: Richard Cochran To: Cc: , intel-wired-lan@lists.osuosl.org, Andre Guedes , Anna-Maria Gleixner , David Miller , Henrik Austad , Jesus Sanchez-Palencia , John Stultz , Thomas Gleixner , Vinicius Costa Gomes Subject: [PATCH RFC V1 net-next 0/6] Time based packet transmission Date: Mon, 18 Sep 2017 09:41:15 +0200 Message-Id: X-Mailer: git-send-email 1.7.10.4 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This series is an early RFC that introduces a new socket option allowing time based transmission of packets. This option will be useful in implementing various real time protocols over Ethernet, including but not limited to P802.1Qbv, which is currently finding its way into 802.1Q. * Open questions about SO_TXTIME semantics - What should the kernel do if the dialed Tx time is in the past? Should the packet be sent ASAP, or should we throw an error? - Should the kernel inform the user if it detects a missed deadline, via the error queue for example? - What should the timescale be for the dialed Tx time? Should the kernel select UTC when using the SW Qdisc and the HW time otherwise? Or should the socket option include a clockid_t? * Things todo - Design a Qdisc for purpose of configuring SO_TXTIME. There should be one option to dial HW offloading or SW best effort. - Implement the SW best effort variant. Here is my back of the napkin sketch. Each interface has its own timerqueue keeping the TXTIME packets in order and a FIFO for all other traffic. A guard window starts at the earliest deadline minus the maximum MTU minus a configurable fudge factor. The Qdisc uses a hrtimer to transmit the next packet in the timerqueue. During the guard window, all other traffic is defered unless the next packet can be transmitted before the guard window expires. * Current limitations - The driver does not handle out of order packets. If user space sends a packet with an earlier Tx time, then the code should stop the queue, reshuffle the descriptors accordingly, and then restart the queue. - The driver does not correctly queue up packets in the distant future. The i210 has a limited time window of +/- 0.5 seconds. Packets with a Tx time greater than that should be deferred in order to enqueue them later on. * Performance measurements 1. Prepared a PC and the Device Under Test (DUT) each with an Intel i210 card connected with a crossover cable. 2. The DUT was a Pentium(R) D CPU 2.80GHz running PREEMPT_RT 4.9.40-rt30 with about 50 usec maximum latency under cyclictest. 3. Synchronized the DUT's PHC to the PC's PHC using ptp4l. 4. Synchronized the DUT's system clock to its PHC using phc2sys. 5. Started netperf to produce some network load. 6. Measured the arrival time of the packets at the PC's PHC using hardware time stamping. I ran ten minute tests both with and without using the so_txtime option, with a period was 1 millisecond. I then repeated the so_txtime case but with a 250 microsecond period. The measured offset from the expected period (in nanoseconds) is shown in the following table. | | plain preempt_rt | so_txtime | txtime @ 250 us | |---------+------------------+---------------+-----------------| | min: | +1.940800e+04 | +4.720000e+02 | +4.720000e+02 | | max: | +7.556000e+04 | +5.680000e+02 | +5.760000e+02 | | pk-pk: | +5.615200e+04 | +9.600000e+01 | +1.040000e+02 | | mean: | +3.292776e+04 | +5.072274e+02 | +5.073602e+02 | | stddev: | +6.514709e+03 | +1.310849e+01 | +1.507144e+01 | | count: | 600000 | 600000 | 2400000 | Using so_txtime, the peak to peak jitter is about 100 nanoseconds, independent of the period. In contrast, plain preempt_rt shows a jitter of of 56 microseconds. The average delay of 507 nanoseconds when using so_txtime is explained by the documented input and output delays on the i210 cards. The test program is appended, below. If anyone is interested in reproducing this test, I can provide helper scripts. Thanks, Richard Richard Cochran (6): net: Add a new socket option for a future transmit time. net: skbuff: Add a field to support time based transmission. net: ipv4: raw: Hook into time based transmission. net: ipv4: udp: Hook into time based transmission. net: packet: Hook into time based transmission. net: igb: Implement time based transmission. arch/alpha/include/uapi/asm/socket.h | 3 ++ arch/frv/include/uapi/asm/socket.h | 3 ++ arch/ia64/include/uapi/asm/socket.h | 3 ++ arch/m32r/include/uapi/asm/socket.h | 3 ++ arch/mips/include/uapi/asm/socket.h | 3 ++ arch/mn10300/include/uapi/asm/socket.h | 3 ++ arch/parisc/include/uapi/asm/socket.h | 3 ++ arch/powerpc/include/uapi/asm/socket.h | 3 ++ arch/s390/include/uapi/asm/socket.h | 3 ++ arch/sparc/include/uapi/asm/socket.h | 3 ++ arch/xtensa/include/uapi/asm/socket.h | 3 ++ drivers/net/ethernet/intel/igb/e1000_82575.h | 1 + drivers/net/ethernet/intel/igb/e1000_defines.h | 68 +++++++++++++++++++++++++- drivers/net/ethernet/intel/igb/e1000_regs.h | 5 ++ drivers/net/ethernet/intel/igb/igb.h | 3 +- drivers/net/ethernet/intel/igb/igb_main.c | 68 +++++++++++++++++++++++--- include/linux/skbuff.h | 2 + include/net/sock.h | 2 + include/uapi/asm-generic/socket.h | 3 ++ net/core/sock.c | 12 +++++ net/ipv4/raw.c | 2 + net/ipv4/udp.c | 5 +- net/packet/af_packet.c | 6 +++ 23 files changed, 200 insertions(+), 10 deletions(-)