From patchwork Thu Dec 17 10:32:53 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Michael S. Tsirkin" X-Patchwork-Id: 558200 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 03B861401AD for ; Thu, 17 Dec 2015 21:34:26 +1100 (AEDT) Received: from localhost ([::1]:52096 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a9Vt9-0002GU-UW for incoming@patchwork.ozlabs.org; Thu, 17 Dec 2015 05:34:23 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41465) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a9VsQ-0001K3-BR for qemu-devel@nongnu.org; Thu, 17 Dec 2015 05:34:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a9Vrm-0007i9-BS for qemu-devel@nongnu.org; Thu, 17 Dec 2015 05:33:38 -0500 Received: from mx1.redhat.com ([209.132.183.28]:39821) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a9Vrl-0007gx-8k for qemu-devel@nongnu.org; Thu, 17 Dec 2015 05:32:58 -0500 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by mx1.redhat.com (Postfix) with ESMTPS id C117B8E748; Thu, 17 Dec 2015 10:32:56 +0000 (UTC) Received: from redhat.com (vpn1-7-193.ams2.redhat.com [10.36.7.193]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with SMTP id tBHAWssr023339; Thu, 17 Dec 2015 05:32:55 -0500 Date: Thu, 17 Dec 2015 12:32:53 +0200 From: "Michael S. Tsirkin" To: linux-kernel@vger.kernel.org Message-ID: <1450347932-16325-1-git-send-email-mst@redhat.com> MIME-Version: 1.0 Content-Disposition: inline X-Mutt-Fcc: =sent X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 209.132.183.28 Cc: Jason Wang , qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org Subject: [Qemu-devel] [PATCH] virtio_ring: use smp_store_mb X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org We need a full barrier after writing out event index, using smp_store_mb there seems better than open-coding. As usual, we need a wrapper to account for strong barriers/non smp. It's tempting to use this in vhost as well, for that, we'll need a variant of smp_store_mb that works on __user pointers. Signed-off-by: Michael S. Tsirkin --- Seems to give a speedup on my box but I'm less sure about this one. E.g. as xchng faster than mfence on all/most intel CPUs? Anyone has an opinion? include/linux/virtio_ring.h | 14 ++++++++++++++ drivers/virtio/virtio_ring.c | 15 +++++++++------ 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/include/linux/virtio_ring.h b/include/linux/virtio_ring.h index 0135c16..8912189 100644 --- a/include/linux/virtio_ring.h +++ b/include/linux/virtio_ring.h @@ -47,6 +47,20 @@ static inline void virtio_wmb(bool weak_barriers) wmb(); } +static inline void virtio_store_mb(bool weak_barriers, + __virtio16 *p, __virtio16 v) +{ +#ifdef CONFIG_SMP + if (weak_barriers) + smp_store_mb(*p, v); + else +#endif + { + WRITE_ONCE(*p, v); + mb(); + } +} + static inline __virtio16 virtio_load_acquire(bool weak_barriers, __virtio16 *p) { if (!weak_barriers) { diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f822cab..b0aea67 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -517,10 +517,10 @@ void *virtqueue_get_buf(struct virtqueue *_vq, unsigned int *len) /* If we expect an interrupt for the next entry, tell host * by writing event index and flush out the write before * the read in the next get_buf call. */ - if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { - vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx); - virtio_mb(vq->weak_barriers); - } + if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->vring), + cpu_to_virtio16(_vq->vdev, vq->last_used_idx)); #ifdef DEBUG vq->last_add_time_valid = false; @@ -653,8 +653,11 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_vq) } /* TODO: tune this threshold */ bufs = (u16)(vq->avail_idx_shadow - vq->last_used_idx) * 3 / 4; - vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs); - virtio_mb(vq->weak_barriers); + + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->vring), + cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs)); + if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->vring.used->idx) - vq->last_used_idx) > bufs)) { END_USE(vq); return false;