From patchwork Thu Dec 21 01:48:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1878788 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EgM8k2Gx; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SwYKj2gMBz20Qt for ; Thu, 21 Dec 2023 12:50:32 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rG8BY-00083i-6Y; Wed, 20 Dec 2023 20:49:16 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8BW-00081d-NX for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:14 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8BV-0002fC-67 for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703123352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Z6GWwX5cBUe5INCQZNIbh12/Fu0Hcv2jrD8zOggAzlc=; b=EgM8k2Gx9ZkOrIQhD88SHcLc1Lmh7DC+/ID3P/QftRBYbLjdnOZiFK3wP3WSh2QSRLCw+o bBDHP/XCTJgiAf+A9CbM8t4r1+SbYDsewOdCuLNiWk6/ge1nBPT70sXJc7Cp7ChTNv8ZV2 EUf1rcSQUqFYTI4t/I9isoSNrL2elyE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-695-J2oDjf1hNgyS2vlwzRgPNA-1; Wed, 20 Dec 2023 20:49:08 -0500 X-MC-Unique: J2oDjf1hNgyS2vlwzRgPNA-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 855CC185A780; Thu, 21 Dec 2023 01:49:08 +0000 (UTC) Received: from localhost (unknown [10.39.192.79]) by smtp.corp.redhat.com (Postfix) with ESMTP id DD2322026D66; Thu, 21 Dec 2023 01:49:07 +0000 (UTC) From: Stefan Hajnoczi To: Kevin Wolf , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Leonardo Bras , qemu-block@nongnu.org, Fam Zheng , Paolo Bonzini , Vladimir Sementsov-Ogievskiy , Fabiano Rosas , Eric Blake , Hanna Reitz , Juan Quintela , Peter Xu Subject: [PATCH 1/6] fixup block-coroutine-wrapper: use qemu_get_current_aio_context() Date: Wed, 20 Dec 2023 20:48:58 -0500 Message-ID: <20231221014903.1537962-2-stefanha@redhat.com> In-Reply-To: <20231221014903.1537962-1-stefanha@redhat.com> References: <20231221014903.1537962-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 Received-SPF: pass client-ip=170.10.129.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.063, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org qemu-iotests 051 fails on my machine so the change to 051.pc.out made by the above commit appears to be incorrect, at least against the current QEMU source tree. Revert it so that 051 passes again. Signed-off-by: Stefan Hajnoczi --- tests/qemu-iotests/051.pc.out | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/051.pc.out b/tests/qemu-iotests/051.pc.out index 86a3e113c4..7e10c5fa1b 100644 --- a/tests/qemu-iotests/051.pc.out +++ b/tests/qemu-iotests/051.pc.out @@ -177,11 +177,11 @@ QEMU X.Y.Z monitor - type 'help' for more information Testing: -drive file=TEST_DIR/t.qcow2,if=none,node-name=disk -object iothread,id=thread0 -device virtio-scsi,iothread=thread0,id=virtio-scsi0 -device scsi-hd,bus=virtio-scsi0.0,drive=disk,share-rw=on -device lsi53c895a,id=lsi0 -device scsi-hd,bus=lsi0.0,drive=disk,share-rw=on QEMU X.Y.Z monitor - type 'help' for more information -QEMU_PROG: -device scsi-hd,bus=lsi0.0,drive=disk,share-rw=on: HBA does not support iothreads +(qemu) QEMU_PROG: -device scsi-hd,bus=lsi0.0,drive=disk,share-rw=on: HBA does not support iothreads Testing: -drive file=TEST_DIR/t.qcow2,if=none,node-name=disk -object iothread,id=thread0 -device virtio-scsi,iothread=thread0,id=virtio-scsi0 -device scsi-hd,bus=virtio-scsi0.0,drive=disk,share-rw=on -device virtio-scsi,id=virtio-scsi1 -device scsi-hd,bus=virtio-scsi1.0,drive=disk,share-rw=on QEMU X.Y.Z monitor - type 'help' for more information -QEMU_PROG: -device scsi-hd,bus=virtio-scsi1.0,drive=disk,share-rw=on: Cannot change iothread of active block backend +(qemu) QEMU_PROG: -device scsi-hd,bus=virtio-scsi1.0,drive=disk,share-rw=on: Cannot change iothread of active block backend Testing: -drive file=TEST_DIR/t.qcow2,if=none,node-name=disk -object iothread,id=thread0 -device virtio-scsi,iothread=thread0,id=virtio-scsi0 -device scsi-hd,bus=virtio-scsi0.0,drive=disk,share-rw=on -device virtio-blk-pci,drive=disk,iothread=thread0,share-rw=on QEMU X.Y.Z monitor - type 'help' for more information From patchwork Thu Dec 21 01:48:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1878785 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=emVduDGB; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SwYKj2JXxz1ydg for ; Thu, 21 Dec 2023 12:50:32 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rG8Bc-00085m-GU; Wed, 20 Dec 2023 20:49:20 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Ba-00084J-2R for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:18 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8BY-0002gG-Mc for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703123356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mLpAPhOLexRIuQHAy6nH/DLSJaRi5up7/+uWx1QOq3Y=; b=emVduDGBYPEEaUKcR9Cz7CPFzRJYG21kyBQH5sOaGfddp35nNluHBId36ORgSevV9Rabk8 20vhAQxsErQoX2AElzX0/DfyBregZzwbW4X63L69gJRi3b/UvmCO5RfJ+uRXaoSk9Pt05W qvdbA9eflzzmaXIQ3hU7+g0wCiptXpo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-656-9Bjqdhu0OkaVh7I9ctXvmw-1; Wed, 20 Dec 2023 20:49:11 -0500 X-MC-Unique: 9Bjqdhu0OkaVh7I9ctXvmw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1A1901C05EA0; Thu, 21 Dec 2023 01:49:11 +0000 (UTC) Received: from localhost (unknown [10.39.192.79]) by smtp.corp.redhat.com (Postfix) with ESMTP id 71E322026D66; Thu, 21 Dec 2023 01:49:10 +0000 (UTC) From: Stefan Hajnoczi To: Kevin Wolf , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Leonardo Bras , qemu-block@nongnu.org, Fam Zheng , Paolo Bonzini , Vladimir Sementsov-Ogievskiy , Fabiano Rosas , Eric Blake , Hanna Reitz , Juan Quintela , Peter Xu Subject: [PATCH 2/6] fixup block: remove AioContext locking Date: Wed, 20 Dec 2023 20:48:59 -0500 Message-ID: <20231221014903.1537962-3-stefanha@redhat.com> In-Reply-To: <20231221014903.1537962-1-stefanha@redhat.com> References: <20231221014903.1537962-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.4 Received-SPF: pass client-ip=170.10.133.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.063, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Signed-off-by: Stefan Hajnoczi --- migration/block.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/migration/block.c b/migration/block.c index 2bcfcbfdf6..6ec6a1d6e6 100644 --- a/migration/block.c +++ b/migration/block.c @@ -311,10 +311,17 @@ static int mig_save_device_bulk(QEMUFile *f, BlkMigDevState *bmds) block_mig_state.submitted++; blk_mig_unlock(); + /* + * The migration thread does not have an AioContext. Lock the BQL so that + * I/O runs in the main loop AioContext (see + * qemu_get_current_aio_context()). + */ + qemu_mutex_lock_iothread(); bdrv_reset_dirty_bitmap(bmds->dirty_bitmap, cur_sector * BDRV_SECTOR_SIZE, nr_sectors * BDRV_SECTOR_SIZE); blk->aiocb = blk_aio_preadv(bb, cur_sector * BDRV_SECTOR_SIZE, &blk->qiov, 0, blk_mig_read_cb, blk); + qemu_mutex_unlock_iothread(); bmds->cur_sector = cur_sector + nr_sectors; return (bmds->cur_sector >= total_sectors); From patchwork Thu Dec 21 01:49:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1878789 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FSbiSLWQ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SwYKk3wtPz20Qv for ; Thu, 21 Dec 2023 12:50:34 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rG8Bb-00085T-IF; Wed, 20 Dec 2023 20:49:19 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Ba-00085L-Th for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:18 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8BZ-0002gP-IK for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703123357; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Hfh2gOX+NIq1CVs5MWxn7jOurKj+miGZbn1OE/3GOcY=; b=FSbiSLWQl3yj938BfIrKwx4A8yKAzgJl7Z/eIcKNctkCGs0H9J4hxT+H4VDO4I5AAWEbsN S+k4lnMmHu6rPBbRWCFxsHzg2pCI5KhpJMFhOdUGMcywQxSAsi1y5GOuJwKrDrVOqfwycm JXaQeBKBtJZLDr5SPoJ/eHn1k93ivYQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-668-1ggOgb_1PoCiFPfwvSyeSA-1; Wed, 20 Dec 2023 20:49:13 -0500 X-MC-Unique: 1ggOgb_1PoCiFPfwvSyeSA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5905D185A780; Thu, 21 Dec 2023 01:49:13 +0000 (UTC) Received: from localhost (unknown [10.39.192.79]) by smtp.corp.redhat.com (Postfix) with ESMTP id B57311121306; Thu, 21 Dec 2023 01:49:12 +0000 (UTC) From: Stefan Hajnoczi To: Kevin Wolf , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Leonardo Bras , qemu-block@nongnu.org, Fam Zheng , Paolo Bonzini , Vladimir Sementsov-Ogievskiy , Fabiano Rosas , Eric Blake , Hanna Reitz , Juan Quintela , Peter Xu Subject: [PATCH 3/6] fixup scsi: only access SCSIDevice->requests from one thread Date: Wed, 20 Dec 2023 20:49:00 -0500 Message-ID: <20231221014903.1537962-4-stefanha@redhat.com> In-Reply-To: <20231221014903.1537962-1-stefanha@redhat.com> References: <20231221014903.1537962-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.063, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Signed-off-by: Stefan Hajnoczi Reviewed-by: Paolo Bonzini --- hw/scsi/scsi-bus.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c index df68a44b6a..5b08cbf60a 100644 --- a/hw/scsi/scsi-bus.c +++ b/hw/scsi/scsi-bus.c @@ -127,7 +127,8 @@ static void scsi_device_for_each_req_async_bh(void *opaque) */ ctx = blk_get_aio_context(s->conf.blk); if (ctx != qemu_get_current_aio_context()) { - aio_bh_schedule_oneshot(ctx, scsi_device_for_each_req_async_bh, data); + aio_bh_schedule_oneshot(ctx, scsi_device_for_each_req_async_bh, + g_steal_pointer(&data)); return; } From patchwork Thu Dec 21 01:49:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1878786 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=YrPtnGzF; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SwYKj2NFmz20Gb for ; Thu, 21 Dec 2023 12:50:32 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rG8Bl-00087u-Cy; Wed, 20 Dec 2023 20:49:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Be-00086f-FX for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:22 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Bd-0002hB-1t for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703123360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qruMacLlFsPykvuMWRHnESpS+zRnMwzUPQmLEI3rCE8=; b=YrPtnGzFqQ0+NSFw7e2M3hax1GCtwvOvo5whpfnkoQN7ApuVdEPHMXOIUc4IqZg/OnIj1L doDHgYJ+CFOU9ApcYp68wdA0pwdSdy4NZffhtfE/lQ6Nz9QlDYybdCyUEOfLtnmM/5icmu GCFN/+/7R8nKx0+r1f46ZHRS2FlYYek= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-687-4tWsDGHQOpisg6JAfnZvdg-1; Wed, 20 Dec 2023 20:49:15 -0500 X-MC-Unique: 4tWsDGHQOpisg6JAfnZvdg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 974091C05EA0; Thu, 21 Dec 2023 01:49:15 +0000 (UTC) Received: from localhost (unknown [10.39.192.79]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0157D3C25; Thu, 21 Dec 2023 01:49:14 +0000 (UTC) From: Stefan Hajnoczi To: Kevin Wolf , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Leonardo Bras , qemu-block@nongnu.org, Fam Zheng , Paolo Bonzini , Vladimir Sementsov-Ogievskiy , Fabiano Rosas , Eric Blake , Hanna Reitz , Juan Quintela , Peter Xu Subject: [PATCH 4/6] nbd/server: avoid per-NBDRequest nbd_client_get/put() Date: Wed, 20 Dec 2023 20:49:01 -0500 Message-ID: <20231221014903.1537962-5-stefanha@redhat.com> In-Reply-To: <20231221014903.1537962-1-stefanha@redhat.com> References: <20231221014903.1537962-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 Received-SPF: pass client-ip=170.10.129.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.063, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org nbd_trip() processes a single NBD request from start to finish and holds an NBDClient reference throughout. NBDRequest does not outlive the scope of nbd_trip(). Therefore it is unnecessary to ref/unref NBDClient for each NBDRequest. Removing these nbd_client_get()/nbd_client_put() calls will make thread-safety easier in the commits that follow. Signed-off-by: Stefan Hajnoczi Reviewed-by: Paolo Bonzini --- nbd/server.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/nbd/server.c b/nbd/server.c index 895cf0a752..0b09ccc8dc 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -1557,7 +1557,6 @@ static NBDRequestData *nbd_request_get(NBDClient *client) client->nb_requests++; req = g_new0(NBDRequestData, 1); - nbd_client_get(client); req->client = client; return req; } @@ -1578,8 +1577,6 @@ static void nbd_request_put(NBDRequestData *req) } nbd_client_receive_next_request(client); - - nbd_client_put(client); } static void blk_aio_attached(AioContext *ctx, void *opaque) From patchwork Thu Dec 21 01:49:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1878790 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=CBqFcrst; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SwYKy2pR2z1ydg for ; Thu, 21 Dec 2023 12:50:46 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rG8Bo-00088k-47; Wed, 20 Dec 2023 20:49:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Be-00086z-Nu for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:22 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Bd-0002h0-4q for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703123360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9UzJmAR6RzllYvvvzKA6qWdFzydxG0i59/3dV2bf+uM=; b=CBqFcrstFPjDZWGxeraDKCYUCif9W38Wzep8OUaI2nBUaqoU5pWbA149DwoU+5YWpLlAR1 2ZK+se0f7HydIHZvbM/lratka7QeQAfZEy3eEOo/dEYLNqW3M2fw+cTvC8N48e9Yj5AUs7 W+y46Oi1UhxP9ROaiV81B7xTgbmE1cI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-662-VzXZR_5PPLGvU_a7R4dx6Q-1; Wed, 20 Dec 2023 20:49:18 -0500 X-MC-Unique: VzXZR_5PPLGvU_a7R4dx6Q-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2C4FE879840; Thu, 21 Dec 2023 01:49:18 +0000 (UTC) Received: from localhost (unknown [10.39.192.79]) by smtp.corp.redhat.com (Postfix) with ESMTP id 824A040C6EB9; Thu, 21 Dec 2023 01:49:17 +0000 (UTC) From: Stefan Hajnoczi To: Kevin Wolf , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Leonardo Bras , qemu-block@nongnu.org, Fam Zheng , Paolo Bonzini , Vladimir Sementsov-Ogievskiy , Fabiano Rosas , Eric Blake , Hanna Reitz , Juan Quintela , Peter Xu Subject: [PATCH 5/6] nbd/server: only traverse NBDExport->clients from main loop thread Date: Wed, 20 Dec 2023 20:49:02 -0500 Message-ID: <20231221014903.1537962-6-stefanha@redhat.com> In-Reply-To: <20231221014903.1537962-1-stefanha@redhat.com> References: <20231221014903.1537962-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.2 Received-SPF: pass client-ip=170.10.133.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.063, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org The NBD clients list is currently accessed from both the export AioContext and the main loop thread. When the AioContext lock is removed there will be nothing protecting the clients list. Adding a lock around the clients list is tricky because NBDClient structs are refcounted and may be freed from the export AioContext or the main loop thread. nbd_export_request_shutdown() -> client_close() -> nbd_client_put() is also tricky because the list lock would be held while indirectly dropping references to NDBClients. A simpler approach is to only allow nbd_client_put() and client_close() calls from the main loop thread. Then the NBD clients list is only accessed from the main loop thread and no fancy locking is needed. nbd_trip() just needs to reschedule itself in the main loop AioContext before calling nbd_client_put() and client_close(). This costs more CPU cycles per NBD request but is needed for thread-safety when the AioContext lock is removed. Note that nbd_client_get() can still be called from either thread, so make NBDClient->refcount atomic. Signed-off-by: Stefan Hajnoczi --- nbd/server.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/nbd/server.c b/nbd/server.c index 0b09ccc8dc..527fbdab4a 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -122,7 +122,7 @@ struct NBDMetaContexts { }; struct NBDClient { - int refcount; + int refcount; /* atomic */ void (*close_fn)(NBDClient *client, bool negotiated); NBDExport *exp; @@ -1501,14 +1501,17 @@ static int coroutine_fn nbd_receive_request(NBDClient *client, NBDRequest *reque #define MAX_NBD_REQUESTS 16 +/* Runs in export AioContext and main loop thread */ void nbd_client_get(NBDClient *client) { - client->refcount++; + qatomic_inc(&client->refcount); } void nbd_client_put(NBDClient *client) { - if (--client->refcount == 0) { + assert(qemu_in_main_thread()); + + if (qatomic_fetch_dec(&client->refcount) == 1) { /* The last reference should be dropped by client->close, * which is called by client_close. */ @@ -1531,6 +1534,8 @@ void nbd_client_put(NBDClient *client) static void client_close(NBDClient *client, bool negotiated) { + assert(qemu_in_main_thread()); + if (client->closing) { return; } @@ -2938,8 +2943,15 @@ static coroutine_fn void nbd_trip(void *opaque) int ret; Error *local_err = NULL; + /* + * Note that nbd_client_put() and client_close() must be called from the + * main loop thread. Use aio_co_reschedule_self() to switch AioContext + * before calling these functions. + */ + trace_nbd_trip(); if (client->closing) { + aio_co_reschedule_self(qemu_get_aio_context()); nbd_client_put(client); return; } @@ -2949,6 +2961,7 @@ static coroutine_fn void nbd_trip(void *opaque) * We're switching between AIO contexts. Don't attempt to receive a new * request and kick the main context which may be waiting for us. */ + aio_co_reschedule_self(qemu_get_aio_context()); nbd_client_put(client); client->recv_coroutine = NULL; aio_wait_kick(); @@ -3013,6 +3026,8 @@ static coroutine_fn void nbd_trip(void *opaque) qio_channel_set_cork(client->ioc, false); done: nbd_request_put(req); + + aio_co_reschedule_self(qemu_get_aio_context()); nbd_client_put(client); return; @@ -3021,6 +3036,8 @@ disconnect: error_reportf_err(local_err, "Disconnect client, due to: "); } nbd_request_put(req); + + aio_co_reschedule_self(qemu_get_aio_context()); client_close(client, true); nbd_client_put(client); } From patchwork Thu Dec 21 01:49:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 1878787 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=NbdUktOl; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SwYKj2YKKz20Qs for ; Thu, 21 Dec 2023 12:50:32 +1100 (AEDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rG8Bs-0008A9-0z; Wed, 20 Dec 2023 20:49:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Bj-00088C-4U for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:28 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG8Bh-0002i5-5G for qemu-devel@nongnu.org; Wed, 20 Dec 2023 20:49:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703123364; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rGfjh/YlozNNkXclMsnlpr3zJG/93/87HIvPn823zgw=; b=NbdUktOlqUDRYhvjvUmpSbIfz0bVJ9pC0SaDhR6u3YnWi1/aHId07ZEoVqrSgEzKbVwmSw ifgqi+3HKnchQ1j8E2ziQZbsyYikcMM/WfPsoCLMx+vPHes5lEUt8xgSLWJiL3Oawr4rXb EpOVzAmahaOt/+FTcrYFY/SCAIafSfc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-612-Cl6mdu6cNPyxat6YUURKDQ-1; Wed, 20 Dec 2023 20:49:21 -0500 X-MC-Unique: Cl6mdu6cNPyxat6YUURKDQ-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 809638370E2; Thu, 21 Dec 2023 01:49:20 +0000 (UTC) Received: from localhost (unknown [10.39.192.79]) by smtp.corp.redhat.com (Postfix) with ESMTP id ED78A3C25; Thu, 21 Dec 2023 01:49:19 +0000 (UTC) From: Stefan Hajnoczi To: Kevin Wolf , qemu-devel@nongnu.org Cc: Stefan Hajnoczi , Leonardo Bras , qemu-block@nongnu.org, Fam Zheng , Paolo Bonzini , Vladimir Sementsov-Ogievskiy , Fabiano Rosas , Eric Blake , Hanna Reitz , Juan Quintela , Peter Xu Subject: [PATCH 6/6] nbd/server: introduce NBDClient->lock to protect fields Date: Wed, 20 Dec 2023 20:49:03 -0500 Message-ID: <20231221014903.1537962-7-stefanha@redhat.com> In-Reply-To: <20231221014903.1537962-1-stefanha@redhat.com> References: <20231221014903.1537962-1-stefanha@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 Received-SPF: pass client-ip=170.10.133.124; envelope-from=stefanha@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.063, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org NBDClient has a number of fields that are accessed by both the export AioContext and the main loop thread. When the AioContext lock is removed these fields will need another form of protection. Add NBDClient->lock and protect fields that are accessed by both threads. Also add assertions where possible and otherwise add doc comments stating assumptions about which thread and lock holding. Note this patch moves the client->recv_coroutine assertion from nbd_co_receive_request() to nbd_trip() where client->lock is held. Signed-off-by: Stefan Hajnoczi --- nbd/server.c | 128 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 94 insertions(+), 34 deletions(-) diff --git a/nbd/server.c b/nbd/server.c index 527fbdab4a..4008ec7df9 100644 --- a/nbd/server.c +++ b/nbd/server.c @@ -125,23 +125,25 @@ struct NBDClient { int refcount; /* atomic */ void (*close_fn)(NBDClient *client, bool negotiated); + QemuMutex lock; + NBDExport *exp; QCryptoTLSCreds *tlscreds; char *tlsauthz; QIOChannelSocket *sioc; /* The underlying data channel */ QIOChannel *ioc; /* The current I/O channel which may differ (eg TLS) */ - Coroutine *recv_coroutine; + Coroutine *recv_coroutine; /* protected by lock */ CoMutex send_lock; Coroutine *send_coroutine; - bool read_yielding; - bool quiescing; + bool read_yielding; /* protected by lock */ + bool quiescing; /* protected by lock */ QTAILQ_ENTRY(NBDClient) next; - int nb_requests; - bool closing; + int nb_requests; /* protected by lock */ + bool closing; /* protected by lock */ uint32_t check_align; /* If non-zero, check for aligned client requests */ @@ -1415,11 +1417,18 @@ nbd_read_eof(NBDClient *client, void *buffer, size_t size, Error **errp) len = qio_channel_readv(client->ioc, &iov, 1, errp); if (len == QIO_CHANNEL_ERR_BLOCK) { - client->read_yielding = true; + WITH_QEMU_LOCK_GUARD(&client->lock) { + if (client->quiescing) { + return -EAGAIN; + } + client->read_yielding = true; + } qio_channel_yield(client->ioc, G_IO_IN); - client->read_yielding = false; - if (client->quiescing) { - return -EAGAIN; + WITH_QEMU_LOCK_GUARD(&client->lock) { + client->read_yielding = false; + if (client->quiescing) { + return -EAGAIN; + } } continue; } else if (len < 0) { @@ -1528,6 +1537,7 @@ void nbd_client_put(NBDClient *client) blk_exp_unref(&client->exp->common); } g_free(client->contexts.bitmaps); + qemu_mutex_destroy(&client->lock); g_free(client); } } @@ -1536,11 +1546,13 @@ static void client_close(NBDClient *client, bool negotiated) { assert(qemu_in_main_thread()); - if (client->closing) { - return; - } + WITH_QEMU_LOCK_GUARD(&client->lock) { + if (client->closing) { + return; + } - client->closing = true; + client->closing = true; + } /* Force requests to finish. They will drop their own references, * then we'll close the socket and free the NBDClient. @@ -1554,6 +1566,7 @@ static void client_close(NBDClient *client, bool negotiated) } } +/* Runs in export AioContext with client->lock held */ static NBDRequestData *nbd_request_get(NBDClient *client) { NBDRequestData *req; @@ -1566,6 +1579,7 @@ static NBDRequestData *nbd_request_get(NBDClient *client) return req; } +/* Runs in export AioContext with client->lock held */ static void nbd_request_put(NBDRequestData *req) { NBDClient *client = req->client; @@ -1589,14 +1603,18 @@ static void blk_aio_attached(AioContext *ctx, void *opaque) NBDExport *exp = opaque; NBDClient *client; + assert(qemu_in_main_thread()); + trace_nbd_blk_aio_attached(exp->name, ctx); exp->common.ctx = ctx; QTAILQ_FOREACH(client, &exp->clients, next) { - assert(client->nb_requests == 0); - assert(client->recv_coroutine == NULL); - assert(client->send_coroutine == NULL); + WITH_QEMU_LOCK_GUARD(&client->lock) { + assert(client->nb_requests == 0); + assert(client->recv_coroutine == NULL); + assert(client->send_coroutine == NULL); + } } } @@ -1604,6 +1622,8 @@ static void blk_aio_detach(void *opaque) { NBDExport *exp = opaque; + assert(qemu_in_main_thread()); + trace_nbd_blk_aio_detach(exp->name, exp->common.ctx); exp->common.ctx = NULL; @@ -1614,8 +1634,12 @@ static void nbd_drained_begin(void *opaque) NBDExport *exp = opaque; NBDClient *client; + assert(qemu_in_main_thread()); + QTAILQ_FOREACH(client, &exp->clients, next) { - client->quiescing = true; + WITH_QEMU_LOCK_GUARD(&client->lock) { + client->quiescing = true; + } } } @@ -1624,9 +1648,13 @@ static void nbd_drained_end(void *opaque) NBDExport *exp = opaque; NBDClient *client; + assert(qemu_in_main_thread()); + QTAILQ_FOREACH(client, &exp->clients, next) { - client->quiescing = false; - nbd_client_receive_next_request(client); + WITH_QEMU_LOCK_GUARD(&client->lock) { + client->quiescing = false; + nbd_client_receive_next_request(client); + } } } @@ -1635,17 +1663,21 @@ static bool nbd_drained_poll(void *opaque) NBDExport *exp = opaque; NBDClient *client; + assert(qemu_in_main_thread()); + QTAILQ_FOREACH(client, &exp->clients, next) { - if (client->nb_requests != 0) { - /* - * If there's a coroutine waiting for a request on nbd_read_eof() - * enter it here so we don't depend on the client to wake it up. - */ - if (client->recv_coroutine != NULL && client->read_yielding) { - qio_channel_wake_read(client->ioc); + WITH_QEMU_LOCK_GUARD(&client->lock) { + if (client->nb_requests != 0) { + /* + * If there's a coroutine waiting for a request on nbd_read_eof() + * enter it here so we don't depend on the client to wake it up. + */ + if (client->recv_coroutine != NULL && client->read_yielding) { + qio_channel_wake_read(client->ioc); + } + + return true; } - - return true; } } @@ -1656,6 +1688,8 @@ static void nbd_eject_notifier(Notifier *n, void *data) { NBDExport *exp = container_of(n, NBDExport, eject_notifier); + assert(qemu_in_main_thread()); + blk_exp_request_shutdown(&exp->common); } @@ -2541,7 +2575,6 @@ static int coroutine_fn nbd_co_receive_request(NBDRequestData *req, int ret; g_assert(qemu_in_coroutine()); - assert(client->recv_coroutine == qemu_coroutine_self()); ret = nbd_receive_request(client, request, errp); if (ret < 0) { return ret; @@ -2950,7 +2983,11 @@ static coroutine_fn void nbd_trip(void *opaque) */ trace_nbd_trip(); + + qemu_mutex_lock(&client->lock); + if (client->closing) { + qemu_mutex_unlock(&client->lock); aio_co_reschedule_self(qemu_get_aio_context()); nbd_client_put(client); return; @@ -2961,15 +2998,24 @@ static coroutine_fn void nbd_trip(void *opaque) * We're switching between AIO contexts. Don't attempt to receive a new * request and kick the main context which may be waiting for us. */ - aio_co_reschedule_self(qemu_get_aio_context()); - nbd_client_put(client); client->recv_coroutine = NULL; + qemu_mutex_unlock(&client->lock); aio_wait_kick(); + + aio_co_reschedule_self(qemu_get_aio_context()); + nbd_client_put(client); return; } req = nbd_request_get(client); - ret = nbd_co_receive_request(req, &request, &local_err); + + do { + assert(client->recv_coroutine == qemu_coroutine_self()); + qemu_mutex_unlock(&client->lock); + ret = nbd_co_receive_request(req, &request, &local_err); + qemu_mutex_lock(&client->lock); + } while (ret == -EAGAIN && !client->quiescing); + client->recv_coroutine = NULL; if (client->closing) { @@ -2981,11 +3027,13 @@ static coroutine_fn void nbd_trip(void *opaque) } if (ret == -EAGAIN) { - assert(client->quiescing); goto done; } nbd_client_receive_next_request(client); + + qemu_mutex_unlock(&client->lock); + if (ret == -EIO) { goto disconnect; } @@ -3024,8 +3072,10 @@ static coroutine_fn void nbd_trip(void *opaque) } qio_channel_set_cork(client->ioc, false); + qemu_mutex_lock(&client->lock); done: nbd_request_put(req); + qemu_mutex_unlock(&client->lock); aio_co_reschedule_self(qemu_get_aio_context()); nbd_client_put(client); @@ -3035,13 +3085,20 @@ disconnect: if (local_err) { error_reportf_err(local_err, "Disconnect client, due to: "); } + + qemu_mutex_lock(&client->lock); nbd_request_put(req); + qemu_mutex_unlock(&client->lock); aio_co_reschedule_self(qemu_get_aio_context()); client_close(client, true); nbd_client_put(client); } +/* + * Runs in export AioContext and main loop thread. Caller must hold + * client->lock. + */ static void nbd_client_receive_next_request(NBDClient *client) { if (!client->recv_coroutine && client->nb_requests < MAX_NBD_REQUESTS && @@ -3067,7 +3124,9 @@ static coroutine_fn void nbd_co_client_start(void *opaque) return; } - nbd_client_receive_next_request(client); + WITH_QEMU_LOCK_GUARD(&client->lock) { + nbd_client_receive_next_request(client); + } } /* @@ -3084,6 +3143,7 @@ void nbd_client_new(QIOChannelSocket *sioc, Coroutine *co; client = g_new0(NBDClient, 1); + qemu_mutex_init(&client->lock); client->refcount = 1; client->tlscreds = tlscreds; if (tlscreds) {