From patchwork Thu Mar 31 15:08:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611814 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=KBvWnWH7; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn8J5bcLz9sGP for ; Fri, 1 Apr 2022 02:21:36 +1100 (AEDT) Received: from localhost ([::1]:49400 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwcA-0005L2-QT for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:21:34 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52876) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwR6-0004VJ-Et for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:08 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:60780) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwR4-0001eI-TC for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739406; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QTrDs9GixbZVy8Cnd3HQuKCvJ4SqLTdPVQ5e8X7M4T0=; b=KBvWnWH7fgVtMEd9vZ4miuBd7DykDT3e0WaZ1+wGJCyGLPrl0w0rkKSpSR3WYgF/CEWGeY xNdiHDNEQLG6vUWMu3lz4YLGq+DGYgs3gTMJ+R9D7GESMb5d9BFXNJQMNy08Nl9ulefIxs DC8OxgJIiaF+R2iW1znBQxDjeDcBWPc= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-613-aLZGbjS_O1STRBPYU66AuA-1; Thu, 31 Mar 2022 11:09:00 -0400 X-MC-Unique: aLZGbjS_O1STRBPYU66AuA-1 Received: by mail-qv1-f71.google.com with SMTP id z1-20020ad44781000000b00440ded04b09so18735670qvy.22 for ; Thu, 31 Mar 2022 08:09:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QTrDs9GixbZVy8Cnd3HQuKCvJ4SqLTdPVQ5e8X7M4T0=; b=q2Cawh/ULxa8gA68DWeedQy4MBrksRN7k+k8vAhk3xVtyQX6yd3C9Dnt2coDisBAmV Q7tZa9G86t3mH5iy9RGaor5t/Dj7di2dwUERzGsMGNojd26WunWtRgBwHs/F+gqKbsSd jzkDaiIKoWd0sTZcbZXfkzBeH/jjWZnoArUVcz4CSprylAzEdqmP6hmSTAt6CHAlFScF JB8xEY3k17zCIN0wZksVN4B1WaDYEvL9zNVO8vz34PcyxiSpwgULo468lsR1HbRtyldF /2FLYTWH8ncCE/2HLsxtzF2+0+6SbQUi8XDgEVrCrMW5DQd9h7PNFt9nqy9+hsRFj4pC wU5Q== X-Gm-Message-State: AOAM532mqR5i4q7nBJz109EJqDvXc3PSfne3AOCZA301WAsPxCCetYJf OEI6rHbKa/AbuByuZFPb3ZBh9wdf8ABIwMWWz9PgdbuZ+FZbhYnUJMTvIs6yrvMIXtYZN+FzqGG Ov7Ku8ucxUawlhaUiGZfOWAeDXuHdz7QnZoHUI9I/msB5FgV4p4E4bz28b3SB68b2 X-Received: by 2002:a05:6214:4017:b0:440:d478:c167 with SMTP id kd23-20020a056214401700b00440d478c167mr4515234qvb.79.1648739339979; Thu, 31 Mar 2022 08:08:59 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxSicRRFPsc6xixqGWuDZeyyTC8jhMHugUR7ViWnyHFbuTtpMUY3BOWyPaSEYrvsrrTO4oYwg== X-Received: by 2002:a05:6214:4017:b0:440:d478:c167 with SMTP id kd23-20020a056214401700b00440d478c167mr4515192qvb.79.1648739339531; Thu, 31 Mar 2022 08:08:59 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.08.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:08:59 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 01/19] migration: Postpone releasing MigrationState.hostname Date: Thu, 31 Mar 2022 11:08:39 -0400 Message-Id: <20220331150857.74406-2-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We used to release it right after migrate_fd_connect(). That's not good enough when there're more than one socket pair required, because it'll be needed to establish TLS connection for the rest channels. One example is multifd, where we copied over the hostname for each channel but that's actually not needed. Keeping the hostname until the cleanup phase of migration. Cc: Daniel P. Berrange Signed-off-by: Peter Xu Reviewed-by: Dr. David Alan Gilbert --- migration/channel.c | 1 - migration/migration.c | 5 +++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/migration/channel.c b/migration/channel.c index c4fc000a1a..c6a8dcf1d7 100644 --- a/migration/channel.c +++ b/migration/channel.c @@ -96,6 +96,5 @@ void migration_channel_connect(MigrationState *s, } } migrate_fd_connect(s, error); - g_free(s->hostname); error_free(error); } diff --git a/migration/migration.c b/migration/migration.c index 695f0f2900..281d33326b 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1809,6 +1809,11 @@ static void migrate_fd_cleanup(MigrationState *s) qemu_bh_delete(s->cleanup_bh); s->cleanup_bh = NULL; + if (s->hostname) { + g_free(s->hostname); + s->hostname = NULL; + } + qemu_savevm_state_cleanup(); if (s->to_dst_file) { From patchwork Thu Mar 31 15:08:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611819 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=C8EuhXje; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTnDM6zmZz9sGP for ; Fri, 1 Apr 2022 02:25:07 +1100 (AEDT) Received: from localhost ([::1]:57170 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwfY-0002CR-4a for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:25:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52960) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwRJ-0004hz-PT for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:45160) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwRI-0001fF-3f for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739419; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2+PDhrpWKbHP/PkTW5UAcTW+ZMb1mT4D36GE2ydAWcc=; b=C8EuhXjeGIocQDurpR+uo8372ok1N70lf6Y8fQgch11VPcXA4J1I6PKkEDMNTnaSGN0hd8 5b/n6gMHOYVCtBU0kQPbHPzy/qlnXRr8wFyLbKCcvSVOB1VMxICxHR0eMBu7Hw2YzGL82i 9wnNqaFvfh4EwuM1sc6IBrhyUSekvf0= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-625-1QgCgfWFPUmufmUJW0DMFw-1; Thu, 31 Mar 2022 11:09:02 -0400 X-MC-Unique: 1QgCgfWFPUmufmUJW0DMFw-1 Received: by mail-qv1-f72.google.com with SMTP id k20-20020ad44714000000b00440fd2c4a0aso18747435qvz.20 for ; Thu, 31 Mar 2022 08:09:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2+PDhrpWKbHP/PkTW5UAcTW+ZMb1mT4D36GE2ydAWcc=; b=n1XGte17SW29mARGRa5txzOBiFIJK3F52ekCN6f79BNh6LKZhRVNhocdWzugUVplZO tzPwtozRnIVFBPwrvWh/+Bk+p5ztWYR/RsGKRXhFUnxYeqVke4Fw7xUEsUNKlG2UN3px YbXO914j5R5hBP7uyd9WAvv704laMaAVXkkrk+eTlqyST1P9xpRotPUbde8f8n47YWfg cXRm/5yafahSuxd4ttqDk2fIaeCZnO//gyShueSLCHiO6L2Bbccx0+XcvYGWkZPRqQ1R Ka9j6GZFFfUBQW2jIydW6onYHCXqIu2pIjwp3nCJiEE7l5IjwaieRGte/s89BiqPhj77 BkTw== X-Gm-Message-State: AOAM530k3zqvNrHmKmV792npRoGLb4i+HNh3hc/IWbLmY7ATlbDt7s6w XxkDerWbrfHqQjrXe4ixVUKXgucHzLevC0WQeRSB2bZeE0+2HFNavTIyrv64lICe7IntAZukuYh U2VnlGdpptX4jx9aBRG305/OkvUUOQRrOl8SZLlQ1Fk4PmtpRrfyzIoxKKOVa+wcA X-Received: by 2002:a0c:d688:0:b0:42c:3cb0:3923 with SMTP id k8-20020a0cd688000000b0042c3cb03923mr4158600qvi.69.1648739341252; Thu, 31 Mar 2022 08:09:01 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw1paY5f6hELjrpmSwCSUeZxd0dfSp9Soy7kG6/nElhBBkExk7d1ej/nOvMAKACcaQrBnRc4w== X-Received: by 2002:a0c:d688:0:b0:42c:3cb0:3923 with SMTP id k8-20020a0cd688000000b0042c3cb03923mr4158553qvi.69.1648739340851; Thu, 31 Mar 2022 08:09:00 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.08.59 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:00 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 02/19] migration: Drop multifd tls_hostname cache Date: Thu, 31 Mar 2022 11:08:40 -0400 Message-Id: <20220331150857.74406-3-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" The hostname is cached N times, N equals to the multifd channels. Drop that cache because after previous patch we've got s->hostname being alive for the whole lifecycle of migration procedure. Cc: Juan Quintela Cc: Daniel P. Berrange Signed-off-by: Peter Xu Reviewed-by: Dr. David Alan Gilbert Reviewed-by: Daniel P. Berrangé --- migration/multifd.c | 10 +++------- migration/multifd.h | 2 -- 2 files changed, 3 insertions(+), 9 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index 76b57a7177..1be4ab5d17 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -542,8 +542,6 @@ void multifd_save_cleanup(void) qemu_sem_destroy(&p->sem_sync); g_free(p->name); p->name = NULL; - g_free(p->tls_hostname); - p->tls_hostname = NULL; multifd_pages_clear(p->pages); p->pages = NULL; p->packet_len = 0; @@ -763,7 +761,7 @@ static void multifd_tls_channel_connect(MultiFDSendParams *p, Error **errp) { MigrationState *s = migrate_get_current(); - const char *hostname = p->tls_hostname; + const char *hostname = s->hostname; QIOChannelTLS *tioc; tioc = migration_tls_client_create(s, ioc, hostname, errp); @@ -787,7 +785,8 @@ static bool multifd_channel_connect(MultiFDSendParams *p, MigrationState *s = migrate_get_current(); trace_multifd_set_outgoing_channel( - ioc, object_get_typename(OBJECT(ioc)), p->tls_hostname, error); + ioc, object_get_typename(OBJECT(ioc)), + migrate_get_current()->hostname, error); if (!error) { if (s->parameters.tls_creds && @@ -874,7 +873,6 @@ int multifd_save_setup(Error **errp) int thread_count; uint32_t page_count = MULTIFD_PACKET_SIZE / qemu_target_page_size(); uint8_t i; - MigrationState *s; if (!migrate_use_multifd()) { return 0; @@ -884,7 +882,6 @@ int multifd_save_setup(Error **errp) return -1; } - s = migrate_get_current(); thread_count = migrate_multifd_channels(); multifd_send_state = g_malloc0(sizeof(*multifd_send_state)); multifd_send_state->params = g_new0(MultiFDSendParams, thread_count); @@ -909,7 +906,6 @@ int multifd_save_setup(Error **errp) p->packet->magic = cpu_to_be32(MULTIFD_MAGIC); p->packet->version = cpu_to_be32(MULTIFD_VERSION); p->name = g_strdup_printf("multifdsend_%d", i); - p->tls_hostname = g_strdup(s->hostname); /* We need one extra place for the packet header */ p->iov = g_new0(struct iovec, page_count + 1); p->normal = g_new0(ram_addr_t, page_count); diff --git a/migration/multifd.h b/migration/multifd.h index 4dda900a0b..3d577b98b7 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -72,8 +72,6 @@ typedef struct { uint8_t id; /* channel thread name */ char *name; - /* tls hostname */ - char *tls_hostname; /* channel thread id */ QemuThread thread; /* communication channel */ From patchwork Thu Mar 31 15:08:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611824 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=e9ijK2x1; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTnKD0dwkz9sGP for ; Fri, 1 Apr 2022 02:29:20 +1100 (AEDT) Received: from localhost ([::1]:43524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwjd-0003gF-G7 for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:29:17 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52794) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQr-0003yd-Ch for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:53 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:54891) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQp-0001R7-Pe for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739391; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LAPpR8IngvwRIk9gz5ypRjMNFfZ9AK4pYA9cueZ///o=; b=e9ijK2x1x/+SKl4Q8HpeDQycIp9QDXYphV+fqQwFu8pww2p/Obg8UpOMzzGCjrvx66Qxtc Yxql1ZUb7nDkcIT+X8rQp/7QbSgNWYDvZD2xsbKDmyi1o1Q26X2D6KJdXmfdMJ05UFqRU4 dOz/WtnRyFNiJizp3lTfb3ceaSM8nF0= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-153-2c659Ta4P1mp6HYRkW512g-1; Thu, 31 Mar 2022 11:09:04 -0400 X-MC-Unique: 2c659Ta4P1mp6HYRkW512g-1 Received: by mail-qk1-f199.google.com with SMTP id bj2-20020a05620a190200b005084968bb24so14966996qkb.23 for ; Thu, 31 Mar 2022 08:09:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LAPpR8IngvwRIk9gz5ypRjMNFfZ9AK4pYA9cueZ///o=; b=tBfARVGguRkk7J9u9Ihccm3VBiaTLjXIRgdfuKUKpoXmq7KFoWZ6hXvzem2q8guews ItyDrrgyIYHJUqu0m3+qRH5XsiqGFp7BNbSbyAE2iJ9vvMMVJi3YM/YQiBYnXU+hTxNp mYChEWPuZ9kYudCP7MKF8UUN+vnErr91g4Yq2hvk2jitGfzlIzGlS6Fg411rI8Zn5lQX SWLUsdDiEd7wOrK59lAiahdpdzCOavu6ND6ASvDCOvXSaKBfuQOmJY10lwKhoef72C1B Ptf06+8F8HMqwXCfFBABiyITOIOvqtacUZqz38rmH1l2iKqMmtNC+f/pSiJsrMmQQaYZ pLHA== X-Gm-Message-State: AOAM532Ihga7AOyh4SxzneQbtnFuz1za6lU2D3zUeJTB8cZ600DGr4Na B9Vy6Kpn+Z61FynylJOlqthmcvW73x3OnaV0ZLB+CYH9AgJ/V6Lzzek0Yay5Gyzj3ObmBySrC2F tHI2jNCM9+yonJilcwZ45LO1s+oThigU0DRDS7AnikywXoA7ztiTGqO+rQIi6GHTr X-Received: by 2002:a05:620a:4305:b0:67e:8b39:201d with SMTP id u5-20020a05620a430500b0067e8b39201dmr3549513qko.741.1648739342491; Thu, 31 Mar 2022 08:09:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyKmSREqRik0P4Le3TkHxf1SXMHp52hrdTAmI5/RF2bUdfd6ZJklHGTzvau93ZYfWNjtjc98Q== X-Received: by 2002:a05:620a:4305:b0:67e:8b39:201d with SMTP id u5-20020a05620a430500b0067e8b39201dmr3549465qko.741.1648739342010; Thu, 31 Mar 2022 08:09:02 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.00 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:01 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 03/19] migration: Add pss.postcopy_requested status Date: Thu, 31 Mar 2022 11:08:41 -0400 Message-Id: <20220331150857.74406-4-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This boolean flag shows whether the current page during migration is triggered by postcopy or not. Then in ram_save_host_page() and deeper stack we'll be able to have a reference on the priority of this page. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Reviewed-by: Daniel P. Berrangé --- migration/ram.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/migration/ram.c b/migration/ram.c index 3532f64ecb..bfcd45a36e 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -414,6 +414,8 @@ struct PageSearchStatus { unsigned long page; /* Set once we wrap around */ bool complete_round; + /* Whether current page is explicitly requested by postcopy */ + bool postcopy_requested; }; typedef struct PageSearchStatus PageSearchStatus; @@ -1487,6 +1489,9 @@ retry: */ static bool find_dirty_block(RAMState *rs, PageSearchStatus *pss, bool *again) { + /* This is not a postcopy requested page */ + pss->postcopy_requested = false; + pss->page = migration_bitmap_find_dirty(rs, pss->block, pss->page); if (pss->complete_round && pss->block == rs->last_seen_block && pss->page >= rs->last_page) { @@ -1981,6 +1986,7 @@ static bool get_queued_page(RAMState *rs, PageSearchStatus *pss) * really rare. */ pss->complete_round = false; + pss->postcopy_requested = true; } return !!block; From patchwork Thu Mar 31 15:08:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611796 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=CYgrpIj6; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTmtx4zBNz9sGF for ; Fri, 1 Apr 2022 02:10:01 +1100 (AEDT) Received: from localhost ([::1]:49040 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwQx-0002hE-Ob for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:09:59 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52368) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQ7-00023f-VK for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:08 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:23552) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQ6-0001L5-A6 for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739345; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=x/Zjh6Mlx8sOOfdBb0H4Xbql+QDN1t5yKGdHO6e9oDI=; b=CYgrpIj65GCVQrkjfG8vnXL5zj63NpKHGUROTAhJZnD2PsKh8Eejyn92xbR8Z50KVEAwBY xPgDwThMnAfOkAE+tGws3flfXRlAaNYGLFVfh36JMl0e4vaeEi1FLgaGHqjFhm6eF043/+ cDMkIhGSPALiIhAL5PK7ATYSeii3z8Q= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-131-AE_FGQ_tNIWDUIuZS9n9jw-1; Thu, 31 Mar 2022 11:09:04 -0400 X-MC-Unique: AE_FGQ_tNIWDUIuZS9n9jw-1 Received: by mail-qv1-f70.google.com with SMTP id z2-20020a056214060200b00440d1bc7815so18792999qvw.1 for ; Thu, 31 Mar 2022 08:09:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=x/Zjh6Mlx8sOOfdBb0H4Xbql+QDN1t5yKGdHO6e9oDI=; b=WLVlaIHiTjV00FWzXx13dcid3WREYgm9RtpYFD15RrCLkPjef9qraPoa6wAZBfmYE4 JL0vTP3Gv0CQL1GMPgvOfnZILHp6ViSHTGo58JT7V2LsyKFYLtDUCC1qd2q1AKZMvzXd BHbYQM6DssnWspbciViY3fPY1KZv+hrY1QGvBtOLh5A6ApJMSoCUP4oggBFyrS9mB1Ud snPXITSEiTZq3iHasGng9J7hWMGLxQXvGLsWsDkiM/33Z7ScrJXPPz5wkX3c0ri/0kDi GW8QU74023dgE5yavT6DTRbH5SKTUBPP937TdCGkgecrAsUjrRMrX7Q+mXOBXFzs7UD3 aUsw== X-Gm-Message-State: AOAM530pMYAjidR1avlV5Vuq/Wz4/tMjAe9sgIVTE7wRBGju52V40IBG QrDcFZlpnkkNeg4TAJO21qh7yQJRNGy8cR8ajTTiFbo5dSvcgWoYnCPvsDn7TDisjTFzAR9QEu5 yYWBKCFbT4ZETLRtd3qIQUBagFnxP/iAgo0uYxkx6XdsAa0Ly2SaxsGe04YwEq/KK X-Received: by 2002:a05:620a:46a1:b0:67e:6d5a:3845 with SMTP id bq33-20020a05620a46a100b0067e6d5a3845mr3740748qkb.271.1648739343652; Thu, 31 Mar 2022 08:09:03 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyN/6l1OwKy3fcMqj0G/OYIO7+OW3Xzzwcyac42BTvLf7RlY44lnxZ894DoSi1MLuwvDKDP0A== X-Received: by 2002:a05:620a:46a1:b0:67e:6d5a:3845 with SMTP id bq33-20020a05620a46a100b0067e6d5a3845mr3740719qkb.271.1648739343291; Thu, 31 Mar 2022 08:09:03 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:03 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 04/19] migration: Move migrate_allow_multifd and helpers into migration.c Date: Thu, 31 Mar 2022 11:08:42 -0400 Message-Id: <20220331150857.74406-5-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This variable, along with its helpers, is used to detect whether multiple channel will be supported for migration. In follow up patches, there'll be other capability that requires multi-channels. Hence move it outside multifd specific code and make it public. Meanwhile rename it from "multifd" to "multi_channels" to show its real meaning. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 22 +++++++++++++++++----- migration/migration.h | 3 +++ migration/multifd.c | 19 ++++--------------- migration/multifd.h | 2 -- 4 files changed, 24 insertions(+), 22 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 281d33326b..596d3d30b4 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -180,6 +180,18 @@ static int migration_maybe_pause(MigrationState *s, int new_state); static void migrate_fd_cancel(MigrationState *s); +static bool migrate_allow_multi_channels = true; + +void migrate_protocol_allow_multi_channels(bool allow) +{ + migrate_allow_multi_channels = allow; +} + +bool migrate_multi_channels_is_allowed(void) +{ + return migrate_allow_multi_channels; +} + static gint page_request_addr_cmp(gconstpointer ap, gconstpointer bp) { uintptr_t a = (uintptr_t) ap, b = (uintptr_t) bp; @@ -469,12 +481,12 @@ static void qemu_start_incoming_migration(const char *uri, Error **errp) { const char *p = NULL; - migrate_protocol_allow_multifd(false); /* reset it anyway */ + migrate_protocol_allow_multi_channels(false); /* reset it anyway */ qapi_event_send_migration(MIGRATION_STATUS_SETUP); if (strstart(uri, "tcp:", &p) || strstart(uri, "unix:", NULL) || strstart(uri, "vsock:", NULL)) { - migrate_protocol_allow_multifd(true); + migrate_protocol_allow_multi_channels(true); socket_start_incoming_migration(p ? p : uri, errp); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { @@ -1261,7 +1273,7 @@ static bool migrate_caps_check(bool *cap_list, /* incoming side only */ if (runstate_check(RUN_STATE_INMIGRATE) && - !migrate_multifd_is_allowed() && + !migrate_multi_channels_is_allowed() && cap_list[MIGRATION_CAPABILITY_MULTIFD]) { error_setg(errp, "multifd is not supported by current protocol"); return false; @@ -2324,11 +2336,11 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, } } - migrate_protocol_allow_multifd(false); + migrate_protocol_allow_multi_channels(false); if (strstart(uri, "tcp:", &p) || strstart(uri, "unix:", NULL) || strstart(uri, "vsock:", NULL)) { - migrate_protocol_allow_multifd(true); + migrate_protocol_allow_multi_channels(true); socket_start_outgoing_migration(s, p ? p : uri, &local_err); #ifdef CONFIG_RDMA } else if (strstart(uri, "rdma:", &p)) { diff --git a/migration/migration.h b/migration/migration.h index 2de861df01..f17ccc657c 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -430,4 +430,7 @@ void migration_cancel(const Error *error); void populate_vfio_info(MigrationInfo *info); void postcopy_temp_page_reset(PostcopyTmpPage *tmp_page); +bool migrate_multi_channels_is_allowed(void); +void migrate_protocol_allow_multi_channels(bool allow); + #endif diff --git a/migration/multifd.c b/migration/multifd.c index 1be4ab5d17..9ea4f581e2 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -517,7 +517,7 @@ void multifd_save_cleanup(void) { int i; - if (!migrate_use_multifd() || !migrate_multifd_is_allowed()) { + if (!migrate_use_multifd() || !migrate_multi_channels_is_allowed()) { return; } multifd_send_terminate_threads(NULL); @@ -857,17 +857,6 @@ cleanup: multifd_new_send_channel_cleanup(p, sioc, local_err); } -static bool migrate_allow_multifd = true; -void migrate_protocol_allow_multifd(bool allow) -{ - migrate_allow_multifd = allow; -} - -bool migrate_multifd_is_allowed(void) -{ - return migrate_allow_multifd; -} - int multifd_save_setup(Error **errp) { int thread_count; @@ -877,7 +866,7 @@ int multifd_save_setup(Error **errp) if (!migrate_use_multifd()) { return 0; } - if (!migrate_multifd_is_allowed()) { + if (!migrate_multi_channels_is_allowed()) { error_setg(errp, "multifd is not supported by current protocol"); return -1; } @@ -976,7 +965,7 @@ int multifd_load_cleanup(Error **errp) { int i; - if (!migrate_use_multifd() || !migrate_multifd_is_allowed()) { + if (!migrate_use_multifd() || !migrate_multi_channels_is_allowed()) { return 0; } multifd_recv_terminate_threads(NULL); @@ -1125,7 +1114,7 @@ int multifd_load_setup(Error **errp) if (!migrate_use_multifd()) { return 0; } - if (!migrate_multifd_is_allowed()) { + if (!migrate_multi_channels_is_allowed()) { error_setg(errp, "multifd is not supported by current protocol"); return -1; } diff --git a/migration/multifd.h b/migration/multifd.h index 3d577b98b7..7d0effcb03 100644 --- a/migration/multifd.h +++ b/migration/multifd.h @@ -13,8 +13,6 @@ #ifndef QEMU_MIGRATION_MULTIFD_H #define QEMU_MIGRATION_MULTIFD_H -bool migrate_multifd_is_allowed(void); -void migrate_protocol_allow_multifd(bool allow); int multifd_save_setup(Error **errp); void multifd_save_cleanup(void); int multifd_load_setup(Error **errp); From patchwork Thu Mar 31 15:08:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611795 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EI7n4+Mp; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTmtn492wz9sGF for ; Fri, 1 Apr 2022 02:09:53 +1100 (AEDT) Received: from localhost ([::1]:48314 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwQp-0002Aq-KK for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:09:51 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52440) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQE-00027T-8Y for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:26607) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQ6-0001LG-Rc for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739346; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WuNDDv29rDBOfHxmW0v67zbZ/KMeTKCpZ84LsWOMvjI=; b=EI7n4+MpGO5T8bsKWKX5pOx70iW7DjExZMFQkTDQ8qXC0yjPlFoMRcYVcZXPtW2g5UbD6b lXymbmIc7RjZviCIxXuC58rsoty3P16khM63aMOudRjWVUaVGfB6zqPq35yDjzhEAWmolT uOPprcZZFU1YWRfhCSw+ar7/G6yRMkc= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-454-FHDOu2RANS6F429jnBTO3Q-1; Thu, 31 Mar 2022 11:09:05 -0400 X-MC-Unique: FHDOu2RANS6F429jnBTO3Q-1 Received: by mail-qv1-f69.google.com with SMTP id cl19-20020a05621404f300b0044103143211so18774785qvb.10 for ; Thu, 31 Mar 2022 08:09:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WuNDDv29rDBOfHxmW0v67zbZ/KMeTKCpZ84LsWOMvjI=; b=40AaKs5htUNbcbeaorYLfcLlHe9Js3/tDhrSqerP0aTa4TjADx9cNO0o3ixPCLNCqC k18T0CfcKp3WKs5ordfo2WcShAP3Le4xUuQ3oX+K1pxKdCLwXu+oTz/GekgivyceGlx/ 4zUhjypJN2wZtrWWSH4MSEboTjoqoYvIN8eVX0BifBSJ9+4k9QUwIsbBYjw0w3p3GkoD alFkegl1SinO/Cqh+NIa5QSILkG2C73bE7XyptDm6wR1UCzd5GCsodcsq0yl9dq2Zwar Sop295L5GNa6xgoAlg5tVTjH7/9F/92DZJ1SmsTRSPOm0hwp9Nw5DJe3DrCR4htYu40b Sf8A== X-Gm-Message-State: AOAM533zZ1CzvGkKF6ops0zbAThAng2dPyFjdtxKWm899+PFjImvhrq4 C+dYE/4b9u3dhBPEV60ug1PoES1rMhxGlXUHwjb3sTT+8gNjggWZhzO6k0yXGt4FR7npex3NiFD +qpqZXhJYxJufzQTiFboahUxWWf8pJ8QuVYnklxTcFgkkc32Ai/8psHws1KueY8LY X-Received: by 2002:a05:622a:58e:b0:2e1:cdf1:d428 with SMTP id c14-20020a05622a058e00b002e1cdf1d428mr4592316qtb.254.1648739344548; Thu, 31 Mar 2022 08:09:04 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw+LduFbJjgYdqYC5bQoWxJBRkfof3BqC5xKsNZpsFbDAdzbOFL8WJO+/va89BpMFq1faHw+A== X-Received: by 2002:a05:622a:58e:b0:2e1:cdf1:d428 with SMTP id c14-20020a05622a058e00b002e1cdf1d428mr4592275qtb.254.1648739344199; Thu, 31 Mar 2022 08:09:04 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.03 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:03 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 05/19] migration: Export ram_load_postcopy() Date: Thu, 31 Mar 2022 11:08:43 -0400 Message-Id: <20220331150857.74406-6-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Will be reused in postcopy fast load thread. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Reviewed-by: Daniel P. Berrangé --- migration/ram.c | 2 +- migration/ram.h | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/migration/ram.c b/migration/ram.c index bfcd45a36e..253fe4b756 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3645,7 +3645,7 @@ int ram_postcopy_incoming_init(MigrationIncomingState *mis) * * @f: QEMUFile where to send the data */ -static int ram_load_postcopy(QEMUFile *f) +int ram_load_postcopy(QEMUFile *f) { int flags = 0, ret = 0; bool place_needed = false; diff --git a/migration/ram.h b/migration/ram.h index 2c6dc3675d..ded0a3a086 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -61,6 +61,7 @@ void ram_postcopy_send_discard_bitmap(MigrationState *ms); /* For incoming postcopy discard */ int ram_discard_range(const char *block_name, uint64_t start, size_t length); int ram_postcopy_incoming_init(MigrationIncomingState *mis); +int ram_load_postcopy(QEMUFile *f); void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); From patchwork Thu Mar 31 15:08:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611798 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=i/V9NQdO; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTmzB04tbz9sGP for ; Fri, 1 Apr 2022 02:13:42 +1100 (AEDT) Received: from localhost ([::1]:57612 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwUW-0008WN-36 for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:13:40 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52390) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQ9-00024S-K4 for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:59816) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQ8-0001LL-10 for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739347; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=muISGSqC4VLW6KPUHb9CY6sFKOFruXP+msv79+xlWRw=; b=i/V9NQdOz0+AE/xbg5xBbE8lKjH5FO0ltt+ZpMFTyFETmU+N3P0YyDRz1qBZGNo2jKi+7E I1kcM7k1g2n2aOV6yRet47Qp0isubxOXo/ITXRxg/NM17TBViCqNcUFeelSPr0xzoi/BtO YuKApU77LxUS78oaraVeCNGeUZXFn10= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-329-CHclpZ32MTi4xUpm2lDtRg-1; Thu, 31 Mar 2022 11:09:06 -0400 X-MC-Unique: CHclpZ32MTi4xUpm2lDtRg-1 Received: by mail-qt1-f197.google.com with SMTP id f3-20020ac84983000000b002e22396acfbso20237367qtq.18 for ; Thu, 31 Mar 2022 08:09:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=muISGSqC4VLW6KPUHb9CY6sFKOFruXP+msv79+xlWRw=; b=5f4SDsU5M9MlTsKXcXn50cYr6JBWnQ6kIl+AaP/z+MEczjlZuE/+W1AmctAC1peQ1m wQrxnl4S9yAaoqhggIRtwiBuNPzlsyrIQEudM7X8minWALGzBD0A5U4w1+zgC+F1FTJc WD0E+UnKs+HMbGDCfS775P+jmMOK9EVGkXG95r6AhDCJToNTpyzl/DschhjhGQAl9dD8 lUQIZdFrf/5VadXeLtqdRn2wvcoGYxD9IWp8b78r9/a4nP+cxZrpnIzg4Cje8Et4ifZM E1cbEB9jTFpS+5EWiUxXQLFihe4dQteorasBbdcp53V7N3yJSDB/LQTd93eUjbvK6hSl llRA== X-Gm-Message-State: AOAM531bccqLn6eLudqDc4kqDv8VK5oiBVi9mkdgj4rDyRsAu+hBopgJ UfVpUM6wawuEOq9eoqE2wNDLRjnpedNI9iUdj8uop5zAOPJ74FEcjJpf31Xfcasegr+i0mYC0Kp BuL6YkNna5NzXUsl5Q2B+WuVA2UlEtyx/UiZ80zP9unpA0LFic/usUL/3yAFTnm8t X-Received: by 2002:a05:620a:2892:b0:67e:be24:e5d7 with SMTP id j18-20020a05620a289200b0067ebe24e5d7mr3590970qkp.762.1648739345748; Thu, 31 Mar 2022 08:09:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwQZD9a4p3NxNCkqYi7waHR7En2ta75aR+IxOT0DIpCa+zZsiDRHMBJpIC68PaTswppMh71ag== X-Received: by 2002:a05:620a:2892:b0:67e:be24:e5d7 with SMTP id j18-20020a05620a289200b0067ebe24e5d7mr3590935qkp.762.1648739345355; Thu, 31 Mar 2022 08:09:05 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.04 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:04 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 06/19] migration: Move channel setup out of postcopy_try_recover() Date: Thu, 31 Mar 2022 11:08:44 -0400 Message-Id: <20220331150857.74406-7-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We used to use postcopy_try_recover() to replace migration_incoming_setup() to setup incoming channels. That's fine for the old world, but in the new world there can be more than one channels that need setup. Better move the channel setup out of it so that postcopy_try_recover() only handles the last phase of switching to the recovery phase. To do that in migration_fd_process_incoming(), move the postcopy_try_recover() call to be after migration_incoming_setup(), which will setup the channels. While in migration_ioc_process_incoming(), postpone the recover() routine right before we'll jump into migration_incoming_process(). A side benefit is we don't need to pass in QEMUFile* to postcopy_try_recover() anymore. Remove it. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Reviewed-by: Daniel P. Berrangé --- migration/migration.c | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 596d3d30b4..8ecf78f2c7 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -671,19 +671,20 @@ void migration_incoming_process(void) } /* Returns true if recovered from a paused migration, otherwise false */ -static bool postcopy_try_recover(QEMUFile *f) +static bool postcopy_try_recover(void) { MigrationIncomingState *mis = migration_incoming_get_current(); if (mis->state == MIGRATION_STATUS_POSTCOPY_PAUSED) { /* Resumed from a paused postcopy migration */ - mis->from_src_file = f; + /* This should be set already in migration_incoming_setup() */ + assert(mis->from_src_file); /* Postcopy has standalone thread to do vm load */ - qemu_file_set_blocking(f, true); + qemu_file_set_blocking(mis->from_src_file, true); /* Re-configure the return path */ - mis->to_src_file = qemu_file_get_return_path(f); + mis->to_src_file = qemu_file_get_return_path(mis->from_src_file); migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_PAUSED, MIGRATION_STATUS_POSTCOPY_RECOVER); @@ -704,11 +705,10 @@ static bool postcopy_try_recover(QEMUFile *f) void migration_fd_process_incoming(QEMUFile *f, Error **errp) { - if (postcopy_try_recover(f)) { + if (!migration_incoming_setup(f, errp)) { return; } - - if (!migration_incoming_setup(f, errp)) { + if (postcopy_try_recover()) { return; } migration_incoming_process(); @@ -724,11 +724,6 @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp) /* The first connection (multifd may have multiple) */ QEMUFile *f = qemu_fopen_channel_input(ioc); - /* If it's a recovery, we're done */ - if (postcopy_try_recover(f)) { - return; - } - if (!migration_incoming_setup(f, errp)) { return; } @@ -749,6 +744,10 @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp) } if (start_migration) { + /* If it's a recovery, we're done */ + if (postcopy_try_recover()) { + return; + } migration_incoming_process(); } } From patchwork Thu Mar 31 15:08:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611797 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=aYfLGcQh; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTmyq19b8z9sGP for ; Fri, 1 Apr 2022 02:13:21 +1100 (AEDT) Received: from localhost ([::1]:56876 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwUA-00081u-0x for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:13:18 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52438) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQE-00027S-7n for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:14 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:59538) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQB-0001Lk-1q for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739350; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TVKNqyq4weVAvwWKRPYdQpiVnLIi/a9PNfjk/yWRalU=; b=aYfLGcQhyApmoEntNr43vj9jiLK3KMXaYvfPXrY4XRwWDaqKQZvcRijYoXHTtd0P+dkMG4 gb4k4NKlBOeUaK4rlyU82wI0H6/HceiLgNO386pmmcEEMbmWO6bbsdx841q6VUo9jybWRO 0UC+wPGvpRhIabsr4Q+lMFlg0jn18VM= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-448-iiFexE61MlaGiKE-9H6FnQ-1; Thu, 31 Mar 2022 11:09:07 -0400 X-MC-Unique: iiFexE61MlaGiKE-9H6FnQ-1 Received: by mail-qt1-f197.google.com with SMTP id h11-20020ac87d4b000000b002e1c9a3ae51so20330425qtb.0 for ; Thu, 31 Mar 2022 08:09:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TVKNqyq4weVAvwWKRPYdQpiVnLIi/a9PNfjk/yWRalU=; b=wOlB2P89DIEub8if5dairTABqfcfr0vfeQ+aDBUpabbZXvB8nBsQ6WFCJGiJPPWhkR OngppxNTrkx2AL+O1aosz5gxcsMM+7qp541juS9BU8kvJtalpubTyNQJlnfkr/PFJ5RW FtrZeGeptKPJR3hT3NJzYoRJ7QSVbpvh2fBYnNsR9d5uEtrXQRFv4QTzqwOM+I5RqUY4 6cyK4pcYyvBMjrSmWCFKApIZubQ8ITX92ii5ppm6Iw7bZ9Xx8NWX7iKdJ04WCXbWescp gxXCTP9XWBeH8OsnQC6QRX8D0t/rES7oFcIgzPAMVGW4ePkLoDMfYPJDW5qMyBZYffbn zcng== X-Gm-Message-State: AOAM533q9OULAeUGQp4f7pB3ilPHFWrl9Z9wU9cC8M4SQk1plbtruaWU qgcA2Ix35W3le8OzVdLLbpvR2xPfzM33FSfDNQ0vSCkEkxFOF+YGBRTSoNRhyehnAxlUNhdxq8m M4Hem4v1Vr3gch3NUCn6TIL02bszcseCAACUXa2gEGB+qhXNBAumnfWsTUwtl3Pa3 X-Received: by 2002:a05:620a:4452:b0:67d:b94a:8c34 with SMTP id w18-20020a05620a445200b0067db94a8c34mr3655225qkp.167.1648739346811; Thu, 31 Mar 2022 08:09:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxCvz5Uvw0evkcyqodn5yY0IMEf8nFv5vWrKL0Mc5nwUlMJ8/spPr0SQbCYqunPgFUXHV2CqA== X-Received: by 2002:a05:620a:4452:b0:67d:b94a:8c34 with SMTP id w18-20020a05620a445200b0067db94a8c34mr3655190qkp.167.1648739346475; Thu, 31 Mar 2022 08:09:06 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.05 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:06 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 07/19] migration: Allow migrate-recover to run multiple times Date: Thu, 31 Mar 2022 11:08:45 -0400 Message-Id: <20220331150857.74406-8-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Previously migration didn't have an easy way to cleanup the listening transport, migrate recovery only allows to execute once. That's done with a trick flag in postcopy_recover_triggered. Now the facility is already there. Drop postcopy_recover_triggered and instead allows a new migrate-recover to release the previous listener transport. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu Reviewed-by: Daniel P. Berrangé --- migration/migration.c | 13 ++----------- migration/migration.h | 1 - migration/savevm.c | 3 --- 3 files changed, 2 insertions(+), 15 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 8ecf78f2c7..21fcf5102f 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -2164,11 +2164,8 @@ void qmp_migrate_recover(const char *uri, Error **errp) return; } - if (qatomic_cmpxchg(&mis->postcopy_recover_triggered, - false, true) == true) { - error_setg(errp, "Migrate recovery is triggered already"); - return; - } + /* If there's an existing transport, release it */ + migration_incoming_transport_cleanup(mis); /* * Note that this call will never start a real migration; it will @@ -2176,12 +2173,6 @@ void qmp_migrate_recover(const char *uri, Error **errp) * to continue using that newly established channel. */ qemu_start_incoming_migration(uri, errp); - - /* Safe to dereference with the assert above */ - if (*errp) { - /* Reset the flag so user could still retry */ - qatomic_set(&mis->postcopy_recover_triggered, false); - } } void qmp_migrate_pause(Error **errp) diff --git a/migration/migration.h b/migration/migration.h index f17ccc657c..a863032b71 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -139,7 +139,6 @@ struct MigrationIncomingState { struct PostcopyBlocktimeContext *blocktime_ctx; /* notify PAUSED postcopy incoming migrations to try to continue */ - bool postcopy_recover_triggered; QemuSemaphore postcopy_pause_sem_dst; QemuSemaphore postcopy_pause_sem_fault; diff --git a/migration/savevm.c b/migration/savevm.c index 02ed94c180..d9076897b8 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2589,9 +2589,6 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) assert(migrate_postcopy_ram()); - /* Clear the triggered bit to allow one recovery */ - mis->postcopy_recover_triggered = false; - /* * Unregister yank with either from/to src would work, since ioc behind it * is the same From patchwork Thu Mar 31 15:08:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611804 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QBUaFv3A; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn222hCbz9sGP for ; Fri, 1 Apr 2022 02:16:10 +1100 (AEDT) Received: from localhost ([::1]:37158 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwWu-0005KS-B3 for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:16:08 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52470) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQG-000290-3u for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:54918) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQD-0001Mj-Vc for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pLFazjM2d/7k3yDS8eU700zmkV3N78vq83O/0EL3Xg8=; b=QBUaFv3Af/GRJlxHA7ySdzBrBuMH1vHiPlNtEmgl5pqxHhau2+qItcsDhQjd0qYoTFB5Y8 PEc8tSpGREpLlKKCSG2wb0YUr9qStLOXLdbIPGprXLzWAiicR9a5p1G6plBhLLqc4i63nv XMy4iSt4wHRmox3KiKDZI6/wL2DYVBo= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-586-czXO_HnoPjeFgXdVmJK4UA-1; Thu, 31 Mar 2022 11:09:08 -0400 X-MC-Unique: czXO_HnoPjeFgXdVmJK4UA-1 Received: by mail-qk1-f197.google.com with SMTP id d12-20020a379b0c000000b0067d8cda1aaaso14997813qke.8 for ; Thu, 31 Mar 2022 08:09:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pLFazjM2d/7k3yDS8eU700zmkV3N78vq83O/0EL3Xg8=; b=kBb4K6zhCa6PXarvQt+ZWTFWBgKXHUlFfDBNpkXRVkdvrttVDlkxJW8Y229N4TNsWb ja/M+krTc8AIiucDUIf/1wZa8PHVKCYGuYkHqxaNfchcKN/tdw0hLtO6DW4YoC+ghXGQ tIYRJDaRRf+yl8OU7zDsH/FW76VbPzaDipnGHju4WsnTX9s4g76X1t/wf7Ru/9lVx4fu d+F56ySj8LujktlC9uiHQ/RrBpiIzMHhKYiY9CNA0dca9eOarfoaDxrPm5BKGs6OX+Uq X6oabopF1xPROHSKEXvV4hJoKSshH4QZi3moIkOgGKZwSI/7voDrL7bVQNhYz8Li5s4U SJLQ== X-Gm-Message-State: AOAM530RZQPJpOL/0X9iYIg13HmfNjdYCfwu1tibgIZDfyl1uAcreO9h fDZ1LH3hkIQg/+4GbY+KJCex1GXNdrTLtRnQlalU0uBuZHtcdIRK0qlipnM65PQdAk+fEYTpOx7 ItxWfmyqPflqZXxcKoNSNkEo39mXwylHfSBd60uQJSNjF9uxPvuWpnYhYHe4oMHkP X-Received: by 2002:a37:a310:0:b0:67d:3749:16e7 with SMTP id m16-20020a37a310000000b0067d374916e7mr3619559qke.159.1648739347937; Thu, 31 Mar 2022 08:09:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJylfeRaYsO/ezB6ob0QqbPGvkV3dSAcJXvKI+Bjfc6GMPwRFocUPawtGG2MJ73FV/hcKjPytw== X-Received: by 2002:a37:a310:0:b0:67d:3749:16e7 with SMTP id m16-20020a37a310000000b0067d374916e7mr3619526qke.159.1648739347529; Thu, 31 Mar 2022 08:09:07 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:07 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 08/19] migration: Add postcopy-preempt capability Date: Thu, 31 Mar 2022 11:08:46 -0400 Message-Id: <20220331150857.74406-9-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Firstly, postcopy already preempts precopy due to the fact that we do unqueue_page() first before looking into dirty bits. However that's not enough, e.g., when there're host huge page enabled, when sending a precopy huge page, a postcopy request needs to wait until the whole huge page that is sending to finish. That could introduce quite some delay, the bigger the huge page is the larger delay it'll bring. This patch adds a new capability to allow postcopy requests to preempt existing precopy page during sending a huge page, so that postcopy requests can be serviced even faster. Meanwhile to send it even faster, bypass the precopy stream by providing a standalone postcopy socket for sending requested pages. Since the new behavior will not be compatible with the old behavior, this will not be the default, it's enabled only when the new capability is set on both src/dst QEMUs. This patch only adds the capability itself, the logic will be added in follow up patches. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 23 +++++++++++++++++++++++ migration/migration.h | 1 + qapi/migration.json | 8 +++++++- 3 files changed, 31 insertions(+), 1 deletion(-) diff --git a/migration/migration.c b/migration/migration.c index 21fcf5102f..76e6ada524 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -1235,6 +1235,11 @@ static bool migrate_caps_check(bool *cap_list, error_setg(errp, "Postcopy is not compatible with ignore-shared"); return false; } + + if (cap_list[MIGRATION_CAPABILITY_MULTIFD]) { + error_setg(errp, "Multifd is not supported in postcopy"); + return false; + } } if (cap_list[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT]) { @@ -1278,6 +1283,13 @@ static bool migrate_caps_check(bool *cap_list, return false; } + if (cap_list[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT]) { + if (!cap_list[MIGRATION_CAPABILITY_POSTCOPY_RAM]) { + error_setg(errp, "Postcopy preempt requires postcopy-ram"); + return false; + } + } + return true; } @@ -2627,6 +2639,15 @@ bool migrate_background_snapshot(void) return s->enabled_capabilities[MIGRATION_CAPABILITY_BACKGROUND_SNAPSHOT]; } +bool migrate_postcopy_preempt(void) +{ + MigrationState *s; + + s = migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_POSTCOPY_PREEMPT]; +} + /* migration thread support */ /* * Something bad happened to the RP stream, mark an error @@ -4237,6 +4258,8 @@ static Property migration_properties[] = { DEFINE_PROP_MIG_CAP("x-compress", MIGRATION_CAPABILITY_COMPRESS), DEFINE_PROP_MIG_CAP("x-events", MIGRATION_CAPABILITY_EVENTS), DEFINE_PROP_MIG_CAP("x-postcopy-ram", MIGRATION_CAPABILITY_POSTCOPY_RAM), + DEFINE_PROP_MIG_CAP("x-postcopy-preempt", + MIGRATION_CAPABILITY_POSTCOPY_PREEMPT), DEFINE_PROP_MIG_CAP("x-colo", MIGRATION_CAPABILITY_X_COLO), DEFINE_PROP_MIG_CAP("x-release-ram", MIGRATION_CAPABILITY_RELEASE_RAM), DEFINE_PROP_MIG_CAP("x-block", MIGRATION_CAPABILITY_BLOCK), diff --git a/migration/migration.h b/migration/migration.h index a863032b71..af4bcb19c2 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -394,6 +394,7 @@ int migrate_decompress_threads(void); bool migrate_use_events(void); bool migrate_postcopy_blocktime(void); bool migrate_background_snapshot(void); +bool migrate_postcopy_preempt(void); /* Sending on the return path - generic and then for each message type */ void migrate_send_rp_shut(MigrationIncomingState *mis, diff --git a/qapi/migration.json b/qapi/migration.json index 18e2610e88..3523f23386 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -463,6 +463,12 @@ # procedure starts. The VM RAM is saved with running VM. # (since 6.0) # +# @postcopy-preempt: If enabled, the migration process will allow postcopy +# requests to preempt precopy stream, so postcopy requests +# will be handled faster. This is a performance feature and +# should not affect the correctness of postcopy migration. +# (since 7.0) +# # Features: # @unstable: Members @x-colo and @x-ignore-shared are experimental. # @@ -476,7 +482,7 @@ 'block', 'return-path', 'pause-before-switchover', 'multifd', 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate', { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] }, - 'validate-uuid', 'background-snapshot'] } + 'validate-uuid', 'background-snapshot', 'postcopy-preempt'] } ## # @MigrationCapabilityStatus: From patchwork Thu Mar 31 15:08:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611809 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QfgsWHqe; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn6b4jQNz9sGP for ; Fri, 1 Apr 2022 02:20:06 +1100 (AEDT) Received: from localhost ([::1]:45558 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwaf-0002ge-UN for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:20:01 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52468) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQG-00028z-42 for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:40703) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQC-0001MJ-8T for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739351; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oLh7AWTsxENQn04JHEx/vmWZbl1/dnGUdXevqLuUURg=; b=QfgsWHqeFXG/NrD5zuwRwhsSNJUcDEW7C2jEYUygUnsHwl5RXRdXsrZ5rSidpbXs2ESfsg QLY5cRkAlqJlqqEEytzRJ/m70Z61UyurE5OspSKOdFPayKJOv+16A7CkLmJcgSx4N9gZGk VuuU7Lu2b2BInX3atNZMSg73sRGU48s= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-232-SUidXvfHMjS17mDPGf7jhQ-1; Thu, 31 Mar 2022 11:09:10 -0400 X-MC-Unique: SUidXvfHMjS17mDPGf7jhQ-1 Received: by mail-qv1-f71.google.com with SMTP id q4-20020a0cf5c4000000b0044346ee3627so10600260qvm.16 for ; Thu, 31 Mar 2022 08:09:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oLh7AWTsxENQn04JHEx/vmWZbl1/dnGUdXevqLuUURg=; b=385Inn5RsgaWCkK1vqq3lj+kIbniH99Oo6eU+uRBuJ/nrd7CF2A2nhEdDR7otk8QDD lrDxFvPJMvc7XT9JhkKcGl0jHaOqaCuolJsGUz0GbU7z+to1m/qOAwzgM76sDuc9HOcv EhYRlZZBsXhlkBH8vrJpvw0Tor1LE0VrV9kcDG3iptpL6nrLC+2K3ZwliUUmNPaghkfd nY1Lqp53OaBUs+pDd3mUZT/1LGm+IEzSYq6nnQDV1daUlmxOkf8yp6Elu7BVUFC23gzX Bl9Zxu/QVJRjhChf7ZxNqSAFbNesGvlV+JqKerHLFU5keThMpKoXa9NVXYbWhrjozv6C 0/2g== X-Gm-Message-State: AOAM533xVYmaixs0X8ZbQnWeAvINEs5jslAA6DkolDXG8dyhhcuN3Po/ +5x12Nt7gfpxPe3kiUrGRVxP/Exk7cpNUDx2BN7/My9Uv2Nvc3WxQjzYD1wt8a4kr7RLy+qZdHM sNhEFKhPk4YqWxz3Je2tXnuSseqTLlXoPSlVoMzWMfa4PGXMbt9VN5jC1IbTP4IR7 X-Received: by 2002:a05:620a:15af:b0:67e:9b09:697e with SMTP id f15-20020a05620a15af00b0067e9b09697emr3660630qkk.139.1648739349572; Thu, 31 Mar 2022 08:09:09 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzzvERvWeslpBIzma1MmwicKvGyPDHQFo4F4anuRCAFyMGATkmoDWuA5E429jAk/iGkl2Nlmg== X-Received: by 2002:a05:620a:15af:b0:67e:9b09:697e with SMTP id f15-20020a05620a15af00b0067e9b09697emr3660577qkk.139.1648739348894; Thu, 31 Mar 2022 08:09:08 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:08 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 09/19] migration: Postcopy preemption preparation on channel creation Date: Thu, 31 Mar 2022 11:08:47 -0400 Message-Id: <20220331150857.74406-10-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URG_BIZ=0.573 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Create a new socket for postcopy to be prepared to send postcopy requested pages via this specific channel, so as to not get blocked by precopy pages. A new thread is also created on dest qemu to receive data from this new channel based on the ram_load_postcopy() routine. The ram_load_postcopy(POSTCOPY) branch and the thread has not started to function, and that'll be done in follow up patches. Cleanup the new sockets on both src/dst QEMUs, meanwhile look after the new thread too to make sure it'll be recycled properly. Signed-off-by: Peter Xu Reviewed-by: Daniel P. Berrangé --- migration/migration.c | 62 +++++++++++++++++++++++---- migration/migration.h | 8 ++++ migration/postcopy-ram.c | 92 ++++++++++++++++++++++++++++++++++++++-- migration/postcopy-ram.h | 10 +++++ migration/ram.c | 25 ++++++++--- migration/ram.h | 4 +- migration/savevm.c | 20 ++++----- migration/socket.c | 22 +++++++++- migration/socket.h | 1 + migration/trace-events | 5 ++- 10 files changed, 218 insertions(+), 31 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 76e6ada524..01b882494d 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -321,6 +321,12 @@ void migration_incoming_state_destroy(void) mis->page_requested = NULL; } + if (mis->postcopy_qemufile_dst) { + migration_ioc_unregister_yank_from_file(mis->postcopy_qemufile_dst); + qemu_fclose(mis->postcopy_qemufile_dst); + mis->postcopy_qemufile_dst = NULL; + } + yank_unregister_instance(MIGRATION_YANK_INSTANCE); } @@ -714,15 +720,21 @@ void migration_fd_process_incoming(QEMUFile *f, Error **errp) migration_incoming_process(); } +static bool migration_needs_multiple_sockets(void) +{ + return migrate_use_multifd() || migrate_postcopy_preempt(); +} + void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp) { MigrationIncomingState *mis = migration_incoming_get_current(); Error *local_err = NULL; bool start_migration; + QEMUFile *f; if (!mis->from_src_file) { /* The first connection (multifd may have multiple) */ - QEMUFile *f = qemu_fopen_channel_input(ioc); + f = qemu_fopen_channel_input(ioc); if (!migration_incoming_setup(f, errp)) { return; @@ -730,13 +742,18 @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp) /* * Common migration only needs one channel, so we can start - * right now. Multifd needs more than one channel, we wait. + * right now. Some features need more than one channel, we wait. */ - start_migration = !migrate_use_multifd(); + start_migration = !migration_needs_multiple_sockets(); } else { /* Multiple connections */ - assert(migrate_use_multifd()); - start_migration = multifd_recv_new_channel(ioc, &local_err); + assert(migration_needs_multiple_sockets()); + if (migrate_use_multifd()) { + start_migration = multifd_recv_new_channel(ioc, &local_err); + } else if (migrate_postcopy_preempt()) { + f = qemu_fopen_channel_input(ioc); + start_migration = postcopy_preempt_new_channel(mis, f); + } if (local_err) { error_propagate(errp, local_err); return; @@ -761,11 +778,20 @@ void migration_ioc_process_incoming(QIOChannel *ioc, Error **errp) bool migration_has_all_channels(void) { MigrationIncomingState *mis = migration_incoming_get_current(); - bool all_channels; - all_channels = multifd_recv_all_channels_created(); + if (!mis->from_src_file) { + return false; + } + + if (migrate_use_multifd()) { + return multifd_recv_all_channels_created(); + } + + if (migrate_postcopy_preempt()) { + return mis->postcopy_qemufile_dst != NULL; + } - return all_channels && mis->from_src_file != NULL; + return true; } /* @@ -1863,6 +1889,12 @@ static void migrate_fd_cleanup(MigrationState *s) qemu_fclose(tmp); } + if (s->postcopy_qemufile_src) { + migration_ioc_unregister_yank_from_file(s->postcopy_qemufile_src); + qemu_fclose(s->postcopy_qemufile_src); + s->postcopy_qemufile_src = NULL; + } + assert(!migration_is_active(s)); if (s->state == MIGRATION_STATUS_CANCELLING) { @@ -3238,6 +3270,11 @@ static void migration_completion(MigrationState *s) qemu_savevm_state_complete_postcopy(s->to_dst_file); qemu_mutex_unlock_iothread(); + /* Shutdown the postcopy fast path thread */ + if (migrate_postcopy_preempt()) { + postcopy_preempt_shutdown_file(s); + } + trace_migration_completion_postcopy_end_after_complete(); } else { goto fail; @@ -4125,6 +4162,15 @@ void migrate_fd_connect(MigrationState *s, Error *error_in) } } + /* This needs to be done before resuming a postcopy */ + if (postcopy_preempt_setup(s, &local_err)) { + error_report_err(local_err); + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP, + MIGRATION_STATUS_FAILED); + migrate_fd_cleanup(s); + return; + } + if (resume) { /* Wakeup the main migration thread to do the recovery */ migrate_set_state(&s->state, MIGRATION_STATUS_POSTCOPY_PAUSED, diff --git a/migration/migration.h b/migration/migration.h index af4bcb19c2..caa910d956 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -23,6 +23,7 @@ #include "io/channel-buffer.h" #include "net/announce.h" #include "qom/object.h" +#include "postcopy-ram.h" struct PostcopyBlocktimeContext; @@ -112,6 +113,11 @@ struct MigrationIncomingState { * enabled. */ unsigned int postcopy_channels; + /* QEMUFile for postcopy only; it'll be handled by a separate thread */ + QEMUFile *postcopy_qemufile_dst; + /* Postcopy priority thread is used to receive postcopy requested pages */ + QemuThread postcopy_prio_thread; + bool postcopy_prio_thread_created; /* * An array of temp host huge pages to be used, one for each postcopy * channel. @@ -192,6 +198,8 @@ struct MigrationState { QEMUBH *cleanup_bh; /* Protected by qemu_file_lock */ QEMUFile *to_dst_file; + /* Postcopy specific transfer channel */ + QEMUFile *postcopy_qemufile_src; QIOChannelBuffer *bioc; /* * Protects to_dst_file/from_dst_file pointers. We need to make sure we diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index 32c52f4b1d..df0c02f729 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -33,6 +33,9 @@ #include "trace.h" #include "hw/boards.h" #include "exec/ramblock.h" +#include "socket.h" +#include "qemu-file-channel.h" +#include "yank_functions.h" /* Arbitrary limit on size of each discard command, * keeps them around ~200 bytes @@ -567,6 +570,11 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis) { trace_postcopy_ram_incoming_cleanup_entry(); + if (mis->postcopy_prio_thread_created) { + qemu_thread_join(&mis->postcopy_prio_thread); + mis->postcopy_prio_thread_created = false; + } + if (mis->have_fault_thread) { Error *local_err = NULL; @@ -1102,8 +1110,13 @@ static int postcopy_temp_pages_setup(MigrationIncomingState *mis) int err, i, channels; void *temp_page; - /* TODO: will be boosted when enable postcopy preemption */ - mis->postcopy_channels = 1; + if (migrate_postcopy_preempt()) { + /* If preemption enabled, need extra channel for urgent requests */ + mis->postcopy_channels = RAM_CHANNEL_MAX; + } else { + /* Both precopy/postcopy on the same channel */ + mis->postcopy_channels = 1; + } channels = mis->postcopy_channels; mis->postcopy_tmp_pages = g_malloc0_n(sizeof(PostcopyTmpPage), channels); @@ -1170,7 +1183,7 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) return -1; } - postcopy_thread_create(mis, &mis->fault_thread, "postcopy/fault", + postcopy_thread_create(mis, &mis->fault_thread, "fault-default", postcopy_ram_fault_thread, QEMU_THREAD_JOINABLE); mis->have_fault_thread = true; @@ -1185,6 +1198,16 @@ int postcopy_ram_incoming_setup(MigrationIncomingState *mis) return -1; } + if (migrate_postcopy_preempt()) { + /* + * This thread needs to be created after the temp pages because it'll fetch + * RAM_CHANNEL_POSTCOPY PostcopyTmpPage immediately. + */ + postcopy_thread_create(mis, &mis->postcopy_prio_thread, "fault-fast", + postcopy_preempt_thread, QEMU_THREAD_JOINABLE); + mis->postcopy_prio_thread_created = true; + } + trace_postcopy_ram_enable_notify(); return 0; @@ -1514,3 +1537,66 @@ void postcopy_unregister_shared_ufd(struct PostCopyFD *pcfd) } } } + +bool postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file) +{ + /* + * The new loading channel has its own threads, so it needs to be + * blocked too. It's by default true, just be explicit. + */ + qemu_file_set_blocking(file, true); + mis->postcopy_qemufile_dst = file; + trace_postcopy_preempt_new_channel(); + + /* Start the migration immediately */ + return true; +} + +int postcopy_preempt_setup(MigrationState *s, Error **errp) +{ + QIOChannel *ioc; + + if (!migrate_postcopy_preempt()) { + return 0; + } + + if (!migrate_multi_channels_is_allowed()) { + error_setg(errp, "Postcopy preempt is not supported as current " + "migration stream does not support multi-channels."); + return -1; + } + + ioc = socket_send_channel_create_sync(errp); + + if (ioc == NULL) { + return -1; + } + + migration_ioc_register_yank(ioc); + s->postcopy_qemufile_src = qemu_fopen_channel_output(ioc); + + trace_postcopy_preempt_new_channel(); + + return 0; +} + +void *postcopy_preempt_thread(void *opaque) +{ + MigrationIncomingState *mis = opaque; + int ret; + + trace_postcopy_preempt_thread_entry(); + + rcu_register_thread(); + + qemu_sem_post(&mis->thread_sync_sem); + + /* Sending RAM_SAVE_FLAG_EOS to terminate this thread */ + ret = ram_load_postcopy(mis->postcopy_qemufile_dst, RAM_CHANNEL_POSTCOPY); + + rcu_unregister_thread(); + + trace_postcopy_preempt_thread_exit(); + + return ret == 0 ? NULL : (void *)-1; +} diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 07684c0e1d..34b1080cde 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -183,4 +183,14 @@ int postcopy_wake_shared(struct PostCopyFD *pcfd, uint64_t client_addr, int postcopy_request_shared_page(struct PostCopyFD *pcfd, RAMBlock *rb, uint64_t client_addr, uint64_t offset); +/* Hard-code channels for now for postcopy preemption */ +enum PostcopyChannels { + RAM_CHANNEL_PRECOPY = 0, + RAM_CHANNEL_POSTCOPY = 1, + RAM_CHANNEL_MAX, +}; + +bool postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file); +int postcopy_preempt_setup(MigrationState *s, Error **errp); + #endif diff --git a/migration/ram.c b/migration/ram.c index 253fe4b756..c7ea1d9215 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3644,15 +3644,15 @@ int ram_postcopy_incoming_init(MigrationIncomingState *mis) * rcu_read_lock is taken prior to this being called. * * @f: QEMUFile where to send the data + * @channel: the channel to use for loading */ -int ram_load_postcopy(QEMUFile *f) +int ram_load_postcopy(QEMUFile *f, int channel) { int flags = 0, ret = 0; bool place_needed = false; bool matches_target_page_size = false; MigrationIncomingState *mis = migration_incoming_get_current(); - /* Currently we only use channel 0. TODO: use all the channels */ - PostcopyTmpPage *tmp_page = &mis->postcopy_tmp_pages[0]; + PostcopyTmpPage *tmp_page = &mis->postcopy_tmp_pages[channel]; while (!ret && !(flags & RAM_SAVE_FLAG_EOS)) { ram_addr_t addr; @@ -3676,7 +3676,7 @@ int ram_load_postcopy(QEMUFile *f) flags = addr & ~TARGET_PAGE_MASK; addr &= TARGET_PAGE_MASK; - trace_ram_load_postcopy_loop((uint64_t)addr, flags); + trace_ram_load_postcopy_loop(channel, (uint64_t)addr, flags); if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE | RAM_SAVE_FLAG_COMPRESS_PAGE)) { block = ram_block_from_stream(mis, f, flags); @@ -3717,10 +3717,10 @@ int ram_load_postcopy(QEMUFile *f) } else if (tmp_page->host_addr != host_page_from_ram_block_offset(block, addr)) { /* not the 1st TP within the HP */ - error_report("Non-same host page detected. " + error_report("Non-same host page detected on channel %d: " "Target host page %p, received host page %p " "(rb %s offset 0x"RAM_ADDR_FMT" target_pages %d)", - tmp_page->host_addr, + channel, tmp_page->host_addr, host_page_from_ram_block_offset(block, addr), block->idstr, addr, tmp_page->target_pages); ret = -EINVAL; @@ -4107,7 +4107,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id) */ WITH_RCU_READ_LOCK_GUARD() { if (postcopy_running) { - ret = ram_load_postcopy(f); + /* + * Note! Here RAM_CHANNEL_PRECOPY is the precopy channel of + * postcopy migration, we have another RAM_CHANNEL_POSTCOPY to + * service fast page faults. + */ + ret = ram_load_postcopy(f, RAM_CHANNEL_PRECOPY); } else { ret = ram_load_precopy(f); } @@ -4269,6 +4274,12 @@ static int ram_resume_prepare(MigrationState *s, void *opaque) return 0; } +void postcopy_preempt_shutdown_file(MigrationState *s) +{ + qemu_put_be64(s->postcopy_qemufile_src, RAM_SAVE_FLAG_EOS); + qemu_fflush(s->postcopy_qemufile_src); +} + static SaveVMHandlers savevm_ram_handlers = { .save_setup = ram_save_setup, .save_live_iterate = ram_save_iterate, diff --git a/migration/ram.h b/migration/ram.h index ded0a3a086..5d90945a6e 100644 --- a/migration/ram.h +++ b/migration/ram.h @@ -61,7 +61,7 @@ void ram_postcopy_send_discard_bitmap(MigrationState *ms); /* For incoming postcopy discard */ int ram_discard_range(const char *block_name, uint64_t start, size_t length); int ram_postcopy_incoming_init(MigrationIncomingState *mis); -int ram_load_postcopy(QEMUFile *f); +int ram_load_postcopy(QEMUFile *f, int channel); void ram_handle_compressed(void *host, uint8_t ch, uint64_t size); @@ -73,6 +73,8 @@ int64_t ramblock_recv_bitmap_send(QEMUFile *file, const char *block_name); int ram_dirty_bitmap_reload(MigrationState *s, RAMBlock *rb); bool ramblock_page_is_discarded(RAMBlock *rb, ram_addr_t start); +void postcopy_preempt_shutdown_file(MigrationState *s); +void *postcopy_preempt_thread(void *opaque); /* ram cache */ int colo_init_ram_cache(void); diff --git a/migration/savevm.c b/migration/savevm.c index d9076897b8..ecee05e631 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2575,16 +2575,6 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) { int i; - /* - * If network is interrupted, any temp page we received will be useless - * because we didn't mark them as "received" in receivedmap. After a - * proper recovery later (which will sync src dirty bitmap with receivedmap - * on dest) these cached small pages will be resent again. - */ - for (i = 0; i < mis->postcopy_channels; i++) { - postcopy_temp_page_reset(&mis->postcopy_tmp_pages[i]); - } - trace_postcopy_pause_incoming(); assert(migrate_postcopy_ram()); @@ -2613,6 +2603,16 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) /* Notify the fault thread for the invalidated file handle */ postcopy_fault_thread_notify(mis); + /* + * If network is interrupted, any temp page we received will be useless + * because we didn't mark them as "received" in receivedmap. After a + * proper recovery later (which will sync src dirty bitmap with receivedmap + * on dest) these cached small pages will be resent again. + */ + for (i = 0; i < mis->postcopy_channels; i++) { + postcopy_temp_page_reset(&mis->postcopy_tmp_pages[i]); + } + error_report("Detected IO failure for postcopy. " "Migration paused."); diff --git a/migration/socket.c b/migration/socket.c index 05705a32d8..a7f345b353 100644 --- a/migration/socket.c +++ b/migration/socket.c @@ -26,7 +26,7 @@ #include "io/channel-socket.h" #include "io/net-listener.h" #include "trace.h" - +#include "postcopy-ram.h" struct SocketOutgoingArgs { SocketAddress *saddr; @@ -39,6 +39,24 @@ void socket_send_channel_create(QIOTaskFunc f, void *data) f, data, NULL, NULL); } +QIOChannel *socket_send_channel_create_sync(Error **errp) +{ + QIOChannelSocket *sioc = qio_channel_socket_new(); + + if (!outgoing_args.saddr) { + object_unref(OBJECT(sioc)); + error_setg(errp, "Initial sock address not set!"); + return NULL; + } + + if (qio_channel_socket_connect_sync(sioc, outgoing_args.saddr, errp) < 0) { + object_unref(OBJECT(sioc)); + return NULL; + } + + return QIO_CHANNEL(sioc); +} + int socket_send_channel_destroy(QIOChannel *send) { /* Remove channel */ @@ -158,6 +176,8 @@ socket_start_incoming_migration_internal(SocketAddress *saddr, if (migrate_use_multifd()) { num = migrate_multifd_channels(); + } else if (migrate_postcopy_preempt()) { + num = RAM_CHANNEL_MAX; } if (qio_net_listener_open_sync(listener, saddr, num, errp) < 0) { diff --git a/migration/socket.h b/migration/socket.h index 891dbccceb..dc54df4e6c 100644 --- a/migration/socket.h +++ b/migration/socket.h @@ -21,6 +21,7 @@ #include "io/task.h" void socket_send_channel_create(QIOTaskFunc f, void *data); +QIOChannel *socket_send_channel_create_sync(Error **errp); int socket_send_channel_destroy(QIOChannel *send); void socket_start_incoming_migration(const char *str, Error **errp); diff --git a/migration/trace-events b/migration/trace-events index 1aec580e92..1f932782d9 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -91,7 +91,7 @@ migration_bitmap_clear_dirty(char *str, uint64_t start, uint64_t size, unsigned migration_throttle(void) "" ram_discard_range(const char *rbname, uint64_t start, size_t len) "%s: start: %" PRIx64 " %zx" ram_load_loop(const char *rbname, uint64_t addr, int flags, void *host) "%s: addr: 0x%" PRIx64 " flags: 0x%x host: %p" -ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x" +ram_load_postcopy_loop(int channel, uint64_t addr, int flags) "chan=%d addr=%" PRIx64 " flags=%x" ram_postcopy_send_discard_bitmap(void) "" ram_save_page(const char *rbname, uint64_t offset, void *host) "%s: offset: 0x%" PRIx64 " host: %p" ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: 0x%zx len: 0x%zx" @@ -278,6 +278,9 @@ postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t rb_off postcopy_request_shared_page_present(const char *sharer, const char *rb, uint64_t rb_offset) "%s already %s offset 0x%"PRIx64 postcopy_wake_shared(uint64_t client_addr, const char *rb) "at 0x%"PRIx64" in %s" postcopy_page_req_del(void *addr, int count) "resolved page req %p total %d" +postcopy_preempt_new_channel(void) "" +postcopy_preempt_thread_entry(void) "" +postcopy_preempt_thread_exit(void) "" get_mem_fault_cpu_index(int cpu, uint32_t pid) "cpu: %d, pid: %u" From patchwork Thu Mar 31 15:08:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611806 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UN6gmCM6; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn2v4j5Pz9sBy for ; Fri, 1 Apr 2022 02:16:55 +1100 (AEDT) Received: from localhost ([::1]:38292 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwXd-00069n-Ni for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:16:53 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52592) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQK-0002HO-Ux for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:27233) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQE-0001Mf-4d for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739352; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TNH+OWMLIg1wRkLI7itAwvWZSaFu058+HZKJ0ceRpe0=; b=UN6gmCM6urQX7cqsjG5gn+DOHZ6OCd04WyOPVkGZsjkx7mgVhFymCB6B32Fjxy0+LAYNzd ahX301dtMEZUlVDY4jTA+2yZ93rYm1gLmYYGGnpPSU9htCFXPyB7Jd4yP2847uyMg91NiT Eh5qe3Sz3VDMjUixEom/ULrru09swGw= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-298-qzB4tomDM4Gg1hVt8cXPEQ-1; Thu, 31 Mar 2022 11:09:11 -0400 X-MC-Unique: qzB4tomDM4Gg1hVt8cXPEQ-1 Received: by mail-qt1-f200.google.com with SMTP id l19-20020a05622a175300b002e1a85b5b52so20247633qtk.21 for ; Thu, 31 Mar 2022 08:09:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TNH+OWMLIg1wRkLI7itAwvWZSaFu058+HZKJ0ceRpe0=; b=U4atrIxwIN97b5VGUXuY9cFFFZfMIcciRaF3zzd+Nvf4VUsoJ0Orzjm+NRZCS8VZSH wgFi6zvAY/aEjJdTo4+9qUpBQ858s8ouRLaVtYZpuIUb1OXhfBrmRTy01exfPIVtFnCA 6MjwSU+RBI2jmVDVza8Ve5hpcvp/HSeu/Oo2rZUpZ680sDSVfH7w2jgWZUBMxAS3RLd3 zCFFTkPduAOhgxsCyvGCmZaRbQ2XV502Few8EUdFP0E7emZL2iIkzbG0Q4qI7RRSRGZw DhCTtgUVP0D2oE92cel2PWxNmQQcgJw7tl7IYZ2fMhu0P0WntSHQqx/034bU63xotfeY 4SPw== X-Gm-Message-State: AOAM53179ANJIFcTNV7nzapFOIcmsOcwDcd+7jPfOSyGn4O64A5OMFE7 lS8cdVD+jJQ4XtZvTuHoIiHIYXQTvCaiRotoNhUneDyQZi+xpBN+qNcjN13Z6ugUqyjDCJtwTB+ jwY9v0RhY6v94AqmErPBLOn2obOmUCT59+rhgwvaHCxEvVBnpdLYhLQxsmHZJEJMq X-Received: by 2002:a05:620a:2448:b0:67e:cbef:8961 with SMTP id h8-20020a05620a244800b0067ecbef8961mr3611305qkn.392.1648739350447; Thu, 31 Mar 2022 08:09:10 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxgUz49LKpPXY7ovl1pNhs2tteNO9N/k/eUcSF3ONHZFj4T7awvEK1wNnl1KvsYv6v+LJiyxA== X-Received: by 2002:a05:620a:2448:b0:67e:cbef:8961 with SMTP id h8-20020a05620a244800b0067ecbef8961mr3611260qkn.392.1648739349841; Thu, 31 Mar 2022 08:09:09 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:09 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 10/19] migration: Postcopy preemption enablement Date: Thu, 31 Mar 2022 11:08:48 -0400 Message-Id: <20220331150857.74406-11-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch enables postcopy-preempt feature. It contains two major changes to the migration logic: (1) Postcopy requests are now sent via a different socket from precopy background migration stream, so as to be isolated from very high page request delays. (2) For huge page enabled hosts: when there's postcopy requests, they can now intercept a partial sending of huge host pages on src QEMU. After this patch, we'll live migrate a VM with two channels for postcopy: (1) PRECOPY channel, which is the default channel that transfers background pages; and (2) POSTCOPY channel, which only transfers requested pages. There's no strict rule of which channel to use, e.g., if a requested page is already being transferred on precopy channel, then we will keep using the same precopy channel to transfer the page even if it's explicitly requested. In 99% of the cases we'll prioritize the channels so we send requested page via the postcopy channel as long as possible. On the source QEMU, when we found a postcopy request, we'll interrupt the PRECOPY channel sending process and quickly switch to the POSTCOPY channel. After we serviced all the high priority postcopy pages, we'll switch back to PRECOPY channel so that we'll continue to send the interrupted huge page again. There's no new thread introduced on src QEMU. On the destination QEMU, one new thread is introduced to receive page data from the postcopy specific socket (done in the preparation patch). This patch has a side effect: after sending postcopy pages, previously we'll assume the guest will access follow up pages so we'll keep sending from there. Now it's changed. Instead of going on with a postcopy requested page, we'll go back and continue sending the precopy huge page (which can be intercepted by a postcopy request so the huge page can be sent partially before). Whether that's a problem is debatable, because "assuming the guest will continue to access the next page" may not really suite when huge pages are used, especially if the huge page is large (e.g. 1GB pages). So that locality hint is much meaningless if huge pages are used. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 2 + migration/migration.h | 2 +- migration/ram.c | 250 +++++++++++++++++++++++++++++++++++++++-- migration/trace-events | 7 ++ 4 files changed, 252 insertions(+), 9 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 01b882494d..56d54c186b 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -3158,6 +3158,8 @@ static int postcopy_start(MigrationState *ms) MIGRATION_STATUS_FAILED); } + trace_postcopy_preempt_enabled(migrate_postcopy_preempt()); + return ret; fail_closefb: diff --git a/migration/migration.h b/migration/migration.h index caa910d956..b8aacfe3af 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -68,7 +68,7 @@ typedef struct { struct MigrationIncomingState { QEMUFile *from_src_file; /* Previously received RAM's RAMBlock pointer */ - RAMBlock *last_recv_block; + RAMBlock *last_recv_block[RAM_CHANNEL_MAX]; /* A hook to allow cleanup at the end of incoming migration */ void *transport_data; void (*transport_cleanup)(void *data); diff --git a/migration/ram.c b/migration/ram.c index c7ea1d9215..518d511874 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -295,6 +295,20 @@ struct RAMSrcPageRequest { QSIMPLEQ_ENTRY(RAMSrcPageRequest) next_req; }; +typedef struct { + /* + * Cached ramblock/offset values if preempted. They're only meaningful if + * preempted==true below. + */ + RAMBlock *ram_block; + unsigned long ram_page; + /* + * Whether a postcopy preemption just happened. Will be reset after + * precopy recovered to background migration. + */ + bool preempted; +} PostcopyPreemptState; + /* State of RAM for migration */ struct RAMState { /* QEMUFile used for this migration */ @@ -349,6 +363,14 @@ struct RAMState { /* Queue of outstanding page requests from the destination */ QemuMutex src_page_req_mutex; QSIMPLEQ_HEAD(, RAMSrcPageRequest) src_page_requests; + + /* Postcopy preemption informations */ + PostcopyPreemptState postcopy_preempt_state; + /* + * Current channel we're using on src VM. Only valid if postcopy-preempt + * is enabled. + */ + unsigned int postcopy_channel; }; typedef struct RAMState RAMState; @@ -356,6 +378,11 @@ static RAMState *ram_state; static NotifierWithReturnList precopy_notifier_list; +static void postcopy_preempt_reset(RAMState *rs) +{ + memset(&rs->postcopy_preempt_state, 0, sizeof(PostcopyPreemptState)); +} + /* Whether postcopy has queued requests? */ static bool postcopy_has_request(RAMState *rs) { @@ -1947,6 +1974,55 @@ void ram_write_tracking_stop(void) } #endif /* defined(__linux__) */ +/* + * Check whether two addr/offset of the ramblock falls onto the same host huge + * page. Returns true if so, false otherwise. + */ +static bool offset_on_same_huge_page(RAMBlock *rb, uint64_t addr1, + uint64_t addr2) +{ + size_t page_size = qemu_ram_pagesize(rb); + + addr1 = ROUND_DOWN(addr1, page_size); + addr2 = ROUND_DOWN(addr2, page_size); + + return addr1 == addr2; +} + +/* + * Whether a previous preempted precopy huge page contains current requested + * page? Returns true if so, false otherwise. + * + * This should really happen very rarely, because it means when we were sending + * during background migration for postcopy we're sending exactly the page that + * some vcpu got faulted on on dest node. When it happens, we probably don't + * need to do much but drop the request, because we know right after we restore + * the precopy stream it'll be serviced. It'll slightly affect the order of + * postcopy requests to be serviced (e.g. it'll be the same as we move current + * request to the end of the queue) but it shouldn't be a big deal. The most + * imporant thing is we can _never_ try to send a partial-sent huge page on the + * POSTCOPY channel again, otherwise that huge page will got "split brain" on + * two channels (PRECOPY, POSTCOPY). + */ +static bool postcopy_preempted_contains(RAMState *rs, RAMBlock *block, + ram_addr_t offset) +{ + PostcopyPreemptState *state = &rs->postcopy_preempt_state; + + /* No preemption at all? */ + if (!state->preempted) { + return false; + } + + /* Not even the same ramblock? */ + if (state->ram_block != block) { + return false; + } + + return offset_on_same_huge_page(block, offset, + state->ram_page << TARGET_PAGE_BITS); +} + /** * get_queued_page: unqueue a page from the postcopy requests * @@ -1962,9 +2038,17 @@ static bool get_queued_page(RAMState *rs, PageSearchStatus *pss) RAMBlock *block; ram_addr_t offset; +again: block = unqueue_page(rs, &offset); - if (!block) { + if (block) { + /* See comment above postcopy_preempted_contains() */ + if (postcopy_preempted_contains(rs, block, offset)) { + trace_postcopy_preempt_hit(block->idstr, offset); + /* This request is dropped */ + goto again; + } + } else { /* * Poll write faults too if background snapshot is enabled; that's * when we have vcpus got blocked by the write protected pages. @@ -2180,6 +2264,117 @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss) return ram_save_page(rs, pss); } +static bool postcopy_needs_preempt(RAMState *rs, PageSearchStatus *pss) +{ + /* Not enabled eager preempt? Then never do that. */ + if (!migrate_postcopy_preempt()) { + return false; + } + + /* If the ramblock we're sending is a small page? Never bother. */ + if (qemu_ram_pagesize(pss->block) == TARGET_PAGE_SIZE) { + return false; + } + + /* Not in postcopy at all? */ + if (!migration_in_postcopy()) { + return false; + } + + /* + * If we're already handling a postcopy request, don't preempt as this page + * has got the same high priority. + */ + if (pss->postcopy_requested) { + return false; + } + + /* If there's postcopy requests, then check it up! */ + return postcopy_has_request(rs); +} + +/* Returns true if we preempted precopy, false otherwise */ +static void postcopy_do_preempt(RAMState *rs, PageSearchStatus *pss) +{ + PostcopyPreemptState *p_state = &rs->postcopy_preempt_state; + + trace_postcopy_preempt_triggered(pss->block->idstr, pss->page); + + /* + * Time to preempt precopy. Cache current PSS into preempt state, so that + * after handling the postcopy pages we can recover to it. We need to do + * so because the dest VM will have partial of the precopy huge page kept + * over in its tmp huge page caches; better move on with it when we can. + */ + p_state->ram_block = pss->block; + p_state->ram_page = pss->page; + p_state->preempted = true; +} + +/* Whether we're preempted by a postcopy request during sending a huge page */ +static bool postcopy_preempt_triggered(RAMState *rs) +{ + return rs->postcopy_preempt_state.preempted; +} + +static void postcopy_preempt_restore(RAMState *rs, PageSearchStatus *pss) +{ + PostcopyPreemptState *state = &rs->postcopy_preempt_state; + + assert(state->preempted); + + pss->block = state->ram_block; + pss->page = state->ram_page; + /* This is not a postcopy request but restoring previous precopy */ + pss->postcopy_requested = false; + + trace_postcopy_preempt_restored(pss->block->idstr, pss->page); + + /* Reset preempt state, most importantly, set preempted==false */ + postcopy_preempt_reset(rs); +} + +static void postcopy_preempt_choose_channel(RAMState *rs, PageSearchStatus *pss) +{ + MigrationState *s = migrate_get_current(); + unsigned int channel; + QEMUFile *next; + + channel = pss->postcopy_requested ? + RAM_CHANNEL_POSTCOPY : RAM_CHANNEL_PRECOPY; + + if (channel != rs->postcopy_channel) { + if (channel == RAM_CHANNEL_PRECOPY) { + next = s->to_dst_file; + } else { + next = s->postcopy_qemufile_src; + } + /* Update and cache the current channel */ + rs->f = next; + rs->postcopy_channel = channel; + + /* + * If channel switched, reset last_sent_block since the old sent block + * may not be on the same channel. + */ + rs->last_sent_block = NULL; + + trace_postcopy_preempt_switch_channel(channel); + } + + trace_postcopy_preempt_send_host_page(pss->block->idstr, pss->page); +} + +/* We need to make sure rs->f always points to the default channel elsewhere */ +static void postcopy_preempt_reset_channel(RAMState *rs) +{ + if (migrate_postcopy_preempt() && migration_in_postcopy()) { + rs->postcopy_channel = RAM_CHANNEL_PRECOPY; + rs->f = migrate_get_current()->to_dst_file; + trace_postcopy_preempt_reset_channel(); + } +} + /** * ram_save_host_page: save a whole host page * @@ -2211,7 +2406,16 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss) return 0; } + if (migrate_postcopy_preempt() && migration_in_postcopy()) { + postcopy_preempt_choose_channel(rs, pss); + } + do { + if (postcopy_needs_preempt(rs, pss)) { + postcopy_do_preempt(rs, pss); + break; + } + /* Check the pages is dirty and if it is send it */ if (migration_bitmap_clear_dirty(rs, pss->block, pss->page)) { tmppages = ram_save_target_page(rs, pss); @@ -2235,6 +2439,19 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss) /* The offset we leave with is the min boundary of host page and block */ pss->page = MIN(pss->page, hostpage_boundary); + /* + * When with postcopy preempt mode, flush the data as soon as possible for + * postcopy requests, because we've already sent a whole huge page, so the + * dst node should already have enough resource to atomically filling in + * the current missing page. + * + * More importantly, when using separate postcopy channel, we must do + * explicit flush or it won't flush until the buffer is full. + */ + if (migrate_postcopy_preempt() && pss->postcopy_requested) { + qemu_fflush(rs->f); + } + res = ram_save_release_protection(rs, pss, start_page); return (res < 0 ? res : pages); } @@ -2276,8 +2493,17 @@ static int ram_find_and_save_block(RAMState *rs) found = get_queued_page(rs, &pss); if (!found) { - /* priority queue empty, so just search for something dirty */ - found = find_dirty_block(rs, &pss, &again); + /* + * Recover previous precopy ramblock/offset if postcopy has + * preempted precopy. Otherwise find the next dirty bit. + */ + if (postcopy_preempt_triggered(rs)) { + postcopy_preempt_restore(rs, &pss); + found = true; + } else { + /* priority queue empty, so just search for something dirty */ + found = find_dirty_block(rs, &pss, &again); + } } if (found) { @@ -2405,6 +2631,8 @@ static void ram_state_reset(RAMState *rs) rs->last_page = 0; rs->last_version = ram_list.version; rs->xbzrle_enabled = false; + postcopy_preempt_reset(rs); + rs->postcopy_channel = RAM_CHANNEL_PRECOPY; } #define MAX_WAIT 50 /* ms, half buffered_file limit */ @@ -3043,6 +3271,8 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) } qemu_mutex_unlock(&rs->bitmap_mutex); + postcopy_preempt_reset_channel(rs); + /* * Must occur before EOS (or any QEMUFile operation) * because of RDMA protocol. @@ -3112,6 +3342,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque) ram_control_after_iterate(f, RAM_CONTROL_FINISH); } + postcopy_preempt_reset_channel(rs); + if (ret >= 0) { multifd_send_sync_main(rs->f); qemu_put_be64(f, RAM_SAVE_FLAG_EOS); @@ -3194,11 +3426,13 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host) * @mis: the migration incoming state pointer * @f: QEMUFile where to read the data from * @flags: Page flags (mostly to see if it's a continuation of previous block) + * @channel: the channel we're using */ static inline RAMBlock *ram_block_from_stream(MigrationIncomingState *mis, - QEMUFile *f, int flags) + QEMUFile *f, int flags, + int channel) { - RAMBlock *block = mis->last_recv_block; + RAMBlock *block = mis->last_recv_block[channel]; char id[256]; uint8_t len; @@ -3225,7 +3459,7 @@ static inline RAMBlock *ram_block_from_stream(MigrationIncomingState *mis, return NULL; } - mis->last_recv_block = block; + mis->last_recv_block[channel] = block; return block; } @@ -3679,7 +3913,7 @@ int ram_load_postcopy(QEMUFile *f, int channel) trace_ram_load_postcopy_loop(channel, (uint64_t)addr, flags); if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE | RAM_SAVE_FLAG_COMPRESS_PAGE)) { - block = ram_block_from_stream(mis, f, flags); + block = ram_block_from_stream(mis, f, flags, channel); if (!block) { ret = -EINVAL; break; @@ -3930,7 +4164,7 @@ static int ram_load_precopy(QEMUFile *f) if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE | RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { - RAMBlock *block = ram_block_from_stream(mis, f, flags); + RAMBlock *block = ram_block_from_stream(mis, f, flags, RAM_CHANNEL_PRECOPY); host = host_from_ram_block_offset(block, addr); /* diff --git a/migration/trace-events b/migration/trace-events index 1f932782d9..f92793b5f5 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -111,6 +111,12 @@ ram_load_complete(int ret, uint64_t seq_iter) "exit_code %d seq iteration %" PRI ram_write_tracking_ramblock_start(const char *block_id, size_t page_size, void *addr, size_t length) "%s: page_size: %zu addr: %p length: %zu" ram_write_tracking_ramblock_stop(const char *block_id, size_t page_size, void *addr, size_t length) "%s: page_size: %zu addr: %p length: %zu" unqueue_page(char *block, uint64_t offset, bool dirty) "ramblock '%s' offset 0x%"PRIx64" dirty %d" +postcopy_preempt_triggered(char *str, unsigned long page) "during sending ramblock %s offset 0x%lx" +postcopy_preempt_restored(char *str, unsigned long page) "ramblock %s offset 0x%lx" +postcopy_preempt_hit(char *str, uint64_t offset) "ramblock %s offset 0x%"PRIx64 +postcopy_preempt_send_host_page(char *str, uint64_t offset) "ramblock %s offset 0x%"PRIx64 +postcopy_preempt_switch_channel(int channel) "%d" +postcopy_preempt_reset_channel(void) "" # multifd.c multifd_new_send_channel_async(uint8_t id) "channel %u" @@ -176,6 +182,7 @@ migration_thread_low_pending(uint64_t pending) "%" PRIu64 migrate_transferred(uint64_t tranferred, uint64_t time_spent, uint64_t bandwidth, uint64_t size) "transferred %" PRIu64 " time_spent %" PRIu64 " bandwidth %" PRIu64 " max_size %" PRId64 process_incoming_migration_co_end(int ret, int ps) "ret=%d postcopy-state=%d" process_incoming_migration_co_postcopy_end_main(void) "" +postcopy_preempt_enabled(bool value) "%d" # channel.c migration_set_incoming_channel(void *ioc, const char *ioctype) "ioc=%p ioctype=%s" From patchwork Thu Mar 31 15:08:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611799 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZgB/XzAU; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTmzG0XqVz9sGP for ; Fri, 1 Apr 2022 02:13:46 +1100 (AEDT) Received: from localhost ([::1]:57892 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwUa-0000Ii-3E for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:13:44 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52572) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQK-0002Em-6g for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:48229) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQE-0001Mp-4g for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739353; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wQaSTm3KPTilmyUMfVAV0Vk+qNPQBPN0Bd4CtZOcSNE=; b=ZgB/XzAUw4jYSUerzzaDWtRaNx7p5kRQh6x4pole3DSy1C578PHOw2SFbbKWSMafVQxWxb S3aluhK4CTKh53h2SOtaN+Irvhty9F744x7wg6gImtNvIxvwT2G7yizfZCO408GBc5Z/cr TJrBJ//93awyRRkBwpmymb5RrFGO6zI= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-650-nrU_pClHP3Osrg5wtPmYUg-1; Thu, 31 Mar 2022 11:09:12 -0400 X-MC-Unique: nrU_pClHP3Osrg5wtPmYUg-1 Received: by mail-qv1-f70.google.com with SMTP id kl16-20020a056214519000b004439e99963bso1363652qvb.3 for ; Thu, 31 Mar 2022 08:09:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wQaSTm3KPTilmyUMfVAV0Vk+qNPQBPN0Bd4CtZOcSNE=; b=1uSykkA+mfCREyqGWvfXWWtPjk0ioElpPKfX49f3SNR/bYSyktwNbqwrktXmG6dIR6 5LY2zKybV8pMjFWJDqK5nINQqSaK62VeE/ojx77WZZHcs6D9R0aw/dIDZ9V4Z76s6Z2W Jpu+glJgjJKsW0uNKU86oZJAGteNm/2p2QR3fAMZtwz9TRgH/fIoU/WMfO55R8FfArDB ID/Xbg52c2P3icY2ZfIBX5ocy12Homg3BiqF+Oh1PzJrcI+Ke/j8MT1VlZAKvSN4KPhc jMn1b6SBMiWYlzJcmIIHDGmR09tIlzRY9zGHpzQ/Nz3kKWdan758EYYux09erKSlKOTu w/QQ== X-Gm-Message-State: AOAM531LAzQsLsauXKPOPzrmna4iAlQk5RvwBoeh4tgvh96FEG8I+wd2 aNEGw19WYqzIPi1GifRmEwMMKWrJNkUqb55l14RmbIRJup3K+1itmYlAT3+w97n4CHJU+SnFko+ kLnjQQ/WrwWUjPsErnoqpMjh+yUQDRT91Ub22eOdWMRZauDlc/BnTqIHhaGqCKcPK X-Received: by 2002:a05:622a:13cf:b0:2e2:3535:56ae with SMTP id p15-20020a05622a13cf00b002e2353556aemr4663463qtk.373.1648739351345; Thu, 31 Mar 2022 08:09:11 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwhwhazebx+qx/fsSA3mo3Mmt05cvSKUaobuh1jDYoTg9ztMHRwXYuBUHsadsYGsXWZaJH41g== X-Received: by 2002:a05:622a:13cf:b0:2e2:3535:56ae with SMTP id p15-20020a05622a13cf00b002e2353556aemr4663412qtk.373.1648739350777; Thu, 31 Mar 2022 08:09:10 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:10 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 11/19] migration: Postcopy recover with preempt enabled Date: Thu, 31 Mar 2022 11:08:49 -0400 Message-Id: <20220331150857.74406-12-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" To allow postcopy recovery, the ram fast load (preempt-only) dest QEMU thread needs similar handling on fault tolerance. When ram_load_postcopy() fails, instead of stopping the thread it halts with a semaphore, preparing to be kicked again when recovery is detected. A mutex is introduced to make sure there's no concurrent operation upon the socket. To make it simple, the fast ram load thread will take the mutex during its whole procedure, and only release it if it's paused. The fast-path socket will be properly released by the main loading thread safely when there's network failures during postcopy with that mutex held. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 27 +++++++++++++++++++++++---- migration/migration.h | 19 +++++++++++++++++++ migration/postcopy-ram.c | 24 ++++++++++++++++++++++-- migration/qemu-file.c | 27 +++++++++++++++++++++++++++ migration/qemu-file.h | 1 + migration/savevm.c | 26 ++++++++++++++++++++++++-- migration/trace-events | 2 ++ 7 files changed, 118 insertions(+), 8 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 56d54c186b..157a34c844 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -215,9 +215,11 @@ void migration_object_init(void) current_incoming->postcopy_remote_fds = g_array_new(FALSE, TRUE, sizeof(struct PostCopyFD)); qemu_mutex_init(¤t_incoming->rp_mutex); + qemu_mutex_init(¤t_incoming->postcopy_prio_thread_mutex); qemu_event_init(¤t_incoming->main_thread_load_event, false); qemu_sem_init(¤t_incoming->postcopy_pause_sem_dst, 0); qemu_sem_init(¤t_incoming->postcopy_pause_sem_fault, 0); + qemu_sem_init(¤t_incoming->postcopy_pause_sem_fast_load, 0); qemu_mutex_init(¤t_incoming->page_request_mutex); current_incoming->page_requested = g_tree_new(page_request_addr_cmp); @@ -697,9 +699,9 @@ static bool postcopy_try_recover(void) /* * Here, we only wake up the main loading thread (while the - * fault thread will still be waiting), so that we can receive + * rest threads will still be waiting), so that we can receive * commands from source now, and answer it if needed. The - * fault thread will be woken up afterwards until we are sure + * rest threads will be woken up afterwards until we are sure * that source is ready to reply to page requests. */ qemu_sem_post(&mis->postcopy_pause_sem_dst); @@ -3471,6 +3473,18 @@ static MigThrError postcopy_pause(MigrationState *s) qemu_file_shutdown(file); qemu_fclose(file); + /* + * Do the same to postcopy fast path socket too if there is. No + * locking needed because no racer as long as we do this before setting + * status to paused. + */ + if (s->postcopy_qemufile_src) { + migration_ioc_unregister_yank_from_file(s->postcopy_qemufile_src); + qemu_file_shutdown(s->postcopy_qemufile_src); + qemu_fclose(s->postcopy_qemufile_src); + s->postcopy_qemufile_src = NULL; + } + migrate_set_state(&s->state, s->state, MIGRATION_STATUS_POSTCOPY_PAUSED); @@ -3526,8 +3540,13 @@ static MigThrError migration_detect_error(MigrationState *s) return MIG_THR_ERR_FATAL; } - /* Try to detect any file errors */ - ret = qemu_file_get_error_obj(s->to_dst_file, &local_error); + /* + * Try to detect any file errors. Note that postcopy_qemufile_src will + * be NULL when postcopy preempt is not enabled. + */ + ret = qemu_file_get_error_obj_any(s->to_dst_file, + s->postcopy_qemufile_src, + &local_error); if (!ret) { /* Everything is fine */ assert(!local_error); diff --git a/migration/migration.h b/migration/migration.h index b8aacfe3af..91f845e9e4 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -118,6 +118,18 @@ struct MigrationIncomingState { /* Postcopy priority thread is used to receive postcopy requested pages */ QemuThread postcopy_prio_thread; bool postcopy_prio_thread_created; + /* + * Used to sync between the ram load main thread and the fast ram load + * thread. It protects postcopy_qemufile_dst, which is the postcopy + * fast channel. + * + * The ram fast load thread will take it mostly for the whole lifecycle + * because it needs to continuously read data from the channel, and + * it'll only release this mutex if postcopy is interrupted, so that + * the ram load main thread will take this mutex over and properly + * release the broken channel. + */ + QemuMutex postcopy_prio_thread_mutex; /* * An array of temp host huge pages to be used, one for each postcopy * channel. @@ -147,6 +159,13 @@ struct MigrationIncomingState { /* notify PAUSED postcopy incoming migrations to try to continue */ QemuSemaphore postcopy_pause_sem_dst; QemuSemaphore postcopy_pause_sem_fault; + /* + * This semaphore is used to allow the ram fast load thread (only when + * postcopy preempt is enabled) fall into sleep when there's network + * interruption detected. When the recovery is done, the main load + * thread will kick the fast ram load thread using this semaphore. + */ + QemuSemaphore postcopy_pause_sem_fast_load; /* List of listening socket addresses */ SocketAddressList *socket_address_list; diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index df0c02f729..e20305a9e2 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -1580,6 +1580,15 @@ int postcopy_preempt_setup(MigrationState *s, Error **errp) return 0; } +static void postcopy_pause_ram_fast_load(MigrationIncomingState *mis) +{ + trace_postcopy_pause_fast_load(); + qemu_mutex_unlock(&mis->postcopy_prio_thread_mutex); + qemu_sem_wait(&mis->postcopy_pause_sem_fast_load); + qemu_mutex_lock(&mis->postcopy_prio_thread_mutex); + trace_postcopy_pause_fast_load_continued(); +} + void *postcopy_preempt_thread(void *opaque) { MigrationIncomingState *mis = opaque; @@ -1592,11 +1601,22 @@ void *postcopy_preempt_thread(void *opaque) qemu_sem_post(&mis->thread_sync_sem); /* Sending RAM_SAVE_FLAG_EOS to terminate this thread */ - ret = ram_load_postcopy(mis->postcopy_qemufile_dst, RAM_CHANNEL_POSTCOPY); + qemu_mutex_lock(&mis->postcopy_prio_thread_mutex); + while (1) { + ret = ram_load_postcopy(mis->postcopy_qemufile_dst, RAM_CHANNEL_POSTCOPY); + /* If error happened, go into recovery routine */ + if (ret) { + postcopy_pause_ram_fast_load(mis); + } else { + /* We're done */ + break; + } + } + qemu_mutex_unlock(&mis->postcopy_prio_thread_mutex); rcu_unregister_thread(); trace_postcopy_preempt_thread_exit(); - return ret == 0 ? NULL : (void *)-1; + return NULL; } diff --git a/migration/qemu-file.c b/migration/qemu-file.c index 1479cddad9..397652f0ba 100644 --- a/migration/qemu-file.c +++ b/migration/qemu-file.c @@ -139,6 +139,33 @@ int qemu_file_get_error_obj(QEMUFile *f, Error **errp) return f->last_error; } +/* + * Get last error for either stream f1 or f2 with optional Error*. + * The error returned (non-zero) can be either from f1 or f2. + * + * If any of the qemufile* is NULL, then skip the check on that file. + * + * When there is no error on both qemufile, zero is returned. + */ +int qemu_file_get_error_obj_any(QEMUFile *f1, QEMUFile *f2, Error **errp) +{ + int ret = 0; + + if (f1) { + ret = qemu_file_get_error_obj(f1, errp); + /* If there's already error detected, return */ + if (ret) { + return ret; + } + } + + if (f2) { + ret = qemu_file_get_error_obj(f2, errp); + } + + return ret; +} + /* * Set the last error for stream f with optional Error* */ diff --git a/migration/qemu-file.h b/migration/qemu-file.h index 3f36d4dc8c..2564e5e1c7 100644 --- a/migration/qemu-file.h +++ b/migration/qemu-file.h @@ -156,6 +156,7 @@ void qemu_file_update_transfer(QEMUFile *f, int64_t len); void qemu_file_set_rate_limit(QEMUFile *f, int64_t new_rate); int64_t qemu_file_get_rate_limit(QEMUFile *f); int qemu_file_get_error_obj(QEMUFile *f, Error **errp); +int qemu_file_get_error_obj_any(QEMUFile *f1, QEMUFile *f2, Error **errp); void qemu_file_set_error_obj(QEMUFile *f, int ret, Error *err); void qemu_file_set_error(QEMUFile *f, int ret); int qemu_file_shutdown(QEMUFile *f); diff --git a/migration/savevm.c b/migration/savevm.c index ecee05e631..050874650a 100644 --- a/migration/savevm.c +++ b/migration/savevm.c @@ -2152,6 +2152,13 @@ static int loadvm_postcopy_handle_resume(MigrationIncomingState *mis) */ qemu_sem_post(&mis->postcopy_pause_sem_fault); + if (migrate_postcopy_preempt()) { + /* The channel should already be setup again; make sure of it */ + assert(mis->postcopy_qemufile_dst); + /* Kick the fast ram load thread too */ + qemu_sem_post(&mis->postcopy_pause_sem_fast_load); + } + return 0; } @@ -2597,6 +2604,21 @@ static bool postcopy_pause_incoming(MigrationIncomingState *mis) mis->to_src_file = NULL; qemu_mutex_unlock(&mis->rp_mutex); + /* + * NOTE: this must happen before reset the PostcopyTmpPages below, + * otherwise it's racy to reset those fields when the fast load thread + * can be accessing it in parallel. + */ + if (mis->postcopy_qemufile_dst) { + qemu_file_shutdown(mis->postcopy_qemufile_dst); + /* Take the mutex to make sure the fast ram load thread halted */ + qemu_mutex_lock(&mis->postcopy_prio_thread_mutex); + migration_ioc_unregister_yank_from_file(mis->postcopy_qemufile_dst); + qemu_fclose(mis->postcopy_qemufile_dst); + mis->postcopy_qemufile_dst = NULL; + qemu_mutex_unlock(&mis->postcopy_prio_thread_mutex); + } + migrate_set_state(&mis->state, MIGRATION_STATUS_POSTCOPY_ACTIVE, MIGRATION_STATUS_POSTCOPY_PAUSED); @@ -2634,8 +2656,8 @@ retry: while (true) { section_type = qemu_get_byte(f); - if (qemu_file_get_error(f)) { - ret = qemu_file_get_error(f); + ret = qemu_file_get_error_obj_any(f, mis->postcopy_qemufile_dst, NULL); + if (ret) { break; } diff --git a/migration/trace-events b/migration/trace-events index f92793b5f5..b21d5f371f 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -270,6 +270,8 @@ mark_postcopy_blocktime_begin(uint64_t addr, void *dd, uint32_t time, int cpu, i mark_postcopy_blocktime_end(uint64_t addr, void *dd, uint32_t time, int affected_cpu) "addr: 0x%" PRIx64 ", dd: %p, time: %u, affected_cpu: %d" postcopy_pause_fault_thread(void) "" postcopy_pause_fault_thread_continued(void) "" +postcopy_pause_fast_load(void) "" +postcopy_pause_fast_load_continued(void) "" postcopy_ram_fault_thread_entry(void) "" postcopy_ram_fault_thread_exit(void) "" postcopy_ram_fault_thread_fds_core(int baseufd, int quitfd) "ufd: %d quitfd: %d" From patchwork Thu Mar 31 15:08:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611805 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=DnRYmjfT; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn2G4GNSz9sBy for ; Fri, 1 Apr 2022 02:16:22 +1100 (AEDT) Received: from localhost ([::1]:37834 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwX6-0005rS-LO for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:16:20 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52564) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQJ-0002Df-SA for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:47031) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQG-0001NI-Cg for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739355; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ffvhKWyfzJn2iowy5Yr0REXiZWOmBeW+MY7e6yM3urw=; b=DnRYmjfTcSbluyFE/EwQHrltckB2kg4NPVcEMkWl6jp7HWfDykazeGWOC5p8mw5163nL9L IaquoEcm1p69K+gJTUrIJ8mAJenujxh53i8ycyc4NU+hMA5+13+tC9qbI2grlGwMz1HJwv nAg5vr8Nfd6ZwDYL9Sz4Zy33mfF66ng= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-205-c4ZWL_8ePfG8aoXPb3926A-1; Thu, 31 Mar 2022 11:09:14 -0400 X-MC-Unique: c4ZWL_8ePfG8aoXPb3926A-1 Received: by mail-qk1-f197.google.com with SMTP id bk23-20020a05620a1a1700b0067b32f93b90so959269qkb.16 for ; Thu, 31 Mar 2022 08:09:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ffvhKWyfzJn2iowy5Yr0REXiZWOmBeW+MY7e6yM3urw=; b=kT1fqMeZLjdCE7e2GrnYVS2QYkPfL43gffUx0jWuCNrve/QCcwDIeDQbHlSGNsBmOJ wN6n03hRp2r47JPVAPWiI0DDL2vrHUuBLw8pccFCOdJrAxH0+A3C0s023Et2NLutyPV9 RwqhBcRI7NDtejFRLjSruvbFOvfWmp3IN7BgOW8tBum4gcQjDdBYGmsPqpTVff5JER5U 9uPHb3wggmsY1zuVEzGFZbShZNhbM6Dho4lHmTUzgVsGCNO3S4G3SMr9K1VoFqcX/zwY cQtcG+zlKL589SViAX9i74TKOJzx7WXEzqRbBVvNspC9YMbcUYcvm/QJxifZEehciTmm +Z3g== X-Gm-Message-State: AOAM532mz0X7KKbVYvNpgmJDwwBnVvpExcjhFHCI0jZYcoitbEEYfDa6 kzKLvMXbqirl807kudG8xZJ6/sQIM1TTchuJCW02ImtvP5qdFvBmjgBuRvLgELdskKAa/W7KYVQ LdHJztWpJ8pf0P+BJXLQypXbEoc4flKB7sFSTsx+uQpvkH7cQRT8vGtyzpSrY210I X-Received: by 2002:ad4:5dc9:0:b0:441:56ad:8d93 with SMTP id m9-20020ad45dc9000000b0044156ad8d93mr4392119qvh.76.1648739352454; Thu, 31 Mar 2022 08:09:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy4ovHAdBIFnfqHPXTJXqRjgZisFw4gfPbdMN/rUL+RJIGcnqZyg8O/fECZxaAEEsiECZBpHA== X-Received: by 2002:ad4:5dc9:0:b0:441:56ad:8d93 with SMTP id m9-20020ad45dc9000000b0044156ad8d93mr4392071qvh.76.1648739351966; Thu, 31 Mar 2022 08:09:11 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.11 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:11 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 12/19] migration: Create the postcopy preempt channel asynchronously Date: Thu, 31 Mar 2022 11:08:50 -0400 Message-Id: <20220331150857.74406-13-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch allows the postcopy preempt channel to be created asynchronously. The benefit is that when the connection is slow, we won't take the BQL (and potentially block all things like QMP) for a long time without releasing. A function postcopy_preempt_wait_channel() is introduced, allowing the migration thread to be able to wait on the channel creation. The channel is always created by the main thread, in which we'll kick a new semaphore to tell the migration thread that the channel has created. We'll need to wait for the new channel in two places: (1) when there's a new postcopy migration that is starting, or (2) when there's a postcopy migration to resume. For the start of migration, we don't need to wait for this channel until when we want to start postcopy, aka, postcopy_start(). We'll fail the migration if we found that the channel creation failed (which should probably not happen at all in 99% of the cases, because the main channel is using the same network topology). For a postcopy recovery, we'll need to wait in postcopy_pause(). In that case if the channel creation failed, we can't fail the migration or we'll crash the VM, instead we keep in PAUSED state, waiting for yet another recovery. Signed-off-by: Peter Xu --- migration/migration.c | 16 ++++++++++++ migration/migration.h | 7 +++++ migration/postcopy-ram.c | 56 +++++++++++++++++++++++++++++++--------- migration/postcopy-ram.h | 1 + 4 files changed, 68 insertions(+), 12 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 157a34c844..33faa0ff6e 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -3021,6 +3021,12 @@ static int postcopy_start(MigrationState *ms) int64_t bandwidth = migrate_max_postcopy_bandwidth(); bool restart_block = false; int cur_state = MIGRATION_STATUS_ACTIVE; + + if (postcopy_preempt_wait_channel(ms)) { + migrate_set_state(&ms->state, ms->state, MIGRATION_STATUS_FAILED); + return -1; + } + if (!migrate_pause_before_switchover()) { migrate_set_state(&ms->state, MIGRATION_STATUS_ACTIVE, MIGRATION_STATUS_POSTCOPY_ACTIVE); @@ -3502,6 +3508,14 @@ static MigThrError postcopy_pause(MigrationState *s) if (s->state == MIGRATION_STATUS_POSTCOPY_RECOVER) { /* Woken up by a recover procedure. Give it a shot */ + if (postcopy_preempt_wait_channel(s)) { + /* + * Preempt enabled, and new channel create failed; loop + * back to wait for another recovery. + */ + continue; + } + /* * Firstly, let's wake up the return path now, with a new * return path channel. @@ -4361,6 +4375,7 @@ static void migration_instance_finalize(Object *obj) qemu_sem_destroy(&ms->postcopy_pause_sem); qemu_sem_destroy(&ms->postcopy_pause_rp_sem); qemu_sem_destroy(&ms->rp_state.rp_sem); + qemu_sem_destroy(&ms->postcopy_qemufile_src_sem); error_free(ms->error); } @@ -4407,6 +4422,7 @@ static void migration_instance_init(Object *obj) qemu_sem_init(&ms->rp_state.rp_sem, 0); qemu_sem_init(&ms->rate_limit_sem, 0); qemu_sem_init(&ms->wait_unplug_sem, 0); + qemu_sem_init(&ms->postcopy_qemufile_src_sem, 0); qemu_mutex_init(&ms->qemu_file_lock); } diff --git a/migration/migration.h b/migration/migration.h index 91f845e9e4..f898b8547a 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -219,6 +219,13 @@ struct MigrationState { QEMUFile *to_dst_file; /* Postcopy specific transfer channel */ QEMUFile *postcopy_qemufile_src; + /* + * It is posted when the preempt channel is established. Note: this is + * used for both the start or recover of a postcopy migration. We'll + * post to this sem every time a new preempt channel is created in the + * main thread, and we keep post() and wait() in pair. + */ + QemuSemaphore postcopy_qemufile_src_sem; QIOChannelBuffer *bioc; /* * Protects to_dst_file/from_dst_file pointers. We need to make sure we diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index e20305a9e2..ab2a50cf45 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -1552,10 +1552,50 @@ bool postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file) return true; } -int postcopy_preempt_setup(MigrationState *s, Error **errp) +static void +postcopy_preempt_send_channel_new(QIOTask *task, gpointer opaque) { - QIOChannel *ioc; + MigrationState *s = opaque; + QIOChannel *ioc = QIO_CHANNEL(qio_task_get_source(task)); + Error *local_err = NULL; + + if (qio_task_propagate_error(task, &local_err)) { + /* Something wrong happened.. */ + migrate_set_error(s, local_err); + error_free(local_err); + } else { + migration_ioc_register_yank(ioc); + s->postcopy_qemufile_src = qemu_fopen_channel_output(ioc); + trace_postcopy_preempt_new_channel(); + } + + /* + * Kick the waiter in all cases. The waiter should check upon + * postcopy_qemufile_src to know whether it failed or not. + */ + qemu_sem_post(&s->postcopy_qemufile_src_sem); + object_unref(OBJECT(ioc)); +} +/* Returns 0 if channel established, -1 for error. */ +int postcopy_preempt_wait_channel(MigrationState *s) +{ + /* If preempt not enabled, no need to wait */ + if (!migrate_postcopy_preempt()) { + return 0; + } + + /* + * We need the postcopy preempt channel to be established before + * starting doing anything. + */ + qemu_sem_wait(&s->postcopy_qemufile_src_sem); + + return s->postcopy_qemufile_src ? 0 : -1; +} + +int postcopy_preempt_setup(MigrationState *s, Error **errp) +{ if (!migrate_postcopy_preempt()) { return 0; } @@ -1566,16 +1606,8 @@ int postcopy_preempt_setup(MigrationState *s, Error **errp) return -1; } - ioc = socket_send_channel_create_sync(errp); - - if (ioc == NULL) { - return -1; - } - - migration_ioc_register_yank(ioc); - s->postcopy_qemufile_src = qemu_fopen_channel_output(ioc); - - trace_postcopy_preempt_new_channel(); + /* Kick an async task to connect */ + socket_send_channel_create(postcopy_preempt_send_channel_new, s); return 0; } diff --git a/migration/postcopy-ram.h b/migration/postcopy-ram.h index 34b1080cde..6147bf7d1d 100644 --- a/migration/postcopy-ram.h +++ b/migration/postcopy-ram.h @@ -192,5 +192,6 @@ enum PostcopyChannels { bool postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file); int postcopy_preempt_setup(MigrationState *s, Error **errp); +int postcopy_preempt_wait_channel(MigrationState *s); #endif From patchwork Thu Mar 31 15:08:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611807 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EXYVLetT; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn3T49kGz9sBy for ; Fri, 1 Apr 2022 02:17:25 +1100 (AEDT) Received: from localhost ([::1]:40046 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwY7-0007Ku-L5 for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:17:23 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52694) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQV-0002eD-Jp for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:31 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:29176) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQU-0001Pg-21 for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739369; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AjWWDJmru+q4d2CmT6YENR08AaXC5qZccuUY2hfeQGw=; b=EXYVLetTyViPY8KlSOuH7bUYLIAVd0sNZ9IBWOXn8f5dc1ckrH4NHp2XRDdGy9T+Ii2tEP SQ80nIE0jb6bvwVfnGy92q2E4wXz9ae6lkIxOsaQU0mea49+Z2LN4cpLC11fOGnPrOhS6s JFCzM8fPox0klVbmWulBKKao3dGYPRE= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-199-1PtLvYqkMkGuA0DnIYrwUg-1; Thu, 31 Mar 2022 11:09:28 -0400 X-MC-Unique: 1PtLvYqkMkGuA0DnIYrwUg-1 Received: by mail-qv1-f69.google.com with SMTP id 94-20020a0c8167000000b004411702e935so18769226qvc.7 for ; Thu, 31 Mar 2022 08:09:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AjWWDJmru+q4d2CmT6YENR08AaXC5qZccuUY2hfeQGw=; b=AIvw3DcvaYDkc4TXIBRxbbk8nBLlsEPFMRouwchXDbZY0jcr4MG0c6CDGjkSkyq21b nB8PZEyIktbeoxa5yQex0LyoXjrNucE3pAbuX72n3UR2bcLs1tf2f3lqtZPgnBRnw+N7 mEOU+JNWCzdFZBdtR2uHdiMVPxGVvUlZZJ5UMnXXlrHeAH0vYNUoJoUjuY0yDuoQ2XiK 2gWBfhJGtjw8r8OE+NI9HPusOLWypODPKvkTJCy5jnOFlAeH5MlqVy+gB3ZA5fyIWXLs uWYWC3UQvnUj/E+B+pbZZphkU+fZAQqA0+hdhHVFL+8HvxyqcH3brJ67on4GZKe3MAZY u3Jg== X-Gm-Message-State: AOAM533LbpnnoChqcbgSll7Q5Imyn82WnW+da0nI0IOZ8kWZFmxugqWD oyhKizVm9WCFWIYyEK/+p3QIYZ/m/yRWkjE//CI0bfamKHvLr+DGOmm61iPIcWtxJDnso829flD OSRjcAxU8n7KddY+7M8VyppMQz19xq+txMmb9xQfChTNF0nEnSCehrbkoU+Ks5xBe X-Received: by 2002:ad4:5941:0:b0:433:75f:8627 with SMTP id eo1-20020ad45941000000b00433075f8627mr36879189qvb.122.1648739353207; Thu, 31 Mar 2022 08:09:13 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyJmbapVSQvnTEp3vY1tF039e9RuKvChJ+4I8V9YUB8pU/d12UAnKIByFXXYiRuxzVKiHGYlQ== X-Received: by 2002:ad4:5941:0:b0:433:75f:8627 with SMTP id eo1-20020ad45941000000b00433075f8627mr36879154qvb.122.1648739352868; Thu, 31 Mar 2022 08:09:12 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:12 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 13/19] migration: Parameter x-postcopy-preempt-break-huge Date: Thu, 31 Mar 2022 11:08:51 -0400 Message-Id: <20220331150857.74406-14-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add a parameter that can conditionally disable the "break sending huge page" behavior in postcopy preemption. By default it's enabled. It should only be used for debugging purposes, and we should never remove the "x-" prefix. Signed-off-by: Peter Xu --- migration/migration.c | 2 ++ migration/migration.h | 7 +++++++ migration/ram.c | 7 +++++++ 3 files changed, 16 insertions(+) diff --git a/migration/migration.c b/migration/migration.c index 33faa0ff6e..ee3df9e229 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -4330,6 +4330,8 @@ static Property migration_properties[] = { DEFINE_PROP_SIZE("announce-step", MigrationState, parameters.announce_step, DEFAULT_MIGRATE_ANNOUNCE_STEP), + DEFINE_PROP_BOOL("x-postcopy-preempt-break-huge", MigrationState, + postcopy_preempt_break_huge, true), /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), diff --git a/migration/migration.h b/migration/migration.h index f898b8547a..6ee520642f 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -340,6 +340,13 @@ struct MigrationState { bool send_configuration; /* Whether we send section footer during migration */ bool send_section_footer; + /* + * Whether we allow break sending huge pages when postcopy preempt is + * enabled. When disabled, we won't interrupt precopy within sending a + * host huge page, which is the old behavior of vanilla postcopy. + * NOTE: this parameter is ignored if postcopy preempt is not enabled. + */ + bool postcopy_preempt_break_huge; /* Needed by postcopy-pause state */ QemuSemaphore postcopy_pause_sem; diff --git a/migration/ram.c b/migration/ram.c index 518d511874..3400cde6e9 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -2266,11 +2266,18 @@ static int ram_save_target_page(RAMState *rs, PageSearchStatus *pss) static bool postcopy_needs_preempt(RAMState *rs, PageSearchStatus *pss) { + MigrationState *ms = migrate_get_current(); + /* Not enabled eager preempt? Then never do that. */ if (!migrate_postcopy_preempt()) { return false; } + /* If the user explicitly disabled breaking of huge page, skip */ + if (!ms->postcopy_preempt_break_huge) { + return false; + } + /* If the ramblock we're sending is a small page? Never bother. */ if (qemu_ram_pagesize(pss->block) == TARGET_PAGE_SIZE) { return false; From patchwork Thu Mar 31 15:08:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611816 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=G5nf78ZU; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTnCR0y30z9sGP for ; Fri, 1 Apr 2022 02:24:19 +1100 (AEDT) Received: from localhost ([::1]:54354 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwen-0000Fl-5q for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:24:17 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52594) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQL-0002I1-8S for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:42481) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQH-0001NO-Gd for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739356; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1qMQ8LRtoG4NBh/wSkWUaxdpYn6HvLjx1HvsHm6HlNw=; b=G5nf78ZU67qblUNSKJyWlZgqGPyVvgnH47yqneAp0I2gw+XR+LThLUvdYKIEqxQ0DXBiTA nvPM6woBCTikUDqIubFvqSzBFuEJtb8cSSDqD+Wj3ipUzXfqnzkG4vWAfbSaovEuRmvfag L4ckRgCcsrnIaB88ZpD7YeYbCu4iJ8I= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-18-9oqoht7tOC6O_GeSpNKeLw-1; Thu, 31 Mar 2022 11:09:15 -0400 X-MC-Unique: 9oqoht7tOC6O_GeSpNKeLw-1 Received: by mail-qt1-f197.google.com with SMTP id h11-20020a05622a170b00b002e0769b9018so20288899qtk.14 for ; Thu, 31 Mar 2022 08:09:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1qMQ8LRtoG4NBh/wSkWUaxdpYn6HvLjx1HvsHm6HlNw=; b=Gvqg/I4o6QwG+iALGKnwCHGz234ig5l/vFFscKEMLIb5s2yudr5QwXc1Qz7xuYdu3U im3769DrZRo/6B8bjQd/5iD6FjGcpJa1FGc5S7ZxMlByEw5fhRauuTbyjELIo+y4zW6z AmCyO8kUlO4kVMBl2IFDrQ+B5+osyNeH/VE9VKpibZlxRffx/NLQwck4BMv4dbk2JnEo zSgIxZixUp7lfVCFDVbXBG5p0OJbWJIM088YYu3nO4C5MXsd+MMrPNU96F8OZOBBQco4 nlcNuCXPdsBcZtyLTRX4CTDbfQTv1eC2iCe4zjLo4zzWb82xTsTb+nV1UwU/7uWVPrmX pTbA== X-Gm-Message-State: AOAM531chPw8OwMi3j+b4jM4UNLRBaVjtyiLhBmZRylayizYfQD3MUHN FTifhWaCTGUYIoNNaj1OCXaL+jpB6ZJUGKAErJmhzIwINVWRdNnPtHAlplCWesypTOAiP+SZCU4 fKA2W45gjfB0JRAXQcJVEXWY7Zd3r2s/GxLNW6OnwMwOLXiqIgTa4+R9Z4oVAvKnl X-Received: by 2002:a05:6214:21a5:b0:440:f3b8:d0aa with SMTP id t5-20020a05621421a500b00440f3b8d0aamr4093951qvc.61.1648739354562; Thu, 31 Mar 2022 08:09:14 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzYw6u36LH1OfTB/EeT7CvW70MsOxI+4vTlw6W5AOo3+h8Cl7SSDok6hZfoOHKIuv0QFiKJyA== X-Received: by 2002:a05:6214:21a5:b0:440:f3b8:d0aa with SMTP id t5-20020a05621421a500b00440f3b8d0aamr4093902qvc.61.1648739354074; Thu, 31 Mar 2022 08:09:14 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.13 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:13 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 14/19] migration: Add helpers to detect TLS capability Date: Thu, 31 Mar 2022 11:08:52 -0400 Message-Id: <20220331150857.74406-15-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Add migrate_tls_enabled() to detect whether TLS is configured. Add migrate_channel_requires_tls() to detect whether the specific channel requires TLS. No functional change intended. Signed-off-by: Peter Xu --- migration/channel.c | 10 ++-------- migration/migration.c | 17 +++++++++++++++++ migration/migration.h | 4 ++++ migration/multifd.c | 7 +------ 4 files changed, 24 insertions(+), 14 deletions(-) diff --git a/migration/channel.c b/migration/channel.c index c6a8dcf1d7..36e59eaeec 100644 --- a/migration/channel.c +++ b/migration/channel.c @@ -38,10 +38,7 @@ void migration_channel_process_incoming(QIOChannel *ioc) trace_migration_set_incoming_channel( ioc, object_get_typename(OBJECT(ioc))); - if (s->parameters.tls_creds && - *s->parameters.tls_creds && - !object_dynamic_cast(OBJECT(ioc), - TYPE_QIO_CHANNEL_TLS)) { + if (migrate_channel_requires_tls(ioc)) { migration_tls_channel_process_incoming(s, ioc, &local_err); } else { migration_ioc_register_yank(ioc); @@ -71,10 +68,7 @@ void migration_channel_connect(MigrationState *s, ioc, object_get_typename(OBJECT(ioc)), hostname, error); if (!error) { - if (s->parameters.tls_creds && - *s->parameters.tls_creds && - !object_dynamic_cast(OBJECT(ioc), - TYPE_QIO_CHANNEL_TLS)) { + if (migrate_channel_requires_tls(ioc)) { migration_tls_channel_connect(s, ioc, hostname, &error); if (!error) { diff --git a/migration/migration.c b/migration/migration.c index ee3df9e229..899084f993 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -49,6 +49,7 @@ #include "trace.h" #include "exec/target_page.h" #include "io/channel-buffer.h" +#include "io/channel-tls.h" #include "migration/colo.h" #include "hw/boards.h" #include "hw/qdev-properties.h" @@ -4251,6 +4252,22 @@ void migration_global_dump(Monitor *mon) ms->clear_bitmap_shift); } +bool migrate_tls_enabled(void) +{ + MigrationState *s = migrate_get_current(); + + return s->parameters.tls_creds && *s->parameters.tls_creds; +} + +bool migrate_channel_requires_tls(QIOChannel *ioc) +{ + if (!migrate_tls_enabled()) { + return false; + } + + return !object_dynamic_cast(OBJECT(ioc), TYPE_QIO_CHANNEL_TLS); +} + #define DEFINE_PROP_MIG_CAP(name, x) \ DEFINE_PROP_BOOL(name, MigrationState, enabled_capabilities[x], false) diff --git a/migration/migration.h b/migration/migration.h index 6ee520642f..8b9ad7fe31 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -436,6 +436,10 @@ bool migrate_use_events(void); bool migrate_postcopy_blocktime(void); bool migrate_background_snapshot(void); bool migrate_postcopy_preempt(void); +/* Whether TLS is enabled for migration? */ +bool migrate_tls_enabled(void); +/* Whether the QIO channel requires further TLS handshake? */ +bool migrate_channel_requires_tls(QIOChannel *ioc); /* Sending on the return path - generic and then for each message type */ void migrate_send_rp_shut(MigrationIncomingState *mis, diff --git a/migration/multifd.c b/migration/multifd.c index 9ea4f581e2..19e3c44491 100644 --- a/migration/multifd.c +++ b/migration/multifd.c @@ -782,17 +782,12 @@ static bool multifd_channel_connect(MultiFDSendParams *p, QIOChannel *ioc, Error *error) { - MigrationState *s = migrate_get_current(); - trace_multifd_set_outgoing_channel( ioc, object_get_typename(OBJECT(ioc)), migrate_get_current()->hostname, error); if (!error) { - if (s->parameters.tls_creds && - *s->parameters.tls_creds && - !object_dynamic_cast(OBJECT(ioc), - TYPE_QIO_CHANNEL_TLS)) { + if (migrate_channel_requires_tls(ioc)) { multifd_tls_channel_connect(p, ioc, &error); if (!error) { /* From patchwork Thu Mar 31 15:08:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611812 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WemTrl1W; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn790c32z9sGP for ; Fri, 1 Apr 2022 02:20:37 +1100 (AEDT) Received: from localhost ([::1]:46306 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwbD-0003Ag-56 for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:20:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52598) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQL-0002Id-C0 for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:21149) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQI-0001Na-5H for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739357; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cWpfOPDWJiir2kbNPeHaebJZkVlgkAHklNyvbakimFg=; b=WemTrl1WU50a/WfANkmz9jthK7HyoBmL2V5z7NwVHTNQTWuF4Wy0Va0CqvlMdroNDbJSER 2Q6i0GEmJcn/QgBZuSBigxlI9JkmO0UtnnlxDS9X3NctR/+hQ4xWRGQGLAs6tuv5TxJNeZ ODVy8fQUKMDDNi86/iz1IxfnTRw72XQ= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-313-ojeGOXlEMWihP8oNDXNBLQ-1; Thu, 31 Mar 2022 11:09:16 -0400 X-MC-Unique: ojeGOXlEMWihP8oNDXNBLQ-1 Received: by mail-qk1-f199.google.com with SMTP id a66-20020ae9e845000000b0067b308a9f56so14989245qkg.21 for ; Thu, 31 Mar 2022 08:09:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cWpfOPDWJiir2kbNPeHaebJZkVlgkAHklNyvbakimFg=; b=T7FM9Kwja2Jp5SjFxBceop3tbA7gt4JUbgKiK9dmRhL66AA6BHnAz+HJRcP4Gbfcso +7gCW1GHnoxZkLLQZkgDxwmOKVYG51lswYaVuoNbnknE3jRnzXIphYvKkm96Nq+C+DzI LL5JgOy2xobVxUyKOFBnrHnLXSGMciGywk9fbj4gpc0gkDkOF/kP1t/rPLed7Oue+mnT cysKQXJwAfM6jasg9FpGeAq2lQ2/JlgJnpJvcZlEgW2lDGAiyl5v2/qzGC0CUDOzc/Qz Zl/6IivUORlm6pV9tAzdVnbEzGx0X6+nVRmpnXOcomwPnHX+VVYSr7VZZQn2ilTZUIWy VBmA== X-Gm-Message-State: AOAM531NVMWMW/a4j5LI1J+wOmkUrYtMcfiUMgWuhdP3iD4IR65RsatJ aXtKBf7CxicDExe+DQwrV1OK6LSsqEB4xFheGCtCmS8OIfKLvnA6iUJ0tpZjQBM2ARMkgNKBcCj 8CToaJGwJbaQJ6wkOTmPPw1kBxnr2tBQLD9Vx+oUD3Yo8iyqYYkO58htUxQJIXpoB X-Received: by 2002:a0c:fc46:0:b0:440:f78f:f4c4 with SMTP id w6-20020a0cfc46000000b00440f78ff4c4mr35470677qvp.108.1648739355554; Thu, 31 Mar 2022 08:09:15 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxQKeWWRrIroLoP/PNAKI1D+oZqMQQ7jOJjMWX1aTRVNaT72M0Z9XpuHgVAXd5iMSIp264VhA== X-Received: by 2002:a0c:fc46:0:b0:440:f78f:f4c4 with SMTP id w6-20020a0cfc46000000b00440f78ff4c4mr35470643qvp.108.1648739355195; Thu, 31 Mar 2022 08:09:15 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.14 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:14 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 15/19] migration: Export tls-[creds|hostname|authz] params to cmdline too Date: Thu, 31 Mar 2022 11:08:53 -0400 Message-Id: <20220331150857.74406-16-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" It's useful for specifying tls credentials all in the cmdline (along with the -object tls-creds-*), especially for debugging purpose. The trick here is we must remember to not free these fields again in the finalize() function of migration object, otherwise it'll cause double-free. The thing is when destroying an object, we'll first destroy the properties that bound to the object, then the object itself. To be explicit, when destroy the object in object_finalize() we have such sequence of operations: object_property_del_all(obj); object_deinit(obj, ti); So after this change the two fields are properly released already even before reaching the finalize() function but in object_property_del_all(), hence we don't need to free them anymore in finalize() or it's double-free. Signed-off-by: Peter Xu --- migration/migration.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/migration/migration.c b/migration/migration.c index 899084f993..1dc80be1f4 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -4349,6 +4349,9 @@ static Property migration_properties[] = { DEFAULT_MIGRATE_ANNOUNCE_STEP), DEFINE_PROP_BOOL("x-postcopy-preempt-break-huge", MigrationState, postcopy_preempt_break_huge, true), + DEFINE_PROP_STRING("tls-creds", MigrationState, parameters.tls_creds), + DEFINE_PROP_STRING("tls-hostname", MigrationState, parameters.tls_hostname), + DEFINE_PROP_STRING("tls-authz", MigrationState, parameters.tls_authz), /* Migration capabilities */ DEFINE_PROP_MIG_CAP("x-xbzrle", MIGRATION_CAPABILITY_XBZRLE), @@ -4382,12 +4385,9 @@ static void migration_class_init(ObjectClass *klass, void *data) static void migration_instance_finalize(Object *obj) { MigrationState *ms = MIGRATION_OBJ(obj); - MigrationParameters *params = &ms->parameters; qemu_mutex_destroy(&ms->error_mutex); qemu_mutex_destroy(&ms->qemu_file_lock); - g_free(params->tls_hostname); - g_free(params->tls_creds); qemu_sem_destroy(&ms->wait_unplug_sem); qemu_sem_destroy(&ms->rate_limit_sem); qemu_sem_destroy(&ms->pause_sem); From patchwork Thu Mar 31 15:08:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611820 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=b46L06zr; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTnFl3V42z9sGP for ; Fri, 1 Apr 2022 02:26:18 +1100 (AEDT) Received: from localhost ([::1]:59618 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwgi-0003vA-JV for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:26:16 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53002) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwRQ-00052j-3p for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:28 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:29610) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwRO-0001in-Ce for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739425; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hE6sbGR3e5wSAPTOpUg7+HEoCjCHYy/m1agjPPHUXOU=; b=b46L06zrqT+Tc0xwsMTA/r/BoSTM9F3z6wntv135S5u6QWIX1W5qkyOw3s9v0s9e/8Kxo2 wgGlIFZ7jamTMJZ8HDNP/amWKZ24tAydwINoiivNMuIORplmvEGWqZGbPaqJJCnL6Xi+tm xs3u4vq9/QPzexd59bquLPUs/RHF7rY= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-583-0rx_dJRjMDemVgmCGVamCQ-1; Thu, 31 Mar 2022 11:09:17 -0400 X-MC-Unique: 0rx_dJRjMDemVgmCGVamCQ-1 Received: by mail-qk1-f197.google.com with SMTP id d12-20020a379b0c000000b0067d8cda1aaaso14998169qke.8 for ; Thu, 31 Mar 2022 08:09:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hE6sbGR3e5wSAPTOpUg7+HEoCjCHYy/m1agjPPHUXOU=; b=QCUQZyZp4M+yQToXhrCGShAooyDAMK52NP4pqvg33lBAZNqkj9tgSZH4XuRijbcf2Z lp6ctNf8vOmB+HY/b8tWCEhUMovCAeEFgL3OhwLOlKsSSMgbOax5IGLWq1ZAPCrO9NHj WRyYJqqkGuJyO7KwvSvatUSVF5aBtkl0TVCGBXtKd6X9Np3bFAwG0ipf2sWdHWgHZfAx epHX8p7MggY4KbPnpCuZJC2RPeJ2spKg2jU+bfodwzU9d1hz97fN+H5iNOTBFXxtr/Gy u8OI7RtDo8ODgzc8hjx6D01+R7JSVyHN+YZCQDYnQwhde9emKthoBPdnZJb32D3sPxCO U5Ww== X-Gm-Message-State: AOAM531e7xPQsYy3JkR+LmW2atKRht6UvmqDFwtAFUL2BsiWpAkzZLTI O6mvHgjnipg+HDj0ULgyMd6wkqk+b0WyMaMT214Sr6qePZetd+1lYnV/JlNa2YsXyDfDWuxNCSo 1Vi70mD8KpXI3aUvyvQZuAST9Pk/6xvSMA5cg0JtlAUP7WbnDkLoWLTOc0BFu1sHx X-Received: by 2002:a0c:d688:0:b0:42c:3cb0:3923 with SMTP id k8-20020a0cd688000000b0042c3cb03923mr4159897qvi.69.1648739356858; Thu, 31 Mar 2022 08:09:16 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwp43QTEAS4xv7ImcffRR+dyXGNGDeWQqvVZTdy/jDghcigRZnlZoF9XXf7VKp22L1AdnjAdQ== X-Received: by 2002:a0c:d688:0:b0:42c:3cb0:3923 with SMTP id k8-20020a0cd688000000b0042c3cb03923mr4159859qvi.69.1648739356531; Thu, 31 Mar 2022 08:09:16 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.15 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:16 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 16/19] migration: Enable TLS for preempt channel Date: Thu, 31 Mar 2022 11:08:54 -0400 Message-Id: <20220331150857.74406-17-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch is based on the async preempt channel creation. It continues wiring up the new channel with TLS handshake to destionation when enabled. Note that only the src QEMU needs such operation; the dest QEMU does not need any change for TLS support due to the fact that all channels are established synchronously there, so all the TLS magic is already properly handled by migration_tls_channel_process_incoming(). Signed-off-by: Peter Xu --- migration/postcopy-ram.c | 60 +++++++++++++++++++++++++++++++++++----- migration/trace-events | 1 + 2 files changed, 54 insertions(+), 7 deletions(-) diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c index ab2a50cf45..f5ba176862 100644 --- a/migration/postcopy-ram.c +++ b/migration/postcopy-ram.c @@ -36,6 +36,7 @@ #include "socket.h" #include "qemu-file-channel.h" #include "yank_functions.h" +#include "tls.h" /* Arbitrary limit on size of each discard command, * keeps them around ~200 bytes @@ -1552,15 +1553,15 @@ bool postcopy_preempt_new_channel(MigrationIncomingState *mis, QEMUFile *file) return true; } +/* + * Setup the postcopy preempt channel with the IOC. If ERROR is specified, + * setup the error instead. This helper will free the ERROR if specified. + */ static void -postcopy_preempt_send_channel_new(QIOTask *task, gpointer opaque) +postcopy_preempt_send_channel_done(MigrationState *s, + QIOChannel *ioc, Error *local_err) { - MigrationState *s = opaque; - QIOChannel *ioc = QIO_CHANNEL(qio_task_get_source(task)); - Error *local_err = NULL; - - if (qio_task_propagate_error(task, &local_err)) { - /* Something wrong happened.. */ + if (local_err) { migrate_set_error(s, local_err); error_free(local_err); } else { @@ -1574,6 +1575,51 @@ postcopy_preempt_send_channel_new(QIOTask *task, gpointer opaque) * postcopy_qemufile_src to know whether it failed or not. */ qemu_sem_post(&s->postcopy_qemufile_src_sem); +} + +static void +postcopy_preempt_tls_handshake(QIOTask *task, gpointer opaque) +{ + MigrationState *s = opaque; + QIOChannel *ioc = QIO_CHANNEL(qio_task_get_source(task)); + Error *err = NULL; + + qio_task_propagate_error(task, &err); + postcopy_preempt_send_channel_done(s, ioc, err); + object_unref(OBJECT(ioc)); +} + +static void +postcopy_preempt_send_channel_new(QIOTask *task, gpointer opaque) +{ + MigrationState *s = opaque; + QIOChannel *ioc = QIO_CHANNEL(qio_task_get_source(task)); + QIOChannelTLS *tioc; + Error *local_err = NULL; + + if (qio_task_propagate_error(task, &local_err)) { + assert(local_err); + goto out; + } + + if (migrate_channel_requires_tls(ioc)) { + tioc = migration_tls_client_create(s, ioc, s->hostname, &local_err); + if (!tioc) { + assert(local_err); + goto out; + } + trace_postcopy_preempt_tls_handshake(); + qio_channel_set_name(QIO_CHANNEL(tioc), "migration-tls-preempt"); + qio_channel_tls_handshake(tioc, postcopy_preempt_tls_handshake, + s, NULL, NULL); + /* Setup the channel until TLS handshake finished */ + object_unref(OBJECT(ioc)); + return; + } + +out: + /* This handles both good and error cases */ + postcopy_preempt_send_channel_done(s, ioc, local_err); object_unref(OBJECT(ioc)); } diff --git a/migration/trace-events b/migration/trace-events index b21d5f371f..00ab2e1b96 100644 --- a/migration/trace-events +++ b/migration/trace-events @@ -287,6 +287,7 @@ postcopy_request_shared_page(const char *sharer, const char *rb, uint64_t rb_off postcopy_request_shared_page_present(const char *sharer, const char *rb, uint64_t rb_offset) "%s already %s offset 0x%"PRIx64 postcopy_wake_shared(uint64_t client_addr, const char *rb) "at 0x%"PRIx64" in %s" postcopy_page_req_del(void *addr, int count) "resolved page req %p total %d" +postcopy_preempt_tls_handshake(void) "" postcopy_preempt_new_channel(void) "" postcopy_preempt_thread_entry(void) "" postcopy_preempt_thread_exit(void) "" From patchwork Thu Mar 31 15:08:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611821 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=cE8cdQCg; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTnG61T9Tz9sGP for ; Fri, 1 Apr 2022 02:26:38 +1100 (AEDT) Received: from localhost ([::1]:33208 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwh2-00056x-8y for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:26:36 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52628) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQN-0002MN-IX for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:55643) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQK-0001OC-Li for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739359; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=adScMC+H9c/2TDao3xFbYMKyXHCGuT9u1P39HtUdouo=; b=cE8cdQCg+Ckx5bdAVHXUJnXXjimNGjY2xW3Ct5U7tCurLn7Urw5igtxKdl+r1NOD7ORM3z Wxu2bKZFN03/m3Hq1SbEaf90PXkxoqAZr8iv9SI0NNFxbSr62cmi94x5QFjqceFc8fWrHS BhfGtXaotnqYUTiKbmeoZKi3J16+Scg= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-449-6jTdchTfO2Ob3Dkm6baDJQ-1; Thu, 31 Mar 2022 11:09:18 -0400 X-MC-Unique: 6jTdchTfO2Ob3Dkm6baDJQ-1 Received: by mail-qk1-f200.google.com with SMTP id h68-20020a376c47000000b0067e05dade89so15041940qkc.2 for ; Thu, 31 Mar 2022 08:09:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=adScMC+H9c/2TDao3xFbYMKyXHCGuT9u1P39HtUdouo=; b=th3pQs57/0q0LQrokbydAsHTtRg110MK/pYUbFOhJSkdCmfXz17YpTZ9hES8Elkboy zXdDXJnZY/DoxkD4hHqnXESnlYv6PBPMs+OgkMMbJmqAdMmvtGMJvCyOmvTNAdbfb1EO qtXElrsT2e9EEEcsQXOcJONGIkeyA8iwJ7cR4IALlAphIQFDcgx7O7KUMJJa1m1y6fvQ s2FMI5YWAVhltIzM/0Ewl1dRdZ1rqRqx0TQcrv9/L3HK+4r1I0hTaqL1aJvlGsid5OXr qPxBw8u9186glIQEzN0WRNkPgAb7+hgWGAQaeun6mQB2S8Ls7a44lOES2zMcnk0rlGiK b4XQ== X-Gm-Message-State: AOAM533RIOOBxIoxUakg+iiGzHN2Krz7jb0l5XvJZzNYj93JRIa5OWQI nvknmoH2S8+omJPYblMj8wITbTVtBNDQGSHYKbLp6zdMxmkHbF4KEHCfXvZ4zet9ivPzeT80lPQ Mb1ky6hl1HKUwqpmuTuD7OMqTnaA0vp8OJ6zdveM7z8z5VjMY6p6bGTmAm+RWuO/G X-Received: by 2002:a05:620a:4554:b0:67d:6750:6a3 with SMTP id u20-20020a05620a455400b0067d675006a3mr3583335qkp.439.1648739358011; Thu, 31 Mar 2022 08:09:18 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw0RtQXAzwSu759paQcZqke1m4OQe/C54UCNcj6fqIoTjylzevUpI8dbjSIfhE5MmuvCLVJFw== X-Received: by 2002:a05:620a:4554:b0:67d:6750:6a3 with SMTP id u20-20020a05620a455400b0067d675006a3mr3583297qkp.439.1648739357610; Thu, 31 Mar 2022 08:09:17 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:17 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 17/19] tests: Add postcopy tls migration test Date: Thu, 31 Mar 2022 11:08:55 -0400 Message-Id: <20220331150857.74406-18-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" We just added TLS tests for precopy but not postcopy. Add the corresponding test for vanilla postcopy. Signed-off-by: Peter Xu --- tests/qtest/migration-test.c | 43 +++++++++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 6 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index d9f444ea14..80c4244871 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -481,6 +481,10 @@ typedef struct { bool only_target; /* Use dirty ring if true; dirty logging otherwise */ bool use_dirty_ring; + /* Whether use TLS channels for postcopy test? */ + bool postcopy_tls; + /* Used only if postcopy_tls==true, to cache the data object */ + void *postcopy_tls_data; const char *opts_source; const char *opts_target; } MigrateStart; @@ -980,6 +984,10 @@ static int migrate_postcopy_prepare(QTestState **from_ptr, return -1; } + if (args->postcopy_tls) { + args->postcopy_tls_data = test_migrate_tls_psk_start_match(from, to); + } + migrate_set_capability(from, "postcopy-ram", true); migrate_set_capability(to, "postcopy-ram", true); migrate_set_capability(to, "postcopy-blocktime", true); @@ -1004,7 +1012,8 @@ static int migrate_postcopy_prepare(QTestState **from_ptr, return 0; } -static void migrate_postcopy_complete(QTestState *from, QTestState *to) +static void migrate_postcopy_complete(QTestState *from, QTestState *to, + MigrateStart *args) { wait_for_migration_complete(from); @@ -1015,19 +1024,38 @@ static void migrate_postcopy_complete(QTestState *from, QTestState *to) read_blocktime(to); } + if (args->postcopy_tls) { + assert(args->postcopy_tls_data); + test_migrate_tls_psk_finish(from, to, args->postcopy_tls_data); + args->postcopy_tls_data = NULL; + } + test_migrate_end(from, to, true); } -static void test_postcopy(void) +static void test_postcopy_common(MigrateStart *args) { - MigrateStart args = {}; QTestState *from, *to; - if (migrate_postcopy_prepare(&from, &to, &args)) { + if (migrate_postcopy_prepare(&from, &to, args)) { return; } migrate_postcopy_start(from, to); - migrate_postcopy_complete(from, to); + migrate_postcopy_complete(from, to, args); +} + +static void test_postcopy(void) +{ + MigrateStart args = { }; + + test_postcopy_common(&args); +} + +static void test_postcopy_tls(void) +{ + MigrateStart args = { .postcopy_tls = true }; + + test_postcopy_common(&args); } static void test_postcopy_recovery(void) @@ -1089,7 +1117,7 @@ static void test_postcopy_recovery(void) /* Restore the postcopy bandwidth to unlimited */ migrate_set_parameter_int(from, "max-postcopy-bandwidth", 0); - migrate_postcopy_complete(from, to); + migrate_postcopy_complete(from, to, &args); } static void test_baddest(void) @@ -2134,6 +2162,9 @@ int main(int argc, char **argv) qtest_add_func("/migration/postcopy/unix", test_postcopy); qtest_add_func("/migration/postcopy/recovery", test_postcopy_recovery); +#ifdef CONFIG_GNUTLS + qtest_add_func("/migration/postcopy/tls", test_postcopy_tls); +#endif /* CONFIG_GNUTLS */ qtest_add_func("/migration/bad_dest", test_baddest); qtest_add_func("/migration/precopy/unix/plain", test_precopy_unix_plain); qtest_add_func("/migration/precopy/unix/xbzrle", test_precopy_unix_xbzrle); From patchwork Thu Mar 31 15:08:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611800 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dprOq74x; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTmzn2Ds8z9sGP for ; Fri, 1 Apr 2022 02:14:13 +1100 (AEDT) Received: from localhost ([::1]:59652 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwV1-0001Tv-Bk for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:14:11 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52662) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQP-0002RI-DE for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:45127) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwQL-0001Oc-K7 for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:09:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739361; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D7cg24gbIDVfatkh+h2kS8W8/k5egsPyKjSZg4lvrLM=; b=dprOq74xpGGXhFgPMIHHdMG7otrny2drNU9iq02RqHHxqLIDg14ADO3uoZfuE9i+13T+Br foXNQbZ4JxTwvNIMdXNGn9e7zmvItON8cfaqYqSLHNOqztB/jj3/l+eyScPZrLbhPk3ZMN 0jdx+8MDp8wMCK8Mcls8y8uwIIGLeDE= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-124-jN2INtwMMqG65qymQs7rXQ-1; Thu, 31 Mar 2022 11:09:19 -0400 X-MC-Unique: jN2INtwMMqG65qymQs7rXQ-1 Received: by mail-qk1-f200.google.com with SMTP id bj2-20020a05620a190200b005084968bb24so14967665qkb.23 for ; Thu, 31 Mar 2022 08:09:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=D7cg24gbIDVfatkh+h2kS8W8/k5egsPyKjSZg4lvrLM=; b=zTRJoSAUxjvEGbQLCQ9HEJ4a6OgyH1rinF76GG7p+zSQVllHRZfIdgw0UZuc5me5+q OvyHup4DL7XLl692KhGy3MGdGZaX2urJW5tuG1E63Il7GZALW8XjBMKX1rm+Fpaj5Umj GleaBOnIwF4tBbWxp92sgp0GuMNHxmzHmYFgDp5+B/j6eHud+DR3bNWA1oHeKAnDpgX4 HA45qAauAgr20vluV9yz0fnyomIIbBKHh1S4zJwijZN31Iqcq+pkGhGMpDn/V2Uuz4z3 se6m2DXUTcG2QnKlEPpK1D3eqyANMqS/i2C/7Ywgw9yjdWi+VCqe5r5SjXbN7iqAOODL hXGQ== X-Gm-Message-State: AOAM531NKDtXASAto+oxlANFUEiFaZaSrAzZ43aLs+DgNJT3MQHUNi9f nZiQiCrYtoITsESJ2vRCt4Dcn+I9Nl7gbdje991zlwobxbkF7ZMANa/HDJr1FHmDULzkGSfHFuQ HM+0Igezu4q72kOa5rFys/KFVmgtfaOdtmbxvwdT5eIDbFWMFy9zbu6HtCnwRilMi X-Received: by 2002:a05:620a:6c3:b0:67d:7694:875d with SMTP id 3-20020a05620a06c300b0067d7694875dmr3625120qky.394.1648739359001; Thu, 31 Mar 2022 08:09:19 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyNkOItIxxGQgPdznEUgpdjs27kVOtF527qkC2hwhpd5xP4AI2DIgKcDWX4bJYay3ioA09HdQ== X-Received: by 2002:a05:620a:6c3:b0:67d:7694:875d with SMTP id 3-20020a05620a06c300b0067d7694875dmr3625095qky.394.1648739358710; Thu, 31 Mar 2022 08:09:18 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:18 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 18/19] tests: Add postcopy tls recovery migration test Date: Thu, 31 Mar 2022 11:08:56 -0400 Message-Id: <20220331150857.74406-19-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" It's easy to build this upon the postcopy tls test. Signed-off-by: Peter Xu --- tests/qtest/migration-test.c | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 80c4244871..7288c64e97 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -1058,15 +1058,15 @@ static void test_postcopy_tls(void) test_postcopy_common(&args); } -static void test_postcopy_recovery(void) +static void test_postcopy_recovery_common(MigrateStart *args) { - MigrateStart args = { - .hide_stderr = true, - }; QTestState *from, *to; g_autofree char *uri = NULL; - if (migrate_postcopy_prepare(&from, &to, &args)) { + /* Always hide errors for postcopy recover tests since they're expected */ + args->hide_stderr = true; + + if (migrate_postcopy_prepare(&from, &to, args)) { return; } @@ -1117,7 +1117,21 @@ static void test_postcopy_recovery(void) /* Restore the postcopy bandwidth to unlimited */ migrate_set_parameter_int(from, "max-postcopy-bandwidth", 0); - migrate_postcopy_complete(from, to, &args); + migrate_postcopy_complete(from, to, args); +} + +static void test_postcopy_recovery(void) +{ + MigrateStart args = { }; + + test_postcopy_recovery_common(&args); +} + +static void test_postcopy_recovery_tls(void) +{ + MigrateStart args = { .postcopy_tls = true }; + + test_postcopy_recovery_common(&args); } static void test_baddest(void) @@ -2164,6 +2178,7 @@ int main(int argc, char **argv) qtest_add_func("/migration/postcopy/recovery", test_postcopy_recovery); #ifdef CONFIG_GNUTLS qtest_add_func("/migration/postcopy/tls", test_postcopy_tls); + qtest_add_func("/migration/postcopy/tls/recovery", test_postcopy_recovery_tls); #endif /* CONFIG_GNUTLS */ qtest_add_func("/migration/bad_dest", test_baddest); qtest_add_func("/migration/precopy/unix/plain", test_precopy_unix_plain); From patchwork Thu Mar 31 15:08:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 1611813 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=OP1K+N3x; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KTn7K75N1z9sGP for ; Fri, 1 Apr 2022 02:20:45 +1100 (AEDT) Received: from localhost ([::1]:46854 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nZwbM-0003XB-2O for incoming@patchwork.ozlabs.org; Thu, 31 Mar 2022 11:20:44 -0400 Received: from eggs.gnu.org ([209.51.188.92]:52988) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwRO-0004um-6a for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:26 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:21309) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nZwRM-0001hS-Gb for qemu-devel@nongnu.org; Thu, 31 Mar 2022 11:10:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648739423; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aebwCcvLCMGoglNFedm/LJowYh+slbC5rlPrkZcazGg=; b=OP1K+N3x8vkOlnQjWoDnGIMvdJFPbLU6UidXW/2g3hPolQIlveZkQM6007cOpTX7waDCQG WDJ97xhY/C6EXTeNLdzdDvzGKizGiN19RUtt9TPqRBGMLbTZ/Q4m8l9TZs8YpBMnKSFMhK GxmP2KOKdf0GDCict2K3GtwgzQk7zBM= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-665-oex6XXEDPLSls4kGKPsXfQ-1; Thu, 31 Mar 2022 11:09:21 -0400 X-MC-Unique: oex6XXEDPLSls4kGKPsXfQ-1 Received: by mail-qv1-f72.google.com with SMTP id cl19-20020a05621404f300b0044103143211so18775477qvb.10 for ; Thu, 31 Mar 2022 08:09:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aebwCcvLCMGoglNFedm/LJowYh+slbC5rlPrkZcazGg=; b=yjy/eGoXgkyvK47RO5FFsrVNuyyde0CuCPGWopFStiZKokzDa5oib4lagbLEMM0YW6 kJjI9bo9vKChQSi+8UBGerFAMk0YgzDL+gMPKAJoMdNZdProW6mRfJycj6Kw/paqdtMw Gehd2bvSSMGXglu2+kXf8lUV+h1/i0ZOtaf3tpoyQQTiaaKh9QQt8cFLlBKFpMW/EJri 3DMZUSLjwbI+UpKZ9m1lhYNdS6893UfMSZjBCTRAnFlU+4Cx3qPoTUg92HKW1Y9+b81V 6JifllcNgWGPUYHjVJu74pDbAETrkZTYxCvXXjfZHytd8iscpvOWcR71VBJNHP8jSDMI dpFA== X-Gm-Message-State: AOAM530bdeSi0YxgttB+V4pt2Q2uaAsvIMW5+Fdeu9OeFRBJcgmxmY47 W7onXeUOHSpwJ/sjcRDvG4Gg9y9IKcd8C+P8Azm9W155C3uusMU21y87iPWSheIE4qnpqQJtAye WgTITJYXbyBlZVLoGxgiTFN4hQv/uDY+aGtBgEqbuloQl5fNbnFXNVHRo/eDeJyi+ X-Received: by 2002:a05:620a:4706:b0:67d:a135:c912 with SMTP id bs6-20020a05620a470600b0067da135c912mr3743068qkb.344.1648739360230; Thu, 31 Mar 2022 08:09:20 -0700 (PDT) X-Google-Smtp-Source: ABdhPJx4pDE6tcjBazwJl+hzOTQRpTJYXaFAEUStydpfGxxIslZPSe2u0seSQ0midBST+AYFflqnkQ== X-Received: by 2002:a05:620a:4706:b0:67d:a135:c912 with SMTP id bs6-20020a05620a470600b0067da135c912mr3743031qkb.344.1648739359904; Thu, 31 Mar 2022 08:09:19 -0700 (PDT) Received: from localhost.localdomain (cpec09435e3e0ee-cmc09435e3e0ec.cpe.net.cable.rogers.com. [99.241.198.116]) by smtp.gmail.com with ESMTPSA id 21-20020ac85715000000b002e1ce9605ffsm20246871qtw.65.2022.03.31.08.09.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 31 Mar 2022 08:09:19 -0700 (PDT) From: Peter Xu To: qemu-devel@nongnu.org Subject: [PATCH v4 19/19] tests: Add postcopy preempt tests Date: Thu, 31 Mar 2022 11:08:57 -0400 Message-Id: <20220331150857.74406-20-peterx@redhat.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220331150857.74406-1-peterx@redhat.com> References: <20220331150857.74406-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leonardo Bras Soares Passos , "Daniel P . Berrange" , "Dr . David Alan Gilbert" , peterx@redhat.com, Juan Quintela Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Four tests are added for preempt mode: - Postcopy default - Postcopy tls - Postcopy recovery - Postcopy tls+recovery Signed-off-by: Peter Xu --- tests/qtest/migration-test.c | 49 ++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 7288c64e97..7188503ae1 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -477,6 +477,7 @@ typedef struct { */ bool hide_stderr; bool use_shmem; + bool postcopy_preempt; /* only launch the target process */ bool only_target; /* Use dirty ring if true; dirty logging otherwise */ @@ -992,6 +993,11 @@ static int migrate_postcopy_prepare(QTestState **from_ptr, migrate_set_capability(to, "postcopy-ram", true); migrate_set_capability(to, "postcopy-blocktime", true); + if (args->postcopy_preempt) { + migrate_set_capability(from, "postcopy-preempt", true); + migrate_set_capability(to, "postcopy-preempt", true); + } + /* We want to pick a speed slow enough that the test completes * quickly, but that it doesn't complete precopy even on a slow * machine, so also set the downtime. @@ -1058,6 +1064,25 @@ static void test_postcopy_tls(void) test_postcopy_common(&args); } +static void test_postcopy_preempt(void) +{ + MigrateStart args = { + .postcopy_preempt = true, + }; + + test_postcopy_common(&args); +} + +static void test_postcopy_preempt_tls(void) +{ + MigrateStart args = { + .postcopy_preempt = true, + .postcopy_tls = true, + }; + + test_postcopy_common(&args); +} + static void test_postcopy_recovery_common(MigrateStart *args) { QTestState *from, *to; @@ -1134,6 +1159,24 @@ static void test_postcopy_recovery_tls(void) test_postcopy_recovery_common(&args); } +static void test_postcopy_preempt_recovery(void) +{ + MigrateStart args = { .postcopy_preempt = true }; + + test_postcopy_recovery_common(&args); +} + +/* This contains preempt+recovery+tls test altogether */ +static void test_postcopy_preempt_all(void) +{ + MigrateStart args = { + .postcopy_preempt = true, + .postcopy_tls = true, + }; + + test_postcopy_recovery_common(&args); +} + static void test_baddest(void) { MigrateStart args = { @@ -2176,6 +2219,12 @@ int main(int argc, char **argv) qtest_add_func("/migration/postcopy/unix", test_postcopy); qtest_add_func("/migration/postcopy/recovery", test_postcopy_recovery); + qtest_add_func("/migration/postcopy/preempt/unix", test_postcopy_preempt); + qtest_add_func("/migration/postcopy/preempt/recovery", + test_postcopy_preempt_recovery); + qtest_add_func("/migration/postcopy/preempt/tls", test_postcopy_preempt_tls); + qtest_add_func("/migration/postcopy/preempt/tls+recovery", + test_postcopy_preempt_all); #ifdef CONFIG_GNUTLS qtest_add_func("/migration/postcopy/tls", test_postcopy_tls); qtest_add_func("/migration/postcopy/tls/recovery", test_postcopy_recovery_tls);