From patchwork Wed Apr 10 04:29:37 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: mrhines@linux.vnet.ibm.com X-Patchwork-Id: 235289 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id A4E1E2C007A for ; Wed, 10 Apr 2013 14:39:50 +1000 (EST) Received: from localhost ([::1]:33542 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UPmjR-0005j0-9N for incoming@patchwork.ozlabs.org; Wed, 10 Apr 2013 00:34:01 -0400 Received: from eggs.gnu.org ([208.118.235.92]:51760) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UPmft-0000w4-G0 for qemu-devel@nongnu.org; Wed, 10 Apr 2013 00:30:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UPmfq-0001Gp-DJ for qemu-devel@nongnu.org; Wed, 10 Apr 2013 00:30:21 -0400 Received: from e33.co.us.ibm.com ([32.97.110.151]:47389) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UPmfq-0001Gi-5g for qemu-devel@nongnu.org; Wed, 10 Apr 2013 00:30:18 -0400 Received: from /spool/local by e33.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 9 Apr 2013 22:30:17 -0600 Received: from d03dlp03.boulder.ibm.com (9.17.202.179) by e33.co.us.ibm.com (192.168.1.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 9 Apr 2013 22:30:15 -0600 Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227]) by d03dlp03.boulder.ibm.com (Postfix) with ESMTP id 8DAF619D8043 for ; Tue, 9 Apr 2013 22:30:10 -0600 (MDT) Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r3A4UCF9129780 for ; Tue, 9 Apr 2013 22:30:13 -0600 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r3A4UAek022233 for ; Tue, 9 Apr 2013 22:30:10 -0600 Received: from mrhinesdev.klabtestbed.com (klinux.watson.ibm.com [9.2.208.21]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id r3A4U3RK021687; Tue, 9 Apr 2013 22:30:09 -0600 From: mrhines@linux.vnet.ibm.com To: qemu-devel@nongnu.org Date: Wed, 10 Apr 2013 00:29:37 -0400 Message-Id: <1365568180-19593-5-git-send-email-mrhines@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.10.4 In-Reply-To: <1365568180-19593-1-git-send-email-mrhines@linux.vnet.ibm.com> References: <1365568180-19593-1-git-send-email-mrhines@linux.vnet.ibm.com> X-TM-AS-MML: No X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13041004-2398-0000-0000-0000131F6AEF X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.4.x-2.6.x [generic] X-Received-From: 32.97.110.151 Cc: aliguori@us.ibm.com, mst@redhat.com, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com, pbonzini@redhat.com Subject: [Qemu-devel] [RFC PATCH RDMA support v6: 4/7] Introduce two new capabilities X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: "Michael R. Hines" RDMA performs very slowly with zero-page checking. Without the ability to disable it, RDMA throughput and latency promises and high performance links cannot be fully realized. On the other hand, dynamic page registration support is also included in the RDMA protocol. This second capability also cannot be fully realized without the ability to enable zero page scanning. So, we have two new capabilities which work together: 1. migrate_set_capability check_for_zero on|off (default on) 2. migrate_set_capability chunk_register_destination on|off (default off) Signed-off-by: Michael R. Hines --- include/migration/qemu-file.h | 15 +++++++++++++++ migration.c | 33 +++++++++++++++++++++++++++++++-- qapi-schema.json | 2 +- 3 files changed, 47 insertions(+), 3 deletions(-) diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h index 623c434..b6f3256 100644 --- a/include/migration/qemu-file.h +++ b/include/migration/qemu-file.h @@ -57,12 +57,15 @@ typedef int (QEMUFileGetFD)(void *opaque); typedef ssize_t (QEMUFileWritevBufferFunc)(void *opaque, struct iovec *iov, int iovcnt); +typedef struct QEMURamControlOps QEMURamControlOps; + typedef struct QEMUFileOps { QEMUFilePutBufferFunc *put_buffer; QEMUFileGetBufferFunc *get_buffer; QEMUFileCloseFunc *close; QEMUFileGetFD *get_fd; QEMUFileWritevBufferFunc *writev_buffer; + const QEMURamControlOps *ram_control; } QEMUFileOps; QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops); @@ -80,6 +83,18 @@ void qemu_put_byte(QEMUFile *f, int v); * The buffer should be available till it is sent asynchronously. */ void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, int size); +void qemu_file_set_error(QEMUFile *f, int ret); + +void qemu_rdma_cleanup(void *opaque); +int qemu_rdma_close(void *opaque); +int qemu_rdma_get_fd(void *opaque); +int qemu_rdma_get_buffer(void *opaque, uint8_t *buf, int64_t pos, int size); +int qemu_rdma_put_buffer(void *opaque, const uint8_t *buf, + int64_t pos, int size); +bool qemu_file_mode_is_not_valid(const char * mode); + +extern const QEMUFileOps rdma_read_ops; +extern const QEMUFileOps rdma_write_ops; static inline void qemu_put_ubyte(QEMUFile *f, unsigned int v) { diff --git a/migration.c b/migration.c index 3b4b467..875cee3 100644 --- a/migration.c +++ b/migration.c @@ -66,6 +66,7 @@ MigrationState *migrate_get_current(void) .state = MIG_STATE_SETUP, .bandwidth_limit = MAX_THROTTLE, .xbzrle_cache_size = DEFAULT_MIGRATE_CACHE_SIZE, + .enabled_capabilities[MIGRATION_CAPABILITY_CHECK_FOR_ZERO] = true, }; return ¤t_migration; @@ -77,6 +78,10 @@ void qemu_start_incoming_migration(const char *uri, Error **errp) if (strstart(uri, "tcp:", &p)) tcp_start_incoming_migration(p, errp); +#ifdef CONFIG_RDMA + else if (strstart(uri, "rdma:", &p)) + rdma_start_incoming_migration(p, errp); +#endif #if !defined(WIN32) else if (strstart(uri, "exec:", &p)) exec_start_incoming_migration(p, errp); @@ -120,8 +125,10 @@ void process_incoming_migration(QEMUFile *f) Coroutine *co = qemu_coroutine_create(process_incoming_migration_co); int fd = qemu_get_fd(f); - assert(fd != -1); - qemu_set_nonblock(fd); + if(fd != -2) { /* rdma returns -2 */ + assert(fd != -1); + qemu_set_nonblock(fd); + } qemu_coroutine_enter(co, f); } @@ -405,6 +412,10 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk, if (strstart(uri, "tcp:", &p)) { tcp_start_outgoing_migration(s, p, &local_err); +#ifdef CONFIG_RDMA + } else if (strstart(uri, "rdma:", &p)) { + rdma_start_outgoing_migration(s, p, &local_err); +#endif #if !defined(WIN32) } else if (strstart(uri, "exec:", &p)) { exec_start_outgoing_migration(s, p, &local_err); @@ -474,6 +485,24 @@ void qmp_migrate_set_downtime(double value, Error **errp) max_downtime = (uint64_t)value; } +bool migrate_chunk_register_destination(void) +{ + MigrationState *s; + + s = migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_CHUNK_REGISTER_DESTINATION]; +} + +bool migrate_check_for_zero(void) +{ + MigrationState *s; + + s = migrate_get_current(); + + return s->enabled_capabilities[MIGRATION_CAPABILITY_CHECK_FOR_ZERO]; +} + int migrate_use_xbzrle(void) { MigrationState *s; diff --git a/qapi-schema.json b/qapi-schema.json index db542f6..7ebcf99 100644 --- a/qapi-schema.json +++ b/qapi-schema.json @@ -602,7 +602,7 @@ # Since: 1.2 ## { 'enum': 'MigrationCapability', - 'data': ['xbzrle'] } + 'data': ['xbzrle', 'check_for_zero', 'chunk_register_destination'] } ## # @MigrationCapabilityStatus