Patchwork Migration via unix sockets.

login
register
mail settings
Submitter Chris Lalancette
Date Aug. 11, 2009, 9:15 a.m.
Message ID <4A8136CF.5040304@redhat.com>
Download mbox | patch
Permalink /patch/31126/
State Superseded
Headers show

Comments

Chris Lalancette - Aug. 11, 2009, 9:15 a.m.
Avi Kivity wrote:
> On 08/10/2009 01:23 PM, Chris Lalancette wrote:
>> Chris Lalancette wrote:
>>    
>>> Implement migration via unix sockets.  While you can fake this using
>>> exec and netcat, this involves forking another process and is
>>> generally not very nice.  By doing this directly in qemu, we can avoid
>>> the copy through the external nc command.  This is useful for
>>> implementations (such as libvirt) that want to do "secure" migration;
>>> we pipe the data on the sending side into the unix socket, libvirt
>>> picks it up, encrypts it, and transports it, and then on the remote
>>> side libvirt decrypts it, dumps it to another unix socket, and
>>> feeds it into qemu.
>>>
>>> The implementation is straightforward and looks very similar to
>>> migration-exec.c and migration-tcp.c
>>>      
>> ping?
>>    
> 
> It would be nice to support migration via arbitrary fd using the recent 
> SCM_RIGHTS support.

A possible implementation of that is attached.  I have to say, though, that it
is way more clumsy to use than the unix implementation I posted earlier.  On the
outgoing side, you have to use the "getfd" monitor command to pass the fd using
SCM_RIGHTS, then you have to issue another monitor command to start the migration:

(qemu) getfd migration # passes opened fd via SCM_RIGHTS
(qemu) migrate -d fd:migration

On the incoming side, you have to pass the fd via the command-line.  That means
that you have to first arrange for it not to be closed on exec, and it also
means that qemu is now depending on the external program to correctly set up the
incoming socket so that qemu can just do the accept() on it.

None of these problems are insurmountable, but they do make it cumbersome to use
in general, and very difficult to use from the command-line.

The other option is that I've misunderstood your intent, and if that is the
case, please correct me where I'm wrong :).
Avi Kivity - Aug. 11, 2009, 9:35 a.m.
On 08/11/2009 12:15 PM, Chris Lalancette wrote:
> A possible implementation of that is attached.  I have to say, though, that it
> is way more clumsy to use than the unix implementation I posted earlier.  On the
> outgoing side, you have to use the "getfd" monitor command to pass the fd using
> SCM_RIGHTS, then you have to issue another monitor command to start the migration:
>
> (qemu) getfd migration # passes opened fd via SCM_RIGHTS
> (qemu) migrate -d fd:migration
>    

That doesn't seem so bad, especially if you're a program.

> On the incoming side, you have to pass the fd via the command-line.  That means
> that you have to first arrange for it not to be closed on exec, and it also
> means that qemu is now depending on the external program to correctly set up the
> incoming socket so that qemu can just do the accept() on it.
>
>    

I'd like to see a qemu monitor incoming migration command.

> None of these problems are insurmountable, but they do make it cumbersome to use
> in general, and very difficult to use from the command-line.
>    

These are intended for qemu control programs, not humans.

> The other option is that I've misunderstood your intent, and if that is the
> case, please correct me where I'm wrong :).
>    

No, I think you got it.
Paolo Bonzini - Aug. 11, 2009, 9:44 a.m.
> That means that you have to first arrange for it not to be closed on
 > exec, and it also means that qemu is now depending on the external
 > program to correctly set up the incoming socket so that qemu can just
 > do the accept() on it.

At this point, I wonder if it isn't better to move the accept to the 
external program, so that it could even use a pipe.  The accept can be 
replaced by

struct pollfd pfd;
pfd.fd = fd;
pfd.events = POLLIN;
poll (&pfd, 1, INFTIM);
if (pfd.revents & (POLLERR | POLLHUP | POLLNVAL))
   handle_error ();

or the equivalent using select.

Paolo
Avi Kivity - Aug. 11, 2009, 10:06 a.m.
On 08/11/2009 12:44 PM, Paolo Bonzini wrote:
> > That means that you have to first arrange for it not to be closed on
> > exec, and it also means that qemu is now depending on the external
> > program to correctly set up the incoming socket so that qemu can just
> > do the accept() on it.
>
> At this point, I wonder if it isn't better to move the accept to the 
> external program, so that it could even use a pipe.  The accept can be 
> replaced by
>
> struct pollfd pfd;
> pfd.fd = fd;
> pfd.events = POLLIN;
> poll (&pfd, 1, INFTIM);
> if (pfd.revents & (POLLERR | POLLHUP | POLLNVAL))
>   handle_error ();
>
> or the equivalent using select.

Yes, arbitrary fd definitely needs to skip accept().
Daniel P. Berrange - Aug. 11, 2009, 10:20 a.m.
On Tue, Aug 11, 2009 at 11:15:59AM +0200, Chris Lalancette wrote:
> +static void fd_accept_incoming_migration(void *opaque)
> +{
> +    struct sockaddr addr;
> +    socklen_t addrlen = sizeof(addr);
> +    int s = (unsigned long)opaque;
> +    QEMUFile *f;
> +    int c, ret;
> +
> +    do {
> +        c = accept(s, &addr, &addrlen);
> +    } while (c == -1 && socket_error() == EINTR);

This bit doesn't make sense if we're just passing an open FD, since
it may not be a socket. Just declare that if passing a socket FD,
the FD must be a pre-accepted client, and remove this line.

Regards,
Daniel

Patch

diff --git a/Makefile b/Makefile
index c5763b7..a8cd0c9 100644
--- a/Makefile
+++ b/Makefile
@@ -110,7 +110,7 @@  obj-$(CONFIG_BRLAPI) += baum.o
 LIBS+=$(BRLAPI_LIBS)
 
 obj-$(CONFIG_WIN32) += tap-win32.o
-obj-$(CONFIG_POSIX) += migration-exec.o
+obj-$(CONFIG_POSIX) += migration-exec.o migration-fd.o
 
 ifdef CONFIG_COREAUDIO
 AUDIO_PT = y
diff --git a/migration-fd.c b/migration-fd.c
new file mode 100644
index 0000000..fea5a43
--- /dev/null
+++ b/migration-fd.c
@@ -0,0 +1,146 @@ 
+/*
+ * QEMU live migration via generic fd
+ *
+ * Copyright Red Hat, Inc. 2009
+ *
+ * Authors:
+ *  Chris Lalancette <clalance@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "qemu_socket.h"
+#include "migration.h"
+#include "monitor.h"
+#include "qemu-char.h"
+#include "sysemu.h"
+#include "buffered_file.h"
+#include "block.h"
+#include "qemu_socket.h"
+
+//#define DEBUG_MIGRATION_FD
+
+#ifdef DEBUG_MIGRATION_FD
+#define dprintf(fmt, ...) \
+    do { printf("migration-fd: " fmt, ## __VA_ARGS__); } while (0)
+#else
+#define dprintf(fmt, ...) \
+    do { } while (0)
+#endif
+
+static int fd_errno(FdMigrationState *s)
+{
+    return errno;
+}
+
+static int fd_write(FdMigrationState *s, const void * buf, size_t size)
+{
+    return write(s->fd, buf, size);
+}
+
+static int fd_close(FdMigrationState *s)
+{
+    dprintf("fd_close\n");
+    if (s->fd != -1) {
+        close(s->fd);
+        s->fd = -1;
+    }
+    return 0;
+}
+
+MigrationState *fd_start_outgoing_migration(Monitor *mon,
+					    const char *fdname,
+					    int64_t bandwidth_limit,
+					    int detach)
+{
+    FdMigrationState *s;
+
+    s = qemu_mallocz(sizeof(*s));
+
+    s->fd = monitor_get_fd(mon, fdname);
+    if (s->fd == -1) {
+        dprintf("fd_migration: no file descriptor supplied via SCM_RIGHTS\n");
+	qemu_free(s);
+        return NULL;
+    }
+
+    socket_set_nonblock(s->fd);
+
+    s->get_error = fd_errno;
+    s->write = fd_write;
+    s->close = fd_close;
+    s->mig_state.cancel = migrate_fd_cancel;
+    s->mig_state.get_status = migrate_fd_get_status;
+    s->mig_state.release = migrate_fd_release;
+
+    s->state = MIG_STATE_ACTIVE;
+    s->mon_resume = NULL;
+    s->bandwidth_limit = bandwidth_limit;
+
+    if (!detach)
+        migrate_fd_monitor_suspend(s);
+
+    migrate_fd_connect(s);
+
+    return &s->mig_state;
+}
+
+static void fd_accept_incoming_migration(void *opaque)
+{
+    struct sockaddr addr;
+    socklen_t addrlen = sizeof(addr);
+    int s = (unsigned long)opaque;
+    QEMUFile *f;
+    int c, ret;
+
+    do {
+        c = accept(s, &addr, &addrlen);
+    } while (c == -1 && socket_error() == EINTR);
+
+    if (c == -1) {
+        fprintf(stderr, "could not accept migration connection\n");
+        return;
+    }
+
+    dprintf("accepted migration\n");
+
+    f = qemu_fopen_socket(c);
+    if (f == NULL) {
+        fprintf(stderr, "could not qemu_fopen socket\n");
+        goto out;
+    }
+
+    ret = qemu_loadvm_state(f);
+    if (ret < 0) {
+        fprintf(stderr, "load of migration failed\n");
+        goto out_fopen;
+    }
+    qemu_announce_self();
+    dprintf("successfully loaded vm state\n");
+
+    /* we've successfully migrated, close the server socket */
+    qemu_set_fd_handler2(s, NULL, NULL, NULL, NULL);
+    close(s);
+
+out_fopen:
+    qemu_fclose(f);
+out:
+    close(c);
+}
+
+int fd_start_incoming_migration(const char *infd)
+{
+    int fd;
+
+    dprintf("Attempting to start an fd incoming migration\n");
+
+    fd = strtol(infd, NULL, 0);
+
+    qemu_set_fd_handler2(fd, NULL, fd_accept_incoming_migration, NULL,
+			 (void *)(unsigned long)fd);
+
+    return 0;
+}
diff --git a/migration.c b/migration.c
index 34e2bc1..c18d595 100644
--- a/migration.c
+++ b/migration.c
@@ -45,4 +45,6 @@  void qemu_start_incoming_migration(const char *uri)
         exec_start_incoming_migration(p);
+    else if (strstart(uri, "fd:", &p))
+        fd_start_incoming_migration(p);
 #endif
     else
         fprintf(stderr, "unknown migration protocol: %s\n", uri);
@@ -62,4 +64,6 @@  void do_migrate(Monitor *mon, int detach, const char *uri)
         s = exec_start_outgoing_migration(p, max_throttle, detach);
+    else if (strstart(uri, "fd:", &p))
+        s = fd_start_outgoing_migration(mon, p, max_throttle, detach);
 #endif
     else
         monitor_printf(mon, "unknown migration protocol: %s\n", uri);
diff --git a/migration.h b/migration.h
index 0ed1fcb..96dad38 100644
--- a/migration.h
+++ b/migration.h
@@ -79,6 +79,13 @@  MigrationState *unix_start_outgoing_migration(const char *path,
 					      int64_t bandwidth_limit,
 					      int detach);
 
+int fd_start_incoming_migration(const char *path);
+
+MigrationState *fd_start_outgoing_migration(Monitor *mon,
+					    const char *fdname,
+					    int64_t bandwidth_limit,
+					    int detach);
+
 void migrate_fd_monitor_suspend(FdMigrationState *s);
 
 void migrate_fd_error(FdMigrationState *s);