Patchwork [07/18] migration: make writes blocking

login
register
mail settings
Submitter Juan Quintela
Date Oct. 29, 2012, 2:11 p.m.
Message ID <1351519903-26607-8-git-send-email-quintela@redhat.com>
Download mbox | patch
Permalink /patch/195019/
State New
Headers show

Comments

Juan Quintela - Oct. 29, 2012, 2:11 p.m.
Move all the writes to the migration_thread, and make writings
blocking.  Notice that are still using the iothread for everything
that we do.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration-exec.c |  1 -
 migration-fd.c   |  1 -
 migration.c      | 21 ---------------------
 qemu-file.h      |  5 -----
 qemu-sockets.c   |  4 ----
 savevm.c         |  5 -----
 6 files changed, 37 deletions(-)
Markus Armbruster - Oct. 29, 2012, 3:51 p.m.
Juan Quintela <quintela@redhat.com> writes:

> Move all the writes to the migration_thread, and make writings
> blocking.  Notice that are still using the iothread for everything
> that we do.
[...]
> diff --git a/qemu-sockets.c b/qemu-sockets.c
> index cfed9c5..61b6e95 100644
> --- a/qemu-sockets.c
> +++ b/qemu-sockets.c
> @@ -276,9 +276,6 @@ static int inet_connect_addr(struct addrinfo *addr, bool *in_progress,
>          return -1;
>      }
>      qemu_setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
> -    if (connect_state != NULL) {
> -        socket_set_nonblock(sock);
> -    }
>      /* connect to peer */
>      do {
>          rc = 0;
> @@ -732,7 +729,6 @@ int unix_connect_opts(QemuOpts *opts, Error **errp,
>          connect_state = g_malloc0(sizeof(*connect_state));
>          connect_state->callback = callback;
>          connect_state->opaque = opaque;
> -        socket_set_nonblock(sock);
>      }
>
>      memset(&un, 0, sizeof(un));

Doesn't this break inet_nonblocking_connect() and
unix_nonblocking_connect()?

In your cover letter, you wrote:

    Note: Writes has become blocking, and I have to change the "remove"
    the feature now in qemu-sockets.c.  Checked that migration was the
    only user of that feature.  If new users appear, they just need to add
    the socket_set_nonblock() by hand.

Yes, migration-{tcp,unix} are their only users, but if they want a
blocking socket now, why not use inet_connect() and unix_connect()?

New users can't "just add socket_set_nonblock()".  They'd have to add it
right where you deleted it: between qemu_socket() and connect().  Else
the connect() is blocking.

[...]
Juan Quintela - Oct. 29, 2012, 5:32 p.m.
Markus Armbruster <armbru@redhat.com> wrote:
> Juan Quintela <quintela@redhat.com> writes:
>
>> Move all the writes to the migration_thread, and make writings
>> blocking.  Notice that are still using the iothread for everything
>> that we do.
> [...]
>> diff --git a/qemu-sockets.c b/qemu-sockets.c
>> index cfed9c5..61b6e95 100644
>> --- a/qemu-sockets.c
>> +++ b/qemu-sockets.c
>> @@ -276,9 +276,6 @@ static int inet_connect_addr(struct addrinfo *addr, bool *in_progress,
>>          return -1;
>>      }
>>      qemu_setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
>> -    if (connect_state != NULL) {
>> -        socket_set_nonblock(sock);
>> -    }
>>      /* connect to peer */
>>      do {
>>          rc = 0;
>> @@ -732,7 +729,6 @@ int unix_connect_opts(QemuOpts *opts, Error **errp,
>>          connect_state = g_malloc0(sizeof(*connect_state));
>>          connect_state->callback = callback;
>>          connect_state->opaque = opaque;
>> -        socket_set_nonblock(sock);
>>      }
>>
>>      memset(&un, 0, sizeof(un));
>
> Doesn't this break inet_nonblocking_connect() and
> unix_nonblocking_connect()?
>
> In your cover letter, you wrote:
>
>     Note: Writes has become blocking, and I have to change the "remove"
>     the feature now in qemu-sockets.c.  Checked that migration was the
>     only user of that feature.  If new users appear, they just need to add
>     the socket_set_nonblock() by hand.
>
> Yes, migration-{tcp,unix} are their only users, but if they want a
> blocking socket now, why not use inet_connect() and unix_connect()?
>
> New users can't "just add socket_set_nonblock()".  They'd have to add it
> right where you deleted it: between qemu_socket() and connect().  Else
> the connect() is blocking.

Grrr.

So, is there any way to make a connection that is non-blocking, but then
writes are blocking?

Later, Juan.
Paolo Bonzini - Oct. 29, 2012, 10:13 p.m.
Il 29/10/2012 18:32, Juan Quintela ha scritto:
>> > New users can't "just add socket_set_nonblock()".  They'd have to add it
>> > right where you deleted it: between qemu_socket() and connect().  Else
>> > the connect() is blocking.
> Grrr.
> 
> So, is there any way to make a connection that is non-blocking, but then
> writes are blocking?

socket_set_block.

Paolo
Markus Armbruster - Oct. 30, 2012, 7:36 a.m.
Juan Quintela <quintela@redhat.com> writes:

> Markus Armbruster <armbru@redhat.com> wrote:
>> Juan Quintela <quintela@redhat.com> writes:
>>
>>> Move all the writes to the migration_thread, and make writings
>>> blocking.  Notice that are still using the iothread for everything
>>> that we do.
>> [...]
>>> diff --git a/qemu-sockets.c b/qemu-sockets.c
>>> index cfed9c5..61b6e95 100644
>>> --- a/qemu-sockets.c
>>> +++ b/qemu-sockets.c
>>> @@ -276,9 +276,6 @@ static int inet_connect_addr(struct addrinfo *addr, bool *in_progress,
>>>          return -1;
>>>      }
>>>      qemu_setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
>>> -    if (connect_state != NULL) {
>>> -        socket_set_nonblock(sock);
>>> -    }
>>>      /* connect to peer */
>>>      do {
>>>          rc = 0;
>>> @@ -732,7 +729,6 @@ int unix_connect_opts(QemuOpts *opts, Error **errp,
>>>          connect_state = g_malloc0(sizeof(*connect_state));
>>>          connect_state->callback = callback;
>>>          connect_state->opaque = opaque;
>>> -        socket_set_nonblock(sock);
>>>      }
>>>
>>>      memset(&un, 0, sizeof(un));
>>
>> Doesn't this break inet_nonblocking_connect() and
>> unix_nonblocking_connect()?
>>
>> In your cover letter, you wrote:
>>
>>     Note: Writes has become blocking, and I have to change the "remove"
>>     the feature now in qemu-sockets.c.  Checked that migration was the
>>     only user of that feature.  If new users appear, they just need to add
>>     the socket_set_nonblock() by hand.
>>
>> Yes, migration-{tcp,unix} are their only users, but if they want a
>> blocking socket now, why not use inet_connect() and unix_connect()?
>>
>> New users can't "just add socket_set_nonblock()".  They'd have to add it
>> right where you deleted it: between qemu_socket() and connect().  Else
>> the connect() is blocking.
>
> Grrr.
>
> So, is there any way to make a connection that is non-blocking, but then
> writes are blocking?

Which operations on the migration socket do you need to block, and which
ones do you need not to block?

If connect() should block, use inet_connect() / unix_connect().  The
returned socket will be blocking.  You can then switch to non-blocking
mode (and possibly back) at appropriate times.

If connect() should not block, use inet_nonblocking_connect() and so
forth.
Juan Quintela - Oct. 30, 2012, 8:23 a.m.
Markus Armbruster <armbru@redhat.com> wrote:
> Juan Quintela <quintela@redhat.com> writes:
>
>> Markus Armbruster <armbru@redhat.com> wrote:
>>> Juan Quintela <quintela@redhat.com> writes:
>>>
>>>> Move all the writes to the migration_thread, and make writings
>>>> blocking.  Notice that are still using the iothread for everything
>>>> that we do.
>>> [...]
>>>> diff --git a/qemu-sockets.c b/qemu-sockets.c
>>>> index cfed9c5..61b6e95 100644
>>>> --- a/qemu-sockets.c
>>>> +++ b/qemu-sockets.c
>>>> @@ -276,9 +276,6 @@ static int inet_connect_addr(struct addrinfo *addr, bool *in_progress,
>>>>          return -1;
>>>>      }
>>>>      qemu_setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
>>>> -    if (connect_state != NULL) {
>>>> -        socket_set_nonblock(sock);
>>>> -    }
>>>>      /* connect to peer */
>>>>      do {
>>>>          rc = 0;
>>>> @@ -732,7 +729,6 @@ int unix_connect_opts(QemuOpts *opts, Error **errp,
>>>>          connect_state = g_malloc0(sizeof(*connect_state));
>>>>          connect_state->callback = callback;
>>>>          connect_state->opaque = opaque;
>>>> -        socket_set_nonblock(sock);
>>>>      }
>>>>
>>>>      memset(&un, 0, sizeof(un));
>>>
>>> Doesn't this break inet_nonblocking_connect() and
>>> unix_nonblocking_connect()?
>>>
>>> In your cover letter, you wrote:
>>>
>>>     Note: Writes has become blocking, and I have to change the "remove"
>>>     the feature now in qemu-sockets.c.  Checked that migration was the
>>>     only user of that feature.  If new users appear, they just need to add
>>>     the socket_set_nonblock() by hand.
>>>
>>> Yes, migration-{tcp,unix} are their only users, but if they want a
>>> blocking socket now, why not use inet_connect() and unix_connect()?
>>>
>>> New users can't "just add socket_set_nonblock()".  They'd have to add it
>>> right where you deleted it: between qemu_socket() and connect().  Else
>>> the connect() is blocking.
>>
>> Grrr.
>>
>> So, is there any way to make a connection that is non-blocking, but then
>> writes are blocking?
>
> Which operations on the migration socket do you need to block, and which
> ones do you need not to block?

connect: not blocking (done on the iothread)
writes: blocking, done in the migration thread.

I think thet socket_set_block() that paolo says is the right solution.

>
> If connect() should block, use inet_connect() / unix_connect().  The
> returned socket will be blocking.  You can then switch to non-blocking
> mode (and possibly back) at appropriate times.
>
> If connect() should not block, use inet_nonblocking_connect() and so
> forth.

Thanks, Juan.
Markus Armbruster - Oct. 30, 2012, 10:50 a.m.
Juan Quintela <quintela@redhat.com> writes:

> Markus Armbruster <armbru@redhat.com> wrote:
>> Juan Quintela <quintela@redhat.com> writes:
>>
>>> Markus Armbruster <armbru@redhat.com> wrote:
>>>> Juan Quintela <quintela@redhat.com> writes:
>>>>
>>>>> Move all the writes to the migration_thread, and make writings
>>>>> blocking.  Notice that are still using the iothread for everything
>>>>> that we do.
>>>> [...]
>>>>> diff --git a/qemu-sockets.c b/qemu-sockets.c
>>>>> index cfed9c5..61b6e95 100644
>>>>> --- a/qemu-sockets.c
>>>>> +++ b/qemu-sockets.c
>>>>> @@ -276,9 +276,6 @@ static int inet_connect_addr(struct addrinfo *addr, bool *in_progress,
>>>>>          return -1;
>>>>>      }
>>>>>      qemu_setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
>>>>> -    if (connect_state != NULL) {
>>>>> -        socket_set_nonblock(sock);
>>>>> -    }
>>>>>      /* connect to peer */
>>>>>      do {
>>>>>          rc = 0;
>>>>> @@ -732,7 +729,6 @@ int unix_connect_opts(QemuOpts *opts, Error **errp,
>>>>>          connect_state = g_malloc0(sizeof(*connect_state));
>>>>>          connect_state->callback = callback;
>>>>>          connect_state->opaque = opaque;
>>>>> -        socket_set_nonblock(sock);
>>>>>      }
>>>>>
>>>>>      memset(&un, 0, sizeof(un));
>>>>
>>>> Doesn't this break inet_nonblocking_connect() and
>>>> unix_nonblocking_connect()?
>>>>
>>>> In your cover letter, you wrote:
>>>>
>>>>     Note: Writes has become blocking, and I have to change the "remove"
>>>>     the feature now in qemu-sockets.c.  Checked that migration was the
>>>>     only user of that feature.  If new users appear, they just need to add
>>>>     the socket_set_nonblock() by hand.
>>>>
>>>> Yes, migration-{tcp,unix} are their only users, but if they want a
>>>> blocking socket now, why not use inet_connect() and unix_connect()?
>>>>
>>>> New users can't "just add socket_set_nonblock()".  They'd have to add it
>>>> right where you deleted it: between qemu_socket() and connect().  Else
>>>> the connect() is blocking.
>>>
>>> Grrr.
>>>
>>> So, is there any way to make a connection that is non-blocking, but then
>>> writes are blocking?
>>
>> Which operations on the migration socket do you need to block, and which
>> ones do you need not to block?
>
> connect: not blocking (done on the iothread)
> writes: blocking, done in the migration thread.
>
> I think thet socket_set_block() that paolo says is the right solution.

Sounds good.

By the way, I probably would've missed this had you not pointed to it in
the cover letter.  Smart move on your part.
Juan Quintela - Oct. 30, 2012, 11:08 a.m.
Markus Armbruster <armbru@redhat.com> wrote:
> Juan Quintela <quintela@redhat.com> writes:
>>>>>
>>>>> Doesn't this break inet_nonblocking_connect() and
>>>>> unix_nonblocking_connect()?
>>>>>
>>>>> In your cover letter, you wrote:
>>>>>
>>>>>     Note: Writes has become blocking, and I have to change the "remove"
>>>>>     the feature now in qemu-sockets.c.  Checked that migration was the
>>>>>     only user of that feature.  If new users appear, they just need to add
>>>>>     the socket_set_nonblock() by hand.
>>>>>
>>>>> Yes, migration-{tcp,unix} are their only users, but if they want a
>>>>> blocking socket now, why not use inet_connect() and unix_connect()?
>>>>>
>>>>> New users can't "just add socket_set_nonblock()".  They'd have to add it
>>>>> right where you deleted it: between qemu_socket() and connect().  Else
>>>>> the connect() is blocking.
>>>>
>>>> Grrr.
>>>>
>>>> So, is there any way to make a connection that is non-blocking, but then
>>>> writes are blocking?
>>>
>>> Which operations on the migration socket do you need to block, and which
>>> ones do you need not to block?
>>
>> connect: not blocking (done on the iothread)
>> writes: blocking, done in the migration thread.
>>
>> I think thet socket_set_block() that paolo says is the right solution.
>
> Sounds good.
>
> By the way, I probably would've missed this had you not pointed to it in
> the cover letter.  Smart move on your part.

I knew somebody would have noticed, so it was supposed to be a "priori"
apologize ....

thanks, Juan.

Patch

diff --git a/migration-exec.c b/migration-exec.c
index 519af57..ecc0f00 100644
--- a/migration-exec.c
+++ b/migration-exec.c
@@ -72,7 +72,6 @@  void exec_start_outgoing_migration(MigrationState *s, const char *command, Error

     s->fd = fileno(f);
     assert(s->fd != -1);
-    socket_set_nonblock(s->fd);

     s->opaque = qemu_popen(f, "w");

diff --git a/migration-fd.c b/migration-fd.c
index ce6932d..9398b91 100644
--- a/migration-fd.c
+++ b/migration-fd.c
@@ -80,7 +80,6 @@  void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
         return;
     }

-    fcntl(s->fd, F_SETFL, O_NONBLOCK);
     s->get_error = fd_errno;
     s->write = fd_write;
     s->close = fd_close;
diff --git a/migration.c b/migration.c
index d5c178d..23a7974 100644
--- a/migration.c
+++ b/migration.c
@@ -244,10 +244,6 @@  static int migrate_fd_cleanup(MigrationState *s)
 {
     int ret = 0;

-    if (s->fd != -1) {
-        qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
-    }
-
     if (s->file) {
         DPRINTF("closing file\n");
         ret = qemu_fclose(s->file);
@@ -282,18 +278,6 @@  static void migrate_fd_completed(MigrationState *s)
     notifier_list_notify(&migration_state_notifiers, s);
 }

-static void migrate_fd_put_notify(void *opaque)
-{
-    MigrationState *s = opaque;
-    int ret;
-
-    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
-    ret = qemu_file_put_notify(s->file);
-    if (ret) {
-        migrate_fd_error(s);
-    }
-}
-
 ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
                               size_t size)
 {
@@ -310,10 +294,6 @@  ssize_t migrate_fd_put_buffer(MigrationState *s, const void *data,
     if (ret == -1)
         ret = -(s->get_error(s));

-    if (ret == -EAGAIN) {
-        qemu_set_fd_handler2(s->fd, NULL, NULL, migrate_fd_put_notify, s);
-    }
-
     return ret;
 }

@@ -409,7 +389,6 @@  int migrate_fd_wait_for_unfreeze(MigrationState *s)

 int migrate_fd_close(MigrationState *s)
 {
-    qemu_set_fd_handler2(s->fd, NULL, NULL, NULL, NULL);
     return s->close(s);
 }

diff --git a/qemu-file.h b/qemu-file.h
index 9c8985b..e88892c 100644
--- a/qemu-file.h
+++ b/qemu-file.h
@@ -104,11 +104,6 @@  int64_t qemu_file_set_rate_limit(QEMUFile *f, int64_t new_rate);
 int64_t qemu_file_get_rate_limit(QEMUFile *f);
 int qemu_file_get_error(QEMUFile *f);

-/* Try to send any outstanding data.  This function is useful when output is
- * halted due to rate limiting or EAGAIN errors occur as it can be used to
- * resume output. */
-int qemu_file_put_notify(QEMUFile *f);
-
 static inline void qemu_put_be64s(QEMUFile *f, const uint64_t *pv)
 {
     qemu_put_be64(f, *pv);
diff --git a/qemu-sockets.c b/qemu-sockets.c
index cfed9c5..61b6e95 100644
--- a/qemu-sockets.c
+++ b/qemu-sockets.c
@@ -276,9 +276,6 @@  static int inet_connect_addr(struct addrinfo *addr, bool *in_progress,
         return -1;
     }
     qemu_setsockopt(sock, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on));
-    if (connect_state != NULL) {
-        socket_set_nonblock(sock);
-    }
     /* connect to peer */
     do {
         rc = 0;
@@ -732,7 +729,6 @@  int unix_connect_opts(QemuOpts *opts, Error **errp,
         connect_state = g_malloc0(sizeof(*connect_state));
         connect_state->callback = callback;
         connect_state->opaque = opaque;
-        socket_set_nonblock(sock);
     }

     memset(&un, 0, sizeof(un));
diff --git a/savevm.c b/savevm.c
index b080d37..69f1768 100644
--- a/savevm.c
+++ b/savevm.c
@@ -523,11 +523,6 @@  int qemu_fclose(QEMUFile *f)
     return ret;
 }

-int qemu_file_put_notify(QEMUFile *f)
-{
-    return f->put_buffer(f->opaque, NULL, 0, 0);
-}
-
 void qemu_put_buffer(QEMUFile *f, const uint8_t *buf, int size)
 {
     int l;