| Message ID | 20180410124913.10832-3-peterx@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | qemu-thread: allow cur_mon be per thread | expand |
On 04/10/2018 07:49 AM, Peter Xu wrote: > cur_mon was only used in main loop so we don't really need that to be > per-thread variable. Now it's possible that we have more than one > thread to operate on it. Let's start to let it be per-thread variable. > > In case we'll create threads within a valid cur_mon setup, we'd better > let the child threads to inherit the cur_mon from parent thread too. Do > that for both posix and win32 threads. > > Signed-off-by: Peter Xu <peterx@redhat.com> > --- > include/monitor/monitor.h | 2 +- > include/qemu/thread-win32.h | 1 + > monitor.c | 2 +- > stubs/monitor.c | 2 +- > tests/test-util-sockets.c | 2 +- > util/qemu-thread-posix.c | 6 ++++++ > util/qemu-thread-win32.c | 6 ++++++ > 7 files changed, 17 insertions(+), 4 deletions(-) > > @@ -494,6 +496,9 @@ static void *qemu_thread_start(void *args) > void *(*start_routine)(void *) = qemu_thread_args->start_routine; > void *arg = qemu_thread_args->arg; > > + /* Inherit the cur_mon pointer from father thread */ More typical as s/father/parent/ > +++ b/util/qemu-thread-win32.c > @@ -339,6 +341,9 @@ static unsigned __stdcall win32_start_routine(void *arg) > void *(*start_routine)(void *) = data->start_routine; > void *thread_arg = data->arg; > > + /* Inherit the cur_mon pointer from father thread */ > + cur_mon = data->current_monitor; Otherwise makes sense to me. I agree with your analysis that the set of existing OOB commands (just 'x-oob-test') has no direct use of cur_mon. I'm a little fuzzier on whether the OOB changes can cause cur_mon to be modified by two threads in parallel (monitor_qmp_dispatch_one() is futzing around with 'cur_mon' around the call to qmp_dispatch(), and at least qmp_human_monitor_command() is also futzing around with it; is there a case where handling qmp_human_monitor_command() in the dispatch thread in parallel with more input on the main thread could break?) Thus I'm not sure whether this is needed for 2.12 to avoid a regression.
On Tue, Apr 10, 2018 at 08:49:13PM +0800, Peter Xu wrote: > cur_mon was only used in main loop so we don't really need that to be > per-thread variable. Now it's possible that we have more than one > thread to operate on it. Let's start to let it be per-thread variable. Trying to understand the reason for this patch: Are there any users of per-thread cur_mon? or Does this patch fix a bug? > In case we'll create threads within a valid cur_mon setup, we'd better > let the child threads to inherit the cur_mon from parent thread too. Do > that for both posix and win32 threads. Without actual users I don't like this. It sounds like "let's make it global just in case something needs it some day". It's ugly for QEMU's thread API to know about the monitor - that's a layering violation. If there's a legitimate need I think this patch might be necessary, but I don't see enough justification to do this yet. Stefan
On Tue, Apr 10, 2018 at 08:54:31AM -0500, Eric Blake wrote: > On 04/10/2018 07:49 AM, Peter Xu wrote: > > cur_mon was only used in main loop so we don't really need that to be > > per-thread variable. Now it's possible that we have more than one > > thread to operate on it. Let's start to let it be per-thread variable. > > > > In case we'll create threads within a valid cur_mon setup, we'd better > > let the child threads to inherit the cur_mon from parent thread too. Do > > that for both posix and win32 threads. > > > > Signed-off-by: Peter Xu <peterx@redhat.com> > > --- > > include/monitor/monitor.h | 2 +- > > include/qemu/thread-win32.h | 1 + > > monitor.c | 2 +- > > stubs/monitor.c | 2 +- > > tests/test-util-sockets.c | 2 +- > > util/qemu-thread-posix.c | 6 ++++++ > > util/qemu-thread-win32.c | 6 ++++++ > > 7 files changed, 17 insertions(+), 4 deletions(-) > > > > > @@ -494,6 +496,9 @@ static void *qemu_thread_start(void *args) > > void *(*start_routine)(void *) = qemu_thread_args->start_routine; > > void *arg = qemu_thread_args->arg; > > > > + /* Inherit the cur_mon pointer from father thread */ > > More typical as s/father/parent/ Fixed. > > > +++ b/util/qemu-thread-win32.c > > > @@ -339,6 +341,9 @@ static unsigned __stdcall win32_start_routine(void *arg) > > void *(*start_routine)(void *) = data->start_routine; > > void *thread_arg = data->arg; > > > > + /* Inherit the cur_mon pointer from father thread */ > > + cur_mon = data->current_monitor; > > Otherwise makes sense to me. > > I agree with your analysis that the set of existing OOB commands (just > 'x-oob-test') has no direct use of cur_mon. I'm a little fuzzier on > whether the OOB changes can cause cur_mon to be modified by two threads > in parallel (monitor_qmp_dispatch_one() is futzing around with 'cur_mon' > around the call to qmp_dispatch(), and at least > qmp_human_monitor_command() is also futzing around with it; is there a > case where handling qmp_human_monitor_command() in the dispatch thread > in parallel with more input on the main thread could break?) Thus I'm > not sure whether this is needed for 2.12 to avoid a regression. Could I ask what's the "more input on the main thread"? AFAIU, if we don't take x-oob-test into account, then still only the main thread will touch (not only modify, even read) the cur_mon variable. And as long as that's true, we are keeping the old behavior as when we are without Out-Of-Band, then IMHO we'll be fine. qmp_human_monitor_command() is only one QMP command handler, now it can only be run within main thread. So even we can have the stack like monitor_qmp_dispatch_one (which modifies cur_mon) calls qmp_human_monitor_command (which modifies cur_mon again), still we'll be fine too as long as they are in the same thread, just like before. That's why I think it's not a 2.12 regression. We just need to be prepared for what might come. Thanks,
On Wed, Apr 11, 2018 at 09:45:32AM +0800, Stefan Hajnoczi wrote: > On Tue, Apr 10, 2018 at 08:49:13PM +0800, Peter Xu wrote: > > cur_mon was only used in main loop so we don't really need that to be > > per-thread variable. Now it's possible that we have more than one > > thread to operate on it. Let's start to let it be per-thread variable. > > Trying to understand the reason for this patch: > > Are there any users of per-thread cur_mon? Currently no. But if considering future OOB-capable commands, they will modify cur_mon in monitor IOThread at least. > > or > > Does this patch fix a bug? No; currently we have no bug. But we have encounter the bug when we start to add more OOB commands. Here is the problem (Marc-Andre reported this, and I'll try to summarize): after we have valid OOB commands, monitor_qmp_dispatch_one() can be run not only in main thread, but also in monitor iothread. When that happens, both of them (main thread, and monitor iothread) can be modifying the cur_mon variable at the same time. [1] Considering that cur_mon is only used "just like" a stack variable, it should be perfectly fine we just make it as a per thread variable, hence this patch. > > > In case we'll create threads within a valid cur_mon setup, we'd better > > let the child threads to inherit the cur_mon from parent thread too. Do > > that for both posix and win32 threads. > > Without actual users I don't like this. It sounds like "let's make it > global just in case something needs it some day". > > It's ugly for QEMU's thread API to know about the monitor - that's a > layering violation. Yes, I'm sorry about it. Actually I don't like it too. But that seems to be an efficient and simple solution to me now. The ideal solution should be totally removing cur_mon, which is non-trivial. And for sure we can try to avoid layer violation. For example, we can have something like qemu_thread_register_variable(pthread_key_t), then monitor code can register the cur_mon pthread_key to the qemu thread module. That'll somehow achieve isolation between modules but I'm not sure whether that would be necessary for now, hence I chose the simple. > > If there's a legitimate need I think this patch might be necessary, but > I don't see enough justification to do this yet. The problem was described at [1]. Thanks,
On 11/04/2018 05:49, Peter Xu wrote: > On Wed, Apr 11, 2018 at 09:45:32AM +0800, Stefan Hajnoczi wrote: >> On Tue, Apr 10, 2018 at 08:49:13PM +0800, Peter Xu wrote: >>> cur_mon was only used in main loop so we don't really need that to be >>> per-thread variable. Now it's possible that we have more than one >>> thread to operate on it. Let's start to let it be per-thread variable. >> Trying to understand the reason for this patch: >> >> Are there any users of per-thread cur_mon? > > Currently no. But if considering future OOB-capable commands, they > will modify cur_mon in monitor IOThread at least. That's fine, but it shouldn't need the inheritance part. The monitor IOThread can set cur_mon when it starts. In general, relying on cur_mon should be avoided. Paolo
On Wed, Apr 11, 2018 at 11:23:57AM +0200, Paolo Bonzini wrote: > On 11/04/2018 05:49, Peter Xu wrote: > > On Wed, Apr 11, 2018 at 09:45:32AM +0800, Stefan Hajnoczi wrote: > >> On Tue, Apr 10, 2018 at 08:49:13PM +0800, Peter Xu wrote: > >>> cur_mon was only used in main loop so we don't really need that to be > >>> per-thread variable. Now it's possible that we have more than one > >>> thread to operate on it. Let's start to let it be per-thread variable. > >> Trying to understand the reason for this patch: > >> > >> Are there any users of per-thread cur_mon? > > > > Currently no. But if considering future OOB-capable commands, they > > will modify cur_mon in monitor IOThread at least. > > That's fine, but it shouldn't need the inheritance part. The monitor > IOThread can set cur_mon when it starts. Yeah, the inheritance will only make sure cur_mon be initialized always with correct value just like when we are without Out-Of-Band. For example, it's still possible a thread is created within a QMP handler. If without current change, the cur_mon in the new thread would be NULL. AFAIU even if cur_mon==NULL we should mostly be fine (e.g., error_vprintf will handle that case well). If any of you can help me confirm this, then I agree that this patch is not really needed. If so, maybe even we don't need to setup cur_mon at entry of monitor iothread, since cur_mon is always used in the way like: old_mon = cur_mon; cur_mon = xxx; ... (do something) cur_mon = old_mon; And it'll be fine old_mon==NULL here. Then IMHO the only thing we need to do is to mark cur_mon as per-thread and we're done. Thanks,
On 11/04/2018 11:35, Peter Xu wrote: > Yeah, the inheritance will only make sure cur_mon be initialized > always with correct value just like when we are without Out-Of-Band. > For example, it's still possible a thread is created within a QMP > handler. If without current change, the cur_mon in the new thread > would be NULL. > > AFAIU even if cur_mon==NULL we should mostly be fine (e.g., > error_vprintf will handle that case well). If any of you can help me > confirm this, then I agree that this patch is not really needed. Yes, though the solution is to not use error_vprintf from auxiliary threads. :) Just make sure all the communication happens through a struct that's passed in the thread entry point, and possibly bottom halves from the auxiliary thread to the monitor iothread. Paolo > If so, maybe even we don't need to setup cur_mon at entry of monitor > iothread, since cur_mon is always used in the way like:
On Wed, Apr 11, 2018 at 11:38:58AM +0200, Paolo Bonzini wrote: > On 11/04/2018 11:35, Peter Xu wrote: > > Yeah, the inheritance will only make sure cur_mon be initialized > > always with correct value just like when we are without Out-Of-Band. > > For example, it's still possible a thread is created within a QMP > > handler. If without current change, the cur_mon in the new thread > > would be NULL. > > > > AFAIU even if cur_mon==NULL we should mostly be fine (e.g., > > error_vprintf will handle that case well). If any of you can help me > > confirm this, then I agree that this patch is not really needed. > > Yes, though the solution is to not use error_vprintf from auxiliary > threads. :) Just make sure all the communication happens through a > struct that's passed in the thread entry point, and possibly bottom > halves from the auxiliary thread to the monitor iothread. Okay. :) Thanks for confirming. Then let me repost this patch without touching the qemu-threads. Btw, do you want me to repost the first patch separately too, or keep the code as is? I believe it depends on whether you treat that one as a cleanup or not. Thanks,
On 04/11/2018 04:48 AM, Peter Xu wrote: > Okay. :) Thanks for confirming. Then let me repost this patch without > touching the qemu-threads. > > Btw, do you want me to repost the first patch separately too, or keep > the code as is? I believe it depends on whether you treat that one as > a cleanup or not. Thanks, The first patch is no longer necessary for your new approach, but as it is a cleanup and you've already written it, it does not hurt to still send it as a separate cleanup patch useful in its own right.
On Wed, Apr 11, 2018 at 08:06:04AM -0500, Eric Blake wrote: > On 04/11/2018 04:48 AM, Peter Xu wrote: > > > Okay. :) Thanks for confirming. Then let me repost this patch without > > touching the qemu-threads. > > > > Btw, do you want me to repost the first patch separately too, or keep > > the code as is? I believe it depends on whether you treat that one as > > a cleanup or not. Thanks, > > The first patch is no longer necessary for your new approach, but as it > is a cleanup and you've already written it, it does not hurt to still > send it as a separate cleanup patch useful in its own right. Thank you, Eric. I will repost.
diff --git a/include/monitor/monitor.h b/include/monitor/monitor.h index d6ab70cae2..2ef5e04b37 100644 --- a/include/monitor/monitor.h +++ b/include/monitor/monitor.h @@ -6,7 +6,7 @@ #include "qapi/qapi-types-misc.h" #include "qemu/readline.h" -extern Monitor *cur_mon; +extern __thread Monitor *cur_mon; /* flags for monitor_init */ /* 0x01 unused */ diff --git a/include/qemu/thread-win32.h b/include/qemu/thread-win32.h index 3a05e3b3aa..f4d4cd96a1 100644 --- a/include/qemu/thread-win32.h +++ b/include/qemu/thread-win32.h @@ -39,6 +39,7 @@ typedef struct QemuThreadData QemuThreadData; struct QemuThread { QemuThreadData *data; unsigned tid; + Monitor *current_monitor; }; /* Only valid for joinable threads. */ diff --git a/monitor.c b/monitor.c index 51f4cf480f..5035e42364 100644 --- a/monitor.c +++ b/monitor.c @@ -266,7 +266,7 @@ static mon_cmd_t info_cmds[]; QmpCommandList qmp_commands, qmp_cap_negotiation_commands; -Monitor *cur_mon; +__thread Monitor *cur_mon; static QEMUClockType event_clock_type = QEMU_CLOCK_REALTIME; diff --git a/stubs/monitor.c b/stubs/monitor.c index e018c8f594..3890771bb5 100644 --- a/stubs/monitor.c +++ b/stubs/monitor.c @@ -3,7 +3,7 @@ #include "qemu-common.h" #include "monitor/monitor.h" -Monitor *cur_mon = NULL; +__thread Monitor *cur_mon; int monitor_get_fd(Monitor *mon, const char *name, Error **errp) { diff --git a/tests/test-util-sockets.c b/tests/test-util-sockets.c index acadd85e8f..6195a3ac36 100644 --- a/tests/test-util-sockets.c +++ b/tests/test-util-sockets.c @@ -69,7 +69,7 @@ int monitor_get_fd(Monitor *mon, const char *fdname, Error **errp) * stubs/monitor.c is defined, to make sure monitor.o is discarded * otherwise we get duplicate syms at link time. */ -Monitor *cur_mon; +__thread Monitor *cur_mon; void monitor_init(Chardev *chr, int flags) {} diff --git a/util/qemu-thread-posix.c b/util/qemu-thread-posix.c index 3ae96210d6..8d13da1b09 100644 --- a/util/qemu-thread-posix.c +++ b/util/qemu-thread-posix.c @@ -14,6 +14,7 @@ #include "qemu/thread.h" #include "qemu/atomic.h" #include "qemu/notify.h" +#include "monitor/monitor.h" #include "trace.h" static bool name_threads; @@ -486,6 +487,7 @@ typedef struct { void *(*start_routine)(void *); void *arg; char *name; + Monitor *current_monitor; } QemuThreadArgs; static void *qemu_thread_start(void *args) @@ -494,6 +496,9 @@ static void *qemu_thread_start(void *args) void *(*start_routine)(void *) = qemu_thread_args->start_routine; void *arg = qemu_thread_args->arg; + /* Inherit the cur_mon pointer from father thread */ + cur_mon = qemu_thread_args->current_monitor; + /* Attempt to set the threads name; note that this is for debug, so * we're not going to fail if we can't set it. */ @@ -533,6 +538,7 @@ void qemu_thread_create(QemuThread *thread, const char *name, qemu_thread_args->name = g_strdup(name); qemu_thread_args->start_routine = start_routine; qemu_thread_args->arg = arg; + qemu_thread_args->current_monitor = cur_mon; err = pthread_create(&thread->thread, &attr, qemu_thread_start, qemu_thread_args); diff --git a/util/qemu-thread-win32.c b/util/qemu-thread-win32.c index ab60c0d557..b5197dbc78 100644 --- a/util/qemu-thread-win32.c +++ b/util/qemu-thread-win32.c @@ -19,6 +19,7 @@ #include "qemu-common.h" #include "qemu/thread.h" #include "qemu/notify.h" +#include "monitor/monitor.h" #include "trace.h" #include <process.h> @@ -298,6 +299,7 @@ struct QemuThreadData { void *arg; short mode; NotifierList exit; + Monitor *current_monitor; /* Only used for joinable threads. */ bool exited; @@ -339,6 +341,9 @@ static unsigned __stdcall win32_start_routine(void *arg) void *(*start_routine)(void *) = data->start_routine; void *thread_arg = data->arg; + /* Inherit the cur_mon pointer from father thread */ + cur_mon = data->current_monitor; + qemu_thread_data = data; qemu_thread_exit(start_routine(thread_arg)); abort(); @@ -401,6 +406,7 @@ void qemu_thread_create(QemuThread *thread, const char *name, data->arg = arg; data->mode = mode; data->exited = false; + data->current_monitor = cur_mon; notifier_list_init(&data->exit); if (data->mode != QEMU_THREAD_DETACHED) {
cur_mon was only used in main loop so we don't really need that to be per-thread variable. Now it's possible that we have more than one thread to operate on it. Let's start to let it be per-thread variable. In case we'll create threads within a valid cur_mon setup, we'd better let the child threads to inherit the cur_mon from parent thread too. Do that for both posix and win32 threads. Signed-off-by: Peter Xu <peterx@redhat.com> --- include/monitor/monitor.h | 2 +- include/qemu/thread-win32.h | 1 + monitor.c | 2 +- stubs/monitor.c | 2 +- tests/test-util-sockets.c | 2 +- util/qemu-thread-posix.c | 6 ++++++ util/qemu-thread-win32.c | 6 ++++++ 7 files changed, 17 insertions(+), 4 deletions(-)