| Message ID | 20240522150657.2378330-1-alexander.ivanov@virtuozzo.com |
|---|---|
| State | New |
| Headers | show |
| Series | qga: Add an interactive mode to guest-exec via VSOCK for Linux | expand |
There are two python scripts in the attachment: vsock_guest_exec_simple.py - simple example of a client; vsock_guest_exec_test.py - tests with different payload size. The last file should be copied to a guest VM. Edit SRV_PATH variable in the host copy of the script - there should be path to the directory containing a copy of the script in VM. Execute the host script with net arguments: ./vsock_guest_exec_test.py srv <VM_NAME>
On 5/22/24 17:06, Alexander Ivanov wrote: > Add an interactive mode to the guest-exec command in the QEMU Guest Agent > using the VSOCK communication mechanism. It enables interactive sessions > with the executed command in the guest, allowing real-time input/output. > > Introduce "interactive" mode in the GuestExecCaptureOutputMode enumeration > and add optional "cid" and "port" fields to the guest-exec response. In > such a way user can execute guest-exec command, get CID and port number > from the response and connect to the guest server. After connection user > can communicate with the started process. All the data transmitted to the > server is redirected to stdin. Data from stdout and stderr is redirected > to the client. All data blocks are preceded by 32-bit headers (network > byte order): most significant bit contains a sign of stream (stdout - 0, > stderr - 1), all the other bits contain the payload size. > > Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> > --- > qga/commands.c | 272 +++++++++++++++++++++++++++++++++++++++++-- > qga/qapi-schema.json | 11 +- > 2 files changed, 273 insertions(+), 10 deletions(-) FYI, libvirt recently added support for SSH over VSOCK: https://libvirt.org/ssh-proxy.html and it'll be released soon: https://libvirt.org/news.html Though, it requires (soon to be released) systemd to automatically set up SSHD to listen on VSOCK (but it can be configured manually). Michal
On Wed, May 22, 2024 at 05:06:57PM +0200, Alexander Ivanov wrote: > Add an interactive mode to the guest-exec command in the QEMU Guest Agent > using the VSOCK communication mechanism. It enables interactive sessions > with the executed command in the guest, allowing real-time input/output. > > Introduce "interactive" mode in the GuestExecCaptureOutputMode enumeration > and add optional "cid" and "port" fields to the guest-exec response. In > such a way user can execute guest-exec command, get CID and port number > from the response and connect to the guest server. After connection user > can communicate with the started process. All the data transmitted to the > server is redirected to stdin. Data from stdout and stderr is redirected > to the client. All data blocks are preceded by 32-bit headers (network > byte order): most significant bit contains a sign of stream (stdout - 0, > stderr - 1), all the other bits contain the payload size. Every patch to 'guest-exec' takes us torwards re-inventing yet more SSH/telnet functionality, but a poor simulation of it. For exmaple this still lacks any separation of stdout/stderr streams, just interleaving all their data back to the host. There is also zero access control facilities beyond turning off the 'guest-exec' command entirely. IMHO we should really consider "arbitrary command execution" to be something to be handled by a separate process. Let the guest OS admin decide separately from running QEMU GA, whether they want to enable arbitrary host processes to have a trival privileged backdoor into their guest. systemd now supports exposing SSH over VSOCK, and provides an SSH proxy in the host to connect to VMs, while libvirt also has added its own host SSH proxy to allow SSH based on libvirt VM name. For windows guests, there is something called PowerShell Direct which exposes PowerShell over vmbus under HyperV. Possibly that can be enabled in QEMU too if someone understands windows & vmbus enough... ? With regards, Daniel
On 5/22/24 18:10, Daniel P. Berrangé wrote: > On Wed, May 22, 2024 at 05:06:57PM +0200, Alexander Ivanov wrote: >> Add an interactive mode to the guest-exec command in the QEMU Guest Agent >> using the VSOCK communication mechanism. It enables interactive sessions >> with the executed command in the guest, allowing real-time input/output. >> >> Introduce "interactive" mode in the GuestExecCaptureOutputMode enumeration >> and add optional "cid" and "port" fields to the guest-exec response. In >> such a way user can execute guest-exec command, get CID and port number >> from the response and connect to the guest server. After connection user >> can communicate with the started process. All the data transmitted to the >> server is redirected to stdin. Data from stdout and stderr is redirected >> to the client. All data blocks are preceded by 32-bit headers (network >> byte order): most significant bit contains a sign of stream (stdout - 0, >> stderr - 1), all the other bits contain the payload size. > Every patch to 'guest-exec' takes us torwards re-inventing yet more > SSH/telnet functionality, but a poor simulation of it. For exmaple > this still lacks any separation of stdout/stderr streams, just > interleaving all their data back to the host. There is also zero > access control facilities beyond turning off the 'guest-exec' > command entirely. > > IMHO we should really consider "arbitrary command execution" to be > something to be handled by a separate process. Let the guest OS admin > decide separately from running QEMU GA, whether they want to enable > arbitrary host processes to have a trival privileged backdoor into > their guest. > > systemd now supports exposing SSH over VSOCK, and provides an SSH > proxy in the host to connect to VMs, while libvirt also has added > its own host SSH proxy to allow SSH based on libvirt VM name. > > For windows guests, there is something called PowerShell Direct > which exposes PowerShell over vmbus under HyperV. Possibly that > can be enabled in QEMU too if someone understands windows & vmbus > enough... ? > > With regards, > Daniel That makes a lot of sense. Why to support something that is already written. Though I have a note about Windows. The approach could be exactly the same - OpenSSH port for Windows is already known and on top of that VirtIO VSock driver is available too. Why not? Den
On Thu, May 23, 2024 at 09:12:51AM +0200, Denis V. Lunev wrote: > On 5/22/24 18:10, Daniel P. Berrangé wrote: > > On Wed, May 22, 2024 at 05:06:57PM +0200, Alexander Ivanov wrote: > > > Add an interactive mode to the guest-exec command in the QEMU Guest Agent > > > using the VSOCK communication mechanism. It enables interactive sessions > > > with the executed command in the guest, allowing real-time input/output. > > > > > > Introduce "interactive" mode in the GuestExecCaptureOutputMode enumeration > > > and add optional "cid" and "port" fields to the guest-exec response. In > > > such a way user can execute guest-exec command, get CID and port number > > > from the response and connect to the guest server. After connection user > > > can communicate with the started process. All the data transmitted to the > > > server is redirected to stdin. Data from stdout and stderr is redirected > > > to the client. All data blocks are preceded by 32-bit headers (network > > > byte order): most significant bit contains a sign of stream (stdout - 0, > > > stderr - 1), all the other bits contain the payload size. > > Every patch to 'guest-exec' takes us torwards re-inventing yet more > > SSH/telnet functionality, but a poor simulation of it. For exmaple > > this still lacks any separation of stdout/stderr streams, just > > interleaving all their data back to the host. There is also zero > > access control facilities beyond turning off the 'guest-exec' > > command entirely. > > > > IMHO we should really consider "arbitrary command execution" to be > > something to be handled by a separate process. Let the guest OS admin > > decide separately from running QEMU GA, whether they want to enable > > arbitrary host processes to have a trival privileged backdoor into > > their guest. > > > > systemd now supports exposing SSH over VSOCK, and provides an SSH > > proxy in the host to connect to VMs, while libvirt also has added > > its own host SSH proxy to allow SSH based on libvirt VM name. > > > > For windows guests, there is something called PowerShell Direct > > which exposes PowerShell over vmbus under HyperV. Possibly that > > can be enabled in QEMU too if someone understands windows & vmbus > > enough... ? > > > > That makes a lot of sense. Why to support something that is > already written. Though I have a note about Windows. The > approach could be exactly the same - OpenSSH port for Windows > is already known and on top of that VirtIO VSock driver is > available too. Why not? I've not tested it myself, but I would assume (hope) that the Powershell Direct feature is available in Windows guests "out of the box". The OpenSSH + VSock option would require extra user install work. I tend to favour things which will "just work" without extra config in the guest. With regards, Daniel
On 5/22/24 18:10, Daniel P. Berrangé wrote: > On Wed, May 22, 2024 at 05:06:57PM +0200, Alexander Ivanov wrote: >> Add an interactive mode to the guest-exec command in the QEMU Guest Agent >> using the VSOCK communication mechanism. It enables interactive sessions >> with the executed command in the guest, allowing real-time input/output. >> >> Introduce "interactive" mode in the GuestExecCaptureOutputMode enumeration >> and add optional "cid" and "port" fields to the guest-exec response. In >> such a way user can execute guest-exec command, get CID and port number >> from the response and connect to the guest server. After connection user >> can communicate with the started process. All the data transmitted to the >> server is redirected to stdin. Data from stdout and stderr is redirected >> to the client. All data blocks are preceded by 32-bit headers (network >> byte order): most significant bit contains a sign of stream (stdout - 0, >> stderr - 1), all the other bits contain the payload size. > Every patch to 'guest-exec' takes us torwards re-inventing yet more > SSH/telnet functionality, but a poor simulation of it. For exmaple > this still lacks any separation of stdout/stderr streams, just There IS separation of stdout/stderr. Receiving data on the host you can see from which stream it is. > interleaving all their data back to the host. There is also zero > access control facilities beyond turning off the 'guest-exec' > command entirely. > > IMHO we should really consider "arbitrary command execution" to be > something to be handled by a separate process. Let the guest OS admin > decide separately from running QEMU GA, whether they want to enable > arbitrary host processes to have a trival privileged backdoor into > their guest. > > systemd now supports exposing SSH over VSOCK, and provides an SSH > proxy in the host to connect to VMs, while libvirt also has added > its own host SSH proxy to allow SSH based on libvirt VM name. > > For windows guests, there is something called PowerShell Direct > which exposes PowerShell over vmbus under HyperV. Possibly that > can be enabled in QEMU too if someone understands windows & vmbus > enough... ? > > With regards, > Daniel Otherwise, you are right, it makes sense to use SSH over VSOCK. Thank you.
Alexander Ivanov <alexander.ivanov@virtuozzo.com> writes: > There are two python scripts in the attachment: > vsock_guest_exec_simple.py - simple example of a client; > vsock_guest_exec_test.py - tests with different payload size. > > The last file should be copied to a guest VM. Edit SRV_PATH variable > in the host copy of the script - there should be path to the directory > containing a copy of the script in VM. Execute the host script with > net arguments: > ./vsock_guest_exec_test.py srv <VM_NAME> Maybe these would best live in contrib/vsock with a README?
diff --git a/qga/commands.c b/qga/commands.c index 88c1c99fe5..377a79c816 100644 --- a/qga/commands.c +++ b/qga/commands.c @@ -20,6 +20,11 @@ #include "qemu/cutils.h" #include "commands-common.h" +#ifdef CONFIG_LINUX +#include <sys/ioctl.h> +#include <linux/vm_sockets.h> +#endif + /* Maximum captured guest-exec out_data/err_data - 16MB */ #define GUEST_EXEC_MAX_OUTPUT (16 * 1024 * 1024) /* Allocation and I/O buffer for reading guest-exec out_data/err_data - 4KB */ @@ -92,6 +97,27 @@ struct GuestExecIOData { }; typedef struct GuestExecIOData GuestExecIOData; +#define GE_INT_IO_SIZE (256 * 1024) +#define GE_INT_STREAM_MASK 0x80000000 + +struct GEIntPacket { + uint32_t header; + gchar buf[GE_INT_IO_SIZE]; +} __attribute__((aligned(1))); +typedef struct GEIntPacket GEIntPacket; + +struct GEIntData { + unsigned int cid; + unsigned int port; + GIOChannel *ch_srv; + GIOChannel *ch_clt; + GIOChannel *ch_in; + GIOChannel *ch_out; + GIOChannel *ch_err; + GEIntPacket packet; +}; +typedef struct GEIntData GEIntData; + struct GuestExecInfo { GPid pid; int64_t pid_numeric; @@ -101,6 +127,7 @@ struct GuestExecInfo { GuestExecIOData in; GuestExecIOData out; GuestExecIOData err; + GEIntData *int_data; QTAILQ_ENTRY(GuestExecInfo) next; }; typedef struct GuestExecInfo GuestExecInfo; @@ -257,6 +284,194 @@ static char **guest_exec_get_args(const strList *entry, bool log) return args; } +#ifdef CONFIG_LINUX +static void guest_exec_close_channel(GIOChannel *ch) +{ + g_io_channel_shutdown(ch, true, NULL); + g_io_channel_unref(ch); +} + +static void guest_exec_interactive_cleanup(GuestExecInfo *gei) +{ + GEIntData *data = gei->int_data; + + guest_exec_close_channel(data->ch_clt); + guest_exec_close_channel(data->ch_srv); + guest_exec_close_channel(data->ch_in); + guest_exec_close_channel(data->ch_out); + guest_exec_close_channel(data->ch_err); + + g_free(data); + gei->int_data = NULL; +} + +static gboolean guest_exec_interactive_watch(GIOChannel *ch, GIOCondition cond, + gpointer data_) +{ + GuestExecInfo *gei = (GuestExecInfo *)data_; + GEIntData *data = gei->int_data; + gsize size, bytes_written; + GIOStatus gstatus; + GError *gerr = NULL; + GIOChannel *dst_ch; + gchar *p; + + if (data == NULL) { + return false; + } + + if (cond == G_IO_HUP || cond == G_IO_ERR) { + goto close; + } + + gstatus = g_io_channel_read_chars(ch, data->packet.buf, + sizeof(data->packet.buf), &size, NULL); + + if (gstatus == G_IO_STATUS_EOF || gstatus == G_IO_STATUS_ERROR) { + if (gerr) { + g_warning("qga: i/o error reading from a channel: %s", + gerr->message); + g_error_free(gerr); + } + goto close; + } + + if (ch == data->ch_clt) { + dst_ch = data->ch_in; + p = data->packet.buf; + } else { + assert(size < GE_INT_STREAM_MASK); + + dst_ch = data->ch_clt; + p = (gchar *)&(data->packet); + data->packet.header = htonl(size); + if (ch == data->ch_err) { + data->packet.header |= htonl(GE_INT_STREAM_MASK); + } + size += sizeof(data->packet.header); + } + + do { + gstatus = g_io_channel_write_chars(dst_ch, p, size, + &bytes_written, &gerr); + + if (gstatus == G_IO_STATUS_EOF || gstatus == G_IO_STATUS_ERROR) { + if (gerr) { + g_warning("qga: i/o error writing to a channel: %s", + gerr->message); + g_error_free(gerr); + } + goto close; + } + size -= bytes_written; + p += bytes_written; + } while (size > 0); + + return true; + +close: + guest_exec_interactive_cleanup(gei); + return false; +} + +static gboolean +guest_exec_interactive_accept_watch(GIOChannel *ch, GIOCondition cond, + gpointer data_) +{ + GuestExecInfo *gei = (GuestExecInfo *)data_; + GEIntData *data = gei->int_data; + int fd; + + if (cond == G_IO_HUP || cond == G_IO_ERR) { + goto close; + } + + fd = accept(g_io_channel_unix_get_fd(ch), NULL, NULL); + if (fd < 0) { + goto close; + } + + data->ch_clt = g_io_channel_unix_new(fd); + g_io_channel_set_encoding(data->ch_clt, NULL, NULL); + g_io_channel_set_buffered(data->ch_clt, false); + g_io_channel_set_close_on_unref(data->ch_clt, true); + + g_io_add_watch(data->ch_clt, G_IO_IN | G_IO_HUP, + guest_exec_interactive_watch, gei); + g_io_add_watch(data->ch_out, G_IO_IN | G_IO_HUP, + guest_exec_interactive_watch, gei); + g_io_add_watch(data->ch_err, G_IO_IN | G_IO_HUP, + guest_exec_interactive_watch, gei); + return false; + +close: + guest_exec_interactive_cleanup(gei); + return false; +} + +static int get_cid(unsigned int *cid) +{ + int fd, ret; + fd = open("/dev/vsock", O_RDONLY); + if (fd == -1) { + return errno; + } + ret = ioctl(fd, IOCTL_VM_SOCKETS_GET_LOCAL_CID, cid); + close(fd); + return ret; +} + +static int guest_exec_interactive_listen(GuestExecInfo *gei) +{ + struct sockaddr_vm server_addr; + socklen_t len; + int fd, res; + GEIntData *data = (GEIntData *)gei->int_data; + + if (get_cid(&data->cid) != 0) { + slog("Can't get CID: %s", strerror(errno)); + return -1; + } + + fd = socket(AF_VSOCK, SOCK_STREAM, 0); + if (fd == -1) { + slog("Socket creation error: %s", strerror(errno)); + return -1; + } + + memset(&server_addr, 0, sizeof(server_addr)); + server_addr.svm_family = AF_VSOCK; + server_addr.svm_port = VMADDR_PORT_ANY; + server_addr.svm_cid = VMADDR_CID_ANY; + + if (bind(fd, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) { + slog("Bind error: %s", strerror(errno)); + goto err; + } + + len = sizeof(struct sockaddr_vm); + res = getsockname(fd, (struct sockaddr *)&server_addr, &len); + if (res == -1) { + slog("Can't get port: %s", strerror(errno)); + goto err; + } + + if (listen(fd, 1) == -1) { + slog("Can't listen port %d: %s", server_addr.svm_port, strerror(errno)); + goto err; + } + + data->port = server_addr.svm_port; + data->ch_srv = g_io_channel_unix_new(fd); + g_io_add_watch(data->ch_srv, G_IO_IN | G_IO_HUP, + guest_exec_interactive_accept_watch, gei); + return 0; +err: + close(fd); + return -1; +} +#endif + static void guest_exec_child_watch(GPid pid, gint status, gpointer data) { GuestExecInfo *gei = (GuestExecInfo *)data; @@ -424,6 +639,8 @@ GuestExec *qmp_guest_exec(const char *path, GSpawnFlags flags; bool has_output = false; bool has_merge = false; + bool interactive = false; + GuestExecCaptureOutputMode output_mode; g_autofree uint8_t *input = NULL; size_t ninput = 0; @@ -465,6 +682,11 @@ GuestExec *qmp_guest_exec(const char *path, has_output = true; has_merge = true; break; +#endif +#ifdef CONFIG_LINUX + case GUEST_EXEC_CAPTURE_OUTPUT_MODE_INTERACTIVE: + interactive = true; + break; #endif case GUEST_EXEC_CAPTURE_OUTPUT_MODE__MAX: /* Silence warning; impossible branch */ @@ -472,8 +694,10 @@ GuestExec *qmp_guest_exec(const char *path, } ret = g_spawn_async_with_pipes(NULL, argv, envp, flags, - guest_exec_task_setup, &has_merge, &pid, input_data ? &in_fd : NULL, - has_output ? &out_fd : NULL, has_output ? &err_fd : NULL, &gerr); + guest_exec_task_setup, &has_merge, &pid, + (input_data || interactive) ? &in_fd : NULL, + (has_output || interactive) ? &out_fd : NULL, + (has_output || interactive) ? &err_fd : NULL, &gerr); if (!ret) { error_setg(errp, QERR_QGA_COMMAND_FAILED, gerr->message); g_error_free(gerr); @@ -485,9 +709,14 @@ GuestExec *qmp_guest_exec(const char *path, gei = guest_exec_info_add(pid); gei->has_output = has_output; + g_child_watch_add(pid, guest_exec_child_watch, gei); - if (input_data) { + if (interactive) { + gei->int_data = g_malloc0(sizeof(GEIntData)); + } + + if (input_data || interactive) { gei->in.data = g_steal_pointer(&input); gei->in.size = ninput; #ifdef G_OS_WIN32 @@ -499,10 +728,14 @@ GuestExec *qmp_guest_exec(const char *path, g_io_channel_set_buffered(in_ch, false); g_io_channel_set_flags(in_ch, G_IO_FLAG_NONBLOCK, NULL); g_io_channel_set_close_on_unref(in_ch, true); - g_io_add_watch(in_ch, G_IO_OUT, guest_exec_input_watch, &gei->in); + if (interactive) { + gei->int_data->ch_in = in_ch; + } else { + g_io_add_watch(in_ch, G_IO_OUT, guest_exec_input_watch, &gei->in); + } } - if (has_output) { + if (has_output || interactive) { #ifdef G_OS_WIN32 out_ch = g_io_channel_win32_new_fd(out_fd); err_ch = g_io_channel_win32_new_fd(err_fd); @@ -516,12 +749,33 @@ GuestExec *qmp_guest_exec(const char *path, g_io_channel_set_buffered(err_ch, false); g_io_channel_set_close_on_unref(out_ch, true); g_io_channel_set_close_on_unref(err_ch, true); - g_io_add_watch(out_ch, G_IO_IN | G_IO_HUP, - guest_exec_output_watch, &gei->out); - g_io_add_watch(err_ch, G_IO_IN | G_IO_HUP, - guest_exec_output_watch, &gei->err); + + if (interactive) { + gei->int_data->ch_out = out_ch; + gei->int_data->ch_err = err_ch; + } else { + g_io_add_watch(out_ch, G_IO_IN | G_IO_HUP, + guest_exec_output_watch, &gei->out); + g_io_add_watch(err_ch, G_IO_IN | G_IO_HUP, + guest_exec_output_watch, &gei->err); + } } +#ifdef CONFIG_LINUX + if (interactive) { + if (guest_exec_interactive_listen(gei) != 0) { + QTAILQ_REMOVE(&guest_exec_state.processes, gei, next); + g_free(gei->int_data); + g_free(gei); + goto done; + } + ge->has_cid = true; + ge->cid = gei->int_data->cid; + ge->has_port = true; + ge->port = gei->int_data->port; + } +#endif + done: g_free(argv); g_free(envp); diff --git a/qga/qapi-schema.json b/qga/qapi-schema.json index d5af155007..77efa847af 100644 --- a/qga/qapi-schema.json +++ b/qga/qapi-schema.json @@ -1262,10 +1262,16 @@ # # @pid: pid of child process in guest OS # +# @cid: context identifier for interactive mode +# +# @port: port number for interactive mode +# # Since: 2.5 ## { 'struct': 'GuestExec', - 'data': { 'pid': 'int'} } + 'data': { 'pid': 'int', + '*cid': 'int', + '*port': 'int' } } ## # @GuestExecCaptureOutputMode: @@ -1284,10 +1290,13 @@ # @merged: capture both stdout and stderr, but merge together into # out-data. Not effective on windows guests. # +# @interactive: stdin, stdout and stderr are transmitted via Vsock +# # Since: 8.0 ## { 'enum': 'GuestExecCaptureOutputMode', 'data': [ 'none', 'stdout', 'stderr', 'separated', + { 'name': 'interactive', 'if': 'CONFIG_LINUX' }, { 'name': 'merged', 'if': { 'not': 'CONFIG_WIN32' } } ] } ##
Add an interactive mode to the guest-exec command in the QEMU Guest Agent using the VSOCK communication mechanism. It enables interactive sessions with the executed command in the guest, allowing real-time input/output. Introduce "interactive" mode in the GuestExecCaptureOutputMode enumeration and add optional "cid" and "port" fields to the guest-exec response. In such a way user can execute guest-exec command, get CID and port number from the response and connect to the guest server. After connection user can communicate with the started process. All the data transmitted to the server is redirected to stdin. Data from stdout and stderr is redirected to the client. All data blocks are preceded by 32-bit headers (network byte order): most significant bit contains a sign of stream (stdout - 0, stderr - 1), all the other bits contain the payload size. Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> --- qga/commands.c | 272 +++++++++++++++++++++++++++++++++++++++++-- qga/qapi-schema.json | 11 +- 2 files changed, 273 insertions(+), 10 deletions(-)