Patchwork [RFC] pyembed: integer python into QEMU

login
register
mail settings
Submitter Anthony Liguori
Date Jan. 16, 2012, 6:22 p.m.
Message ID <1326738174-14771-1-git-send-email-aliguori@us.ibm.com>
Download mbox | patch
Permalink /patch/136341/
State New
Headers show

Comments

Anthony Liguori - Jan. 16, 2012, 6:22 p.m.
This is something I started during 1.0-rc on a lark and spent some time last
night actually making work.  I'm sending it only to show that (1) it's possible
and (2) to get some input about what other people think as a longer term
direction.

I see a couple possibilities here:

 1. We could code portions of QEMU as call-outs to Python.  Command line
    parsing, GUI interface, etc. are all good candidates to be moved to Python.

 2. We could support loading of user-supplied Python modules.  This would be
    convenient as an extension mechanism since a HIL binding provide a clean
    interface that we could make more stable than just loading a random .so or
    including a .c file into the build.

 3. We could use this as an super tracing mechanism.  I can imagine using
    something like this in a production environment to trouble shoot a complex
    problem.

I'm not totally decided here myself on whether any of this is a good idea but
since this has come up so many times in the past, I figured it's at least worth
exploring.
---
 Makefile.objs |    2 +
 configure     |   32 ++++++++
 hw/hw.h       |   11 +++
 pyembed.c     |  244 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 pyembed.h     |    8 ++
 savevm.c      |   17 ++++
 vl.c          |    7 ++
 7 files changed, 321 insertions(+), 0 deletions(-)
 create mode 100644 pyembed.c
 create mode 100644 pyembed.h
Alex Bradbury - Jan. 16, 2012, 6:35 p.m.
On 16 January 2012 18:22, Anthony Liguori <aliguori@us.ibm.com> wrote:
> This is something I started during 1.0-rc on a lark and spent some time last
> night actually making work.  I'm sending it only to show that (1) it's possible
> and (2) to get some input about what other people think as a longer term
> direction.

At the risk of starting a language flame-war, have you considered
embedding something like Lua?The source of the core Lua interpreter is
small enough that it could be imported into the Qemu repository. There
may also be advantages to having access to LuaJIT (on supported
platforms) for its speed and FFI library [1] which could make it
feasible to prototype new device models in a high level language with
good performance.

Alex

[1] http://luajit.org/ext_ffi.html
Anthony Liguori - Jan. 16, 2012, 7:23 p.m.
On 01/16/2012 12:35 PM, Alex Bradbury wrote:
> On 16 January 2012 18:22, Anthony Liguori<aliguori@us.ibm.com>  wrote:
>> This is something I started during 1.0-rc on a lark and spent some time last
>> night actually making work.  I'm sending it only to show that (1) it's possible
>> and (2) to get some input about what other people think as a longer term
>> direction.
>
> At the risk of starting a language flame-war, have you considered
> embedding something like Lua?The source of the core Lua interpreter is
> small enough that it could be imported into the Qemu repository.

Generally, I don't think importing external source code is a Good Idea.  Part of 
the appeal to me about Python is the rich library that it brings in.

I'm less interested in Python for it's list comprehension syntax and more 
interested in it for it's config parsing library, RPC infrastructure, etc.

Plus, I'm reasonably confident that most QEMU developers have some experience 
with Python.  I'd wager that very few people have any practical experience with LUA.

So in terms of embedding, I think Python is the only reasonable path forward (if 
we think we should even head in this direction).

> There
> may also be advantages to having access to LuaJIT (on supported
> platforms) for its speed and FFI library [1] which could make it
> feasible to prototype new device models in a high level language with
> good performance.

Python has a simple FFI module (ctypes).

Regards,

Anthony Liguori


> Alex
>
> [1] http://luajit.org/ext_ffi.html
>
Stefan Weil - Jan. 16, 2012, 7:45 p.m.
Am 16.01.2012 20:23, schrieb Anthony Liguori:
> On 01/16/2012 12:35 PM, Alex Bradbury wrote:
>> On 16 January 2012 18:22, Anthony Liguori<aliguori@us.ibm.com>  wrote:
>>> This is something I started during 1.0-rc on a lark and spent some 
>>> time last
>>> night actually making work.  I'm sending it only to show that (1) 
>>> it's possible
>>> and (2) to get some input about what other people think as a longer 
>>> term
>>> direction.
>>
>> At the risk of starting a language flame-war, have you considered
>> embedding something like Lua?The source of the core Lua interpreter is
>> small enough that it could be imported into the Qemu repository.
>
> Generally, I don't think importing external source code is a Good 
> Idea.  Part of the appeal to me about Python is the rich library that 
> it brings in.
>
> I'm less interested in Python for it's list comprehension syntax and 
> more interested in it for it's config parsing library, RPC 
> infrastructure, etc.
>
> Plus, I'm reasonably confident that most QEMU developers have some 
> experience with Python.  I'd wager that very few people have any 
> practical experience with LUA.
>
> So in terms of embedding, I think Python is the only reasonable path 
> forward (if we think we should even head in this direction).
>

5 years ago, I experimented with an embedded Ruby interpreter. Although 
it worked
(at least to some degree), that's not a reasonable path. Standalone Ruby 
is a
good interpreter language, but embedding a Ruby interpreter is horrible.

Embedding TCL works well, but TCL is a little old fashioned, so that's 
also not a
reasonable path.

If we need an embedded interpreter, Python is a good choice, but Lua is 
also an excellent
embedded interpreter which would be useful for QEMU.

I'd prefer a competition with both of them instead of a single "only 
reasonable path".
The overhead of supporting two interpreters won't be very high, and it 
will allow more
people to contribute scripts in their favorite language.

Cheers,
Stefan Weil

Patch

diff --git a/Makefile.objs b/Makefile.objs
index d7a6539..710c0ec 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -293,6 +293,8 @@  hw-obj-$(CONFIG_DP8393X) += dp8393x.o
 hw-obj-$(CONFIG_DS1225Y) += ds1225y.o
 hw-obj-$(CONFIG_MIPSNET) += mipsnet.o
 
+hw-obj-$(CONFIG_PYTHON) += pyembed.o
+
 # Sound
 sound-obj-y =
 sound-obj-$(CONFIG_SB16) += sb16.o
diff --git a/configure b/configure
index f033438..381b387 100755
--- a/configure
+++ b/configure
@@ -185,6 +185,7 @@  opengl=""
 zlib="yes"
 guest_agent="yes"
 libiscsi=""
+libpython=""
 
 # parse CC options first
 for opt do
@@ -668,6 +669,10 @@  for opt do
   ;;
   --enable-libiscsi) libiscsi="yes"
   ;;
+  --disable-python) libpython="no"
+  ;;
+  --enable-python) libpython="yes"
+  ;;
   --enable-profiler) profiler="yes"
   ;;
   --enable-cocoa)
@@ -1061,6 +1066,8 @@  echo "  --enable-spice           enable spice"
 echo "  --enable-rbd             enable building the rados block device (rbd)"
 echo "  --disable-libiscsi       disable iscsi support"
 echo "  --enable-libiscsi        enable iscsi support"
+echo "  --disable-python         disable Python support"
+echo "  --enable-python          enable Python support"
 echo "  --disable-smartcard      disable smartcard support"
 echo "  --enable-smartcard       enable smartcard support"
 echo "  --disable-smartcard-nss  disable smartcard nss support"
@@ -2418,6 +2425,26 @@  fi
 
 
 ##########################################
+# Python support probe
+if test "$libpython" != "no"; then
+    if $pkg_config --modversion python > /dev/null 2>&1 ; then
+	libpython="yes"
+        python_cflags=`$pkg_config --cflags python 2>/dev/null`
+        python_libs=`$pkg_config --libs python 2>/dev/null`
+	pygtk_cflags=`$pkg_config --cflags pygobject-2.0 2>/dev/null`
+	pygtk_libs=`$pkg_config --libs pygobject-2.0 2>/dev/null`
+	pygtk_libs="$pygtk_libs -lpyglib-2.0-python2.7"
+	QEMU_CFLAGS="$python_cflags $pygtk_cflags $QEMU_CFLAGS"
+        LIBS="$python_libs $pygtk_libs $LIBS"
+    else
+	if test "$libpython" = "yes"; then
+	    feature_not_found "python"
+	fi
+	libpython="no"
+    fi
+fi
+
+##########################################
 # Do we need librt
 cat > $TMPC <<EOF
 #include <signal.h>
@@ -2829,6 +2856,7 @@  echo "nss used          $smartcard_nss"
 echo "usb net redir     $usb_redir"
 echo "OpenGL support    $opengl"
 echo "libiscsi support  $libiscsi"
+echo "Python support    $libpython"
 echo "build guest agent $guest_agent"
 
 if test "$sdl_too_old" = "yes"; then
@@ -3146,6 +3174,10 @@  if test "$libiscsi" = "yes" ; then
   echo "CONFIG_LIBISCSI=y" >> $config_host_mak
 fi
 
+if test "$libpython" = "yes" ; then
+  echo "CONFIG_PYTHON=y" >> $config_host_mak
+fi
+
 # XXX: suppress that
 if [ "$bsd" = "yes" ] ; then
   echo "CONFIG_BSD=y" >> $config_host_mak
diff --git a/hw/hw.h b/hw/hw.h
index ed20f5a..1b9076b 100644
--- a/hw/hw.h
+++ b/hw/hw.h
@@ -943,4 +943,15 @@  int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
                                    int required_for_version);
 void vmstate_unregister(DeviceState *dev, const VMStateDescription *vmsd,
                         void *opaque);
+
+typedef struct VMState
+{
+    const char *name;
+    int instance_id;
+    VMStateDescription *vmsd;
+    void *object;
+} VMState;
+
+GSList *vmstate_get_all(void);
+
 #endif
diff --git a/pyembed.c b/pyembed.c
new file mode 100644
index 0000000..9184973
--- /dev/null
+++ b/pyembed.c
@@ -0,0 +1,244 @@ 
+#include <Python.h>
+#include "qemu-common.h"
+#include "hw/hw.h"
+#include "pyembed.h"
+#include <pyglib.h>
+
+typedef struct OpaqueData
+{
+    void *read;
+    void *write;
+} OpaqueData;
+
+static OpaqueData opaque_data[64 * 1024];
+
+static void pyembed_ioport_write_trampoline(void *opaque, uint32_t address, int size, uint32_t value)
+{
+    OpaqueData *data = opaque;
+    PyObject *callback = data->write;
+    PyObject *arglist, *result;
+
+    arglist = Py_BuildValue("(IiI)", address, size, value);
+    result = PyObject_CallObject(callback, arglist);
+
+    Py_DECREF(arglist);
+    Py_DECREF(result);
+}
+
+static void pyembed_ioport_write_trampoline1(void *opaque, uint32_t address, uint32_t value)
+{
+    return pyembed_ioport_write_trampoline(opaque, address, 1, value);
+}
+
+static void pyembed_ioport_write_trampoline2(void *opaque, uint32_t address, uint32_t value)
+{
+    return pyembed_ioport_write_trampoline(opaque, address, 2, value);
+}
+
+static void pyembed_ioport_write_trampoline4(void *opaque, uint32_t address, uint32_t value)
+{
+    return pyembed_ioport_write_trampoline(opaque, address, 4, value);
+}
+
+static PyObject *pyembed_register_ioport_write(PyObject *self, PyObject *args)
+{
+    pio_addr_t addr;
+    int length;
+    PyObject *callback;
+    int ret;
+    
+    if (!PyArg_ParseTuple(args, "iiO", &addr, &length, &callback)) {
+        return NULL;
+    }
+
+    if (!PyCallable_Check(callback)) {
+        PyErr_SetString(PyExc_TypeError, "parameter must be callable");
+        return NULL;
+    }
+
+    Py_XINCREF(callback);
+
+    opaque_data[addr].write = callback;
+
+    ret = register_ioport_write(addr, length, 1,
+                                pyembed_ioport_write_trampoline1, &opaque_data[addr]);
+    ret = register_ioport_write(addr, length, 2,
+                                pyembed_ioport_write_trampoline2, &opaque_data[addr]);
+    ret = register_ioport_write(addr, length, 4,
+                                pyembed_ioport_write_trampoline4, &opaque_data[addr]);
+
+    if (ret == -1) {
+        Py_XDECREF(callback);
+    }
+
+    return Py_BuildValue("i", ret);
+}
+
+static uint32_t pyembed_ioport_read_trampoline(void *opaque, uint32_t address, int size)
+{
+    OpaqueData *data = opaque;
+    PyObject *callback = data->read;
+    PyObject *arglist, *result;
+    uint32_t ret = -1U;
+
+    arglist = Py_BuildValue("(Ii)", address, size);
+
+    result = PyObject_CallObject(callback, arglist);
+    if (!PyInt_Check(result)) {
+        PyErr_SetString(PyExc_TypeError, "return value must be an integer");
+        goto out;
+    }
+
+    ret = PyInt_AsUnsignedLongMask(result);
+
+out:
+    Py_DECREF(result);
+    Py_DECREF(arglist);
+    return ret;
+}
+
+static uint32_t pyembed_ioport_read_trampoline1(void *opaque, uint32_t address)
+{
+    return pyembed_ioport_read_trampoline(opaque, address, 1);
+}
+
+static uint32_t pyembed_ioport_read_trampoline2(void *opaque, uint32_t address)
+{
+    return pyembed_ioport_read_trampoline(opaque, address, 2);
+}
+
+static uint32_t pyembed_ioport_read_trampoline4(void *opaque, uint32_t address)
+{
+    return pyembed_ioport_read_trampoline(opaque, address, 4);
+}
+
+static PyObject *pyembed_register_ioport_read(PyObject *self, PyObject *args)
+{
+    pio_addr_t addr;
+    int length;
+    PyObject *callback;
+    int ret;
+    
+    if (!PyArg_ParseTuple(args, "iiO", &addr, &length, &callback)) {
+        PyErr_SetString(PyExc_TypeError, "invalid arguments");
+        return NULL;
+    }
+
+    if (!PyCallable_Check(callback)) {
+        PyErr_SetString(PyExc_TypeError, "parameter must be callable");
+        return NULL;
+    }
+
+    Py_XINCREF(callback);
+
+    opaque_data[addr].read = callback;
+
+    ret = register_ioport_read(addr, length, 1,
+                               pyembed_ioport_read_trampoline1, &opaque_data[addr]);
+    ret = register_ioport_read(addr, length, 2,
+                               pyembed_ioport_read_trampoline2, &opaque_data[addr]);
+    ret = register_ioport_read(addr, length, 4,
+                               pyembed_ioport_read_trampoline4, &opaque_data[addr]);
+
+    if (ret == -1) {
+        Py_XDECREF(callback);
+    }
+
+    return Py_BuildValue("i", ret);
+}
+
+static PyObject *pyembed_vmstate_get_all(PyObject *self, PyObject *args)
+{
+    PyObject *result;
+    GSList *lst;
+
+    result = PyDict_New();
+
+    for (lst = vmstate_get_all(); lst; lst = lst->next) {
+        VMState *vms = lst->data;
+        PyObject *entry;
+        VMStateField *field;
+        gchar *name;
+
+        if (vms->vmsd == NULL) {
+            continue;
+        }
+
+        entry = PyDict_New();
+        for (field = vms->vmsd->fields; field && field->name; field++) {
+            PyObject *value = NULL;
+
+            if (field->info == NULL) {
+                continue;
+            }
+
+            if (strcmp(field->info->name, "int8") == 0) {
+                value = PyInt_FromLong(*(int8_t *)(vms->object + field->offset));
+            } else if (strcmp(field->info->name, "int16") == 0) {
+                value = PyInt_FromLong(*(int16_t *)(vms->object + field->offset));
+            } else if (strcmp(field->info->name, "int32") == 0) {
+                value = PyInt_FromLong(*(int32_t *)(vms->object + field->offset));
+            } else if (strcmp(field->info->name, "int64") == 0) {
+                value = PyInt_FromLong(*(int64_t *)(vms->object + field->offset));
+            } else if (strcmp(field->info->name, "uint8") == 0) {
+                value = PyInt_FromLong(*(uint8_t *)(vms->object + field->offset));
+            } else if (strcmp(field->info->name, "uint16") == 0) {
+                value = PyInt_FromLong(*(uint16_t *)(vms->object + field->offset));
+            } else if (strcmp(field->info->name, "uint32") == 0) {
+                value = PyInt_FromLong(*(uint32_t *)(vms->object + field->offset));
+            } else if (strcmp(field->info->name, "uint64") == 0) {
+                value = PyInt_FromLong(*(uint64_t *)(vms->object + field->offset));
+            }
+
+            if (value) {
+                PyDict_SetItemString(entry, field->name, value);
+            }
+
+        }
+
+        name = g_strdup_printf("%s[%d]", vms->name, vms->instance_id);
+        PyDict_SetItemString(result, name, entry);
+        g_free(name);
+    }
+
+
+    return result;
+}    
+
+static PyMethodDef qemu_methods[] = {
+    { "register_ioport_read", pyembed_register_ioport_read, METH_VARARGS,
+      "Register to handle ioport reads" },
+    { "register_ioport_write", pyembed_register_ioport_write, METH_VARARGS,
+      "Register to handle ioport writes" },
+    { "vmstate_get_all", pyembed_vmstate_get_all, METH_VARARGS,
+      "Get VMState data for all devices" },
+    { },
+};
+
+void python_init(void)
+{
+    Py_Initialize();
+
+    Py_InitModule("qemu", qemu_methods);
+
+    pyglib_init();
+}
+
+void python_load(const char *filename)
+{
+    PyObject *name;
+    PyObject *module;
+
+    name = PyString_FromString(filename);
+    if (name == NULL) {
+        return;
+    }
+
+    module = PyImport_Import(name);
+    Py_DECREF(name);
+}
+
+void python_cleanup(void)
+{
+    Py_Finalize();
+}
diff --git a/pyembed.h b/pyembed.h
new file mode 100644
index 0000000..46c187e
--- /dev/null
+++ b/pyembed.h
@@ -0,0 +1,8 @@ 
+#ifndef QEMU_PYEMBED_H
+#define QEMU_PYEMBED_H
+
+void python_init(void);
+void python_load(const char *filename);
+void python_cleanup(void);
+
+#endif
diff --git a/savevm.c b/savevm.c
index f53cd4c..546475e 100644
--- a/savevm.c
+++ b/savevm.c
@@ -1468,6 +1468,23 @@  static int vmstate_load(QEMUFile *f, SaveStateEntry *se, int version_id)
     return vmstate_load_state(f, se->vmsd, se->opaque, version_id);
 }
 
+GSList *vmstate_get_all(void)
+{
+    SaveStateEntry *se;
+    GSList *lst = NULL;
+
+    QTAILQ_FOREACH(se, &savevm_handlers, entry) {
+        VMState *vms = g_malloc(sizeof(*vms));
+        vms->name = se->idstr;
+        vms->instance_id = se->instance_id;
+        vms->vmsd = se->vmsd;
+        vms->object = se->opaque;
+        lst = g_slist_append(lst, vms);
+    }
+
+    return lst;
+}
+
 static void vmstate_save(QEMUFile *f, SaveStateEntry *se)
 {
     if (!se->vmsd) {         /* Old style */
diff --git a/vl.c b/vl.c
index f5afed4..9255691 100644
--- a/vl.c
+++ b/vl.c
@@ -166,6 +166,7 @@  int main(int argc, char **argv)
 #include "arch_init.h"
 
 #include "ui/qemu-spice.h"
+#include "pyembed.h"
 
 //#define DEBUG_NET
 //#define DEBUG_SLIRP
@@ -2187,6 +2188,8 @@  int main(int argc, char **argv, char **envp)
     atexit(qemu_run_exit_notifiers);
     error_set_progname(argv[0]);
 
+    python_init();
+
     g_mem_set_vtable(&mem_trace);
     if (!g_thread_supported()) {
         g_thread_init(NULL);
@@ -3347,6 +3350,8 @@  int main(int argc, char **argv, char **envp)
     }
     qemu_add_globals();
 
+    python_load("myext");
+
     machine->init(ram_size, boot_devices,
                   kernel_filename, kernel_cmdline, initrd_filename, cpu_model);
 
@@ -3487,5 +3492,7 @@  int main(int argc, char **argv, char **envp)
     net_cleanup();
     res_free();
 
+    python_cleanup();
+
     return 0;
 }