RFC: DSO (dynamic shared objects) support

Hello.

I looked at what's needed to support DSO (dynamic shared objects)
in qemu, in order to be able to split functionality into loadable
"plugins".  It isn't exactly difficult, but a few steps are needed
still.

The whole thing is about splitting the functionality into plugins
from one, single, qemu build, for now, not about allowing 3rd-party
plugins to be distributed outside of qemu.

The plan is to keep the plugins in a single directory (or several
subdirectories, by type of plugins - for block, machine and other),
and load them during startup.


First of all, we need a global config switch/check for this kind of
service.  On a POSIX system this should make the executable to be
linked with -rdynamic option, in order to make all symbols in the
executable to be available to plugins.

This is an interesting case.  We may end up in some symbols from
qemu libraries not linked in when building the executable, but
that symbol may be used in some plugin.  Such situations, if will
happen, will need to be deal with on a case-by-case basis somehow.


Next, the executable should be linked with -ldl (again, for a POSIX
system), to be able to dynamically load anything.  This goes on the
same line as -rdynamic -- once we enable plugins, both linker options
are enabled.


Next, the module loading support.  On POSIX it is <dlfcn.h> and
dlopen(3), this one is trivial.

I think the best is to hook up into util/module.c.  Currently,
we explicitly call modue_call_init(type) for module types the
executable is interested with.  So in that call we may also
load all plugins of the given type just before running all the
module init functions.

For modules themselves, nothing changes -- __attribute__((constructor))
does the same thing be it in a DSO or statically linked into
executable.


Next, and this is the most complex part.  The build system for
modules, and configuring it.   I heard there were plans to use
something like kbuild system for that, has anything been done
in this context?

With current config/build system, the following changes are
needed:

 o split individual libs from libs_softmmu into their own
   variables.

 o allow obj-m (and similar) in addition to obj-y, with build
   flags and rules to produce an .so.

 o ./configure --{enable,disable}-foo -- in addition to
   that, I think --module-foo will be ok.  If not, well,
   any suggestion how to name it?

 o modules installing

 o switching some modules to allow building them modular

 o adding more types of modules, for example a new DISPLAY
   type.


Below is a sample implementation of the loading part (without
configure checks, just POC).

Thanks,

/mjt

On 06/18/13 13:37, Michael Tokarev wrote:

> First of all, we need a global config switch/check for this kind of
> service.  On a POSIX system this should make the executable to be
> linked with -rdynamic option, in order to make all symbols in the
> executable to be available to plugins.
> 
> This is an interesting case.  We may end up in some symbols from
> qemu libraries not linked in when building the executable, but
> that symbol may be used in some plugin.  Such situations, if will
> happen, will need to be deal with on a case-by-case basis somehow.

       -rdynamic
           Pass the flag -export-dynamic to the ELF linker, on
           targets that support it. This instructs the linker to
           add all symbols, not only used ones, to the dynamic
           symbol table. This option is needed for some uses of
           "dlopen" or to allow obtaining backtraces from within
           a program.

I think the problem you describe doesn't threaten therefore. OTOH link
time and executable size could easily grow.


> Next, the executable should be linked with -ldl (again, for a POSIX
> system), to be able to dynamically load anything.  This goes on the
> same line as -rdynamic -- once we enable plugins, both linker options
> are enabled.
> 
> 
> Next, the module loading support.  On POSIX it is <dlfcn.h> and
> dlopen(3), this one is trivial.

dlopen(), yes, dlsym(), yes; type-casting the dlsym()-returned (void*)
to the correct function pointer type, or the pointer-to-struct type,
where the struct has a bunch of function pointers: not so trivial; some
new type declarations would be necessary.


> I think the best is to hook up into util/module.c.  Currently,
> we explicitly call modue_call_init(type) for module types the
> executable is interested with.  So in that call we may also
> load all plugins of the given type just before running all the
> module init functions.
> 
> For modules themselves, nothing changes -- __attribute__((constructor))
> does the same thing be it in a DSO or statically linked into
> executable.

Ah. Basically, you wouldn't use dlsym() at all -- modules would register
their stuff in their constructors, and the main executable would search
by name?


> +        if (dlopen(path, RTLD_NOW) == NULL)

For inter-module sybmol resolution (motivated by any functional
dependencies, of course), RTLD_GLOBAL would be needed too (and then of
course order of loading would matter).


> +        {
> +            fprintf(stderr, "warning: unable to load plugin %s\n", path);

dlerror() is standard, and it provides very good error messages on
GNU/Linux.

Laszlo

On 06/18/13 13:37, Michael Tokarev wrote:

> Next, and this is the most complex part.  The build system for
> modules, and configuring it.   I heard there were plans to use
> something like kbuild system for that, has anything been done
> in this context?

Sorry for responding separately... Some module/symbol versioning will be
necessary, as the binding will be delayed until runtime, and users
*will* mix and match qemu and modules, and get random crashes. AFAIK
versioning is a good way to avoid that (you would get an error in
dlopen(), and dlerror() would pinpoint problem).

No idea how to implement this though, except I expect there's no chance
to do it halfway portably without libtool.

POSIX is very laconic regarding shared libraries, for example

  http://pubs.opengroup.org/onlinepubs/9699919799/utilities/c99.html

in SUSv4 doesn't say anything about them.

Laszlo

18.06.2013 16:17, Laszlo Ersek wrote:
> On 06/18/13 13:37, Michael Tokarev wrote:
> 
>> Next, and this is the most complex part.  The build system for
>> modules, and configuring it.   I heard there were plans to use
>> something like kbuild system for that, has anything been done
>> in this context?
> 
> Sorry for responding separately... Some module/symbol versioning will be
> necessary, as the binding will be delayed until runtime, and users
> *will* mix and match qemu and modules, and get random crashes. AFAIK
> versioning is a good way to avoid that (you would get an error in
> dlopen(), and dlerror() would pinpoint problem).
> 
> No idea how to implement this though, except I expect there's no chance
> to do it halfway portably without libtool.

Well, I think this is trivial really.

Duding build, we re-#define register_module_init() to have a version
number (or even a build id), for example register_module_init_1_6_0().
This is a function which each plugin will call inside its constructor.
Being unable to find this symbol in calling qemu it will fail to load.

Thanks,

/mjt

18.06.2013 16:19, Michael Tokarev пишет:
> 18.06.2013 16:17, Laszlo Ersek wrote:
>> On 06/18/13 13:37, Michael Tokarev wrote:
>>
>>> Next, and this is the most complex part.  The build system for
>>> modules, and configuring it.   I heard there were plans to use
>>> something like kbuild system for that, has anything been done
>>> in this context?
>>
>> Sorry for responding separately... Some module/symbol versioning will be
>> necessary, as the binding will be delayed until runtime, and users
>> *will* mix and match qemu and modules, and get random crashes. AFAIK
>> versioning is a good way to avoid that (you would get an error in
>> dlopen(), and dlerror() would pinpoint problem).
>>
>> No idea how to implement this though, except I expect there's no chance
>> to do it halfway portably without libtool.
> 
> Well, I think this is trivial really.
> 
> Duding build, we re-#define register_module_init() to have a version
> number (or even a build id), for example register_module_init_1_6_0().
> This is a function which each plugin will call inside its constructor.
> Being unable to find this symbol in calling qemu it will fail to load.

$ ./x86_64-softmmu/qemu-system-x86_64 -hda http://foo/bar
warning: unable to load plugin plugins/block_curl.so: plugins/block_curl.so: undefined symbol: register_module_init_1_6_0
qemu-system-x86_64: -hda http://foo/bar: could not open disk image http://foo/bar: No such file or directory

Looks clear enough.

Also, we may add version number to the plugin name too.

/mjt

Hi,

On Tue, Jun 18, 2013 at 6:37 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
> Hello.
>
> I looked at what's needed to support DSO (dynamic shared objects)
> in qemu, in order to be able to split functionality into loadable
> "plugins".  It isn't exactly difficult, but a few steps are needed
> still.

As it turns out, this is an area of interest of mine too.  I put up a
quick page describing some ideas a few weeks ago.

http://wiki.qemu.org/Features/Modules

> The whole thing is about splitting the functionality into plugins
> from one, single, qemu build, for now, not about allowing 3rd-party
> plugins to be distributed outside of qemu.

Ack.  Really critical and important distinction.

> The plan is to keep the plugins in a single directory (or several
> subdirectories, by type of plugins - for block, machine and other),
> and load them during startup.

Ack.

> First of all, we need a global config switch/check for this kind of
> service.  On a POSIX system this should make the executable to be
> linked with -rdynamic option, in order to make all symbols in the
> executable to be available to plugins.
>
> This is an interesting case.  We may end up in some symbols from
> qemu libraries not linked in when building the executable, but
> that symbol may be used in some plugin.  Such situations, if will
> happen, will need to be deal with on a case-by-case basis somehow.

I wouldn't worry about this.  There are a lot of things that can be
made modules today that don't export any symbols at all.  The other
case to consider is things that expose QMP commands.  We'll need to
adjust how commands are registered such that the QMP commands are
registered as part of modules.

But that doesn't have to be in the initial implementation.

> Next, the executable should be linked with -ldl (again, for a POSIX
> system), to be able to dynamically load anything.  This goes on the
> same line as -rdynamic -- once we enable plugins, both linker options
> are enabled.
>
>
> Next, the module loading support.  On POSIX it is <dlfcn.h> and
> dlopen(3), this one is trivial.

https://developer.gnome.org/glib/stable/glib-Dynamic-Loading-of-Modules.html

glib has a nice interface for modules that is cross platform.

> I think the best is to hook up into util/module.c.  Currently,
> we explicitly call modue_call_init(type) for module types the
> executable is interested with.  So in that call we may also
> load all plugins of the given type just before running all the
> module init functions.
>
> For modules themselves, nothing changes -- __attribute__((constructor))
> does the same thing be it in a DSO or statically linked into
> executable.

We ought to start out by loading modules *before* doing any module
init calls.  That will keep things nice and simple.  In the long term,
we'll need to look at each subsystem and see what it takes to load
modules after init.

> Next, and this is the most complex part.  The build system for
> modules, and configuring it.   I heard there were plans to use
> something like kbuild system for that, has anything been done
> in this context?

GSoC just kicked off yesterday.  Paolo can perhaps shed some light on
what the plans are.

> With current config/build system, the following changes are
> needed:
>
>  o split individual libs from libs_softmmu into their own
>    variables.
>
>  o allow obj-m (and similar) in addition to obj-y, with build
>    flags and rules to produce an .so.

We also need a way to define the module targets.  My Makefile-fu is
not all that great but I suspect we can do something like:

libqemu-%.ko: $(eval $(obj-%-m))

I don't think that works but that's the rough idea.  We would then
need to define each module target by hand but that's probably
reasonable in the short term until we have kconfig.

>  o ./configure --{enable,disable}-foo -- in addition to
>    that, I think --module-foo will be ok.  If not, well,
>    any suggestion how to name it?

You don't really need to touch configure to start with.  We can just
rely on people modifying the config*.mak file.  We can wait for a
proper menuconfig via kconfig.

>
>  o modules installing
>
>  o switching some modules to allow building them modular

Pretty much any device is a candidate here.

Regards,

Anthony Liguori

>  o adding more types of modules, for example a new DISPLAY
>    type.
>
>
> Below is a sample implementation of the loading part (without
> configure checks, just POC).
>
> Thanks,
>
> /mjt
>
> --- a/util/module.c
> +++ b/util/module.c
> @@ -17,6 +17,62 @@
>  #include "qemu/queue.h"
>  #include "qemu/module.h"
>
> +#define CONFIG_DSO_POSIX
> +
> +#ifdef CONFIG_DSO_POSIX
> +
> +#include <dlfcn.h>
> +#include <dirent.h>
> +
> +static const char *plugins_dir = "plugins";
> +
> +static const char *module_init_types[MODULE_INIT_MAX] = {
> +   "block", "machine", "qapi", "qom"
> +};
> +
> +static void load_modules(module_init_type type)
> +{
> +    DIR *dir;
> +    struct dirent *de;
> +    char path[PATH_MAX];
> +
> +    const char *typestr = module_init_types[type];
> +    const size_t typelen = strlen(typestr);
> +
> +fprintf(stderr, "loading modules of type %s\n", typestr);
> +
> +    dir = opendir(plugins_dir);
> +    if (!dir)
> +    {
> +        return;
> +    }
> +    while((de = readdir(dir)) != NULL)
> +    {
> +        size_t len = strlen(de->d_name);
> +        if (len <= typelen
> +            || memcmp(de->d_name, typestr, typelen) != 0
> +            || de->d_name[typelen] != '_'
> +            || strcmp(de->d_name + len - 3, ".so") != 0)
> +        {
> +            continue;
> +        }
> +        snprintf(path, sizeof(path), "%s/%s", plugins_dir, de->d_name);
> +        if (dlopen(path, RTLD_NOW) == NULL)
> +        {
> +            fprintf(stderr, "warning: unable to load plugin %s\n", path);
> +            continue;
> +        }
> +        fprintf(stderr, "loaded %s\n", path);
> +    }
> +    closedir(dir);
> +}
> +
> +#else
> +
> +static void load_modules(module_init_type type) {}
> +
> +#endif
> +
>  typedef struct ModuleEntry
>  {
>      void (*init)(void);
> @@ -74,6 +130,7 @@ void module_call_init(module_init_type type)
>      ModuleEntry *e;
>
>      l = find_type(type);
> +    load_modules(type);
>
>      QTAILQ_FOREACH(e, l, node) {
>          e->init();
>

Il 18/06/2013 14:05, Laszlo Ersek ha scritto:
>> > +        if (dlopen(path, RTLD_NOW) == NULL)
> For inter-module sybmol resolution (motivated by any functional
> dependencies, of course), RTLD_GLOBAL would be needed too (and then of
> course order of loading would matter).

I think we want to prohibit this specifically and use RTLD_LOCAL, at
least in the beginning.  If you have dependencies, just put them in a
single module.  As Michael mentioned, this is much simpler than the
Linux kernel's module stuff.

Paolo

On 18 June 2013 13:42, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On Tue, Jun 18, 2013 at 6:37 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
>>  o switching some modules to allow building them modular
>
> Pretty much any device is a candidate here.

Before you can do that you need to convert them so we can build
them once rather than once-per-target : target-specific versions
of modules are just going to be hopelessly confusing IMHO.

thanks
-- PMM

Il 18/06/2013 14:42, Anthony Liguori ha scritto:
>> > Next, and this is the most complex part.  The build system for
>> > modules, and configuring it.   I heard there were plans to use
>> > something like kbuild system for that, has anything been done
>> > in this context?
> GSoC just kicked off yesterday.  Paolo can perhaps shed some light on
> what the plans are.

The plans is just to add kconfig, not kbuild, similar to
seabios/busybox/etc.

It would provide configurability by target and board, and in addition
make it obvious which bits of default-configs/ (such as PCI, USB, HPET,
etc.) are actually configurable.

I outlined a possible implementation at the end of
http://wiki.qemu.org/Features/Modules, though something nicer may be
possible (possibly taking inspiration from Kconfig).

>> > With current config/build system, the following changes are
>> > needed:
>> >
>> >  o split individual libs from libs_softmmu into their own
>> >    variables.

Ack.  This could be doable already and independent from everything else.

>> >  o allow obj-m (and similar) in addition to obj-y, with build
>> >    flags and rules to produce an .so.
> We also need a way to define the module targets.  My Makefile-fu is
> not all that great but I suspect we can do something like:
> 
> libqemu-%.ko: $(eval $(obj-%-m))
> 
> I don't think that works but that's the rough idea.  We would then
> need to define each module target by hand but that's probably
> reasonable in the short term until we have kconfig.

You can limit that to multiple-source modules, which is the common case.
 Single-source modules are easily handled.

Paolo

Il 18/06/2013 15:12, Peter Maydell ha scritto:
>>> >>  o switching some modules to allow building them modular
>> >
>> > Pretty much any device is a candidate here.
> Before you can do that you need to convert them so we can build
> them once rather than once-per-target : target-specific versions
> of modules are just going to be hopelessly confusing IMHO.

Absolutely.  The page that Anthony cited even says "Only devices that
are compiled once can be modularized" (I don't remember if it was him or
me who added it).

Paolo

On 06/18/2013 04:37 AM, Michael Tokarev wrote:
> First of all, we need a global config switch/check for this kind of
> service.  On a POSIX system this should make the executable to be
> linked with -rdynamic option, in order to make all symbols in the
> executable to be available to plugins.

The only way you're going to be able to make this work with windows dlls (and,
frankly, a better way to make this work in elf without -export-dynamic), is to
make the "main" part of qemu itself a shared library, so that the modules can
link against it.

This reduces the qemu executable to, more or less, _start +/- main.



r~

On Tue, Jun 18, 2013 at 02:17:08PM +0200, Laszlo Ersek wrote:
> On 06/18/13 13:37, Michael Tokarev wrote:
> 
> > Next, and this is the most complex part.  The build system for
> > modules, and configuring it.   I heard there were plans to use
> > something like kbuild system for that, has anything been done
> > in this context?
> 
> Sorry for responding separately... Some module/symbol versioning will be
> necessary, as the binding will be delayed until runtime, and users
> *will* mix and match qemu and modules, and get random crashes. AFAIK
> versioning is a good way to avoid that (you would get an error in
> dlopen(), and dlerror() would pinpoint problem).
> 
> No idea how to implement this though, except I expect there's no chance
> to do it halfway portably without libtool.

In libvirt we use ELF symbol versioning for all symbols that are
exported. Public symbols for applications to use get tagged with
something like   LIBVIRT_0.9.3 (according to the version the symbol
arrived in), and then private symbols exported solely for use by
loadable modules get tagged  LIBVIRT_PRIVATE_1.0.3. The version number
for the private tags is changed on every single release. This ensures
that loadable modules can only be loaded by a libvirt that was built
from the same version.

# eu-readelf -a /usr/lib64/libvirt.so | grep LIBVIRT | awk '{print $8}' | less
virPCIDeviceFileIterate@@LIBVIRT_PRIVATE_1.0.3
virDomainObjGetState@@LIBVIRT_PRIVATE_1.0.3
virXPathLongLong@@LIBVIRT_PRIVATE_1.0.3
virNetworkLookupByUUIDString@@LIBVIRT_0.2.0
sexpr_node_copy@@LIBVIRT_PRIVATE_1.0.3
virSecurityManagerPreFork@@LIBVIRT_PRIVATE_1.0.3
virDomainSaveStatus@@LIBVIRT_PRIVATE_1.0.3
virXPathULongLong@@LIBVIRT_PRIVATE_1.0.3
virDomainPinVcpu@@LIBVIRT_0.1.4
virCommandClearCaps@@LIBVIRT_PRIVATE_1.0.3


By having all the private symbols change their version tag on every
release, we make it a total PITA for anyone to provide closed source,
out of tree, third party modules. Which is nice a feature :-)

Symbol versioning like this isn't portable to Windows, but it works
on Solaris and Linux at least. I'm not sure about BSD support for
symbol versioning.

The other nice thing about symbol versioning (for RPM based distros)
is that RPM will auto-add dependancies for symbol versions. So if
you distribute loadable modules in a separate RPM from the main
binary RPM, the dependancies will ensure you always have matched
versions installed.

Regards,
Daniel

On Tue, Jun 18, 2013 at 07:42:11AM -0500, Anthony Liguori wrote:
> Hi,
> 
> On Tue, Jun 18, 2013 at 6:37 AM, Michael Tokarev <mjt@tls.msk.ru> wrote:
> > Hello.
> >
> > I looked at what's needed to support DSO (dynamic shared objects)
> > in qemu, in order to be able to split functionality into loadable
> > "plugins".  It isn't exactly difficult, but a few steps are needed
> > still.
> 
> As it turns out, this is an area of interest of mine too.  I put up a
> quick page describing some ideas a few weeks ago.
> 
> http://wiki.qemu.org/Features/Modules
> 
> > The whole thing is about splitting the functionality into plugins
> > from one, single, qemu build, for now, not about allowing 3rd-party
> > plugins to be distributed outside of qemu.
> 
> Ack.  Really critical and important distinction.

Agreed, if we introduce a module system to QEMU we should clearly
and explicitly document that closed source out of tree modules are
considered to be forbidden & a license violation. We should also
make use of any technical measures available to make out of tree
modules impratical to use. For example, use ELF symbol versioning
and make the version tags change on every single minor release so
you can't easily provide pre-compiled 3rd party modules. Obviously
don't install header files anywhere in /usr/include, etc

Daniel

On 18 June 2013 21:19, Daniel P. Berrange <berrange@redhat.com> wrote:
> For example, use ELF symbol versioning
> and make the version tags change on every single minor release

Change on every git commit hash, please. Otherwise I bet there
are going to be a lot of random weird failures in development
due to things getting silently out of sync.

-- PMM

On 06/18/2013 01:15 PM, Daniel P. Berrange wrote:
> By having all the private symbols change their version tag on every
> release, we make it a total PITA for anyone to provide closed source,
> out of tree, third party modules. Which is nice a feature :-)

Using a symbol version file also means that symbols that aren't
explicitly exported are not present in the dynamic symbol table.

Which isn't *quite* as good as using compiler-level symbol visibility
markup, but the linker can sometimes optimize code sequences in place
or at minimum reduce a full symbol relocation to a relative relocation.

> Symbol versioning like this isn't portable to Windows, but it works
> on Solaris and Linux at least. I'm not sure about BSD support for
> symbol versioning.

Given that symbol versioning is mostly a forward compatible extension
to ELF, assuming the linker doesn't date from the dark ages one should
at least be able to take advantage of visibility improvements, even if
actual symbol versions aren't enforced.


r~