[2/7] python/qemu: formalize as package

NB: I am choosing Python 3.6 here. Although our minimum requirement is
3.5, this code is used only by iotests (so far) under which we have been
using a minimum version of 3.6.

3.6 is being preferred here for variable type hint capability, which
enables us to use mypy for this package.

RFC: This uses the version tags of the parent tree here, so packages
will be installed as e.g. 5.0.0, 5.1.0-rc0, etc.

Pros:
 - Easy to tell which versions of QEMU it supports
 - Simple

Cons:
 - Implies semver, which we do NOT follow for QEMU releases
 - Implies the package is in a stable state


We could also start a separate versioning for just the Python SDK at
e.g. 0.1;

Pros:
 - We can use semver, which is expected of Python packaging
 - Allows us to break compatibility for 0.x releases

Cons:
 - More complex, the mapping from SDK version to QEMU version
   is less obvious
 - Requires someone to manage a secondary version commit for
   the Python SDK.

Or, perhaps, we could start versioning with 0.5.0.0, 0.5.1.0, etc to
combine a bit of both flavors; bumping the major version number only
when incompatible changes to the Python interface itself are made,
treating the major version number more like an epoch.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/README.rst |  6 ++++++
 python/setup.py   | 50 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
 create mode 100644 python/README.rst
 create mode 100755 python/setup.py

03.06.2020 03:15, John Snow wrote:
> NB: I am choosing Python 3.6 here. Although our minimum requirement is
> 3.5, this code is used only by iotests (so far) under which we have been
> using a minimum version of 3.6.
> 
> 3.6 is being preferred here for variable type hint capability, which
> enables us to use mypy for this package.
> 
> RFC: This uses the version tags of the parent tree here, so packages
> will be installed as e.g. 5.0.0, 5.1.0-rc0, etc.
> 
> Pros:
>   - Easy to tell which versions of QEMU it supports
>   - Simple
> 
> Cons:
>   - Implies semver, which we do NOT follow for QEMU releases
>   - Implies the package is in a stable state

Necessarily? Couldn't we state Development Status: Alpha, even with version 5.1.0 ?

> 
> 
> We could also start a separate versioning for just the Python SDK at
> e.g. 0.1;
> 
> Pros:
>   - We can use semver, which is expected of Python packaging
>   - Allows us to break compatibility for 0.x releases
> 
> Cons:
>   - More complex, the mapping from SDK version to QEMU version
>     is less obvious
>   - Requires someone to manage a secondary version commit for
>     the Python SDK.
> 
> Or, perhaps, we could start versioning with 0.5.0.0, 0.5.1.0, etc to
> combine a bit of both flavors; bumping the major version number only
> when incompatible changes to the Python interface itself are made,
> treating the major version number more like an epoch.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   python/README.rst |  6 ++++++
>   python/setup.py   | 50 +++++++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 56 insertions(+)
>   create mode 100644 python/README.rst
>   create mode 100755 python/setup.py
> 
> diff --git a/python/README.rst b/python/README.rst
> new file mode 100644
> index 00000000000..25f6d93fd5f
> --- /dev/null
> +++ b/python/README.rst
> @@ -0,0 +1,6 @@
> +QEMU Python Tooling
> +-------------------
> +
> +This package provides QEMU tooling used by the QEMU project to build,
> +configure, and test QEMU. It is not a fully-fledged SDK and it is subject
> +to change at any time.
> diff --git a/python/setup.py b/python/setup.py
> new file mode 100755
> index 00000000000..f897ceac970
> --- /dev/null
> +++ b/python/setup.py
> @@ -0,0 +1,50 @@
> +#!/usr/bin/env3 python

env python3 you mean

with it fixed:
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

> +"""
> +QEMU tooling installer script
> +Copyright (c) 2020 John Snow for Red Hat, Inc.
> +"""
> +
> +import setuptools
> +
> +def main():
> +    """
> +    QEMU tooling installer
> +    """
> +
> +    kwargs = {
> +        'name': 'qemu',
> +        'use_scm_version': {
> +            'root': '..',
> +            'relative_to': __file__,
> +        },
> +        'maintainer': 'QEMU Developer Team',
> +        'maintainer_email': 'qemu-devel@nongnu.org',
> +        'url': 'https://www.qemu.org/',
> +        'download_url': 'https://www.qemu.org/download/',
> +        'packages': setuptools.find_namespace_packages(),
> +        'description': 'QEMU Python Build, Debug and SDK tooling.',
> +        'classifiers': [
> +            'Development Status :: 5 - Production/Stable',

Could we use "3 - Alpha" ?

> +            'License :: OSI Approved :: GNU General Public License v2 (GPLv2)',
> +            'Natural Language :: English',
> +            'Operating System :: OS Independent',
> +        ],
> +        'platforms': [],
> +        'keywords': [],
> +        'setup_requires': [
> +            'setuptools',
> +            'setuptools_scm',
> +        ],

Hmm, documentation says:

    Warning Using setup_requires is discouraged in favor of PEP-518

did you consider this thing?

> +        'install_requires': [
> +        ],
> +        'python_requires': '>=3.6',
> +        'long_description_content_type': 'text/x-rst',
> +    }
> +
> +    with open("README.rst", "r") as fh:

You use '' for all other strings (except for doc-strings) in this file. Maybe use '' here too?

> +        kwargs['long_description'] = fh.read()
> +
> +    setuptools.setup(**kwargs)
> +
> +if __name__ == '__main__':
> +    main()
> 

Hmm in examples in documentations I always see something like this:

> from setuptools import setup, find_namespace_packages
> 
> setup(
>     name='mynamespace-subpackage-a',
>     ...
>     packages=find_namespace_packages(include=['mynamespace.*'])
> )

How much is it better (or popular) to use __name__ == __main__ style for setup.py?
We are not going to use it as module to import main somewhere...

On 6/5/20 10:40 AM, Vladimir Sementsov-Ogievskiy wrote:
> 03.06.2020 03:15, John Snow wrote:
>> NB: I am choosing Python 3.6 here. Although our minimum requirement is
>> 3.5, this code is used only by iotests (so far) under which we have been
>> using a minimum version of 3.6.
>>
>> 3.6 is being preferred here for variable type hint capability, which
>> enables us to use mypy for this package.
>>
>> RFC: This uses the version tags of the parent tree here, so packages
>> will be installed as e.g. 5.0.0, 5.1.0-rc0, etc.
>>
>> Pros:
>>   - Easy to tell which versions of QEMU it supports
>>   - Simple
>>
>> Cons:
>>   - Implies semver, which we do NOT follow for QEMU releases
>>   - Implies the package is in a stable state
> 
> Necessarily? Couldn't we state Development Status: Alpha, even with
> version 5.1.0 ?
> 

It's a good question, actually...

I had thought that the Python packaging ecosystem implied semver, but I
don't see proof of that right now, actually.

PEP440 https://www.python.org/dev/peps/pep-0440/#semantic-versioning
says only that it considers many of the same cases as python versioning
and that it's more prescriptive.

However, in the "Version scheme" section, PEP440 says:

"All numeric components MAY be zero. Except as described below for the
release segment, a numeric component of zero has no special significance
aside from always being the lowest possible value in the version ordering."

I actually don't know what special significance a release segment of 0
has, though -- I don't see that later in the doc.

Semver is described here: https://semver.org/

"Major version zero (0.y.z) is for initial development. Anything MAY
change at any time. The public API SHOULD NOT be considered stable."

I guess we're not beholden to it. Still, it seems common in the Python
world to gate based on major version. It may present packaging
difficulties. I'll try to research more. Maybe I'll reach out to the
PyPI folks and see if I can get advice here.

>>
>>
>> We could also start a separate versioning for just the Python SDK at
>> e.g. 0.1;
>>
>> Pros:
>>   - We can use semver, which is expected of Python packaging
>>   - Allows us to break compatibility for 0.x releases
>>
>> Cons:
>>   - More complex, the mapping from SDK version to QEMU version
>>     is less obvious
>>   - Requires someone to manage a secondary version commit for
>>     the Python SDK.
>>
>> Or, perhaps, we could start versioning with 0.5.0.0, 0.5.1.0, etc to
>> combine a bit of both flavors; bumping the major version number only
>> when incompatible changes to the Python interface itself are made,
>> treating the major version number more like an epoch.
>>
>> Signed-off-by: John Snow <jsnow@redhat.com>
>> ---
>>   python/README.rst |  6 ++++++
>>   python/setup.py   | 50 +++++++++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 56 insertions(+)
>>   create mode 100644 python/README.rst
>>   create mode 100755 python/setup.py
>>
>> diff --git a/python/README.rst b/python/README.rst
>> new file mode 100644
>> index 00000000000..25f6d93fd5f
>> --- /dev/null
>> +++ b/python/README.rst
>> @@ -0,0 +1,6 @@
>> +QEMU Python Tooling
>> +-------------------
>> +
>> +This package provides QEMU tooling used by the QEMU project to build,
>> +configure, and test QEMU. It is not a fully-fledged SDK and it is
>> subject
>> +to change at any time.
>> diff --git a/python/setup.py b/python/setup.py
>> new file mode 100755
>> index 00000000000..f897ceac970
>> --- /dev/null
>> +++ b/python/setup.py
>> @@ -0,0 +1,50 @@
>> +#!/usr/bin/env3 python
> 
> env python3 you mean
> 

Oh wow. The funny thing is that it works anyway when you use pip!

> with it fixed:
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> 
>> +"""
>> +QEMU tooling installer script
>> +Copyright (c) 2020 John Snow for Red Hat, Inc.
>> +"""
>> +
>> +import setuptools
>> +
>> +def main():
>> +    """
>> +    QEMU tooling installer
>> +    """
>> +
>> +    kwargs = {
>> +        'name': 'qemu',
>> +        'use_scm_version': {
>> +            'root': '..',
>> +            'relative_to': __file__,
>> +        },
>> +        'maintainer': 'QEMU Developer Team',
>> +        'maintainer_email': 'qemu-devel@nongnu.org',
>> +        'url': 'https://www.qemu.org/',
>> +        'download_url': 'https://www.qemu.org/download/',
>> +        'packages': setuptools.find_namespace_packages(),
>> +        'description': 'QEMU Python Build, Debug and SDK tooling.',
>> +        'classifiers': [
>> +            'Development Status :: 5 - Production/Stable',
> 
> Could we use "3 - Alpha" ?
> 

Yes. I used 5 for the placeholder package to explain QEMU's stability,
but I should describe the SDK stability instead.

>> +            'License :: OSI Approved :: GNU General Public License v2
>> (GPLv2)',
>> +            'Natural Language :: English',
>> +            'Operating System :: OS Independent',
>> +        ],
>> +        'platforms': [],
>> +        'keywords': [],
>> +        'setup_requires': [
>> +            'setuptools',
>> +            'setuptools_scm',
>> +        ],
> 
> Hmm, documentation says:
> 
>    Warning Using setup_requires is discouraged in favor of PEP-518
> 
> did you consider this thing?
> 

Oh, the TOML thing ... OK, I need to research this more. I tried using
it a few times and I couldn't observe it to be actually working.

It doesn't seem to be in common, widespread usage in the ecosystem yet.

(Also, and I'm serious: it requires a TOML module that isn't built into
Python, so it has some detriments, I think.)

I'll look into this more ardently.

>> +        'install_requires': [
>> +        ],
>> +        'python_requires': '>=3.6',
>> +        'long_description_content_type': 'text/x-rst',
>> +    }
>> +
>> +    with open("README.rst", "r") as fh:
> 
> You use '' for all other strings (except for doc-strings) in this file.
> Maybe use '' here too?
> 

Yes, I should be consistent.

This is not based on any PEP, but I tend to prefer using single quotes
for static strings, and double quotes for format strings -- except when
convenient to deviate for reasons of avoiding backslashes.

>> +        kwargs['long_description'] = fh.read()
>> +
>> +    setuptools.setup(**kwargs)
>> +
>> +if __name__ == '__main__':
>> +    main()
>>
> 
> Hmm in examples in documentations I always see something like this:
> 
>> from setuptools import setup, find_namespace_packages
>>
>> setup(
>>     name='mynamespace-subpackage-a',
>>     ...
>>     packages=find_namespace_packages(include=['mynamespace.*'])
>> )
> 
> How much is it better (or popular) to use __name__ == __main__ style for
> setup.py?
> We are not going to use it as module to import main somewhere...
> 

Mostly just habit. It's not important here, but if you should so happen
to check your setup.py? with pylint, using the __main__ pattern keeps the
check passing.

On 6/5/20 10:40 AM, Vladimir Sementsov-Ogievskiy wrote:
> 
> Hmm, documentation says:
> 
>    Warning Using setup_requires is discouraged in favor of PEP-518
> 
> did you consider this thing?

I guess the difference here is we start using a pyproject.toml and then
we declare setuptools, wheel and setuptools_scm dependencies there.

In turn, pip will no longer install those projects to the user's
environment, but will create its own virtual environment for the
purposes of the build/installation process.

(It looks like pip re-downloads and re-installs these files every time
you run the install? That seems ... bad?)

It looks like this support was added in pip 10. Not all the platforms we
support at the moment appear to have pip 10 in their distribution
repository, though technically ... you could use pip 9 to install a more
modern pip to your userspace.

I think maybe I'd prefer to just leave this alone for now. We can update
it if there's some compelling reason to upgrade, but it's still a pretty
new feature that doesn't seem to be in widespread usage yet.

setuptools is widely regarded to be "the default", and I believe pip
will assume and support this for a while longer yet.

No hurry to upgrade, I think, but I could be wrong. Python packaging is
convoluted.

--js