[RFC] docs: build-platforms: refine requirements on Python build dependencies

Historically, the critical dependency for both building and running
QEMU has been the distro packages.  Because QEMU is written in C and C's
package management has been tied to distros (at least if you do not want
to bundle libraries with the binary, otherwise I suppose you could use
something like conda or wrapdb), C dependencies of QEMU would target the
version that is shipped in relatively old but still commonly used distros.

For non-C libraries, however, the situation is different, as these
languages have their own package management tool (cpan, pip, gem, npm,
and so on).  For some of these languages, the amount of dependencies
for even a simple program can easily balloon to the point that many
distros have given up on packaging non-C code.  For this reason, it has
become increasingly normal for developers to download dependencies into
a self-contained local environment, instead of relying on distro packages.

Fortunately, this affects QEMU only at build time, as qemu.git does
not package non-C artifacts such as the qemu.qmp package; but still,
as we make more use of Python, we experience a clash between a support
policy that is written for the C world, and dependencies (both direct
and indirect) that increasingly do not care for the distro versions
and are quick at moving past Python runtime versions that are declared
end-of-life.

For example, Python 3.6 has been EOL'd since December 2021 and Meson 0.62
(released the following March) already dropped support for it.  Yet,
Python 3.6 is the default version of the Python runtime for RHEL/CentOS
8 and SLE 15, respectively the penultimate and the most recent version
of two distros that QEMU would like to support.  (It is also the version
used by Ubuntu 18.04, but QEMU stopped supporting it in April 2022).

Fortunately, these long-term support distros do include newer versions of
the Python runtime.  However, these more recent runtimes only come with
a very small subset of the Python packages that the distro includes.
Because most dependencies are optional tests (avocado, mypy, flake8)
and Meson is bundled with QEMU, the most noticeably missing package is
Sphinx (and the readthedocs theme).

Assuming QEMU would like to move forward with the deprecation of
Python 3.6 (for which there are some good reasons: completing the
configure->meson switch, which requires Meson 0.63, or making qapidoc
fully typed which requires newer versions of mypy and also Python due
to PEP563), there are four possibilities:

* we change the support policy and stop supporting CentOS 8 and SLE 15;
  not a good idea since CentOS 8 is not an unreasonable distro for us to
  want to continue to support

* we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being
  supported.  This is a possibility---but we may want to revise the support
  policy anyway because SLE 16 has not even been released, so this would
  mean delaying those desirable reasons for perhaps three years;

* we support Python 3.6 just for building documentation, i.e. we are
  careful not to use Python 3.7+ features in our Sphinx extensions but are
  free to use them elsewhere.  Besides being more complicated to understand
  for developers, this is difficult to apply because qapidoc runs at
  sphinx-build time, and it is one of the areas which would benefit from
  a newer version of the runtime;

* we only support Python 3.7+, which means CentOS 8 CI and users
  have to either install Sphinx from pip or disable documentation.

This proposed update to the support policy chooses the last of these
possibilities.  It does by modifying two aspects of the support policy:

* it introduces different support periods for *native* vs. *non-native*
  dependencies.  Non-native dependencies are currently Python ones only,
  and for simplicity the policy only mentions Python; however, the concept
  generalizes to other languages with a well-known upstream package
  manager, that users of older distributions can fetch dependencies from;

* it limits the support period for non-native dependencies to a fixed
  amount of 4 years.  This is intended to be close to the Python 5-year
  lifecycle while accounting for the time between a distro's feature freeze
  and the day it's released.  This limit applies to all distro versions,
  not just the previous one, in order to cater for the delay of SLE 16.

The 4 year cutoff in practice means that QEMU will be able to drop Python
3.6 support for QEMU 7.1 (RHEL8 becomes 4 year old next May, while SLE
is already over the threshold).

Note that all "non-native" packages are currently build dependencies.
If in the future some non-native packages became runtime dependencies for
parts of QEMU, it would still be possible to choose any of the first
three possibilities for them.

Another possible future change is to the way that these dependencies
have to be obtained by the person building QEMU.  Right now they have to
run pip before the build; it may be desirable for configure to set up a
virtual environment and download them in the same way that it populates
git submodules.  Just like with submodules, this would make things
easier for people that can afford accessing the network in their build
environment; the option to populate the build environment manually with
pip would remain for people whose build machines lack network access.
The change to the support policy neither requires nor forbids this change.

[Thanks to Daniel P. Berrangé, Peter Maydell and others for discussions
 that were copied or summarized in the above commit message]

Cc: Markus Armbruster <armbru@redhat.com>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Cc: John Snow <jsnow@redhat.com>
Cc: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 docs/about/build-platforms.rst | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

On Fri, Feb 17, 2023 at 01:41:50PM +0100, Paolo Bonzini wrote:
> Historically, the critical dependency for both building and running
> QEMU has been the distro packages.  Because QEMU is written in C and C's
> package management has been tied to distros (at least if you do not want
> to bundle libraries with the binary, otherwise I suppose you could use
> something like conda or wrapdb), C dependencies of QEMU would target the
> version that is shipped in relatively old but still commonly used distros.
> 
> For non-C libraries, however, the situation is different, as these
> languages have their own package management tool (cpan, pip, gem, npm,
> and so on).  For some of these languages, the amount of dependencies
> for even a simple program can easily balloon to the point that many
> distros have given up on packaging non-C code.  For this reason, it has
> become increasingly normal for developers to download dependencies into
> a self-contained local environment, instead of relying on distro packages.
> 
> Fortunately, this affects QEMU only at build time, as qemu.git does
> not package non-C artifacts such as the qemu.qmp package; but still,
> as we make more use of Python, we experience a clash between a support
> policy that is written for the C world, and dependencies (both direct
> and indirect) that increasingly do not care for the distro versions
> and are quick at moving past Python runtime versions that are declared
> end-of-life.
> 
> For example, Python 3.6 has been EOL'd since December 2021 and Meson 0.62
> (released the following March) already dropped support for it.  Yet,
> Python 3.6 is the default version of the Python runtime for RHEL/CentOS
> 8 and SLE 15, respectively the penultimate and the most recent version
> of two distros that QEMU would like to support.  (It is also the version
> used by Ubuntu 18.04, but QEMU stopped supporting it in April 2022).
> 
> Fortunately, these long-term support distros do include newer versions of
> the Python runtime.  However, these more recent runtimes only come with
> a very small subset of the Python packages that the distro includes.
> Because most dependencies are optional tests (avocado, mypy, flake8)
> and Meson is bundled with QEMU, the most noticeably missing package is
> Sphinx (and the readthedocs theme).

mypy itself is optional for QEMU, but the implications of new mypy
are not optional. The intent of using new mypy is to leverage
improved python parameter type annotations, and these annotations
are apparently not compatible with py3.6 runtime exection, despite
the fact that they should not conceptually affect runtime.


> Assuming QEMU would like to move forward with the deprecation of
> Python 3.6 (for which there are some good reasons: completing the
> configure->meson switch, which requires Meson 0.63, or making qapidoc
> fully typed which requires newer versions of mypy and also Python due
> to PEP563), there are four possibilities:
> 
> * we change the support policy and stop supporting CentOS 8 and SLE 15;
>   not a good idea since CentOS 8 is not an unreasonable distro for us to
>   want to continue to support

And there's not even a newer SLE 16 available yet, so even less
desirable to drop the only SLE we target.

> * we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being
>   supported.  This is a possibility---but we may want to revise the support
>   policy anyway because SLE 16 has not even been released, so this would
>   mean delaying those desirable reasons for perhaps three years;
> 
> * we support Python 3.6 just for building documentation, i.e. we are
>   careful not to use Python 3.7+ features in our Sphinx extensions but are
>   free to use them elsewhere.  Besides being more complicated to understand
>   for developers, this is difficult to apply because qapidoc runs at
>   sphinx-build time, and it is one of the areas which would benefit from
>   a newer version of the runtime;

Using a mix of python versions is just asking for trouble

> 
> * we only support Python 3.7+, which means CentOS 8 CI and users
>   have to either install Sphinx from pip or disable documentation.
> 
> This proposed update to the support policy chooses the last of these
> possibilities.  It does by modifying two aspects of the support policy:
> 
> * it introduces different support periods for *native* vs. *non-native*
>   dependencies.  Non-native dependencies are currently Python ones only,
>   and for simplicity the policy only mentions Python; however, the concept
>   generalizes to other languages with a well-known upstream package
>   manager, that users of older distributions can fetch dependencies from;

I guess this would mean its applicable to perl/ruby/etc, but really
we should be exclusively using Python for our scripting needs, except
for legacy stuff we already have. So in practice non-native == python.

Potentially if we want to use Rust, we'd want vendoring of deps needed
by Rust code, and that's native deps arguably, rather than non-native.

Still I think it is unlikely we would replace existing working code in
qemu.git with a rust impl, while discarding the original impl. More
likely we'd have a Rust impl in parallel, or use Rust only for a brand
new feature, where we don't need to support all possible old platforms.

> * it limits the support period for non-native dependencies to a fixed
>   amount of 4 years.  This is intended to be close to the Python 5-year
>   lifecycle while accounting for the time between a distro's feature freeze
>   and the day it's released.  This limit applies to all distro versions,
>   not just the previous one, in order to cater for the delay of SLE 16.
> 
> The 4 year cutoff in practice means that QEMU will be able to drop Python
> 3.6 support for QEMU 7.1 (RHEL8 becomes 4 year old next May, while SLE
> is already over the threshold).

So the 4 year timeframe enables us to move off 3.6, but it is still
potentially limiting.

That doesn't explicitly allow for the case where we decide we want
to increae min version of flake8/pylint/mypy to one that is merly
1 year old. As noted, covering multiple versions of these linting
tools is a burden, as the issues reported from new versions are
not always a superset of the old version. So it is somewhat
desirable to fixate on a specific release, or a narrow range of
releases.

I guess we can argue that any optional build time components that
only affect testing don't need to be constrained by our platform
policy in the first, since they are optional and don't affect the
ability to build QEMU. Still if I'm a distro maintainer, and
backporting patches I like to run all static analysis tools to
catch silly python mistake that happen when cherry-picking.
Equally if I'm a contributor working on QEMU I would like to
run all the static linting tests available.


Implicit in the 4 year timeframe is that it is actually easy to
cope with the fallout, because the long life distros support many
newer python runtime versions. The only impact is on the need to
use pip to grab extra pieces.

So a different way of expressing this is to not limit ourselves
by time. Instead declare that we reserve the right to bump the
python runtime version, provided the new version is available
from the disto as an optional alternative to the default python
version.

This would mean we would have had the option to adopt python
3.8 a year ago (or whenever it first became an option in RHEL
/SLES). Today 4 years would suit our immediate needs, but John
probably would have liked to have bumped the version last year
already if it weren't for our support policy, and 4 year cut
off wouldn't have helped last year.

Basically if it is acceptable to require a non-default python
version, it should be ok to require that are more or less
any time, as long as the version is available from the OS
vendor officially.

> Note that all "non-native" packages are currently build dependencies.
> If in the future some non-native packages became runtime dependencies for
> parts of QEMU, it would still be possible to choose any of the first
> three possibilities for them.
> 
> Another possible future change is to the way that these dependencies
> have to be obtained by the person building QEMU.  Right now they have to
> run pip before the build; it may be desirable for configure to set up a
> virtual environment and download them in the same way that it populates
> git submodules.  Just like with submodules, this would make things
> easier for people that can afford accessing the network in their build
> environment; the option to populate the build environment manually with
> pip would remain for people whose build machines lack network access.
> The change to the support policy neither requires nor forbids this change.
> 
> [Thanks to Daniel P. Berrangé, Peter Maydell and others for discussions
>  that were copied or summarized in the above commit message]
> 
> Cc: Markus Armbruster <armbru@redhat.com>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Cc: John Snow <jsnow@redhat.com>
> Cc: Kevin Wolf <kwolf@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  docs/about/build-platforms.rst | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 1c1e7b9e11c3..e1ea09789107 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -86,6 +86,25 @@ respective ports repository, while NetBSD will use the pkgsrc repository.
>  For macOS, `Homebrew`_ will be used, although `MacPorts`_ is expected to carry
>  similar versions.
>  
> +Python build dependencies
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The minimum supported version of Python is currently 3.6.
> +
> +Distributions with long-term support often provide multiple
> +versions of the Python runtime.  QEMU aims to support the default
> +Python runtime for 4 years after the initial release of a new version.
> +Afterwards, you may have to point QEMU to a newer version of the Python
> +runtime using the ``--python`` command line option of the ``configure``
> +script.

My counter proposal is to remove the 4 year marker, and give us
more flexibility:

 Distributions with long-term support often provide multiple
 versions of the Python runtime.  QEMU will initially aim to
 support the default python runtime. QEMU reserves the right
 to increase its minimum version to any newer python that is
 available as an option from the vendor.


> +
> +Some of QEMU's build dependencies are written in Python and available
> +through the Python Package Index (PyPI).  QEMU aims to be compatible
> +with the versions packaged by common Linux distributions for the first
> +4 years after the major release of the distribution.  After 4 years,
> +you may have to use ``pip`` to install some of these build dependencies.

And for this I'd say

  If QEMU bumps its minimum python version to a non-default version,
  then it may be neccessary to fetch python modules from 'pip' to
  build or test QEMU.

With regards,
Daniel

Paolo Bonzini <pbonzini@redhat.com> writes:

> Historically, the critical dependency for both building and running
> QEMU has been the distro packages.  Because QEMU is written in C and C's
> package management has been tied to distros (at least if you do not want
> to bundle libraries with the binary, otherwise I suppose you could use
> something like conda or wrapdb), C dependencies of QEMU would target the
> version that is shipped in relatively old but still commonly used distros.
>
> For non-C libraries, however, the situation is different, as these
> languages have their own package management tool (cpan, pip, gem, npm,
> and so on).  For some of these languages, the amount of dependencies
> for even a simple program can easily balloon to the point that many
> distros have given up on packaging non-C code.  For this reason, it has
> become increasingly normal for developers to download dependencies into
> a self-contained local environment, instead of relying on distro packages.
>
> Fortunately, this affects QEMU only at build time, as qemu.git does
> not package non-C artifacts such as the qemu.qmp package; but still,
> as we make more use of Python, we experience a clash between a support
> policy that is written for the C world, and dependencies (both direct
> and indirect) that increasingly do not care for the distro versions
> and are quick at moving past Python runtime versions that are declared
> end-of-life.
>
> For example, Python 3.6 has been EOL'd since December 2021 and Meson 0.62
> (released the following March) already dropped support for it.  Yet,
> Python 3.6 is the default version of the Python runtime for RHEL/CentOS
> 8 and SLE 15, respectively the penultimate and the most recent version
> of two distros that QEMU would like to support.  (It is also the version
> used by Ubuntu 18.04, but QEMU stopped supporting it in April 2022).
>
> Fortunately, these long-term support distros do include newer versions of
> the Python runtime.  However, these more recent runtimes only come with
> a very small subset of the Python packages that the distro includes.
> Because most dependencies are optional tests (avocado, mypy, flake8)
> and Meson is bundled with QEMU, the most noticeably missing package is
> Sphinx (and the readthedocs theme).
>
> Assuming QEMU would like to move forward with the deprecation of
> Python 3.6 (for which there are some good reasons: completing the
> configure->meson switch, which requires Meson 0.63, or making qapidoc

I think you mean "the QAPI generator".

> fully typed which requires newer versions of mypy and also Python due
> to PEP563), there are four possibilities:
>
> * we change the support policy and stop supporting CentOS 8 and SLE 15;
>   not a good idea since CentOS 8 is not an unreasonable distro for us to
>   want to continue to support
>
> * we keep supporting Python 3.6 until CentOS 8 and SLE 15 stop being
>   supported.  This is a possibility---but we may want to revise the support
>   policy anyway because SLE 16 has not even been released, so this would
>   mean delaying those desirable reasons for perhaps three years;
>
> * we support Python 3.6 just for building documentation, i.e. we are
>   careful not to use Python 3.7+ features in our Sphinx extensions but are
>   free to use them elsewhere.  Besides being more complicated to understand
>   for developers, this is difficult to apply because qapidoc runs at

Suggest "some QAPI generator code runs".

>   sphinx-build time, and it is one of the areas which would benefit from
>   a newer version of the runtime;
>
> * we only support Python 3.7+, which means CentOS 8 CI and users
>   have to either install Sphinx from pip or disable documentation.
>
> This proposed update to the support policy chooses the last of these
> possibilities.  It does by modifying two aspects of the support policy:
>
> * it introduces different support periods for *native* vs. *non-native*
>   dependencies.  Non-native dependencies are currently Python ones only,
>   and for simplicity the policy only mentions Python; however, the concept
>   generalizes to other languages with a well-known upstream package
>   manager, that users of older distributions can fetch dependencies from;
>
> * it limits the support period for non-native dependencies to a fixed
>   amount of 4 years.  This is intended to be close to the Python 5-year
>   lifecycle while accounting for the time between a distro's feature freeze
>   and the day it's released.  This limit applies to all distro versions,
>   not just the previous one, in order to cater for the delay of SLE 16.
>
> The 4 year cutoff in practice means that QEMU will be able to drop Python
> 3.6 support for QEMU 7.1 (RHEL8 becomes 4 year old next May, while SLE

We released 7.1 last August, do you mean 8.0 or 8.1?

> is already over the threshold).
>
> Note that all "non-native" packages are currently build dependencies.
> If in the future some non-native packages became runtime dependencies for
> parts of QEMU, it would still be possible to choose any of the first
> three possibilities for them.
>
> Another possible future change is to the way that these dependencies
> have to be obtained by the person building QEMU.  Right now they have to
> run pip before the build; it may be desirable for configure to set up a
> virtual environment and download them in the same way that it populates
> git submodules.  Just like with submodules, this would make things
> easier for people that can afford accessing the network in their build
> environment; the option to populate the build environment manually with
> pip would remain for people whose build machines lack network access.
> The change to the support policy neither requires nor forbids this change.
>
> [Thanks to Daniel P. Berrangé, Peter Maydell and others for discussions
>  that were copied or summarized in the above commit message]
>
> Cc: Markus Armbruster <armbru@redhat.com>
> Cc: Daniel P. Berrangé <berrange@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Cc: John Snow <jsnow@redhat.com>
> Cc: Kevin Wolf <kwolf@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  docs/about/build-platforms.rst | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
>
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 1c1e7b9e11c3..e1ea09789107 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -86,6 +86,25 @@ respective ports repository, while NetBSD will use the pkgsrc repository.
>  For macOS, `Homebrew`_ will be used, although `MacPorts`_ is expected to carry
>  similar versions.
>  
> +Python build dependencies
> +~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The minimum supported version of Python is currently 3.6.
> +
> +Distributions with long-term support often provide multiple
> +versions of the Python runtime.  QEMU aims to support the default
> +Python runtime for 4 years after the initial release of a new version.

Just to be crystal clear: new version of what?

> +Afterwards, you may have to point QEMU to a newer version of the Python
> +runtime using the ``--python`` command line option of the ``configure``
> +script.
> +
> +Some of QEMU's build dependencies are written in Python and available
> +through the Python Package Index (PyPI).  QEMU aims to be compatible
> +with the versions packaged by common Linux distributions for the first
> +4 years after the major release of the distribution.  After 4 years,
> +you may have to use ``pip`` to install some of these build dependencies.
> +
> +
>  Windows
>  -------

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Fri, Feb 17, 2023 at 01:41:50PM +0100, Paolo Bonzini wrote:

[...]

>> This proposed update to the support policy chooses the last of these
>> possibilities.  It does by modifying two aspects of the support policy:
>> 
>> * it introduces different support periods for *native* vs. *non-native*
>>   dependencies.  Non-native dependencies are currently Python ones only,
>>   and for simplicity the policy only mentions Python; however, the concept
>>   generalizes to other languages with a well-known upstream package
>>   manager, that users of older distributions can fetch dependencies from;
>
> I guess this would mean its applicable to perl/ruby/etc, but really
> we should be exclusively using Python for our scripting needs, except
> for legacy stuff we already have. So in practice non-native == python.
>
> Potentially if we want to use Rust, we'd want vendoring of deps needed
> by Rust code, and that's native deps arguably, rather than non-native.
>
> Still I think it is unlikely we would replace existing working code in
> qemu.git with a rust impl, while discarding the original impl. More
> likely we'd have a Rust impl in parallel, or use Rust only for a brand
> new feature, where we don't need to support all possible old platforms.

I'm not sure.  However, there's no real need to anticipate right now how
we will use Rust; we can update our support policy when we know.

>> * it limits the support period for non-native dependencies to a fixed
>>   amount of 4 years.  This is intended to be close to the Python 5-year
>>   lifecycle while accounting for the time between a distro's feature freeze
>>   and the day it's released.  This limit applies to all distro versions,
>>   not just the previous one, in order to cater for the delay of SLE 16.
>> 
>> The 4 year cutoff in practice means that QEMU will be able to drop Python
>> 3.6 support for QEMU 7.1 (RHEL8 becomes 4 year old next May, while SLE
>> is already over the threshold).
>
> So the 4 year timeframe enables us to move off 3.6, but it is still
> potentially limiting.
>
> That doesn't explicitly allow for the case where we decide we want
> to increae min version of flake8/pylint/mypy to one that is merly
> 1 year old. As noted, covering multiple versions of these linting
> tools is a burden, as the issues reported from new versions are
> not always a superset of the old version. So it is somewhat
> desirable to fixate on a specific release, or a narrow range of
> releases.

Concur.

> I guess we can argue that any optional build time components that
> only affect testing don't need to be constrained by our platform
> policy in the first, since they are optional and don't affect the
> ability to build QEMU. Still if I'm a distro maintainer, and
> backporting patches I like to run all static analysis tools to
> catch silly python mistake that happen when cherry-picking.

Yes, but you want to run the static analysis tools required by the QEMU
you're backporting to, not the one you're backporting from (unless
you're backporting the patches needed to make the newer linters happy,
and then bump the dependencies).

> Equally if I'm a contributor working on QEMU I would like to
> run all the static linting tests available.

This one is valid.  Still, it's a "want", not a "must have".  If it was
a hard "must have", then CentOS 8 support would be in trouble: no mypy
out of the box, as far as I can tell.

We can rely on CI to find the lint.  Delays the feedback, which not
ideal, but also not unlike how we use CI to keep the build working on
all hosts (because not every developer has access to all hosts) and in a
multitude of configurations (requiring all developers to test all
configurations at all times would be impractical, so we don't).

Let's consider the special case mypy.  Right now, you have to run it
manually, but John has been working on integrating it into the build
process.  Now compare two extremes:

(A) Commit to keeping mypy happy for all versions of mypy provided by
    any supported build host

    The build runs mypy if the build host provides it.  Immediate
    feedback then, but only from a single mypy version.

    Developers still have to run all the mypy versions somehow, or else
    punt to maintainers, who may in turn punt to CI.  Delayed feedback
    from all mypy versions but one.

    We operate with the intersection of mypy features and the union of
    mypy bugs, which is bound to complicate the typing job, and may well
    limit typing power.

(B) Pick a single mypy version

    Developers have to run this mypy version, or else punt to
    maintainers.  We'll naturally pick a mypy version the maintainers
    are happy to run, so no need for them to punt to CI.  We can exploit
    the full typing power of that mypy version.  We don't have to work
    around issues older versions may have.

    Developers unable to run this mypy version locally get delayed
    feedback.  Affects the intersection of "people working on Python
    code we check with mypy" and "people developing exclusively on older
    hosts".  If we further intersect with "people doing so on a somewhat
    regular basis", I posit we get the empty set.

I think (A) is pretty much all downsides.  I like (B), but could be too
rigid in practice.  We could also

(C) Pick a narrow range range of mypy versions

> Implicit in the 4 year timeframe is that it is actually easy to
> cope with the fallout, because the long life distros support many
> newer python runtime versions. The only impact is on the need to
> use pip to grab extra pieces.
>
> So a different way of expressing this is to not limit ourselves
> by time. Instead declare that we reserve the right to bump the
> python runtime version, provided the new version is available
> from the disto as an optional alternative to the default python
> version.
>
> This would mean we would have had the option to adopt python
> 3.8 a year ago (or whenever it first became an option in RHEL
> /SLES). Today 4 years would suit our immediate needs, but John
> probably would have liked to have bumped the version last year
> already if it weren't for our support policy, and 4 year cut
> off wouldn't have helped last year.

Good points.

> Basically if it is acceptable to require a non-default python
> version, it should be ok to require that are more or less
> any time, as long as the version is available from the OS
> vendor officially.
>
>> Note that all "non-native" packages are currently build dependencies.
>> If in the future some non-native packages became runtime dependencies for
>> parts of QEMU, it would still be possible to choose any of the first
>> three possibilities for them.
>> 
>> Another possible future change is to the way that these dependencies
>> have to be obtained by the person building QEMU.  Right now they have to
>> run pip before the build; it may be desirable for configure to set up a
>> virtual environment and download them in the same way that it populates
>> git submodules.  Just like with submodules, this would make things
>> easier for people that can afford accessing the network in their build
>> environment; the option to populate the build environment manually with
>> pip would remain for people whose build machines lack network access.
>> The change to the support policy neither requires nor forbids this change.
>> 
>> [Thanks to Daniel P. Berrangé, Peter Maydell and others for discussions
>>  that were copied or summarized in the above commit message]
>> 
>> Cc: Markus Armbruster <armbru@redhat.com>
>> Cc: Daniel P. Berrangé <berrange@redhat.com>
>> Cc: Peter Maydell <peter.maydell@linaro.org>
>> Cc: John Snow <jsnow@redhat.com>
>> Cc: Kevin Wolf <kwolf@redhat.com>
>> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
>> ---
>>  docs/about/build-platforms.rst | 19 +++++++++++++++++++
>>  1 file changed, 19 insertions(+)
>> 
>> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
>> index 1c1e7b9e11c3..e1ea09789107 100644
>> --- a/docs/about/build-platforms.rst
>> +++ b/docs/about/build-platforms.rst
>> @@ -86,6 +86,25 @@ respective ports repository, while NetBSD will use the pkgsrc repository.
>>  For macOS, `Homebrew`_ will be used, although `MacPorts`_ is expected to carry
>>  similar versions.
>>  
>> +Python build dependencies
>> +~~~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The minimum supported version of Python is currently 3.6.
>> +
>> +Distributions with long-term support often provide multiple
>> +versions of the Python runtime.  QEMU aims to support the default
>> +Python runtime for 4 years after the initial release of a new version.
>> +Afterwards, you may have to point QEMU to a newer version of the Python
>> +runtime using the ``--python`` command line option of the ``configure``
>> +script.
>
> My counter proposal is to remove the 4 year marker, and give us
> more flexibility:
>
>  Distributions with long-term support often provide multiple
>  versions of the Python runtime.  QEMU will initially aim to
>  support the default python runtime. QEMU reserves the right
>  to increase its minimum version to any newer python that is
>  available as an option from the vendor.

Sounds good to me.

>> +
>> +Some of QEMU's build dependencies are written in Python and available
>> +through the Python Package Index (PyPI).  QEMU aims to be compatible
>> +with the versions packaged by common Linux distributions for the first
>> +4 years after the major release of the distribution.  After 4 years,
>> +you may have to use ``pip`` to install some of these build dependencies.
>
> And for this I'd say
>
>   If QEMU bumps its minimum python version to a non-default version,
>   then it may be neccessary to fetch python modules from 'pip' to
>   build or test QEMU.

Likewise.

On 2/17/23 16:29, Daniel P. Berrangé wrote:
>> * it introduces different support periods for*native*  vs.*non-native*
>>    dependencies.  Non-native dependencies are currently Python ones only,
>>    and for simplicity the policy only mentions Python; however, the concept
>>    generalizes to other languages with a well-known upstream package
>>    manager, that users of older distributions can fetch dependencies from;
>
> I guess this would mean its applicable to perl/ruby/etc, but really
> we should be exclusively using Python for our scripting needs, except
> for legacy stuff we already have. So in practice non-native == python.

Yeah, we used to have some Perl but it was always pretty basic in terms 
of dependencies.

> Potentially if we want to use Rust, we'd want vendoring of deps needed
> by Rust code, and that's native deps arguably, rather than non-native.

Indeed I was thinking more of Rust/Go than Perl or Ruby.  Distros 
generally don't try to include all possible Rust and Go dependencies or 
to build all Rust/Go packages with a single version of a dependency. 
LWN has had plenty of articles over the years about Debian and Fedora's 
failed attempts to do so.

So I considered those to be non-native dependencies in the sense of 
dependencies that can be fetched by a "non-native" package manager, i.e. 
one that is not rpm- or deb-based.

I'm not sure whether we would vendor the dependencies or download them 
at configure time, but anyway they would probably be vendored in the 
release tarball only, similar to submodules.

Paolo

On 2/18/23 10:27, Markus Armbruster wrote:
> This one is valid.  Still, it's a "want", not a "must have".  If it was
> a hard "must have", then CentOS 8 support would be in trouble: no mypy
> out of the box, as far as I can tell.

Right, and in fact this is by design: these tools move too fast for a 
version packaged for RHEL8 in 2017 to be useful.

> (A) Commit to keeping mypy happy for all versions of mypy provided by
>      any supported build host
> 
> (B) Pick a single mypy version
> 
>      Developers have to run this mypy version, or else punt to
>      maintainers.  We'll naturally pick a mypy version the maintainers
>      are happy to run, so no need for them to punt to CI.  We can exploit
>      the full typing power of that mypy version.  We don't have to work
>      around issues older versions may have.
> 
>      Developers unable to run this mypy version locally get delayed
>      feedback.  Affects the intersection of "people working on Python
>      code we check with mypy" and "people developing exclusively on older
>      hosts".  If we further intersect with "people doing so on a somewhat
>      regular basis", I posit we get the empty set.
> 
> I think (A) is pretty much all downsides.  I like (B), but could be too
> rigid in practice.

I think (B) is feasible.  Right now those developers can use CI and in 
the future we'll have the ability to automatically set up a virtual 
environment for them.

Paolo