Message ID | alpine.DEB.2.20.1808092025590.28500@digraph.polyomino.org.uk |
---|---|
State | New |
Headers | show |
Series | Use gen-libm-test.py to generate ulps table for manual | expand |
Ping. This patch <https://sourceware.org/ml/libc-alpha/2018-08/msg00190.html> is pending review (for the principle of introducing the Python dependency for building the manual, at least).
On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote: > patch.) Note that this introduces a Python dependency for building > the manual, which is thus noted in install.texi and NEWS. This would add python to the bootstrap cycle which is a big problem. Andreas.
On Mon, 13 Aug 2018, Andreas Schwab wrote: > On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote: > > > patch.) Note that this introduces a Python dependency for building > > the manual, which is thus noted in install.texi and NEWS. > > This would add python to the bootstrap cycle which is a big problem. Why is it a big problem? I'd expect it to be at most a one-off addition to an automated bootstrap process. (a) If you are bootstrapping purely from sources and a minimum set of binaries (e.g. starting from GNU Mes), e.g. to defend against Thompson attacks, you'll inevitably need various techniques to break cycles. In this case it's very simple: an early glibc build might not build the manual (or run tests, etc.), but once Python has been built a full glibc build can build the manual and run tests. Even with Python required for the build, such a bootstrap from a minimum set of binaries would probably in practice need to start with building older, simpler versions of various tools with reduced dependencies (GNU Mes does). (b) If you are bootstrapping a new architecture by cross-compilation, (i) Python on the build system is fully sufficient for building the cross-compiled glibc including its manual, and (ii) either you can cross-compile Python, or you can build it natively later. (c) If you are repeatedly rebuilding packages with themselves - in the full configuration building everything from those packages intended to be in a release - to ensure the rebuilds converge to a set of binary packages that are byte-for-byte reproducible when rebuilt with themselves (a stronger version of reproducible builds than the ones where reproducibly building package X in a release depends on having an older version of package Y than the one included in that release), it doesn't even affect the set of packages that depend on each other because Python is already used in glibc testing. For both (a) and (b), having automation of the bootstrap process is clearly strongly desirable, and maybe a one-off change to that automation would be needed to insert a few extra builds. For (c) I'd expect existing automation to be able to work automatically with the build-dependencies so only those need updating for it to determine the updated set of packages to rebuild (and as noted, Python is already a dependency for running the full set of tests). There was clear support in the discussion descending from <https://sourceware.org/ml/libc-alpha/2018-07/msg00549.html> for replacing both perl and awk code in the build with python for the reasons described at <https://sourceware.org/ml/libc-alpha/2018-07/msg00559.html>: it's now better for the current community maintaining glibc. For the bootstrap cases discussed, I don't think Python is significantly different from any other widely-available build dependency - glibc is very careful about cyclic build dependencies in the actual cross-toolchain bootstrap for (b) (glibc built with a static-libgcc-only C-only GCC should be identical to one built in a cycle iterated more times, although you need to rebuild glibc with the shared-libgcc C++-supporting GCC to get a build tree appropriately configured to run the tests), but for tools that only need to be present on the build system to build glibc (not to be built specially for the glibc architecture), and where there aren't strong dependencies on particular versions, I think such bootstrap considerations are really matters for other projects not for glibc.
On Aug 13 2018, Joseph Myers <joseph@codesourcery.com> wrote: > (c) If you are repeatedly rebuilding packages with themselves - in the > full configuration building everything from those packages intended to be > in a release - to ensure the rebuilds converge to a set of binary packages > that are byte-for-byte reproducible when rebuilt with themselves (a > stronger version of reproducible builds than the ones where reproducibly > building package X in a release depends on having an older version of > package Y than the one included in that release), it doesn't even affect > the set of packages that depend on each other because Python is already > used in glibc testing. glibc testing is not part of the bootstrap cycle. Andreas.
On Mon, 13 Aug 2018, Andreas Schwab wrote: > On Aug 13 2018, Joseph Myers <joseph@codesourcery.com> wrote: > > > (c) If you are repeatedly rebuilding packages with themselves - in the > > full configuration building everything from those packages intended to be > > in a release - to ensure the rebuilds converge to a set of binary packages > > that are byte-for-byte reproducible when rebuilt with themselves (a > > stronger version of reproducible builds than the ones where reproducibly > > building package X in a release depends on having an older version of > > package Y than the one included in that release), it doesn't even affect > > the set of packages that depend on each other because Python is already > > used in glibc testing. > > glibc testing is not part of the bootstrap cycle. In that case you could also make the glibc manual build not part of the bootstrap cycle (until we generally move awk scripts to python as well, which was supported in the previous discussion). Or simply include python and its build-dependencies in that bootstrap cycle, much like perl and its build-dependencies (even if we successfully eliminate all direct perl uses in glibc, there will still be the indirect use from makeinfo).
On 08/13/2018 08:46 AM, Andreas Schwab wrote: > On Aug 09 2018, Joseph Myers <joseph@codesourcery.com> wrote: > >> patch.) Note that this introduces a Python dependency for building >> the manual, which is thus noted in install.texi and NEWS. > > This would add python to the bootstrap cycle which is a big problem. Why is this a big problem? Can you expand on your particular use case? Even in Fedora where we do stage1/2/3 boostrapping, we have to build glibc much later to enable manual building because we haven't yet built the tex packages.
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: > Even in Fedora where we do stage1/2/3 boostrapping, we have to build > glibc much later to enable manual building because we haven't yet > built the tex packages. texinfo does not require tex. Currently the following packages are part of the bootstrap cycle: <https://build.opensuse.org/project/repository_state/openSUSE:Factory/standard> acl, attr, audit, autoconf, automake, bc, binutils, bison, bzip2, cpio, cracklib, dejagnu, dwz, e2fsprogs, ed, elfutils, expect, fdupes, file, findutils, flex, gawk, gcc, gcc8, gdbm, gettext-runtime-mini, glibc, gmp, gpm, gzip, help2man, isl, keyutils, krb5-mini, libcap, libcap-ng, libdb-4_8, libnsl, libselinux, libsemanage, libsepol, libtirpc, libtool, libustr, libverto, libzio, linux-glibc-devel, lua-macros, lua53, m4, make, mpc, mpfr, ncurses, pam, patch, pcre, perl, perl-Text-Unidecode, perl-gettext, pkg-config, popt, readline, rpm, screen, shadow, system-user-root, system-users, systemd-rpm-macros, systemtap-headers, sysuser-tools, tcl, tcpd, texinfo, update-alternatives, utempter, util-linux, which, xz, zlib Andreas.
On 08/13/2018 12:01 PM, Andreas Schwab wrote: > On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: > >> Even in Fedora where we do stage1/2/3 boostrapping, we have to build >> glibc much later to enable manual building because we haven't yet >> built the tex packages. > > texinfo does not require tex. Sorry, I meant to write texinfo. > Currently the following packages are part of the bootstrap cycle: > <https://build.opensuse.org/project/repository_state/openSUSE:Factory/standard> > > acl, attr, audit, autoconf, automake, bc, binutils, bison, bzip2, cpio, > cracklib, dejagnu, dwz, e2fsprogs, ed, elfutils, expect, fdupes, file, > findutils, flex, gawk, gcc, gcc8, gdbm, gettext-runtime-mini, glibc, > gmp, gpm, gzip, help2man, isl, keyutils, krb5-mini, libcap, libcap-ng, > libdb-4_8, libnsl, libselinux, libsemanage, libsepol, libtirpc, libtool, > libustr, libverto, libzio, linux-glibc-devel, lua-macros, lua53, m4, > make, mpc, mpfr, ncurses, pam, patch, pcre, perl, perl-Text-Unidecode, > perl-gettext, pkg-config, popt, readline, rpm, screen, shadow, > system-user-root, system-users, systemd-rpm-macros, systemtap-headers, > sysuser-tools, tcl, tcpd, texinfo, update-alternatives, utempter, > util-linux, which, xz, zlib I'm impressed, you've done a really good job keeping this minimal, but what's the problem with including python here? Are you simply disappointed that your minimal list will need to grow a bit? Is there a technical issue at hand?
* Carlos O'Donell: > I'm impressed, you've done a really good job keeping this minimal, but what's > the problem with including python here? It would need a special bootstrap build, reducing external dependencies to the bare minimum. A typical distribution build of Python pulls in X, Java and perhaps more.
On 08/13/2018 04:04 PM, Florian Weimer wrote: > * Carlos O'Donell: > >> I'm impressed, you've done a really good job keeping this minimal, but what's >> the problem with including python here? > > It would need a special bootstrap build, reducing external > dependencies to the bare minimum. A typical distribution build of > Python pulls in X, Java and perhaps more. You still haven't answered the "Why?" Why does your bootstrap need to be smaller? What is wrong with "typical?"
* Carlos O'Donell: > On 08/13/2018 04:04 PM, Florian Weimer wrote: >> * Carlos O'Donell: >> >>> I'm impressed, you've done a really good job keeping this minimal, but what's >>> the problem with including python here? >> >> It would need a special bootstrap build, reducing external >> dependencies to the bare minimum. A typical distribution build of >> Python pulls in X, Java and perhaps more. > > You still haven't answered the "Why?" > > Why does your bootstrap need to be smaller? > > What is wrong with "typical?" When we did this in Fedora, the bootstrap set was pretty much the whole distribution, so fully bootstrapped builds are extremely rare and generally happen only during architecture bringup, and often not even then. Andreas' setup is very good for detecting certain types of build issues, and I wouldn't want to lose that.
On 08/13/2018 04:12 PM, Florian Weimer wrote: > * Carlos O'Donell: > >> On 08/13/2018 04:04 PM, Florian Weimer wrote: >>> * Carlos O'Donell: >>> >>>> I'm impressed, you've done a really good job keeping this minimal, but what's >>>> the problem with including python here? >>> >>> It would need a special bootstrap build, reducing external >>> dependencies to the bare minimum. A typical distribution build of >>> Python pulls in X, Java and perhaps more. >> >> You still haven't answered the "Why?" >> >> Why does your bootstrap need to be smaller? >> >> What is wrong with "typical?" > > When we did this in Fedora, the bootstrap set was pretty much the > whole distribution, so fully bootstrapped builds are extremely rare > and generally happen only during architecture bringup, and often not > even then. It is a question of interest really. Fedora has the technical ability to deliver a bootstrap that happens routinely, but I don't see the motivation to do so. Only mass rebuilds happen routinely when Fedora makes a release (don't have bootstrap issues). > Andreas' setup is very good for detecting certain types of build > issues, and I wouldn't want to lose that. I assume you are arguing that a full bootstrap would take much longer than the set of packages Andreas identifies. Please make such assertions clear. I don't know if Andreas has automation to run the builds in parallel across lots of machines and so it doesn't take that long to add more packages. The honest truth is that I'd like to see Python usable in the build process for glibc, so I think we'd have to bite the bullet on a minimal python at some point. $0.02.
* Carlos O'Donell: > It is a question of interest really. Fedora has the technical ability > to deliver a bootstrap that happens routinely, but I don't see the > motivation to do so. Sorry, this sounds … very unconvincing. This is a capability that deteriorates very quickly if you do not continuously exercise it. We can't even do periodic (say weekly) mass rebuilds in Fedora, and we know that there are uses for that, such as generating accurate markup (be it annobin or CET) based on the latest toolchain fixes. >> Andreas' setup is very good for detecting certain types of build >> issues, and I wouldn't want to lose that. > > I assume you are arguing that a full bootstrap would take much longer > than the set of packages Andreas identifies. Please make such assertions > clear. It's also much more brittle because as the Fedora experience shows, it is very difficult to keep even the core of the distribution in a permanently buildable state. So it's not purely about speed.
On 08/13/2018 04:29 PM, Florian Weimer wrote: > * Carlos O'Donell: > >> It is a question of interest really. Fedora has the technical ability >> to deliver a bootstrap that happens routinely, but I don't see the >> motivation to do so. > > Sorry, this sounds … very unconvincing. Could you clarify what you were not convinced about? The technical ability or lack of motivation? > This is a capability that deteriorates very quickly if you do not > continuously exercise it. Agreed. How does adding python prevent you from continuously exercising the bootstrap? Do you argue that it makes the whole build cycle too long? > We can't even do periodic (say weekly) mass rebuilds in Fedora, and we > know that there are uses for that, such as generating accurate markup > (be it annobin or CET) based on the latest toolchain fixes. Either we can't do it technically (not true, we can). Or we aren't sufficiently motivated to do it (the real case). I don't know why Andreas wants the core bootstrap small. I don't understand Andreas' motivation. If glibc upstream can't understand the reasons why downstream want a minimal bootstrap then we can't ourselves support the position either. You appear to argue that it would be nice if downstream Fedora had a minimal bootstrap (without python) to test: * Annobin (binary annotation work we are doing). * Intel CET (ABI flag day changes for *context() routines). That would be really nice, but the reality is that we don't value that and so we haven't invested in it. Why should glibc upstream value it? Using python in upstream glibc makes it easier to maintain, and the only downside is the bootstrap issue. The solution *today* is to not build the subsystems in glibc that need python (manual, benchtests) during bootstrap. If more things require python then we may need a minimal python for the bootstrap. Investing in that is a good idea only if you value what we have just discussed as a bootstrap. Other more useful bootstraps are IMO something like "Build all C++ programs with the latest C++ compiler changes, but do it in a side tag" which would allow the Fedora toolchain team to test C++ changes. These kinds of CI-esque workflows don't need a minimal python or a minimal bootstrap since the C++ applications you are building will likely already require much of the distro to be present. >>> Andreas' setup is very good for detecting certain types of build >>> issues, and I wouldn't want to lose that. >> >> I assume you are arguing that a full bootstrap would take much longer >> than the set of packages Andreas identifies. Please make such assertions >> clear. > > It's also much more brittle because as the Fedora experience shows, it > is very difficult to keep even the core of the distribution in a > permanently buildable state. So it's not purely about speed. Agreed. This still doesn't convince me that you have a compelling enough use case for minimal bootstraps that upstream glibc must not adopt python as a bootstrap requirement (which it isn't yet). To convince myself that the real solution is a python3-minimal solution I started with the python3 in Fedora Rawhide, and with maybe 5 minutes of spec file hacking I had a Python3 which built (in a clean rawhide mock chroot) *without* all the dependencies enabled. It appears to work for simple bootstrap things. Tests are still running, but languages tests are mostly passing (one _haslib failure). Is there a technical problem with needing python3-minimal? It seems like such a package would solve all sorts of early bootstrap issues with needing python?
* Carlos O'Donell: > On 08/13/2018 04:29 PM, Florian Weimer wrote: >> * Carlos O'Donell: >> >>> It is a question of interest really. Fedora has the technical ability >>> to deliver a bootstrap that happens routinely, but I don't see the >>> motivation to do so. >> >> Sorry, this sounds … very unconvincing. > > Could you clarify what you were not convinced about? The technical ability > or lack of motivation? The ability, both at a technical and organizational level. >> We can't even do periodic (say weekly) mass rebuilds in Fedora, and we >> know that there are uses for that, such as generating accurate markup >> (be it annobin or CET) based on the latest toolchain fixes. > > Either we can't do it technically (not true, we can). I doubt that. It would be a significant engineering effort.
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: > I'm impressed, you've done a really good job keeping this minimal, but what's > the problem with including python here? We want to keep the cycle as small as possible. For example, even though rpmlint is included in every build root and it is written in python, it is not part of the bootstrap cycle because it is replaced by a special package rpmlint-mini that contains copies of the python files and dependencies from the rpmlint package. That shows how important it is to keep python out of the equation. Andreas.
On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: > It is a question of interest really. Fedora has the technical ability > to deliver a bootstrap that happens routinely, but I don't see the > motivation to do so. Only mass rebuilds happen routinely when Fedora > makes a release (don't have bootstrap issues). In openSUSE Factory, every package update goes through a staging build where it is built and tested using the core subset of Factory (see <https://build.opensuse.org/project/dashboard/openSUSE:Factory>). Only after the testing succeeded the package is added to Factory. Andreas.
On Tue, 14 Aug 2018, Andreas Schwab wrote: > On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: > > > I'm impressed, you've done a really good job keeping this minimal, but what's > > the problem with including python here? > > We want to keep the cycle as small as possible. For example, even > though rpmlint is included in every build root and it is written in > python, it is not part of the bootstrap cycle because it is replaced by > a special package rpmlint-mini that contains copies of the python files > and dependencies from the rpmlint package. That shows how important it > is to keep python out of the equation. Given that you're already using special different packages in this bootstrap cycle, a special minimum python package is the obvious way to deal with uses in the glibc build.
On Aug 14 2018, Joseph Myers <joseph@codesourcery.com> wrote: > On Tue, 14 Aug 2018, Andreas Schwab wrote: > >> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: >> >> > I'm impressed, you've done a really good job keeping this minimal, but what's >> > the problem with including python here? >> >> We want to keep the cycle as small as possible. For example, even >> though rpmlint is included in every build root and it is written in >> python, it is not part of the bootstrap cycle because it is replaced by >> a special package rpmlint-mini that contains copies of the python files >> and dependencies from the rpmlint package. That shows how important it >> is to keep python out of the equation. > > Given that you're already using special different packages in this > bootstrap cycle, a special minimum python package is the obvious way to > deal with uses in the glibc build. That already exists, it is called python{3}-base. Andreas.
On Tue, Aug 14, 2018 at 3:10 AM, Andreas Schwab <schwab@suse.de> wrote: > On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: > >> I'm impressed, you've done a really good job keeping this minimal, but what's >> the problem with including python here? > > We want to keep the cycle as small as possible. For example, even > though rpmlint is included in every build root and it is written in > python, it is not part of the bootstrap cycle because it is replaced by > a special package rpmlint-mini that contains copies of the python files > and dependencies from the rpmlint package. That shows how important it > is to keep python out of the equation. I deeply sympathize with the desire to keep the bootstrap dependency graph as small as possible, but I also deeply sympathize with the glibc community's desire to move away from Perl and AWK. Especially AWK. Let me ask you: how many other packages (besides glibc) in the bootstrap depend on Perl? Is it within our reach, perhaps, to *replace* Perl with Python? zw
On Aug 14 2018, Zack Weinberg <zackw@panix.com> wrote: > Let me ask you: how many other packages (besides glibc) in the > bootstrap depend on Perl? Is it within our reach, perhaps, to > *replace* Perl with Python? You would need to rewrite autoconf and automake in python. Also, a lot of testsuites are using perl. Perl is ubiquitous, whereas python is used almost nowhere. Andreas.
On 08/14/2018 09:03 AM, Andreas Schwab wrote: > On Aug 14 2018, Joseph Myers <joseph@codesourcery.com> wrote: > >> On Tue, 14 Aug 2018, Andreas Schwab wrote: >> >>> On Aug 13 2018, Carlos O'Donell <carlos@redhat.com> wrote: >>> >>>> I'm impressed, you've done a really good job keeping this minimal, but what's >>>> the problem with including python here? >>> >>> We want to keep the cycle as small as possible. For example, even >>> though rpmlint is included in every build root and it is written in >>> python, it is not part of the bootstrap cycle because it is replaced by >>> a special package rpmlint-mini that contains copies of the python files >>> and dependencies from the rpmlint package. That shows how important it >>> is to keep python out of the equation. >> >> Given that you're already using special different packages in this >> bootstrap cycle, a special minimum python package is the obvious way to >> deal with uses in the glibc build. > > That already exists, it is called python{3}-base. What is your incremental cost to adding python3-base to the bootstrap?
On Tue, 14 Aug 2018, Zack Weinberg wrote: > Let me ask you: how many other packages (besides glibc) in the > bootstrap depend on Perl? Is it within our reach, perhaps, to > *replace* Perl with Python? We can't even replace it for building the glibc manual (makeinfo is written in Perl). What we can do is eliminate it as a *direct* dependency in glibc (so reducing the number of languages we need to deal with as glibc developers and the number of build-dependencies we need to document).
On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote:
> What is your incremental cost to adding python3-base to the bootstrap?
It's the cost of having another package in the bootstrap cycle, which
means building it at least three times.
Andreas.
On 08/15/2018 03:02 AM, Andreas Schwab wrote: > On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote: > >> What is your incremental cost to adding python3-base to the bootstrap? > > It's the cost of having another package in the bootstrap cycle, which > means building it at least three times. Does python3-base depend on many other things?
On Aug 15 2018, Carlos O'Donell <carlos@redhat.com> wrote: > On 08/15/2018 03:02 AM, Andreas Schwab wrote: >> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote: >> >>> What is your incremental cost to adding python3-base to the bootstrap? >> >> It's the cost of having another package in the bootstrap cycle, which >> means building it at least three times. > > Does python3-base depend on many other things? Yes, it drags in more packages into the bootstrap cycle. Andreas.
On 08/15/2018 10:42 AM, Andreas Schwab wrote: > On Aug 15 2018, Carlos O'Donell <carlos@redhat.com> wrote: > >> On 08/15/2018 03:02 AM, Andreas Schwab wrote: >>> On Aug 14 2018, Carlos O'Donell <carlos@redhat.com> wrote: >>> >>>> What is your incremental cost to adding python3-base to the bootstrap? >>> >>> It's the cost of having another package in the bootstrap cycle, which >>> means building it at least three times. >> >> Does python3-base depend on many other things? > > Yes, it drags in more packages into the bootstrap cycle. How many more packages? How is the boostrap build related to the testing a package has to go through before being added to Factory? Does the core subset of Factory, which is used in the package testing, get rebuilt every time the package is built, just to test that the core is still buildable?
Hi, [answering to a bounced mail, I'm not on libc-alpha@, so keep me CCed] I'm randomly using one mail of this thread as it covers most topics, but I use it for making general points, not only specifically answering to your mail Carlos. On Tue, 14 Aug 2018, Carlos O'Donell wrote: > > This is a capability that deteriorates very quickly if you do not > > continuously exercise it. > > Agreed. How does adding python prevent you from continuously exercising > the bootstrap? Do you argue that it makes the whole build cycle too > long? It's technically not (very) difficult to include python into the bootstrap set. But that's not the most important part. All SUSE distros since forever use this scheme of building the distro, a bootstrap set (iterating builds until stability), and once the bootstrap set is finished building all dependend packages in a topological order (breaking cycles in that other non-bootstrap set, if they exist, at defined points). That implies the following thing: every change in any package of the bootstrap set implies a whole-distro rebuild (any change in any of the other packages implies only a rebuild on the depending packages). So enlarging the bootstrap set makes full distro rebuild more common, and given that this takes about two to four days (depending on architecture) it's something you don't want to do lightly. So changes to packages in the bootstrap set are done more seldom than for other packages. > > We can't even do periodic (say weekly) mass rebuilds in Fedora, and we > > know that there are uses for that, such as generating accurate markup > > (be it annobin or CET) based on the latest toolchain fixes. > > Either we can't do it technically (not true, we can). I bet you theoretically can, in reality you don't. Such bootstrap capability detoriates within weeks if not done regularly (we for instance see this with old enterprise products where this is not regularly done; once you try this you go "oh my god, what is this, what happened here?"). > I don't know why Andreas wants the core bootstrap small. It's not Andreas, it's all (open)SUSE distros. It's the way we do this since ages and it's not small work (occasionally) to keep the bootstrap set small. Which is why we try very hard to avoid extending it, including having these very discussions. > I don't understand Andreas' motivation. I can bring up the original philosophy of how this came to be in the SUSE distros: whenever there's a change in any package, all other packages depending on it (per buildrequires, and transitively so) should be rebuilt (I won't got into the reasoning for this policy here). That requires a topological sort, and as such can't handle cycles. So we allowed exactly one cycle of packages, the bootstrap cycle. That one builds until a fixed point (or well, in reality all packages a max of three times). But all other packages are build in topological order. Obviously it's a good idea to let this cycle be small, so that at least the rest of packages can be built according to the golden rule. So, that's where we're coming from. As you can see in Andreas' list the bootstrap cycle is small (and with minor additions over time contains still the same things since 20 years; though I think we could remove some more things from it with a little work). Over time we of course got more cycles in BuildRequires than this (and the reason is often programmers not taking these things into account, e.g. adding a dependency on java from within gettext; just an example, not necessarily what happened), but still only the bootstrap cycle is iterated and implies a full distro rebuild. > Using python in upstream glibc makes it easier to maintain, and the only > downside is the bootstrap issue. (sidenote: if I were a glibc maintainer I'd disagree; I know perl better than python. Alas, I'm not :) Though one point I'll make is that everybody sufficiently intelligent can learn any reasonable language; and hence the like or dislike for specific languages should never in itself be a reason to choose one or the other. Rather all other reasons to prefer one over the other should trump such personal preferences. Such considerations include: preexisting use of language $X, wideness of availability for language $X interpreters/compilers and so on) Another point I made over at gcc@ is that for base tools like gcc and glibc the list of dependencies should be as minimal as possible, even at the expense of a little more work. In my world the former is of much higher importance than a little convenience for authors of helper scripts. To my liking even depending on perl should be avoided (surely a little text processing can just as well be done with awk). > The solution *today* is to not build the subsystems in glibc that need > python (manual, benchtests) during bootstrap. To note, for all of the above I'm not talking about use of anything for building optional stuff. We do the above, and if python is to be used for building the manual we'll split that off as well from the main glibc build. All my points above only apply to dependencies for building essential part, which for glibc would be, ... well, libc ;-) > If more things require python then we may need a minimal python for the > bootstrap. Investing in that is a good idea only if you value what we > have just discussed as a bootstrap. We split python already into a base and a full python (which requires tk and hence all of X). It might be that we "only" need to add openssl to the bootstrap cycle in addition to python3-base. It would be good if we can avoid all of this though. > Other more useful bootstraps are IMO something like "Build all C++ > programs with the latest C++ compiler changes, but do it in a side tag" > which would allow the Fedora toolchain team to test C++ changes. These > kinds of CI-esque workflows don't need a minimal python or a minimal > bootstrap since the C++ applications you are building will likely > already require much of the distro to be present. Actually at SUSE we do something similar to this as well. All compiler changes are going into a staging project where many packages are rebuilt. A new compiler version (and new glibc and similar things) goes through larger stagings. That is indeed nice, but orthogonal to the bootstrap topic. > Agreed. This still doesn't convince me that you have a compelling enough > use case for minimal bootstraps that upstream glibc must not adopt > python as a bootstrap requirement (which it isn't yet). What I'm always wondering in these discussion about switching languages for helper scripts (of all!) are compelling reasons to switch _to_ new infrastructures. Shouldn't the onus of finding such reasons be on those suggesting a change? (I realize that one stated reason is "easier maintenance", I merely disagree that even if this is true (which is at least questionable) that it'd be a very compelling reason). > To convince myself that the real solution is a python3-minimal solution > I started with the python3 in Fedora Rawhide, and with maybe 5 minutes > of spec file hacking I had a Python3 which built (in a clean rawhide > mock chroot) *without* all the dependencies enabled. As without any dependencies no C compiler would be installed I assume the mock chroot contains a multitude of things by default. So is your experiment for creating a -minimal variant valid? > Is there a technical problem with needing python3-minimal? It seems like > such a package would solve all sorts of early bootstrap issues with > needing python? As we currently don't have any such problems including it wouldn't solve anything. If you chose to create such problem, then yes, including a python3-base will be the technical solution to it. Ciao, Michael.
On Thu, 16 Aug 2018, Michael Matz wrote: > a reason to choose one or the other. Rather all other reasons to prefer > one over the other should trump such personal preferences. Such > considerations include: preexisting use of language $X, wideness of > availability for language $X interpreters/compilers and so on) I think popularity of a language is a perfectly relevant consideration (not the only one, of course). It's not simply among glibc developers that python is preferred to perl; various language rankings show python has much higher popularity than perl nowadays, which serves as external evidence that new contributors as well are more likely to be comfortable with build infrastructure in python than in perl. > Another point I made over at gcc@ is that for base tools like gcc and > glibc the list of dependencies should be as minimal as possible, even at I think the considerations for glibc are separate from those for GCC (given, for example, that glibc is almost certainly built, even when being cross-compiled, on a GNU system for which a range of build tools are likely already available, including python, whereas GCC allows for being built natively on a non-GNU system that may not have such tools, as a starting point for then building such other tools which might not build with random vendor compilers on proprietary Unices). Building glibc is also much more specialized than building GCC (if you're building glibc you're probably either a glibc developer or a system integrator using it as part of building a whole system, rather than just building a newer version locally for your own use, and using a different version of glibc than that coming with the system is tricky, whereas building your own GCC more recent than any that comes with the OS is an expected and supported normal use of GCC).
This discussion seems to have died down. How can we move further towards consensus on the use of Python in building the manual (taking account of the desires also expressed to use it to replace Awk scripts used in the build, and so make it a required build tool for building glibc at all)?
On 08/21/2018 01:20 PM, Joseph Myers wrote: > This discussion seems to have died down. How can we move further towards > consensus on the use of Python in building the manual (taking account of > the desires also expressed to use it to replace Awk scripts used in the > build, and so make it a required build tool for building glibc at all)? https://sourceware.org/glibc/wiki/Consensus#How_do_I_build_consensus.3F 1. Define or reiterate success criteria. Decide on the suitability of Python for core glibc scripts. 2. Identify a technical problems. SUSE does not want python to be used in core glibc scripts because it would increase the size of their bootstrap, costing them time, machine use, etc. 3. Look for resolution on a singular technical problem. SUSE says it will cost them to add python to the bootstrap. I expect that Joseph and I and perhaps others see Python is the strongest viable option to making the core glibc scripts maintainable from now into the future. Python is a growing language, users will be familiar with it, and all of that plays into new developer adoption for the project. The fact that SUSE has python-minimal to use is already indicative of that, and while I'm empathic to SUSE's need here, I think they are going to loose at some point and admit defeat that the minimal bootstrap needs some kind of full-featured interpreted language and that language is Python or Perl. Many professional development environments use LUA for this to limit license and dependency exposure, but in glibc we've been using Python. I'd like to hear that glibc's use of Python is going to cause SUSE "undue hardship" in accommodating this use of Python, otherwise I'd like to see us move forward with using Python. 4. Summarize the resolved list. Nobody has object to non-core uses of Python in glibc. We use python to run microbenchmarks, we use it to rebuild Unicode data sets, etc. The question that remains is the use of python for core glibc scripts.
On Fri, 24 Aug 2018, Carlos O'Donell wrote: > I'd like to hear that glibc's use of Python is going to cause SUSE > "undue hardship" in accommodating this use of Python, otherwise I'd like > to see us move forward with using Python. Do any SUSE people wish to comment further on this issue?
On 08/29/2018 12:56 PM, Joseph Myers wrote: > On Fri, 24 Aug 2018, Carlos O'Donell wrote: > >> I'd like to hear that glibc's use of Python is going to cause SUSE >> "undue hardship" in accommodating this use of Python, otherwise I'd like >> to see us move forward with using Python. > > Do any SUSE people wish to comment further on this issue? Joseph, I will ping this again on Friday, since that will have been a week since my request for information. I'm not sure we can move forward on this issue until Andreas makes his position clear, either a sustained objection (blocks consensus), or a recorded objection to the change (does not block consensus). Paul, I'd like your opinion on this also. Do you have any opinion on the use of python more uniformly within glibc to replace existing awk, perl, and shell scripts? It would require a minimal python in the bootstrap, which is what SUSE is objecting to. Discussion starts here: https://www.sourceware.org/ml/libc-alpha/2018-08/msg00218.html
Hi, On Wed, 29 Aug 2018, Carlos O'Donell wrote: > >> I'd like to hear that glibc's use of Python is going to cause SUSE > >> "undue hardship" in accommodating this use of Python, otherwise I'd > >> like to see us move forward with using Python. > > > > Do any SUSE people wish to comment further on this issue? > > Joseph, > I will ping this again on Friday, since that will have been a week since > my request for information. I'm sorry, I haven't seen the above followup(s) as I'm not on libc-alpha@ and Andreas is on vacation right now. I think we've said all we had to say. As engineers we can deal with most things (though I'd have expected this from new contributors to glibc as well), so we'll manage. Just try to not use python modules that aren't in the base distro of python or require native code parts. Ciao, Michael. P.S: not looking forward to the helper scripts being rewritten into javascript 10 years from now because "it's more popular" ;-/
On 08/30/2018 08:59 AM, Michael Matz wrote: > Hi, > > On Wed, 29 Aug 2018, Carlos O'Donell wrote: > >>>> I'd like to hear that glibc's use of Python is going to cause SUSE >>>> "undue hardship" in accommodating this use of Python, otherwise I'd >>>> like to see us move forward with using Python. >>> >>> Do any SUSE people wish to comment further on this issue? >> >> Joseph, >> I will ping this again on Friday, since that will have been a week since >> my request for information. > > I'm sorry, I haven't seen the above followup(s) as I'm not on libc-alpha@ > and Andreas is on vacation right now. I think we've said all we had to > say. As engineers we can deal with most things (though I'd have expected > this from new contributors to glibc as well), so we'll manage. Just try > to not use python modules that aren't in the base distro of python or > require native code parts. Does your position represent that of SUSE? Andreas is a project steward, and GNU project maintainer, so we still need his input on this issue. We will need to wait for him to come back and respond to this. Lastly, your objection is recorded, and that you will not block consensus on this issue e.g. "deal with most things." Thank you for your input. > P.S: not looking forward to the helper scripts being rewritten into > javascript 10 years from now because "it's more popular" ;-/ Are you kidding? We'll have a native WebAssembly port by then. You'll need a whole browser in the bootstrap ;-)
Hi, On Thu, 30 Aug 2018, Carlos O'Donell wrote: > > I'm sorry, I haven't seen the above followup(s) as I'm not on > > libc-alpha@ and Andreas is on vacation right now. I think we've said > > all we had to say. As engineers we can deal with most things (though > > I'd have expected this from new contributors to glibc as well), so > > we'll manage. Just try to not use python modules that aren't in the > > base distro of python or require native code parts. > > Does your position represent that of SUSE? Effectively, yes. Though of course, for real official positions of a company I guess I'd need to ask our press officer :) (The ones that will have most practical daytoday problems with including python in the bootstrap cycle aren't actually us, the toolchain team, but the build people and project managers trying to fit build schedules for their products; and of course the python packagers who then are under much stricter constraints in updating their package. For them I only speak out of experience not because we have had large meetings about this topic.) > Andreas is a project steward, and GNU project maintainer, so we still > need his input on this issue. We will need to wait for him to come back > and respond to this. Yes, definitely. > Lastly, your objection is recorded, and that you will not block > consensus on this issue e.g. "deal with most things." Thank you for your > input. I will also reserve the right to say "haha, told you so!" five years from now when people start complaining ;-) > > P.S: not looking forward to the helper scripts being rewritten into > > javascript 10 years from now because "it's more popular" ;-/ > > Are you kidding? We'll have a native WebAssembly port by then. You'll > need a whole browser in the bootstrap ;-) /me ogles musl ;) Ciao, Michael.
On 08/30/2018 10:14 AM, Michael Matz wrote: > On Thu, 30 Aug 2018, Carlos O'Donell wrote: >> Are you kidding? We'll have a native WebAssembly port by then. You'll >> need a whole browser in the bootstrap ;-) > > /me ogles musl ;) So will musl... https://github.com/jfbastien/musl
On Thu, 30 Aug 2018, Carlos O'Donell wrote:
> Are you kidding? We'll have a native WebAssembly port by then. You'll need a
It exists (complete with copyright assignments on file at the FSF), just
hasn't been updated lately (nor has the corresponding GCC port, and I
don't know how complete the binutils port - which *is* upstream - is).
https://github.com/pipcet/glibc
https://github.com/pipcet/gcc
I withdraw my objection, under the condition that the use of python is restricted to the base python language. Andreas.
On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote: > > I withdraw my objection, under the condition that the use of python is > restricted to the base python language. Can you define "base python language" more precisely, please? There are a lot of modules in the Python standard library, and many of them depend on externally-maintained libraries, and it's conceivable that we would find a use for 'shelve' for instance... zw
On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote: > On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote: >> >> I withdraw my objection, under the condition that the use of python is >> restricted to the base python language. > > Can you define "base python language" more precisely, please? Basically what is part of the openSUSE python3-base package. Andreas.
On Tue, 4 Sep 2018, Andreas Schwab wrote: > On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote: > > > On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote: > >> > >> I withdraw my objection, under the condition that the use of python is > >> restricted to the base python language. > > > > Can you define "base python language" more precisely, please? > > Basically what is part of the openSUSE python3-base package. Based on the description at <https://software.opensuse.org/package/python3-base> I've updated <https://sourceware.org/glibc/wiki/Style_and_Conventions> to say: * Scripts used in the build of glibc (a standard {{{make}}}, possibly regenerating files in the source tree if timestamps are out of order and the dependencies are present in the normal makefiles) need to avoid modules outside the Python standard library, and standard library modules depending on external libraries (such as XML, database and UI libraries). If a script used in the testsuite has other dependencies (e.g., the pretty printer tests using PExpect), the relevant tests should return UNSUPPORTED if those dependencies are not present. * Scripts used only by glibc developers may have other dependencies (e.g. requiring Python 3), where justified in a particular case. Are there other cases to add of standard library modules depending on external libraries that should be avoided? (There are external library dependencies for data compression modules and hashlib, for example; do we wish to exclude those, or say they are allowed?) Paul, Carlos asked for your input. Do you wish to comment on the use of Python in the glibc build (whether just for the manual, or more generally)? Do we wish to make Python a requirement for the build process, not just for the manual (require it in configure, remove PYTHON conditionals in the makefiles), so that we can use it to replace awk scripts required for the build? (My expectation is that on 1 January 2020 - or rather 1 Feburary 2020, since 1 January is the start of a release freeze - we can cease supporting Python 2 - that we can use the table at <https://devguide.python.org/> as the basis of what Python versions should be supported for building glibc. But we don't need to decide that now.)
On Tue, Sep 4, 2018 at 10:30 AM Joseph Myers <joseph@codesourcery.com> wrote: > On Tue, 4 Sep 2018, Andreas Schwab wrote: > > > On Sep 04 2018, Zack Weinberg <zackw@panix.com> wrote: > > > > > On Tue, Sep 4, 2018 at 9:34 AM Andreas Schwab <schwab@suse.de> wrote: > > >> > > >> I withdraw my objection, under the condition that the use of python is > > >> restricted to the base python language. > > > > > > Can you define "base python language" more precisely, please? > > > > Basically what is part of the openSUSE python3-base package. > > Based on the description at > <https://software.opensuse.org/package/python3-base> I've updated > <https://sourceware.org/glibc/wiki/Style_and_Conventions> to say: > > * Scripts used in the build of glibc (a standard {{{make}}}, possibly > regenerating files in the source tree if timestamps are out of order and > the dependencies are present in the normal makefiles) need to avoid > modules outside the Python standard library, and standard library modules > depending on external libraries (such as XML, database and UI libraries). The Debian equivalent, (lib)python2.7-minimal, has a README with an exact list of modules that may be used. This appears to be a subset of the stuff in SUSE's python3-base I've reproduced the contents of that file (from unstable; specifically, 2.7.15-3_amd64.deb) below. If SUSE has already gotten to the point where Python 2 is not part of their bootstrap, maybe we should ditch Python 2 now rather than waiting for its EOL. Debian currently has both python2.7-minimal and python3.6-minimal packages. The sets of standard library modules available from each appear to be roughly the same, after accounting for all the Python 3 module renames. May I additionally suggest that it's not obvious that this stuff is documented under "Code formatting in python sources", and the section should be renamed to indicate it's general policy for Python usage? zw ---- Contents of the python2.7-minimal package ----------------------------------------- python2.7-minimal consists of a minimum set of modules which may be needed for python scripts used during the boot process. If other packages are needed in these scripts, don't work around the missing module, but file a bug report against this package. The modules in this package are: __builtin__ builtin __future__ module _abcoll module _bisect extension _bytesio extension _codecs builtin _collections extension _fileio extension _functools extension _hashlib extensionx _heapq extension _io extension _locale extension _md5 extension _random extension _sha extension _sha256 extension _sha512 extension _socket extension _sre builtin _ssl extensionx _struct extension _symtable builtin _sysconfigdata module _types builtin _warnings builtin _weakref extension _weakrefset module abc module atexit module ConfigParser module StringIO module UserDict module cPickle extension cStringIO extension array extension base64 module binascii extension bisect module cmath extension calendar module codecs module collections module compileall module contextlib module copy module copy_reg module datetime extension dis module encodings package errno builtin exceptions builtin fcntl extension fnmatch module functools module gc builtin genericpath module getopt module glob module grp extension hashlib module heapq module imp builtin inspect module io module itertools extension keyword module linecache module locale module logging package marshal builtin math extension md5 module opcode module operator extension optparse module os module pickle module platform module popen2 module posix builtin posixpath module pkgutil module pwd builtin py_compile module random module re module repr module runpy module select extension sha module shutil module signal builtin socket module spwd extension sre module sre_compile module sre_constants module sre_parse module ssl module stat module string module strop extension struct module subprocess module sys builtin syslog extension sysconfig module tempfile module textwrap module time extension token module thread builtin token module tokenize module traceback module types module unicodedata extension weakref module warnings module zipimport extension zlib extension Included are as well the codecs and stringprep modules, and the encodings modules for all encodings except the multibyte encodings and the bz2 codec. The following modules are excluded, their import is guarded from the importing module: Used in Excluded ------------ ------------------------------------ os nt ntpath os2 os2emxpath mac macpath riscos riscospath riscosenviron optparse gettext pickle doctest subprocess threading This list was derived by looking at the modules in the perl-base package, then adding python specific "core modules". TODO's ------ - time.strptime cannot be used. The required _strptime module is not included in the -minimal package yet. _strptime, locale, _locale and calendar have to be added. - modules used very often in the testsuite: copy, cPickle, operator.
On Tue, 4 Sep 2018, Zack Weinberg wrote: > The Debian equivalent, (lib)python2.7-minimal, has a README with an > exact list of modules that may be used. This appears to be a subset > of the stuff in SUSE's python3-base I've reproduced the contents of > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below. That list does not include argparse. I think we should be able to use argparse in the glibc build.
On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote: > On Tue, 4 Sep 2018, Zack Weinberg wrote: > > The Debian equivalent, (lib)python2.7-minimal, has a README with an > > exact list of modules that may be used. This appears to be a subset > > of the stuff in SUSE's python3-base I've reproduced the contents of > > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below. > > That list does not include argparse. I think we should be able to use > argparse in the glibc build. I agree. argparse is a pure-Python module and it doesn't look like it depends on any C extensions that aren't already included in the "minimal" subset. I have filed wishlist bugs against python-minimal (#907977) and python3-minimal (#907976) and we'll see what happens. zw
On Tue, 4 Sep 2018, Zack Weinberg wrote: > On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote: > > On Tue, 4 Sep 2018, Zack Weinberg wrote: > > > The Debian equivalent, (lib)python2.7-minimal, has a README with an > > > exact list of modules that may be used. This appears to be a subset > > > of the stuff in SUSE's python3-base I've reproduced the contents of > > > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below. > > > > That list does not include argparse. I think we should be able to use > > argparse in the glibc build. > > I agree. argparse is a pure-Python module and it doesn't look like it > depends on any C extensions that aren't already included in the > "minimal" subset. I have filed wishlist bugs against python-minimal > (#907977) and python3-minimal (#907976) and we'll see what happens. Ubuntu 18.04 /usr/share/doc/python3.6-minimal/README.Debian *does* include argparse. Is that some Ubuntu-specific addition?
On Tue, Sep 4, 2018 at 1:29 PM Joseph Myers <joseph@codesourcery.com> wrote: > On Tue, 4 Sep 2018, Zack Weinberg wrote: > > On Tue, Sep 4, 2018 at 11:55 AM Joseph Myers <joseph@codesourcery.com> wrote: > > > On Tue, 4 Sep 2018, Zack Weinberg wrote: > > > > The Debian equivalent, (lib)python2.7-minimal, has a README with an > > > > exact list of modules that may be used. This appears to be a subset > > > > of the stuff in SUSE's python3-base I've reproduced the contents of > > > > that file (from unstable; specifically, 2.7.15-3_amd64.deb) below. > > > > > > That list does not include argparse. I think we should be able to use > > > argparse in the glibc build. > > > > I agree. argparse is a pure-Python module and it doesn't look like it > > depends on any C extensions that aren't already included in the > > "minimal" subset. I have filed wishlist bugs against python-minimal > > (#907977) and python3-minimal (#907976) and we'll see what happens. > > Ubuntu 18.04 /usr/share/doc/python3.6-minimal/README.Debian *does* include > argparse. Is that some Ubuntu-specific addition? Doh! No, I just missed it in the list. argparse is present in the python3.6-minimal list in version 3.6.6-3 of the package. zw
Dmitry, do you wish to make any comments on the use of Python in building glibc?
On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote: > Dmitry, do you wish to make any comments on the use of Python in building > glibc? My concern with making python a requirement to build glibc is that it will pull in more packages not required to build glibc. If there is a "minimal" subset of python that does not bring indirect dependencies to glibc build, and at the same time is sufficient to build glibc, then adding this subset of python as a requirement to build glibc shouldn't be a problem. I don't know yet whether SUSE or Debian take on the minimal subset pulls in extra dependencies, I'll check this when time permits.
On Mon, 10 Sep 2018, Dmitry V. Levin wrote: > On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote: > > Dmitry, do you wish to make any comments on the use of Python in building > > glibc? > > My concern with making python a requirement to build glibc is that it will > pull in more packages not required to build glibc. If there is a "minimal" > subset of python that does not bring indirect dependencies to glibc build, > and at the same time is sufficient to build glibc, then adding this subset > of python as a requirement to build glibc shouldn't be a problem. > > I don't know yet whether SUSE or Debian take on the minimal subset pulls > in extra dependencies, I'll check this when time permits. Any conclusions there? I'm not sure it really matters for these purposes what a particular distribution's package pulls in, if they are happy with what works for them for the distribution build or bootstrap. The question for us would be more about what constraints this imposes on distributions in general (e.g. if it means libffi will be needed for a native build of glibc, depending on how far the indirect dependencies on ctypes in the Python standard library go), and whether we are OK with those constraints. The dependencies of the minimal distribution packages are simply useful information for us to help in determining those constraints.
On Mon, 17 Sep 2018, Joseph Myers wrote: > On Mon, 10 Sep 2018, Dmitry V. Levin wrote: > > > On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote: > > > Dmitry, do you wish to make any comments on the use of Python in building > > > glibc? > > > > My concern with making python a requirement to build glibc is that it will > > pull in more packages not required to build glibc. If there is a "minimal" > > subset of python that does not bring indirect dependencies to glibc build, > > and at the same time is sufficient to build glibc, then adding this subset > > of python as a requirement to build glibc shouldn't be a problem. > > > > I don't know yet whether SUSE or Debian take on the minimal subset pulls > > in extra dependencies, I'll check this when time permits. > > Any conclusions there? > > I'm not sure it really matters for these purposes what a particular > distribution's package pulls in, if they are happy with what works for > them for the distribution build or bootstrap. The question for us would > be more about what constraints this imposes on distributions in general > (e.g. if it means libffi will be needed for a native build of glibc, > depending on how far the indirect dependencies on ctypes in the Python > standard library go), and whether we are OK with those constraints. The > dependencies of the minimal distribution packages are simply useful > information for us to help in determining those constraints. Ping. Any conclusion on this question about indirect dependencies introduced by use of Python?
On 25/09/18 17:54, Joseph Myers wrote: > On Mon, 17 Sep 2018, Joseph Myers wrote: >> On Mon, 10 Sep 2018, Dmitry V. Levin wrote: >>> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote: >>>> Dmitry, do you wish to make any comments on the use of Python in building >>>> glibc? >>> >>> My concern with making python a requirement to build glibc is that it will >>> pull in more packages not required to build glibc. If there is a "minimal" >>> subset of python that does not bring indirect dependencies to glibc build, >>> and at the same time is sufficient to build glibc, then adding this subset >>> of python as a requirement to build glibc shouldn't be a problem. >>> >>> I don't know yet whether SUSE or Debian take on the minimal subset pulls >>> in extra dependencies, I'll check this when time permits. >> >> Any conclusions there? >> >> I'm not sure it really matters for these purposes what a particular >> distribution's package pulls in, if they are happy with what works for >> them for the distribution build or bootstrap. The question for us would >> be more about what constraints this imposes on distributions in general >> (e.g. if it means libffi will be needed for a native build of glibc, >> depending on how far the indirect dependencies on ctypes in the Python >> standard library go), and whether we are OK with those constraints. The >> dependencies of the minimal distribution packages are simply useful >> information for us to help in determining those constraints. > > Ping. Any conclusion on this question about indirect dependencies > introduced by use of Python? using python3 in the build would be a bit disruptive for me, because the current systems where i build glibc don't have it. last time when make 4.0 requirement was added i had to build it on several machines until our infrastructure got updated to have make 4.0 in the path everywhere. (and our regular testing machines did not produce useful results for several days because of broken glibc build) installing python3 manually is obviously more effort and time, so please do not add such dependency anywhere near release time (like it was done with make 4). i don't have other concerns regarding indirect deps if the python3.4-minimal debian/ubuntu package works.
On Tue, 25 Sep 2018, Joseph Myers wrote: > On Mon, 17 Sep 2018, Joseph Myers wrote: > > > On Mon, 10 Sep 2018, Dmitry V. Levin wrote: > > > > > On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote: > > > > Dmitry, do you wish to make any comments on the use of Python in building > > > > glibc? > > > > > > My concern with making python a requirement to build glibc is that it will > > > pull in more packages not required to build glibc. If there is a "minimal" > > > subset of python that does not bring indirect dependencies to glibc build, > > > and at the same time is sufficient to build glibc, then adding this subset > > > of python as a requirement to build glibc shouldn't be a problem. > > > > > > I don't know yet whether SUSE or Debian take on the minimal subset pulls > > > in extra dependencies, I'll check this when time permits. > > > > Any conclusions there? > > > > I'm not sure it really matters for these purposes what a particular > > distribution's package pulls in, if they are happy with what works for > > them for the distribution build or bootstrap. The question for us would > > be more about what constraints this imposes on distributions in general > > (e.g. if it means libffi will be needed for a native build of glibc, > > depending on how far the indirect dependencies on ctypes in the Python > > standard library go), and whether we are OK with those constraints. The > > dependencies of the minimal distribution packages are simply useful > > information for us to help in determining those constraints. > > Ping. Any conclusion on this question about indirect dependencies > introduced by use of Python? Another ping on this question....
On 10/1/18 11:06 AM, Joseph Myers wrote: > On Tue, 25 Sep 2018, Joseph Myers wrote: > >> On Mon, 17 Sep 2018, Joseph Myers wrote: >> >>> On Mon, 10 Sep 2018, Dmitry V. Levin wrote: >>> >>>> On Mon, Sep 10, 2018 at 12:41:45PM +0000, Joseph Myers wrote: >>>>> Dmitry, do you wish to make any comments on the use of Python in building >>>>> glibc? >>>> >>>> My concern with making python a requirement to build glibc is that it will >>>> pull in more packages not required to build glibc. If there is a "minimal" >>>> subset of python that does not bring indirect dependencies to glibc build, >>>> and at the same time is sufficient to build glibc, then adding this subset >>>> of python as a requirement to build glibc shouldn't be a problem. >>>> >>>> I don't know yet whether SUSE or Debian take on the minimal subset pulls >>>> in extra dependencies, I'll check this when time permits. >>> >>> Any conclusions there? >>> >>> I'm not sure it really matters for these purposes what a particular >>> distribution's package pulls in, if they are happy with what works for >>> them for the distribution build or bootstrap. The question for us would >>> be more about what constraints this imposes on distributions in general >>> (e.g. if it means libffi will be needed for a native build of glibc, >>> depending on how far the indirect dependencies on ctypes in the Python >>> standard library go), and whether we are OK with those constraints. The >>> dependencies of the minimal distribution packages are simply useful >>> information for us to help in determining those constraints. >> >> Ping. Any conclusion on this question about indirect dependencies >> introduced by use of Python? > > Another ping on this question.... Could you help me to gather consensus? If you could write up the following: * Exact version of python to use. * Exact set of imports we need. I'll take that and try to get an acknowledgment from each of the distributions. I'll email all the maintainers and follow up with them (all on libc-alpha) in a new thread starting with the information you will put together for me. I'll also include some of the resolution about the concerns regarding building python on an old distribution.
On Mon, Oct 1, 2018 at 11:21 AM Carlos O'Donell <carlos@redhat.com> wrote: > On 10/1/18 11:06 AM, Joseph Myers wrote: > > On Tue, 25 Sep 2018, Joseph Myers wrote: > >> Ping. Any conclusion on this question about indirect dependencies > >> introduced by use of Python? > > > > Another ping on this question.... > Could you help me to gather consensus? > > If you could write up the following: > > * Exact version of python to use. > > * Exact set of imports we need. > > I'll take that and try to get an acknowledgment from each of the > distributions. I'll email all the maintainers and follow up with > them (all on libc-alpha) in a new thread starting with the information > you will put together for me. > > I'll also include some of the resolution about the concerns regarding > building python on an old distribution. For the record, although I offered some details earlier about how Debian packages Python, I have no official relationship to Debian and cannot speak for any of their teams that might care about this proposed change. As a glibc developer who might well take on some of the work of rewriting existing build logic in Python, I would like to put in a word for requiring Python 3.(>=4) rather than allowing 2.7 as well. The 2.x series is close to end-of-life, writing scripts that work with both major versions of the language is significantly harder than writing 3.x-only code, and I fear 2.x build compatibility will not be adequately tested. 3.4 is the oldest version of the 3.x series still in wide use AFAIK. zw
On Mon, 1 Oct 2018, Carlos O'Donell wrote: > If you could write up the following: > > * Exact version of python to use. Any non-EOL Python 3 series. So 3.4 or later at present, 3.5 or later could be required from 2019-03-16 (but that doesn't mean we have to increase the requirement then, if there aren't any new features we want to use), 3.6 or later could be required from 2020-09-13, as per the table at <https://devguide.python.org/>. > * Exact set of imports we need. Only modules in the Python (3.4) standard library, and excluding modules such as XML, database and UI libraries that depend on external libraries. I don't think we need anything depending on OpenSSL (hashlib appears to work without it, in Python 3.7, and we probably don't need hashlib anyway), but it's possible there may be dependencies on libffi - however, I think it's likely we can avoid ctypes which is what brings in the libffi dependency, and indirect dependencies on it from e.g. multiprocessing. I think trying to list exhaustively exactly which modules may be used is a bad idea, since there are lots of pure-Python modules in the standard library without any external C library dependencies (direct or indirect) and it's hard to predict exactly which could be useful at some point. gen-libm-test.py uses argparse, collections, os (after the present patch that would require it to build the manual) and re. The scripts used in the conform/ tests (not of course strictly relevant to build requirements, exact in regard to what tests can use unconditionally rather than needing to be conditionally UNSUPPORTED) also use shutil, subprocess, tempfile. I expect there would be plenty more pure-Python standard library modules that would turn out to be useful for some particular script.
On Mon, 1 Oct 2018, Carlos O'Donell wrote: > I'll take that and try to get an acknowledgment from each of the > distributions. I'll email all the maintainers and follow up with > them (all on libc-alpha) in a new thread starting with the information > you will put together for me. > > I'll also include some of the resolution about the concerns regarding > building python on an old distribution. Ping? I'd like to be able to move the gen-as-const machinery from awk + sed + makefile logic to python + simpler makefile logic, with a view to it being the starting point of general python infrastructure for extracting constants, macros etc. from C headers (there are various existing tests, e.g. for the syscall list and for signal numbers, that could be rewritten on top of such shared infrastructure, and I'd like to be able to add a lot more tests of consistency with uapi headers using such infrastructure).
On Thu, 11 Oct 2018, Joseph Myers wrote: > On Mon, 1 Oct 2018, Carlos O'Donell wrote: > > > I'll take that and try to get an acknowledgment from each of the > > distributions. I'll email all the maintainers and follow up with > > them (all on libc-alpha) in a new thread starting with the information > > you will put together for me. > > > > I'll also include some of the resolution about the concerns regarding > > building python on an old distribution. > > Ping? I'd like to be able to move the gen-as-const machinery from awk + > sed + makefile logic to python + simpler makefile logic, with a view to it > being the starting point of general python infrastructure for extracting > constants, macros etc. from C headers (there are various existing tests, > e.g. for the syscall list and for signal numbers, that could be rewritten > on top of such shared infrastructure, and I'd like to be able to add a lot > more tests of consistency with uapi headers using such infrastructure). Ping^4? Dmitry, Carlos, any other distributors, any further comments on requiring Python 3.4 (standard library modules not involving external C library dependencies) or later to build glibc?
* Joseph Myers: > Ping^4? Dmitry, Carlos, any other distributors, any further comments on > requiring Python 3.4 (standard library modules not involving external C > library dependencies) or later to build glibc? Fedora and Red Hat Enterprise Linux have sufficiently new versions of Python 3 readily available, so from a distribution perspective, it is not an issue for us. Debian 8/jessie is the oldest (LTS-)supported version, and it has Python 3.4, too. Debian wheezy does not, but I don't think you can reasonably use it for glibc development anyway. Thanks, Florian
On Wed, 17 Oct 2018, Florian Weimer wrote: > * Joseph Myers: > > > Ping^4? Dmitry, Carlos, any other distributors, any further comments on > > requiring Python 3.4 (standard library modules not involving external C > > library dependencies) or later to build glibc? > > Fedora and Red Hat Enterprise Linux have sufficiently new versions of > Python 3 readily available, so from a distribution perspective, it is > not an issue for us. Thanks. Dmitry? Your previous comments were at <https://sourceware.org/ml/libc-alpha/2018-09/msg00107.html>. Have you reached any conclusions regarding dependencies from minimal Python?
On Wed, Oct 17, 2018 at 02:20:45PM +0000, Joseph Myers wrote: > On Thu, 11 Oct 2018, Joseph Myers wrote: > > On Mon, 1 Oct 2018, Carlos O'Donell wrote: > > > > > I'll take that and try to get an acknowledgment from each of the > > > distributions. I'll email all the maintainers and follow up with > > > them (all on libc-alpha) in a new thread starting with the information > > > you will put together for me. > > > > > > I'll also include some of the resolution about the concerns regarding > > > building python on an old distribution. > > > > Ping? I'd like to be able to move the gen-as-const machinery from awk + > > sed + makefile logic to python + simpler makefile logic, with a view to it > > being the starting point of general python infrastructure for extracting > > constants, macros etc. from C headers (there are various existing tests, > > e.g. for the syscall list and for signal numbers, that could be rewritten > > on top of such shared infrastructure, and I'd like to be able to add a lot > > more tests of consistency with uapi headers using such infrastructure). > > Ping^4? Dmitry, Carlos, any other distributors, any further comments on > requiring Python 3.4 (standard library modules not involving external C > library dependencies) or later to build glibc? Thanks, I think this should be fine for us, too.
diff --git a/INSTALL b/INSTALL index 603b23b..518ed58 100644 --- a/INSTALL +++ b/INSTALL @@ -514,8 +514,9 @@ build the GNU C Library: * Python 2.7/3.4 or later - Python is required to run some tests. As of release time, Python - 3.6 is the newest verified to work for testing the GNU C Library. + Python is required to build the GNU C Library manual and to run + some tests. As of release time, Python 3.6 is the newest verified + to work for testing the GNU C Library. * PExpect 4.0 diff --git a/NEWS b/NEWS index 7868c84..b208404 100644 --- a/NEWS +++ b/NEWS @@ -23,7 +23,7 @@ Deprecated and removed features, and other changes affecting compatibility: Changes to build and runtime requirements: - [Add changes to build and runtime requirements here] +* Python is required to build the GNU C Library manual. Security related changes: diff --git a/manual/Makefile b/manual/Makefile index c275664..9c35c9d 100644 --- a/manual/Makefile +++ b/manual/Makefile @@ -99,10 +99,11 @@ $(objpfx)dir-add.texi: xtract-typefun.awk $(texis-path) # The table with the math errors is generated. $(objpfx)libm-err.texi: $(objpfx)stamp-libm-err -$(objpfx)stamp-libm-err: libm-err-tab.pl $(wildcard $(foreach dir,$(sysdirs),\ - $(dir)/libm-test-ulps)) +$(objpfx)stamp-libm-err: $(..)math/gen-libm-test.py \ + $(wildcard $(foreach dir,$(sysdirs),\ + $(dir)/libm-test-ulps)) pwd=`pwd`; \ - $(PERL) $< $$pwd/.. > $(objpfx)libm-err-tmp + $(PYTHON) $< -s $$pwd/.. -m $(objpfx)libm-err-tmp $(move-if-change) $(objpfx)libm-err-tmp $(objpfx)libm-err.texi touch $@ @@ -169,8 +170,9 @@ include ../Rules .PHONY: install subdir_install install-data install-data subdir_install: install -# Generated files requiring perl: libm-err.texi, summary.texi -ifneq ($(PERL),no) +# Generated files requiring python: libm-err.texi +# Generated files requiring perl: summary.texi +ifneq ($(if $(PYTHON),$(PERL),no),no) ifneq ($(strip $(MAKEINFO)),:) install: $(inst_infodir)/libc.info @if $(SHELL) -c '$(INSTALL_INFO) --version' >/dev/null 2>&1; then \ diff --git a/manual/install.texi b/manual/install.texi index df32b39..5372bc4 100644 --- a/manual/install.texi +++ b/manual/install.texi @@ -560,8 +560,9 @@ with any version of @code{sed}. As of release time, @code{sed} version @item Python 2.7/3.4 or later -Python is required to run some tests. As of release time, Python 3.6 -is the newest verified to work for testing @theglibc{}. +Python is required to build the @glibcadj{} manual and to run some +tests. As of release time, Python 3.6 is the newest verified to work +for testing @theglibc{}. @item PExpect 4.0 diff --git a/manual/libm-err-tab.pl b/manual/libm-err-tab.pl deleted file mode 100755 index 768d2da..0000000 --- a/manual/libm-err-tab.pl +++ /dev/null @@ -1,206 +0,0 @@ -#!/usr/bin/perl -w -# Copyright (C) 1999-2018 Free Software Foundation, Inc. -# This file is part of the GNU C Library. -# Contributed by Andreas Jaeger <aj@suse.de>, 1999. - -# The GNU C Library is free software; you can redistribute it and/or -# modify it under the terms of the GNU Lesser General Public -# License as published by the Free Software Foundation; either -# version 2.1 of the License, or (at your option) any later version. - -# The GNU C Library is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -# Lesser General Public License for more details. - -# You should have received a copy of the GNU Lesser General Public -# License along with the GNU C Library; if not, see -# <http://www.gnu.org/licenses/>. - -# Information about tests are stored in: %results -# $results{$test}{"type"} is the result type, e.g. normal or complex. -# In the following description $platform, $type and $float are: -# - $platform is the used platform -# - $type is either "normal", "real" (for the real part of a complex number) -# or "imag" (for the imaginary part # of a complex number). -# - $float is either of float, ifloat, double, idouble, ldouble, ildouble; -# It represents the underlying floating point type (float, double or long -# double) and if inline functions (the leading i stands for inline) -# are used. -# $results{$test}{$platform}{$type}{$float} is defined and has a delta -# or 'fail' as value. - -use File::Find; - -use strict; - -use vars qw ($sources @platforms %pplatforms); -use vars qw (%results @all_floats %suffices %all_functions); - - -# all_floats is in output order and contains all recognised float types that -# we're going to output -@all_floats = ('float', 'double', 'ldouble', 'float128'); -%suffices = - ( 'float' => 'f', - 'double' => '', - 'ldouble' => 'l', - 'float128' => 'f128' - ); - -# Pretty description of platform -%pplatforms = (); - -%all_functions = (); - -if ($#ARGV == 0) { - $sources = $ARGV[0]; -} else { - $sources = '/usr/src/cvs/libc'; -} - -find (\&find_files, $sources); - -@platforms = sort by_platforms @platforms; - -&print_all; - -sub find_files { - if ($_ eq 'libm-test-ulps') { - # print "Parsing $File::Find::name\n"; - push @platforms, $File::Find::dir; - my ($file, $name); - $file = "${File::Find::name}-name"; - open NAME, $file or die ("Can't open $file: $!"); - $name = <NAME>; - chomp $name; - close NAME; - $pplatforms{$File::Find::dir} = $name; - &parse_ulps ($File::Find::name, $File::Find::dir); - } -} - -# Parse ulps file -sub parse_ulps { - my ($file, $platform) = @_; - my ($test, $type, $float, $eps, $ignore_fn); - - # $type has the following values: - # "normal": No complex variable - # "real": Real part of complex result - # "imag": Imaginary part of complex result - open ULP, $file or die ("Can't open $file: $!"); - while (<ULP>) { - chop; - # ignore comments and empty lines - next if /^#/; - next if /^\s*$/; - if (/^Function: /) { - if (/Real part of/) { - s/Real part of //; - $type = 'real'; - } elsif (/Imaginary part of/) { - s/Imaginary part of //; - $type = 'imag'; - } else { - $type = 'normal'; - } - ($test) = ($_ =~ /^Function:\s*\"([a-zA-Z0-9_]+)\"/); - next; - } - if ($test =~ /_(downward|towardzero|upward|vlen)/) { - $ignore_fn = 1; - } else { - $ignore_fn = 0; - $all_functions{$test} = 1; - } - if (/^i?(float|double|ldouble|float128):/) { - ($float, $eps) = split /\s*:\s*/,$_,2; - if ($ignore_fn) { - next; - } elsif ($eps eq 'fail') { - $results{$test}{$platform}{$type}{$float} = 'fail'; - } elsif ($eps eq "0") { - # ignore - next; - } elsif (!exists $results{$test}{$platform}{$type}{$float} - || $results{$test}{$platform}{$type}{$float} ne 'fail') { - $results{$test}{$platform}{$type}{$float} = $eps; - } - if ($type =~ /^real|imag$/) { - $results{$test}{'type'} = 'complex'; - } elsif ($type eq 'normal') { - $results{$test}{'type'} = 'normal'; - } - next; - } - print "Skipping unknown entry: `$_'\n"; - } - close ULP; -} - -sub get_value { - my ($fct, $platform, $type, $float) = @_; - - return (exists $results{$fct}{$platform}{$type}{$float} - ? $results{$fct}{$platform}{$type}{$float} : "0"); -} - -sub print_platforms { - my (@p) = @_; - my ($fct, $platform, $float, $first, $i, $platform_no, $platform_total); - - print '@multitable {nexttowardf} '; - foreach (@p) { - print ' {1000 + i 1000}'; - } - print "\n"; - - print '@item Function '; - foreach (@p) { - print ' @tab '; - print $pplatforms{$_}; - } - print "\n"; - - - foreach $fct (sort keys %all_functions) { - foreach $float (@all_floats) { - print "\@item $fct$suffices{$float} "; - foreach $platform (@p) { - print ' @tab '; - if (exists $results{$fct}{$platform}{'normal'}{$float} - || exists $results{$fct}{$platform}{'real'}{$float} - || exists $results{$fct}{$platform}{'imag'}{$float}) { - if ($results{$fct}{'type'} eq 'complex') { - print &get_value ($fct, $platform, 'real', $float), - ' + i ', &get_value ($fct, $platform, 'imag', $float); - } else { - print $results{$fct}{$platform}{'normal'}{$float}; - } - } else { - print '-'; - } - } - print "\n"; - } - } - - print "\@end multitable\n"; -} - -sub print_all { - my ($i, $max); - - my ($columns) = 5; - - # Print only 5 platforms at a time. - for ($i=0; $i < $#platforms; $i+=$columns) { - $max = $i+$columns-1 > $#platforms ? $#platforms : $i+$columns-1; - print_platforms (@platforms[$i .. $max]); - } -} - -sub by_platforms { - return $pplatforms{$a} cmp $pplatforms{$b}; -} diff --git a/math/gen-libm-test.py b/math/gen-libm-test.py index 8d156ea..0b25a90 100755 --- a/math/gen-libm-test.py +++ b/math/gen-libm-test.py @@ -19,6 +19,7 @@ import argparse from collections import defaultdict +import os import re @@ -32,6 +33,16 @@ ALL_FLOATS_PFX = {'double': 'DBL', 'float': 'FLT', 'float128': 'FLT128'} +# Float types in the order used in the generated ulps tables in the +# manual. +ALL_FLOATS_MANUAL = ('float', 'double', 'ldouble', 'float128') + +# Map float types in ulps files to C function suffix. +ALL_FLOATS_SUFFIX = {'double': '', + 'ldouble': 'l', + 'float': 'f', + 'float128': 'f128'} + # Number of arguments in structure (as opposed to arguments that are # pointers to return values) for an argument descriptor. DESCR_NUM_ARGS = {'f': 1, 'a': 1, 'j': 1, 'i': 1, 'u': 1, 'l': 1, 'L': 1, @@ -161,6 +172,17 @@ class Ulps(object): ulps_dict[ulps_fn][line_first], ulps_val) + def all_functions(self): + """Return the set of functions with ulps and whether they are + complex.""" + funcs = set() + complex = {} + for k_prefix, k_dict in self.ulps_kinds: + for f in k_dict: + funcs.add(f) + complex[f] = True if k_prefix else False + return funcs, complex + def write(self, ulps_file): """Write ulps back out as a sorted ulps file.""" # Output is sorted first by function name, then by (real, @@ -231,6 +253,18 @@ class Ulps(object): f.write(header_text) +def read_all_ulps(srcdir): + """Read all platforms' libm-test-ulps files.""" + all_ulps = {} + for dirpath, dirnames, filenames in os.walk(srcdir): + if 'libm-test-ulps' in filenames: + with open(os.path.join(dirpath, 'libm-test-ulps-name')) as f: + name = f.read().rstrip() + all_ulps[name] = Ulps() + all_ulps[name].read(os.path.join(dirpath, 'libm-test-ulps')) + return all_ulps + + def read_auto_tests(test_file): """Read tests from auto-libm-test-out-<function> (possibly None).""" auto_tests = defaultdict(lambda: defaultdict(dict)) @@ -571,6 +605,60 @@ def generate_testfile(inc_input, auto_tests, c_output): f.write(''.join(test_list)) +def generate_err_table_sub(all_ulps, all_functions, fns_complex, platforms): + """Generate a single table within the overall ulps table section.""" + plat_width = [' {1000 + i 1000}' for p in platforms] + plat_header = [' @tab %s' % p for p in platforms] + table_list = ['@multitable {nexttowardf} %s\n' % ''.join(plat_width), + '@item Function %s\n' % ''.join(plat_header)] + for func in all_functions: + for flt in ALL_FLOATS_MANUAL: + func_ulps = [] + for p in platforms: + p_ulps = all_ulps[p] + if fns_complex[func]: + ulp_real = p_ulps.real[func][flt] + ulp_imag = p_ulps.imag[func][flt] + ulp_str = '%d + i %d' % (ulp_real, ulp_imag) + ulp_str = ulp_str if ulp_real or ulp_imag else '-' + else: + ulp = p_ulps.normal[func][flt] + ulp_str = str(ulp) if ulp else '-' + func_ulps.append(ulp_str) + table_list.append('@item %s%s @tab %s\n' + % (func, ALL_FLOATS_SUFFIX[flt], + ' @tab '.join(func_ulps))) + table_list.append('@end multitable\n') + return ''.join(table_list) + + +def generate_err_table(all_ulps, err_table): + """Generate ulps table for manual.""" + all_platforms = sorted(all_ulps.keys()) + functions_set = set() + functions_complex = {} + for p in all_platforms: + p_functions, p_complex = all_ulps[p].all_functions() + functions_set.update(p_functions) + functions_complex.update(p_complex) + all_functions = sorted([f for f in functions_set + if ('_downward' not in f + and '_towardzero' not in f + and '_upward' not in f + and '_vlen' not in f)]) + err_table_list = [] + # Print five platforms at a time. + num_platforms = len(all_platforms) + for i in range((num_platforms + 4) // 5): + start = i * 5 + end = i * 5 + 5 if num_platforms >= i * 5 + 5 else num_platforms + err_table_list.append(generate_err_table_sub(all_ulps, all_functions, + functions_complex, + all_platforms[start:end])) + with open(err_table, 'w') as f: + f.write(''.join(err_table_list)) + + def main(): """The main entry point.""" parser = argparse.ArgumentParser(description='Generate libm tests.') @@ -580,23 +668,31 @@ def main(): help='input file .inc file with tests') parser.add_argument('-u', dest='ulps_file', metavar='FILE', help='input file with ulps') + parser.add_argument('-s', dest='srcdir', metavar='DIR', + help='input source directory with all ulps') parser.add_argument('-n', dest='ulps_output', metavar='FILE', help='generate sorted ulps file FILE') parser.add_argument('-C', dest='c_output', metavar='FILE', help='generate output C file FILE from .inc file') parser.add_argument('-H', dest='ulps_header', metavar='FILE', help='generate output ulps header FILE') + parser.add_argument('-m', dest='err_table', metavar='FILE', + help='generate output ulps table for manual FILE') args = parser.parse_args() ulps = Ulps() if args.ulps_file is not None: ulps.read(args.ulps_file) auto_tests = read_auto_tests(args.auto_input) + if args.srcdir is not None: + all_ulps = read_all_ulps(args.srcdir) if args.ulps_output is not None: ulps.write(args.ulps_output) if args.ulps_header is not None: ulps.write_header(args.ulps_header) if args.c_output is not None: generate_testfile(args.inc_input, auto_tests, args.c_output) + if args.err_table is not None: + generate_err_table(all_ulps, args.err_table) if __name__ == '__main__':