[1/1] buildroot-test: enable toolchains based on hostarch

- updated toolchain csv to include hostarch
- fixed logic in autobuild-run to correctly select x86_64 vs x86

Signed-off-by: Matt Weber <matt@thewebers.ws>
---
 scripts/autobuild-run                        |  4 +-
 web/toolchains/configs/toolchain-configs.csv | 56 ++++++++++++++--------------
 2 files changed, 31 insertions(+), 29 deletions(-)

Hello,

On Tue, 29 Mar 2016 20:47:25 -0500, Matt Weber wrote:

> +http://autobuild.buildroot.org/toolchains/configs/br-aarch64-glibc.config,x86_64,glibc
> +http://autobuild.buildroot.org/toolchains/configs/br-arc-full-internal.config,x86_64,uclibc

This is not correct. The host arch should be empty, or some magic value
that says "any arch". Ditto for all "internal" toolchain configurations.

Also, I need to think about the deployment: if I deploy the new CSV
file on the server, but the build slaves have not yet updated their
autobuild-run script, what will happen.

I believe it *should* be fine: the old version of the autobuild-run
script will detect the host arch as "x86" so it will only use a subset
of the toolchains, but it will not fail badly or report errors. Then
when the script gets updated, it will properly detect the host arch as
x86_64 for the 64 bits machine, and it will restart using all possible
toolchains.

Thanks!

Thomas

Thomas,

On Wed, Mar 30, 2016 at 9:55 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Tue, 29 Mar 2016 20:47:25 -0500, Matt Weber wrote:
>
>> +http://autobuild.buildroot.org/toolchains/configs/br-aarch64-glibc.config,x86_64,glibc
>> +http://autobuild.buildroot.org/toolchains/configs/br-arc-full-internal.config,x86_64,uclibc
>
> This is not correct. The host arch should be empty, or some magic value
> that says "any arch". Ditto for all "internal" toolchain configurations.

I guess I don't completely follow why "any arch" for the internals, as
that host arch entry should represent the arch of the system the
toolchain was built on, correct?  Or are you saying that because an
internal normally would be built from scratch it would work on any
arch?  If so, maybe we need a test that checks the gcc executable for
arch rather then using this host arch entry?

I do have a bug as we would still want to allow the i386 toolchains to
execute on a x86_64 system.  Currently it excludes those with this
update.

>
> Also, I need to think about the deployment: if I deploy the new CSV
> file on the server, but the build slaves have not yet updated their
> autobuild-run script, what will happen.
>
> I believe it *should* be fine: the old version of the autobuild-run
> script will detect the host arch as "x86" so it will only use a subset
> of the toolchains, but it will not fail badly or report errors. Then
> when the script gets updated, it will properly detect the host arch as
> x86_64 for the 64 bits machine, and it will restart using all possible
> toolchains.
>
> Thanks!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com

Hello,

On Wed, 30 Mar 2016 19:41:00 -0500, Matthew Weber wrote:

> I guess I don't completely follow why "any arch" for the internals, as
> that host arch entry should represent the arch of the system the
> toolchain was built on, correct?  Or are you saying that because an
> internal normally would be built from scratch it would work on any
> arch? 

Precisely: internal toolchain configurations are configurations where
the toolchain is built by Buildroot. Since it's built from scratch, the
architecture of the machine on which we're running the build doesn't
matter.

> If so, maybe we need a test that checks the gcc executable for
> arch rather then using this host arch entry?

The problem is that we need to know *before* choosing the toolchain
configuration whether it will work or not on your machine. Otherwise,
you'll pick up one of the toolchain configuration, generate a random
set of packages, and then start the build. The build will start by
downloading the external toolchain, will attempt to run the external
toolchain gcc, which will fail and abort the build. Of course, at this
point, we could detect if the failure is caused by the gcc not being
for the right architecture, and instead of categorizing this failure as
a build failure, simply move on and generate another configuration. But:

 1/ This seems really wasteful.

 2/ It is not that trivial to distinguish build failures caused by the
    compiler not being built for the right architecture from other
    failures.

> I do have a bug as we would still want to allow the i386 toolchains to
> execute on a x86_64 system.  Currently it excludes those with this
> update.

Correct. i386 toolchains should be considered for x86_64 systems.

Thomas

Thomas,

On Wed, Mar 30, 2016 at 8:00 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Wed, 30 Mar 2016 19:41:00 -0500, Matthew Weber wrote:
>
>> I guess I don't completely follow why "any arch" for the internals, as
>> that host arch entry should represent the arch of the system the
>> toolchain was built on, correct?  Or are you saying that because an
>> internal normally would be built from scratch it would work on any
>> arch?
>
> Precisely: internal toolchain configurations are configurations where
> the toolchain is built by Buildroot. Since it's built from scratch, the
> architecture of the machine on which we're running the build doesn't
> matter.

But for the autobuilder....  wondering how to distinguish host arch
then for internals...

>
>> If so, maybe we need a test that checks the gcc executable for
>> arch rather then using this host arch entry?
>
> The problem is that we need to know *before* choosing the toolchain
> configuration whether it will work or not on your machine. Otherwise,
> you'll pick up one of the toolchain configuration, generate a random
> set of packages, and then start the build. The build will start by
> downloading the external toolchain, will attempt to run the external
> toolchain gcc, which will fail and abort the build. Of course, at this
> point, we could detect if the failure is caused by the gcc not being
> for the right architecture, and instead of categorizing this failure as
> a build failure, simply move on and generate another configuration. But:
>
>  1/ This seems really wasteful.

Agreed

>
>  2/ It is not that trivial to distinguish build failures caused by the
>     compiler not being built for the right architecture from other
>     failures.
>
>> I do have a bug as we would still want to allow the i386 toolchains to
>> execute on a x86_64 system.  Currently it excludes those with this
>> update.
>
> Correct. i386 toolchains should be considered for x86_64 systems.

I'm thinking something like this in the get_toolchain _configs() that
adds the conditional to allow that.
...
    (_, _, _, _, hostarch) = os.uname()
    # ~2015 distros report x86 when on a 32bit install
    if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
        hostarch = 'x86'
    # x86_64 defaults correctly, so not setting

    for row in csv.reader(l):
        config = {}
        config["url"] = row[0]
        config["hostarch"] = row[1]
        # Ignore toolchains that are not built for the appropriate
        # host architecture
        if hostarch != 'x86_64' and hostarch != config["hostarch"]:
            continue
        # Assuming x86_64 can also execute x86 toolchains
        if hostarch == 'x86_64' and config["hostarch"] != 'x86':
            continue
        config["libc"] = row[2]
        with urlopen_closing(config["url"]) as r:
            config["contents"] = decode_byte_list(r.readlines())
        configs.append(config)
    return configs

>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com

Thomas,

On Wed, Mar 30, 2016 at 8:11 PM, Matthew Weber <matt@thewebers.ws> wrote:
> Thomas,
>
> On Wed, Mar 30, 2016 at 8:00 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Hello,
>>
>> On Wed, 30 Mar 2016 19:41:00 -0500, Matthew Weber wrote:
>>
>>> I guess I don't completely follow why "any arch" for the internals, as
>>> that host arch entry should represent the arch of the system the
>>> toolchain was built on, correct?  Or are you saying that because an
>>> internal normally would be built from scratch it would work on any
>>> arch?
>>
>> Precisely: internal toolchain configurations are configurations where
>> the toolchain is built by Buildroot. Since it's built from scratch, the
>> architecture of the machine on which we're running the build doesn't
>> matter.
>
> But for the autobuilder....  wondering how to distinguish host arch
> then for internals...
>
>>
>>> If so, maybe we need a test that checks the gcc executable for
>>> arch rather then using this host arch entry?
>>
>> The problem is that we need to know *before* choosing the toolchain
>> configuration whether it will work or not on your machine. Otherwise,
>> you'll pick up one of the toolchain configuration, generate a random
>> set of packages, and then start the build. The build will start by
>> downloading the external toolchain, will attempt to run the external
>> toolchain gcc, which will fail and abort the build. Of course, at this
>> point, we could detect if the failure is caused by the gcc not being
>> for the right architecture, and instead of categorizing this failure as
>> a build failure, simply move on and generate another configuration. But:
>>
>>  1/ This seems really wasteful.
>
> Agreed
>
>>
>>  2/ It is not that trivial to distinguish build failures caused by the
>>     compiler not being built for the right architecture from other
>>     failures.
>>
>>> I do have a bug as we would still want to allow the i386 toolchains to
>>> execute on a x86_64 system.  Currently it excludes those with this
>>> update.
>>
>> Correct. i386 toolchains should be considered for x86_64 systems.
>
> I'm thinking something like this in the get_toolchain _configs() that
> adds the conditional to allow that.
> ...
>     (_, _, _, _, hostarch) = os.uname()
>     # ~2015 distros report x86 when on a 32bit install
>     if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
>         hostarch = 'x86'
>     # x86_64 defaults correctly, so not setting
>
>     for row in csv.reader(l):
>         config = {}
>         config["url"] = row[0]
>         config["hostarch"] = row[1]
>         # Ignore toolchains that are not built for the appropriate
>         # host architecture
>         if hostarch != 'x86_64' and hostarch != config["hostarch"]:
>             continue
>         # Assuming x86_64 can also execute x86 toolchains
>         if hostarch == 'x86_64' and config["hostarch"] != 'x86':
>             continue
>         config["libc"] = row[2]
>         with urlopen_closing(config["url"]) as r:
>             config["contents"] = decode_byte_list(r.readlines())
>         configs.append(config)
>     return configs
>

Updated to account for 'any' host arch.

    # ~2015 distros report x86 when on a 32bit install
    if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
        hostarch = 'x86'

    for row in csv.reader(l):
        config = {}
        config["url"] = row[0]
        config["hostarch"] = row[1]
        # Ignore toolchains that are not built for the appropriate
        # host architecture
        if hostarch != 'x86_64' and hostarch != config["hostarch"] and
config["hostarch"] != 'any' :
            continue
        # Assuming x86_64 can also execute x86 toolchains
        if hostarch == 'x86_64' and config["hostarch"] != 'x86' and
config["hostarch"] != 'any':
            continue
        config["libc"] = row[2]
        with urlopen_closing(config["url"]) as r:
            config["contents"] = decode_byte_list(r.readlines())
        configs.append(config)
    return configs

>>
>> Thomas
>> --
>> Thomas Petazzoni, CTO, Free Electrons
>> Embedded Linux, Kernel and Android engineering
>> http://free-electrons.com
>
>
>
> --
> Thanks,
> Matt

Hello,

On Wed, 30 Mar 2016 20:24:49 -0500, Matthew Weber wrote:

> Updated to account for 'any' host arch.
> 
>     # ~2015 distros report x86 when on a 32bit install
>     if hostarch == 'i686' or hostarch == 'i386' or hostarch == 'x86':
>         hostarch = 'x86'
> 
>     for row in csv.reader(l):
>         config = {}
>         config["url"] = row[0]
>         config["hostarch"] = row[1]
>         # Ignore toolchains that are not built for the appropriate
>         # host architecture
>         if hostarch != 'x86_64' and hostarch != config["hostarch"] and
> config["hostarch"] != 'any' :
>             continue
>         # Assuming x86_64 can also execute x86 toolchains
>         if hostarch == 'x86_64' and config["hostarch"] != 'x86' and
> config["hostarch"] != 'any':
>             continue
>         config["libc"] = row[2]
>         with urlopen_closing(config["url"]) as r:
>             config["contents"] = decode_byte_list(r.readlines())
>         configs.append(config)
>     return configs

Sounds good, but the conditions look complicated. What about something
like:

	keep = False

	# Keep all toolchain configs that work regardless of the host
	# architecture
	if config['hostarch'] == "any":
		keep = True

	# Keep all toolchain configs that can work on the current host
	# architecture
	if hostarch == config["hostarch"]:
		keep = True

	# Assume that x86 32 bits toolchains work on x86_64 build
	# machines
	if hostarch == 'x86_64' and config["hostarch"] == "x86":
		keep = True

	if not keep:
		continue
		
Thomas

Thomas,

On Wed, Mar 30, 2016 at 8:30 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Wed, 30 Mar 2016 20:24:49 -0500, Matthew Weber wrote:
>
<snip>

> Sounds good, but the conditions look complicated. What about something
> like:
>
>         keep = False
>
>         # Keep all toolchain configs that work regardless of the host
>         # architecture
>         if config['hostarch'] == "any":
>                 keep = True
>
>         # Keep all toolchain configs that can work on the current host
>         # architecture
>         if hostarch == config["hostarch"]:
>                 keep = True
>
>         # Assume that x86 32 bits toolchains work on x86_64 build
>         # machines
>         if hostarch == 'x86_64' and config["hostarch"] == "x86":
>                 keep = True
>
>         if not keep:
>                 continue
>

Appreciate the feedback.  I've pulled that in and will let things run
for the next day with all the changes we've discussed.