[RFC] core: Download all package sources

Everyone,

This is an RFC for a new make option to download all package
sources to aid developers who cannot freely download packages.
This includes limited internet access or restrictive firewalls.
This is a best effort download. Any errors in downloading are
ignored rather than causing a failure.

The main changes in this patch is the addition of an "allsource"
make option that pulls down all of the package sources if a
version string is found. The version check is required because
packages that do not have a default version cause a invalid file
download attempt. I had to abstract the download steps from
pkg-generic.mk so they are callable by both the main Makefile
and the pkg-generic.mk download step.

Implementation details:
 1) The allsource command filters the make .VARIABLES for all
    "*_TARGET_SOURCE" and then strips off the "_TARGET_SOURCE"
    to get the package name. This results in a list of all
    of the packages that can be downloaded.
 2) The newly created PKG_DOWNLOAD define is called on all of
    the package names to download the source. This call does
    not cause the allsource command to fail if there is an
    error. This is only a best effort call.
 3) All of the sources are downloaded to the BR2_DL_DIR folder.

Signed-off-by: Clayton Shotwell <clshotwe@rockwellcollins.com>
---
 Makefile                |    8 +++++++-
 package/pkg-download.mk |   17 +++++++++++++++++
 package/pkg-generic.mk  |    9 +--------
 3 files changed, 25 insertions(+), 9 deletions(-)

>>>>> "Clayton" == Clayton Shotwell <clshotwe@rockwellcollins.com> writes:

 > Everyone,
 > This is an RFC for a new make option to download all package
 > sources to aid developers who cannot freely download packages.
 > This includes limited internet access or restrictive firewalls.
 > This is a best effort download. Any errors in downloading are
 > ignored rather than causing a failure.

 > The main changes in this patch is the addition of an "allsource"
 > make option that pulls down all of the package sources if a
 > version string is found. The version check is required because
 > packages that do not have a default version cause a invalid file
 > download attempt. I had to abstract the download steps from
 > pkg-generic.mk so they are callable by both the main Makefile
 > and the pkg-generic.mk download step.

 > Implementation details:
 >  1) The allsource command filters the make .VARIABLES for all
 >     "*_TARGET_SOURCE" and then strips off the "_TARGET_SOURCE"
 >     to get the package name. This results in a list of all
 >     of the packages that can be downloaded.
 >  2) The newly created PKG_DOWNLOAD define is called on all of
 >     the package names to download the source. This call does
 >     not cause the allsource command to fail if there is an
 >     error. This is only a best effort call.
 >  3) All of the sources are downloaded to the BR2_DL_DIR folder.

 > Signed-off-by: Clayton Shotwell <clshotwe@rockwellcollins.com>

Sorry, how is this different than make allpackageyesconfig; make source?

Dear Peter Korsgaard,

On Wed, 06 Nov 2013 22:46:11 +0100, Peter Korsgaard wrote:

>  > Signed-off-by: Clayton Shotwell <clshotwe@rockwellcollins.com>
> 
> Sorry, how is this different than make allpackageyesconfig; make source?

I'd really like allyespackageconfig to be the solution, because I don't
really like the idea of dumping all make variables, grepping in there,
and using that to download everything.

However, the big problem with allyespackageconfig is that it doesn't
download everything. As soon as you have a "choice" in Config.in (be it
for the package version, whether you want full X.org or KDrive X.org,
or Busybox init vs. sysvinit vs. systemd), then your
allyespackageconfig configuration is only going to contain the packages
that correspond to *one* possible choice selection. That leaves a large
number of packages that will not be downloaded.

Same thing for architecture dependent packages: grub depends on i386 or
x86-64, but ti-gfx depends on arm.

Best regards,

Thomas

Peter and Thomas,

Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote on 11/06/2013 
04:10:48 PM:

> I'd really like allyespackageconfig to be the solution, because I don't
> really like the idea of dumping all make variables, grepping in there,
> and using that to download everything.
> 
> However, the big problem with allyespackageconfig is that it doesn't
> download everything. As soon as you have a "choice" in Config.in (be it
> for the package version, whether you want full X.org or KDrive X.org,
> or Busybox init vs. sysvinit vs. systemd), then your
> allyespackageconfig configuration is only going to contain the packages
> that correspond to *one* possible choice selection. That leaves a large
> number of packages that will not be downloaded.
> 
> Same thing for architecture dependent packages: grub depends on i386 or
> x86-64, but ti-gfx depends on arm.

That is exactly the problem I was running into. As for the grep concern, 
I have not found another way to implement it. I needed to have a full list
of all of the packages but I was not able to find said variable. I spent 
some time digging through the Kconfig stuff to see if there way a way to
get around all of the dependencies but I could not find an easy way. 

The only changes I would like to make would be to display more
information if packages fail to download or can't be downloaded because
of missing information.

Thanks,
Clayton

Clayton Shotwell
Software Engineer
clshotwe@rockwellcollins.com
www.rockwellcollins.com

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Dear Peter Korsgaard,
 > On Wed, 06 Nov 2013 22:46:11 +0100, Peter Korsgaard wrote:

 >> > Signed-off-by: Clayton Shotwell <clshotwe@rockwellcollins.com>
 >> 
 >> Sorry, how is this different than make allpackageyesconfig; make source?

 > I'd really like allyespackageconfig to be the solution, because I don't
 > really like the idea of dumping all make variables, grepping in there,
 > and using that to download everything.

 > However, the big problem with allyespackageconfig is that it doesn't
 > download everything. As soon as you have a "choice" in Config.in (be it
 > for the package version, whether you want full X.org or KDrive X.org,
 > or Busybox init vs. sysvinit vs. systemd), then your
 > allyespackageconfig configuration is only going to contain the packages
 > that correspond to *one* possible choice selection. That leaves a large
 > number of packages that will not be downloaded.

 > Same thing for architecture dependent packages: grub depends on i386 or
 > x86-64, but ti-gfx depends on arm.

Yes, and thats the problem I have with keeping sources.buildroot.net
uptodate. Currently I "work around it" by doing a bunch of randconfig;
make source and hoping I will sooner or later catch them all.

But does this matter for USERS? I would guess users are only interested
in a subset of all the packages (E.G. the ones for the arch they use or
the ones enabled in their .config).

>>>>> "clshotwe" == clshotwe  <clshotwe@rockwellcollins.com> writes:

 >> Same thing for architecture dependent packages: grub depends on i386
 >> or x86-64, but ti-gfx depends on arm.

 > That is exactly the problem I was running into. As for the grep
 > concern, I have not found another way to implement it. I needed to
 > have a full list of all of the packages but I was not able to find
 > said variable. I spent some time digging through the Kconfig stuff to
 > see if there way a way to get around all of the dependencies but I
 > could not find an easy way.

I still don't quite get why you want to download the sources of all
package(-versions) we have in Buildroot. Can you explain in more detail?

Peter,

Peter Korsgaard <jacmet@gmail.com> wrote on 11/06/2013 04:33:58 PM:

>  >> Same thing for architecture dependent packages: grub depends on i386
>  >> or x86-64, but ti-gfx depends on arm.
> 
>  > That is exactly the problem I was running into. As for the grep
>  > concern, I have not found another way to implement it. I needed to
>  > have a full list of all of the packages but I was not able to find
>  > said variable. I spent some time digging through the Kconfig stuff to
>  > see if there way a way to get around all of the dependencies but I
>  > could not find an easy way.
> 
> I still don't quite get why you want to download the sources of all
> package(-versions) we have in Buildroot. Can you explain in more detail?

I do all of my builds behind a very restrictive firewall that does not
allow me to download the packages.  I am able to download them from
another computer and serve up the files behind the firewall.  It is
difficult to keep the sources up to date while tracking the Buildroot
mainline. 

Thanks,
Clayton

Clayton Shotwell
Software Engineer
clshotwe@rockwellcollins.com
www.rockwellcollins.com

>>>>> "clshotwe" == clshotwe  <clshotwe@rockwellcollins.com> writes:

Hi,

 >> I still don't quite get why you want to download the sources of all
 >> package(-versions) we have in Buildroot. Can you explain in more detail?

 > I do all of my builds behind a very restrictive firewall that does not
 > allow me to download the packages.  I am able to download them from
 > another computer and serve up the files behind the firewall.  It is
 > difficult to keep the sources up to date while tracking the Buildroot
 > mainline. 

So you have git access but not generic http access? An alternative could
be to grab the sources from sources.buildroot.net instead (but I only
update that one every wednesday).

You can get the needed tarball names using 'make external-deps'.

Peter, All,

On 2013-11-06 23:32 +0100, Peter Korsgaard spake thusly:
> >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
> 
>  > Dear Peter Korsgaard,
>  > On Wed, 06 Nov 2013 22:46:11 +0100, Peter Korsgaard wrote:
> 
>  >> > Signed-off-by: Clayton Shotwell <clshotwe@rockwellcollins.com>
>  >> 
>  >> Sorry, how is this different than make allpackageyesconfig; make source?
> 
>  > I'd really like allyespackageconfig to be the solution, because I don't
>  > really like the idea of dumping all make variables, grepping in there,
>  > and using that to download everything.
> 
>  > However, the big problem with allyespackageconfig is that it doesn't
>  > download everything. As soon as you have a "choice" in Config.in (be it
>  > for the package version, whether you want full X.org or KDrive X.org,
>  > or Busybox init vs. sysvinit vs. systemd), then your
>  > allyespackageconfig configuration is only going to contain the packages
>  > that correspond to *one* possible choice selection. That leaves a large
>  > number of packages that will not be downloaded.
> 
>  > Same thing for architecture dependent packages: grub depends on i386 or
>  > x86-64, but ti-gfx depends on arm.
> 
> Yes, and thats the problem I have with keeping sources.buildroot.net
> uptodate. Currently I "work around it" by doing a bunch of randconfig;
> make source and hoping I will sooner or later catch them all.
> 
> But does this matter for USERS? I would guess users are only interested
> in a subset of all the packages (E.G. the ones for the arch they use or
> the ones enabled in their .config).

Yes, it does.

Some companies are using a very restrictive firewall, that does not
allow anything but plain http and https (sometime even with MITM), and
no cvs, git, or even ftp...

So, some like to get a computer to another network, do the downloads, and
serve them on the LAN from a mirror.

Sometime, the network is really slow (eg. low BW, or high latency), so
this is also usefull to be able to make a complete mirror.

Of course, that needs resyncing from time to time, but easier done than
coping with a bunch of cursing angry users irrupting in one's office
every now and then! :-)

Regards,
Yann E. MORIN.

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 >> But does this matter for USERS? I would guess users are only interested
 >> in a subset of all the packages (E.G. the ones for the arch they use or
 >> the ones enabled in their .config).

 > Yes, it does.

 > Some companies are using a very restrictive firewall, that does not
 > allow anything but plain http and https (sometime even with MITM), and
 > no cvs, git, or even ftp...

 > So, some like to get a computer to another network, do the downloads, and
 > serve them on the LAN from a mirror.

 > Sometime, the network is really slow (eg. low BW, or high latency), so
 > this is also usefull to be able to make a complete mirror.

 > Of course, that needs resyncing from time to time, but easier done than
 > coping with a bunch of cursing angry users irrupting in one's office
 > every now and then! :-)

Sure, but that's what we have 'make source' (for specific
configurations) or sources.buildroot.net (for local mirror of
everything) for, right?

Peter,

Peter Korsgaard <jacmet@gmail.com> wrote on 11/06/2013 04:42:58 PM:

> From: Peter Korsgaard <jacmet@uclibc.org>
> To: clshotwe@rockwellcollins.com
> Cc: Peter Korsgaard <jacmet@uclibc.org>, buildroot@busybox.net, 
> Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Date: 11/06/2013 04:43 PM
> Subject: Re: [RFC] core: Download all package sources
> Sent by: Peter Korsgaard <jacmet@gmail.com>
> 
> >>>>> "clshotwe" == clshotwe  <clshotwe@rockwellcollins.com> writes:
> 
> Hi,
> 
>  >> I still don't quite get why you want to download the sources of all
>  >> package(-versions) we have in Buildroot. Can you explain in more 
detail?
> 
>  > I do all of my builds behind a very restrictive firewall that does 
not
>  > allow me to download the packages.  I am able to download them from
>  > another computer and serve up the files behind the firewall.  It is
>  > difficult to keep the sources up to date while tracking the Buildroot
>  > mainline. 
> 
> So you have git access but not generic http access? An alternative could
> be to grab the sources from sources.buildroot.net instead (but I only
> update that one every wednesday).
> 
> You can get the needed tarball names using 'make external-deps'.
> 
> -- 
> Bye, Peter Korsgaard

The firewall rules are very difficult to navigate around but we have
ways of getting git access. I could see this feature also being very
helpful for you to grab sources for the sources.buildroot.net.

Thanks,
Clayton

Clayton Shotwell
Software Engineer
clshotwe@rockwellcollins.com
www.rockwellcollins.com

Peter, Yann, Thomas, All,

buildroot-bounces@busybox.net wrote on 11/06/2013 05:07:29 PM:

>  >> But does this matter for USERS? I would guess users are only 
interested
>  >> in a subset of all the packages (E.G. the ones for the arch they use 
or
>  >> the ones enabled in their .config).
> 
>  > Yes, it does.
> 
>  > Some companies are using a very restrictive firewall, that does not
>  > allow anything but plain http and https (sometime even with MITM), 
and
>  > no cvs, git, or even ftp...
> 
>  > So, some like to get a computer to another network, do the downloads, 
and
>  > serve them on the LAN from a mirror.
> 
>  > Sometime, the network is really slow (eg. low BW, or high latency), 
so
>  > this is also usefull to be able to make a complete mirror.
> 
>  > Of course, that needs resyncing from time to time, but easier done 
than
>  > coping with a bunch of cursing angry users irrupting in one's office
>  > every now and then! :-)
> 
> Sure, but that's what we have 'make source' (for specific
> configurations) or sources.buildroot.net (for local mirror of
> everything) for, right?

I don't know what all configurations are being used so I would
rather be safe and pull down all of the packages. This would make
it easier than running make source on a bunch of different configurations.

Thanks,
Clayton

Clayton Shotwell
Software Engineer
clshotwe@rockwellcollins.com
www.rockwellcollins.com

>>>>> "clshotwe" == clshotwe  <clshotwe@rockwellcollins.com> writes:

 >> Sure, but that's what we have 'make source' (for specific
 >> configurations) or sources.buildroot.net (for local mirror of
 >> everything) for, right?

 > I don't know what all configurations are being used so I would
 > rather be safe and pull down all of the packages. This would make
 > it easier than running make source on a bunch of different configurations.

But you still only download a single version per package, so you might
not get all the needed sources (and you most likely download a lot more
than you need).

I personally dislike the patch as it's a bit of a hack, but if there's
really a need for something like this then perhaps - But wouldn't it
make more sense if we instead tried harder to ensure all the package
versions we support were available on our mirror?

if I understand correctly, the question is

"how to build/maintain a local buildroot source mirror" 

I would imagine a local buildroot clone with the default for mirror changed from sources.buildroot.net to whatever the local mirror is

and a simple script to update from the net...

maybe a script in tools/ would make more sense...


Cordialement 

Jérémy Rosen 
+33 (0)1 42 68 28 04

fight key loggers : write some perl using vim 


Open Wide Ingenierie

23, rue Daviel
75012 Paris - France
www.openwide.fr

>>>>> "Peter" == Peter Korsgaard <jacmet@uclibc.org> writes:

Hi,

 > I personally dislike the patch as it's a bit of a hack, but if there's
 > really a need for something like this then perhaps - But wouldn't it
 > make more sense if we instead tried harder to ensure all the package
 > versions we support were available on our mirror?

FYI, I've changed sources.buildroot.net to simply do:

make -k \
 $(find package -name \*.mk|sed 's|.*/\([^/]*\)\.mk|\1-source host-\1-source|'

E.G. foo-source + host-foo-source for all .mk files under package. It
also doesn't cover everything (E.G. packages with version selection),
but it gets the host packages and doesn't need any changes to buildroot
itself.

Peter Korsgaard <peter@korsgaard.com> wrote:
>>>>>> "Peter" == Peter Korsgaard <jacmet@uclibc.org> writes:
>
>Hi,
>
> > I personally dislike the patch as it's a bit of a hack, but if there's
> > really a need for something like this then perhaps - But wouldn't it
> > make more sense if we instead tried harder to ensure all the package
> > versions we support were available on our mirror?
>
>FYI, I've changed sources.buildroot.net to simply do:
>
>make -k \
> $(find package -name \*.mk|sed 's|.*/\([^/]*\)\.mk|\1-source host-\1-source|'
>
>E.G. foo-source + host-foo-source for all .mk files under package. It
>also doesn't cover everything (E.G. packages with version selection),
>but it gets the host packages and doesn't need any changes to buildroot
>itself.
>

You should probably add boot/ there as well.

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin@gmail.com> writes:

Hi,

 >> E.G. foo-source + host-foo-source for all .mk files under package. It
 >> also doesn't cover everything (E.G. packages with version selection),
 >> but it gets the host packages and doesn't need any changes to buildroot
 >> itself.
 >> 

 > You should probably add boot/ there as well.

True. We have version selection options for most of our bootloaders, but
it cannot hurt to try atleast. I'll simply drop 'package' and try all
.mk files.