Proposal: Usage of stow in jenkins build scripts

Message ID 20170515193957.GA2859@yade.chaostreff.at
State New
Headers show

Commit Message

Alexander Huemer May 15, 2017, 7:39 p.m.
Hi,

The osmocom software that I do not install via apt but build myself gets 
installed in $HOME/usr and I do things like:

export PATH="${PATH}:${HOME}/usr/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOME}/usr/lib"
export PKG_CONFIG_PATH=$HOME/usr/lib/pkgconfig
MANPATH="${MANPATH}:${HOME}/usr/man"

Therefore I don't have to install to a destination outside my home 
directory which in turn means I can avoid some sudo calls.
But that actually does not matter so much, you can mentally substitute 
$HOME with '/' and append '/local' in the following if you want.

I don't install to $HOME/usr directly but use stow for 'managing' the 
installations, so an installation goes like this:

        $ mkdir -p $HOME/src/telco/osmo/libosmo-abis
        $ cd $HOME/src/telco/osmo/libosmo-abis
        $ git clone git://git.osmocom.org/libosmo-abis
        $ autoreconf -i
        $ ./configure --prefix=$HOME/usr/stow/libosmo-abis
        $ make
        $ make install
        $ STOW_DIR=$HOME/usr/stow stow libosmo-abis

This has two advantages:
* Good visibility of what is installed by a package.
  $ tree $HOME/usr/stow/libosmocore
* Possibility of uninstalling a package even if the Makefile is not 
  present.

For me those two advantages outweigh the small additional effort of 
using stow.

Because of that workflow I sometimes run into situations in which make 
runs fail because either header files are not found during compilation 
or libraries are not found during linking. The reason is that autotools 
variables pointing to pkgconfig supplied directories point to a 
destination under the prefix path with which configure was called. If a 
dependency was not mentioned in a Makefile.am the required files are not 
found because the other dependencies are in other directories.

For a system on which the osmocom-own dependency libs are installed in 
distinct directories, current openbsc master HEAD requires the attached 
patch for a build to succeed.

I admit that in most situations the dependencies are installed in the 
same directory so this does not matter, but it is still a small bug.

Therefore I propose to modify the jenkins build scripts so that they use 
distinct, stow managed directories for the dependencies. That should 
help to detect oversights, although admittedly developers would have to 
adapt their workflows to use stow as well in their daily hacking.
Otherwise an oversight would be detected by jenkings and not locally 
which in turn would require additional cleanup commits which is not so 
nice.

What is your gut feeling? Does the increased correctness of Makefiles.am 
contents justify the (IMO rather small) additional effort of using stow?

Kind regards,
-Alex

Comments

Neels Hofmeyr May 15, 2017, 8:18 p.m. | #1
The attached patch below indeed looks like errors in our build. Can you please
push this as a patch to gerrit?

And you are meaning to say: if we used stow in our jenkins builds, we would
catch these errors and fail builds if new ones are introduced, right?

What role exactly does stow play here -- do I get the same when I install to
separate '--prefix'es and then add all those prefixes to the PKG_CONFIG_PATH
and LD_LIBRARY_PATH?

The jenkins build scripts for each project are included in the contrib/ dir of
each git tree, using scripts found in the osmo-ci.git (also on gerrit). Feel
free to go ahead and submit patches that use stow, e.g. for the openbsc.git
build to begin with. If it improves our build by catching CFLAGS omissions I'll
happily merge it. I can also install packages that you need for this on the
build slaves.

~N

On Mon, May 15, 2017 at 09:39:57PM +0200, Alexander Huemer wrote:
> From e9fc0ed9c7b6c1acc70714f703983708f5d8a1ee Mon Sep 17 00:00:00 2001
> From: Alexander Huemer <alexander.huemer@xx.vu>
> Date: Mon, 15 May 2017 20:42:47 +0200
> Subject: [PATCH] Add missing _CFLAGS and _LIBS
> 
> These missing pieces go unnoticed if dependencies are not installed in
> distinct directories.
> 
> Change-Id: If8d57b72f63d79cc0d8efba7466c6ec177207cbb
> ---
>  openbsc/src/utils/Makefile.am       | 3 +++
>  openbsc/tests/sndcp_xid/Makefile.am | 3 ++-
>  2 files changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/openbsc/src/utils/Makefile.am b/openbsc/src/utils/Makefile.am
> index 9c3837a36..26494e13d 100644
> --- a/openbsc/src/utils/Makefile.am
> +++ b/openbsc/src/utils/Makefile.am
> @@ -109,6 +109,7 @@ osmo_meas_pcap2db_LDADD = \
>  osmo_meas_pcap2db_CFLAGS = \
>  	$(LIBOSMOCORE_CFLAGS) \
>  	$(LIBOSMOGSM_CFLAGS) \
> +	$(LIBOSMOABIS_CFLAGS) \
>  	$(NULL)
>  
>  osmo_meas_udp2db_SOURCES = \
> @@ -125,6 +126,7 @@ osmo_meas_udp2db_LDADD = \
>  osmo_meas_udp2db_CFLAGS = \
>  	$(LIBOSMOCORE_CFLAGS) \
>  	$(LIBOSMOGSM_CFLAGS) \
> +	$(LIBOSMOABIS_CFLAGS) \
>  	$(NULL)
>  
>  meas_json_SOURCES = \
> @@ -140,5 +142,6 @@ meas_json_LDADD = \
>  meas_json_CFLAGS = \
>  	$(LIBOSMOCORE_CFLAGS) \
>  	$(LIBOSMOGSM_CFLAGS) \
> +	$(LIBOSMOABIS_CFLAGS) \
>  	$(NULL)
>  
> diff --git a/openbsc/tests/sndcp_xid/Makefile.am b/openbsc/tests/sndcp_xid/Makefile.am
> index 99b9d1a4f..d09c41b28 100644
> --- a/openbsc/tests/sndcp_xid/Makefile.am
> +++ b/openbsc/tests/sndcp_xid/Makefile.am
> @@ -15,6 +15,7 @@ sndcp_xid_test_LDADD = \
>  	$(LIBOSMOGB_LIBS) \
>  	$(LIBCARES_LIBS) \
>  	$(LIBCRYPTO_LIBS) \
> -	-lgtp -lrt -lm
> +	$(LIBGTP_LIBS) \
> +	-lrt -lm
>  
>  
> -- 
> 2.11.0
>
Alexander Huemer May 15, 2017, 8:25 p.m. | #2
Hi!

On Mon, May 15, 2017 at 10:18:03PM +0200, Neels Hofmeyr wrote:
> The attached patch below indeed looks like errors in our build. Can you please
> push this as a patch to gerrit?

Will do.
 
> And you are meaning to say: if we used stow in our jenkins builds, we would
> catch these errors and fail builds if new ones are introduced, right?

That is the intention, yes, although stow is a convenience layer, not 
strictly required.

> What role exactly does stow play here -- do I get the same when I install to
> separate '--prefix'es and then add all those prefixes to the PKG_CONFIG_PATH
> and LD_LIBRARY_PATH?

Yes, the effect would be the same. Stow just makes all of that much more 
convenient and straight-forward, as you end up with just one location 
where (symlinks to) libs and so forth have to be searched. stow takes 
care of that.

> The jenkins build scripts for each project are included in the contrib/ dir of
> each git tree, using scripts found in the osmo-ci.git (also on gerrit). Feel
> free to go ahead and submit patches that use stow, e.g. for the openbsc.git
> build to begin with. If it improves our build by catching CFLAGS omissions I'll
> happily merge it. I can also install packages that you need for this on the
> build slaves.

I will take a look into that as well.
 
> On Mon, May 15, 2017 at 09:39:57PM +0200, Alexander Huemer wrote:
> > From e9fc0ed9c7b6c1acc70714f703983708f5d8a1ee Mon Sep 17 00:00:00 2001
> > From: Alexander Huemer <alexander.huemer@xx.vu>
> > Date: Mon, 15 May 2017 20:42:47 +0200
> > Subject: [PATCH] Add missing _CFLAGS and _LIBS
> > 
> > These missing pieces go unnoticed if dependencies are not installed in
> > distinct directories.
> > 
> > Change-Id: If8d57b72f63d79cc0d8efba7466c6ec177207cbb
> > ---
> >  openbsc/src/utils/Makefile.am       | 3 +++
> >  openbsc/tests/sndcp_xid/Makefile.am | 3 ++-
> >  2 files changed, 5 insertions(+), 1 deletion(-)
> > 
> > diff --git a/openbsc/src/utils/Makefile.am b/openbsc/src/utils/Makefile.am
> > index 9c3837a36..26494e13d 100644
> > --- a/openbsc/src/utils/Makefile.am
> > +++ b/openbsc/src/utils/Makefile.am
> > @@ -109,6 +109,7 @@ osmo_meas_pcap2db_LDADD = \
> >  osmo_meas_pcap2db_CFLAGS = \
> >  	$(LIBOSMOCORE_CFLAGS) \
> >  	$(LIBOSMOGSM_CFLAGS) \
> > +	$(LIBOSMOABIS_CFLAGS) \
> >  	$(NULL)
> >  
> >  osmo_meas_udp2db_SOURCES = \
> > @@ -125,6 +126,7 @@ osmo_meas_udp2db_LDADD = \
> >  osmo_meas_udp2db_CFLAGS = \
> >  	$(LIBOSMOCORE_CFLAGS) \
> >  	$(LIBOSMOGSM_CFLAGS) \
> > +	$(LIBOSMOABIS_CFLAGS) \
> >  	$(NULL)
> >  
> >  meas_json_SOURCES = \
> > @@ -140,5 +142,6 @@ meas_json_LDADD = \
> >  meas_json_CFLAGS = \
> >  	$(LIBOSMOCORE_CFLAGS) \
> >  	$(LIBOSMOGSM_CFLAGS) \
> > +	$(LIBOSMOABIS_CFLAGS) \
> >  	$(NULL)
> >  
> > diff --git a/openbsc/tests/sndcp_xid/Makefile.am b/openbsc/tests/sndcp_xid/Makefile.am
> > index 99b9d1a4f..d09c41b28 100644
> > --- a/openbsc/tests/sndcp_xid/Makefile.am
> > +++ b/openbsc/tests/sndcp_xid/Makefile.am
> > @@ -15,6 +15,7 @@ sndcp_xid_test_LDADD = \
> >  	$(LIBOSMOGB_LIBS) \
> >  	$(LIBCARES_LIBS) \
> >  	$(LIBCRYPTO_LIBS) \
> > -	-lgtp -lrt -lm
> > +	$(LIBGTP_LIBS) \
> > +	-lrt -lm
> >  
> >  
> > -- 
> > 2.11.0
> > 
> 
> 
> 
> 
> -- 
> - Neels Hofmeyr <nhofmeyr@sysmocom.de>          http://www.sysmocom.de/
> =======================================================================
> * sysmocom - systems for mobile communications GmbH
> * Alt-Moabit 93
> * 10559 Berlin, Germany
> * Sitz / Registered office: Berlin, HRB 134158 B
> * Geschäftsführer / Managing Directors: Harald Welte

Patch

From e9fc0ed9c7b6c1acc70714f703983708f5d8a1ee Mon Sep 17 00:00:00 2001
From: Alexander Huemer <alexander.huemer@xx.vu>
Date: Mon, 15 May 2017 20:42:47 +0200
Subject: [PATCH] Add missing _CFLAGS and _LIBS

These missing pieces go unnoticed if dependencies are not installed in
distinct directories.

Change-Id: If8d57b72f63d79cc0d8efba7466c6ec177207cbb
---
 openbsc/src/utils/Makefile.am       | 3 +++
 openbsc/tests/sndcp_xid/Makefile.am | 3 ++-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/openbsc/src/utils/Makefile.am b/openbsc/src/utils/Makefile.am
index 9c3837a36..26494e13d 100644
--- a/openbsc/src/utils/Makefile.am
+++ b/openbsc/src/utils/Makefile.am
@@ -109,6 +109,7 @@  osmo_meas_pcap2db_LDADD = \
 osmo_meas_pcap2db_CFLAGS = \
 	$(LIBOSMOCORE_CFLAGS) \
 	$(LIBOSMOGSM_CFLAGS) \
+	$(LIBOSMOABIS_CFLAGS) \
 	$(NULL)
 
 osmo_meas_udp2db_SOURCES = \
@@ -125,6 +126,7 @@  osmo_meas_udp2db_LDADD = \
 osmo_meas_udp2db_CFLAGS = \
 	$(LIBOSMOCORE_CFLAGS) \
 	$(LIBOSMOGSM_CFLAGS) \
+	$(LIBOSMOABIS_CFLAGS) \
 	$(NULL)
 
 meas_json_SOURCES = \
@@ -140,5 +142,6 @@  meas_json_LDADD = \
 meas_json_CFLAGS = \
 	$(LIBOSMOCORE_CFLAGS) \
 	$(LIBOSMOGSM_CFLAGS) \
+	$(LIBOSMOABIS_CFLAGS) \
 	$(NULL)
 
diff --git a/openbsc/tests/sndcp_xid/Makefile.am b/openbsc/tests/sndcp_xid/Makefile.am
index 99b9d1a4f..d09c41b28 100644
--- a/openbsc/tests/sndcp_xid/Makefile.am
+++ b/openbsc/tests/sndcp_xid/Makefile.am
@@ -15,6 +15,7 @@  sndcp_xid_test_LDADD = \
 	$(LIBOSMOGB_LIBS) \
 	$(LIBCARES_LIBS) \
 	$(LIBCRYPTO_LIBS) \
-	-lgtp -lrt -lm
+	$(LIBGTP_LIBS) \
+	-lrt -lm
 
 
-- 
2.11.0