mbox series

[0/1] ci: Run all available tests

Message ID 20210501185351.67197-1-toertel@gmail.com
Headers show
Series ci: Run all available tests | expand

Message

Mark Jonas May 1, 2021, 6:53 p.m. UTC
This patch enables running all available tests, i.e. Makefile targets
'test' and 'acceptance-tests'.

Please note that currently the acceptance-test ValidImageTest for
no_ssl_defconfig will most likely fail. At least this is what happens
in my personal SWUpdate development mirror on GitLab.com.

https://gitlab.com/toertel-contribute/swupdate/swupdate/-/jobs/1228985817

Maybe this is the same Makefile dependency problem we already discussed
on the mailing list in topic "[swupdate] acceptance-tests for
no_ssl_defconfig are failing ValidImageTest".

Mark Jonas (1):
  ci: Run all tests

 ci/setup.sh        | 4 +++-
 ci/test-configs.sh | 2 +-
 2 files changed, 4 insertions(+), 2 deletions(-)

--
2.25.1

Comments

Stefano Babic May 2, 2021, 9:50 a.m. UTC | #1
Hi Mark,

On 01.05.21 20:53, Mark Jonas wrote:
> This patch enables running all available tests, i.e. Makefile targets
> 'test' and 'acceptance-tests'.
> 
> Please note that currently the acceptance-test ValidImageTest for
> no_ssl_defconfig will most likely fail. At least this is what happens
> in my personal SWUpdate development mirror on GitLab.com.
> 
> https://gitlab.com/toertel-contribute/swupdate/swupdate/-/jobs/1228985817
> 
> Maybe this is the same Makefile dependency problem we already discussed
> on the mailing list in topic "[swupdate] acceptance-tests for
> no_ssl_defconfig are failing ValidImageTest".

It is not - this was a race condition and was solved with commit 0f6d01,
"Fix dependencies for acceptance-tests".

In this case, the test cannot be run at all because no crypto library is 
configured, but the test requires hash verification. Running the test 
locally, I see:

make V=1 tests:
--------------

[TRACE] : SWUPDATE running :  [get_common_fields] : Version 0.2.2
[TRACE] : SWUPDATE running :  [parse_hw_compatibility] : Accepted Hw 
Revision : 1
[TRACE] : SWUPDATE running :  [_parse_files] : Found File: hello.txt --> 
/home/hello.txt (ROOTFS)
[ERROR] : SWUPDATE failed [0] ERROR core/parser.c : check_hash_absent : 
52 : hash verification not enabled but hash supplied for hello.txt
[ERROR] : SWUPDATE failed [0] ERROR core/stream_interface.c : 
extract_files : 165 : Compatible SW not found
[ERROR] : SWUPDATE failed [1] Image invalid or corrupted. Not installing ...


make[1]: *** 
[/home/stefano/Projects/swupdate/scripts/acceptance-tests/CheckImage.mk:115: 
ValidImageTest] Error 1
make: *** [Makefile:495: acceptance-tests] Error 2

Best regards,
Stefano
Mark Jonas May 2, 2021, 12:30 p.m. UTC | #2
Hi Stefano,

On Sun, May 2, 2021 at 11:50 AM Stefano Babic <stefano.babic@denx.de> wrote:
>
> Hi Mark,
>
> On 01.05.21 20:53, Mark Jonas wrote:
> > This patch enables running all available tests, i.e. Makefile targets
> > 'test' and 'acceptance-tests'.
> >
> > Please note that currently the acceptance-test ValidImageTest for
> > no_ssl_defconfig will most likely fail. At least this is what happens
> > in my personal SWUpdate development mirror on GitLab.com.
> >
> > https://gitlab.com/toertel-contribute/swupdate/swupdate/-/jobs/1228985817
> >
> > Maybe this is the same Makefile dependency problem we already discussed
> > on the mailing list in topic "[swupdate] acceptance-tests for
> > no_ssl_defconfig are failing ValidImageTest".
>
> It is not - this was a race condition and was solved with commit 0f6d01,
> "Fix dependencies for acceptance-tests".
>
> In this case, the test cannot be run at all because no crypto library is
> configured, but the test requires hash verification. Running the test
> locally, I see:
>
> make V=1 tests:
> --------------
>
> [TRACE] : SWUPDATE running :  [get_common_fields] : Version 0.2.2
> [TRACE] : SWUPDATE running :  [parse_hw_compatibility] : Accepted Hw
> Revision : 1
> [TRACE] : SWUPDATE running :  [_parse_files] : Found File: hello.txt -->
> /home/hello.txt (ROOTFS)
> [ERROR] : SWUPDATE failed [0] ERROR core/parser.c : check_hash_absent :
> 52 : hash verification not enabled but hash supplied for hello.txt
> [ERROR] : SWUPDATE failed [0] ERROR core/stream_interface.c :
> extract_files : 165 : Compatible SW not found
> [ERROR] : SWUPDATE failed [1] Image invalid or corrupted. Not installing ...
>
>
> make[1]: ***
> [/home/stefano/Projects/swupdate/scripts/acceptance-tests/CheckImage.mk:115:
> ValidImageTest] Error 1
> make: *** [Makefile:495: acceptance-tests] Error 2

So would it be ok to add
    CONFIG_HASH_VERIFY=y
to no_ssl_defconfig? That definitely fixes the problem and I can send a patch.

Cheers,
Mark
Stefano Babic May 2, 2021, 12:48 p.m. UTC | #3
Hi Mark,

On 02.05.21 14:30, Mark Jonas wrote:
> Hi Stefano,
> 
> On Sun, May 2, 2021 at 11:50 AM Stefano Babic <stefano.babic@denx.de> wrote:
>>
>> Hi Mark,
>>
>> On 01.05.21 20:53, Mark Jonas wrote:
>>> This patch enables running all available tests, i.e. Makefile targets
>>> 'test' and 'acceptance-tests'.
>>>
>>> Please note that currently the acceptance-test ValidImageTest for
>>> no_ssl_defconfig will most likely fail. At least this is what happens
>>> in my personal SWUpdate development mirror on GitLab.com.
>>>
>>> https://gitlab.com/toertel-contribute/swupdate/swupdate/-/jobs/1228985817
>>>
>>> Maybe this is the same Makefile dependency problem we already discussed
>>> on the mailing list in topic "[swupdate] acceptance-tests for
>>> no_ssl_defconfig are failing ValidImageTest".
>>
>> It is not - this was a race condition and was solved with commit 0f6d01,
>> "Fix dependencies for acceptance-tests".
>>
>> In this case, the test cannot be run at all because no crypto library is
>> configured, but the test requires hash verification. Running the test
>> locally, I see:
>>
>> make V=1 tests:
>> --------------
>>
>> [TRACE] : SWUPDATE running :  [get_common_fields] : Version 0.2.2
>> [TRACE] : SWUPDATE running :  [parse_hw_compatibility] : Accepted Hw
>> Revision : 1
>> [TRACE] : SWUPDATE running :  [_parse_files] : Found File: hello.txt -->
>> /home/hello.txt (ROOTFS)
>> [ERROR] : SWUPDATE failed [0] ERROR core/parser.c : check_hash_absent :
>> 52 : hash verification not enabled but hash supplied for hello.txt
>> [ERROR] : SWUPDATE failed [0] ERROR core/stream_interface.c :
>> extract_files : 165 : Compatible SW not found
>> [ERROR] : SWUPDATE failed [1] Image invalid or corrupted. Not installing ...
>>
>>
>> make[1]: ***
>> [/home/stefano/Projects/swupdate/scripts/acceptance-tests/CheckImage.mk:115:
>> ValidImageTest] Error 1
>> make: *** [Makefile:495: acceptance-tests] Error 2
> 
> So would it be ok to add
>      CONFIG_HASH_VERIFY=y
> to no_ssl_defconfig? That definitely fixes the problem and I can send a patch.

It is ok.

Regards,
Stefano

> 
> Cheers,
> Mark
>