[U-Boot,2/2] configs: sandbox: enable CONFIG_CMD_BOOTEFI_SELFTEST
diff mbox series

Message ID 20191109083902.27903-3-xypron.glpk@gmx.de
State Changes Requested
Delegated to: Simon Glass
Headers show
Series
  • sandbox: enable UEFI unit tests
Related show

Commit Message

Heinrich Schuchardt Nov. 9, 2019, 8:39 a.m. UTC
Activate UEFI unit tests on the sandbox.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
 configs/sandbox64_defconfig        | 1 +
 configs/sandbox_defconfig          | 1 +
 configs/sandbox_flattree_defconfig | 1 +
 configs/sandbox_spl_defconfig      | 1 +
 4 files changed, 4 insertions(+)

--
2.24.0.rc1

Comments

Simon Glass Dec. 10, 2019, 12:58 p.m. UTC | #1
Hi Heinrich,

On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Activate UEFI unit tests on the sandbox.
>
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> ---
>  configs/sandbox64_defconfig        | 1 +
>  configs/sandbox_defconfig          | 1 +
>  configs/sandbox_flattree_defconfig | 1 +
>  configs/sandbox_spl_defconfig      | 1 +
>  4 files changed, 4 insertions(+)

Unfortunately this slows down the testing too much, nearly doubling
the time in my tests.

I think the EFI console tests need to be modified to run in C instead
of all the drain_console() and p.timeout stuff. We need an effort to
speed up the tests, but certainly cannot make them any slower.

Regards,
Simon
Heinrich Schuchardt Dec. 10, 2019, 6:16 p.m. UTC | #2
On 12/10/19 1:58 PM, Simon Glass wrote:
> Hi Heinrich,
>
> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>
>> Activate UEFI unit tests on the sandbox.
>>
>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
>> ---
>>   configs/sandbox64_defconfig        | 1 +
>>   configs/sandbox_defconfig          | 1 +
>>   configs/sandbox_flattree_defconfig | 1 +
>>   configs/sandbox_spl_defconfig      | 1 +
>>   4 files changed, 4 insertions(+)
>
> Unfortunately this slows down the testing too much, nearly doubling
> the time in my tests.
>
> I think the EFI console tests need to be modified to run in C instead
> of all the drain_console() and p.timeout stuff. We need an effort to
> speed up the tests, but certainly cannot make them any slower.

Hello Simon,

thanks for pointing at the excessive timeout.

In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
we call drain_console() a lot indeed.

I am using the Python tests on real hardware. The text input and output
on the test systems uses the serial console. Whether you use C or Python
code to feed the serial adapter will not change the time needed to drain
the console.

We expect less then 50 characters of output per test step. At 9600 baud
draining 50 characters would require 52 ms.

So I would suggest that we add a parameter timeout to drain_console()
which defaults to 1000 ms as currently. But in the test we can set it to
50 ms. p.timeout could be changed likewise.

What are your thoughts?

Best regards

Heinrich
Simon Glass Dec. 11, 2019, 3:43 p.m. UTC | #3
Hi Heinrich,

On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 12/10/19 1:58 PM, Simon Glass wrote:
> > Hi Heinrich,
> >
> > On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >>
> >> Activate UEFI unit tests on the sandbox.
> >>
> >> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> >> ---
> >>   configs/sandbox64_defconfig        | 1 +
> >>   configs/sandbox_defconfig          | 1 +
> >>   configs/sandbox_flattree_defconfig | 1 +
> >>   configs/sandbox_spl_defconfig      | 1 +
> >>   4 files changed, 4 insertions(+)
> >
> > Unfortunately this slows down the testing too much, nearly doubling
> > the time in my tests.
> >
> > I think the EFI console tests need to be modified to run in C instead
> > of all the drain_console() and p.timeout stuff. We need an effort to
> > speed up the tests, but certainly cannot make them any slower.
>
> Hello Simon,
>
> thanks for pointing at the excessive timeout.
>
> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
> we call drain_console() a lot indeed.
>
> I am using the Python tests on real hardware. The text input and output
> on the test systems uses the serial console. Whether you use C or Python
> code to feed the serial adapter will not change the time needed to drain
> the console.
>
> We expect less then 50 characters of output per test step. At 9600 baud
> draining 50 characters would require 52 ms.
>
> So I would suggest that we add a parameter timeout to drain_console()
> which defaults to 1000 ms as currently. But in the test we can set it to
> 50 ms. p.timeout could be changed likewise.
>
> What are your thoughts?

I think you might be missing my point.

I think as a matter of policy we should require that test run mostly
on the device, with just the results being output. By using C I mean
to write the test in C instead of Python - just do the same steps.
E.g. do a lot of run_command() calls and check the results. This
should be much faster.

Regards,
Simon
Heinrich Schuchardt Dec. 11, 2019, 4:59 p.m. UTC | #4
On 12/11/19 4:43 PM, Simon Glass wrote:
> Hi Heinrich,
>
> On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>
>> On 12/10/19 1:58 PM, Simon Glass wrote:
>>> Hi Heinrich,
>>>
>>> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>>>
>>>> Activate UEFI unit tests on the sandbox.
>>>>
>>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
>>>> ---
>>>>    configs/sandbox64_defconfig        | 1 +
>>>>    configs/sandbox_defconfig          | 1 +
>>>>    configs/sandbox_flattree_defconfig | 1 +
>>>>    configs/sandbox_spl_defconfig      | 1 +
>>>>    4 files changed, 4 insertions(+)
>>>
>>> Unfortunately this slows down the testing too much, nearly doubling
>>> the time in my tests.
>>>
>>> I think the EFI console tests need to be modified to run in C instead
>>> of all the drain_console() and p.timeout stuff. We need an effort to
>>> speed up the tests, but certainly cannot make them any slower.
>>
>> Hello Simon,
>>
>> thanks for pointing at the excessive timeout.
>>
>> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
>> we call drain_console() a lot indeed.
>>
>> I am using the Python tests on real hardware. The text input and output
>> on the test systems uses the serial console. Whether you use C or Python
>> code to feed the serial adapter will not change the time needed to drain
>> the console.
>>
>> We expect less then 50 characters of output per test step. At 9600 baud
>> draining 50 characters would require 52 ms.
>>
>> So I would suggest that we add a parameter timeout to drain_console()
>> which defaults to 1000 ms as currently. But in the test we can set it to
>> 50 ms. p.timeout could be changed likewise.
>>
>> What are your thoughts?
>
> I think you might be missing my point.
>
> I think as a matter of policy we should require that test run mostly
> on the device, with just the results being output. By using C I mean
> to write the test in C instead of Python - just do the same steps.
> E.g. do a lot of run_command() calls and check the results. This
> should be much faster.
>

The current test checks if Unicode signs sent over the physical serial
interface are correctly interpreted in U-Boot. One reason for failure
could be that the UART driver uses bit 7 as a parity bit.

U-Boot's internal function run_command() cannot send electrical signals
inbound to the device's UART.

You may implement an additional test with a different scope. But that
should not replace the current test.

Best regards

Heinrich
Simon Glass Dec. 11, 2019, 5:07 p.m. UTC | #5
Hi Heinrich,

On Wed, 11 Dec 2019 at 09:59, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 12/11/19 4:43 PM, Simon Glass wrote:
> > Hi Heinrich,
> >
> > On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >>
> >> On 12/10/19 1:58 PM, Simon Glass wrote:
> >>> Hi Heinrich,
> >>>
> >>> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >>>>
> >>>> Activate UEFI unit tests on the sandbox.
> >>>>
> >>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> >>>> ---
> >>>>    configs/sandbox64_defconfig        | 1 +
> >>>>    configs/sandbox_defconfig          | 1 +
> >>>>    configs/sandbox_flattree_defconfig | 1 +
> >>>>    configs/sandbox_spl_defconfig      | 1 +
> >>>>    4 files changed, 4 insertions(+)
> >>>
> >>> Unfortunately this slows down the testing too much, nearly doubling
> >>> the time in my tests.
> >>>
> >>> I think the EFI console tests need to be modified to run in C instead
> >>> of all the drain_console() and p.timeout stuff. We need an effort to
> >>> speed up the tests, but certainly cannot make them any slower.
> >>
> >> Hello Simon,
> >>
> >> thanks for pointing at the excessive timeout.
> >>
> >> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
> >> we call drain_console() a lot indeed.
> >>
> >> I am using the Python tests on real hardware. The text input and output
> >> on the test systems uses the serial console. Whether you use C or Python
> >> code to feed the serial adapter will not change the time needed to drain
> >> the console.
> >>
> >> We expect less then 50 characters of output per test step. At 9600 baud
> >> draining 50 characters would require 52 ms.
> >>
> >> So I would suggest that we add a parameter timeout to drain_console()
> >> which defaults to 1000 ms as currently. But in the test we can set it to
> >> 50 ms. p.timeout could be changed likewise.
> >>
> >> What are your thoughts?
> >
> > I think you might be missing my point.
> >
> > I think as a matter of policy we should require that test run mostly
> > on the device, with just the results being output. By using C I mean
> > to write the test in C instead of Python - just do the same steps.
> > E.g. do a lot of run_command() calls and check the results. This
> > should be much faster.
> >
>
> The current test checks if Unicode signs sent over the physical serial
> interface are correctly interpreted in U-Boot. One reason for failure
> could be that the UART driver uses bit 7 as a parity bit.

How is this testing U-Boot? This sounds like a test of the user environment?

Even if the test passes on gitlab, if the user is using some sort of
different serial access then it might fail...?

If you want a test for 8-bit serial, you should add one.

>
> U-Boot's internal function run_command() cannot send electrical signals
> inbound to the device's UART.
>
> You may implement an additional test with a different scope. But that
> should not replace the current test.
>
> Best regards
>
> Heinrich

Regards,
Simon

Patch
diff mbox series

diff --git a/configs/sandbox64_defconfig b/configs/sandbox64_defconfig
index 1fea683d89..329645d362 100644
--- a/configs/sandbox64_defconfig
+++ b/configs/sandbox64_defconfig
@@ -21,6 +21,7 @@  CONFIG_DISPLAY_BOARDINFO_LATE=y
 CONFIG_CMD_CPU=y
 CONFIG_CMD_LICENSE=y
 CONFIG_CMD_BOOTZ=y
+CONFIG_CMD_BOOTEFI_SELFTEST=y
 # CONFIG_CMD_ELF is not set
 CONFIG_CMD_ASKENV=y
 CONFIG_CMD_GREPENV=y
diff --git a/configs/sandbox_defconfig b/configs/sandbox_defconfig
index 20ebc68997..b6d0d3f2fa 100644
--- a/configs/sandbox_defconfig
+++ b/configs/sandbox_defconfig
@@ -24,6 +24,7 @@  CONFIG_ANDROID_AB=y
 CONFIG_CMD_CPU=y
 CONFIG_CMD_LICENSE=y
 CONFIG_CMD_BOOTZ=y
+CONFIG_CMD_BOOTEFI_SELFTEST=y
 # CONFIG_CMD_ELF is not set
 CONFIG_CMD_ASKENV=y
 CONFIG_CMD_GREPENV=y
diff --git a/configs/sandbox_flattree_defconfig b/configs/sandbox_flattree_defconfig
index 898815fe53..02bbda071e 100644
--- a/configs/sandbox_flattree_defconfig
+++ b/configs/sandbox_flattree_defconfig
@@ -18,6 +18,7 @@  CONFIG_DISPLAY_BOARDINFO_LATE=y
 CONFIG_CMD_CPU=y
 CONFIG_CMD_LICENSE=y
 CONFIG_CMD_BOOTZ=y
+CONFIG_CMD_BOOTEFI_SELFTEST=y
 # CONFIG_CMD_ELF is not set
 CONFIG_CMD_ASKENV=y
 CONFIG_CMD_GREPENV=y
diff --git a/configs/sandbox_spl_defconfig b/configs/sandbox_spl_defconfig
index 409b8a38d5..9c24ea89ea 100644
--- a/configs/sandbox_spl_defconfig
+++ b/configs/sandbox_spl_defconfig
@@ -27,6 +27,7 @@  CONFIG_SPL_ENV_SUPPORT=y
 CONFIG_CMD_CPU=y
 CONFIG_CMD_LICENSE=y
 CONFIG_CMD_BOOTZ=y
+CONFIG_CMD_BOOTEFI_SELFTEST=y
 # CONFIG_CMD_ELF is not set
 CONFIG_CMD_ASKENV=y
 CONFIG_CMD_GREPENV=y