[U-Boot,2/2] configs: sandbox: enable CONFIG_CMD_BOOTEFI_SELFTEST

Activate UEFI unit tests on the sandbox.

Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
---
 configs/sandbox64_defconfig        | 1 +
 configs/sandbox_defconfig          | 1 +
 configs/sandbox_flattree_defconfig | 1 +
 configs/sandbox_spl_defconfig      | 1 +
 4 files changed, 4 insertions(+)

--
2.24.0.rc1

Hi Heinrich,

On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> Activate UEFI unit tests on the sandbox.
>
> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> ---
>  configs/sandbox64_defconfig        | 1 +
>  configs/sandbox_defconfig          | 1 +
>  configs/sandbox_flattree_defconfig | 1 +
>  configs/sandbox_spl_defconfig      | 1 +
>  4 files changed, 4 insertions(+)

Unfortunately this slows down the testing too much, nearly doubling
the time in my tests.

I think the EFI console tests need to be modified to run in C instead
of all the drain_console() and p.timeout stuff. We need an effort to
speed up the tests, but certainly cannot make them any slower.

Regards,
Simon

On 12/10/19 1:58 PM, Simon Glass wrote:
> Hi Heinrich,
>
> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>
>> Activate UEFI unit tests on the sandbox.
>>
>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
>> ---
>>   configs/sandbox64_defconfig        | 1 +
>>   configs/sandbox_defconfig          | 1 +
>>   configs/sandbox_flattree_defconfig | 1 +
>>   configs/sandbox_spl_defconfig      | 1 +
>>   4 files changed, 4 insertions(+)
>
> Unfortunately this slows down the testing too much, nearly doubling
> the time in my tests.
>
> I think the EFI console tests need to be modified to run in C instead
> of all the drain_console() and p.timeout stuff. We need an effort to
> speed up the tests, but certainly cannot make them any slower.

Hello Simon,

thanks for pointing at the excessive timeout.

In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
we call drain_console() a lot indeed.

I am using the Python tests on real hardware. The text input and output
on the test systems uses the serial console. Whether you use C or Python
code to feed the serial adapter will not change the time needed to drain
the console.

We expect less then 50 characters of output per test step. At 9600 baud
draining 50 characters would require 52 ms.

So I would suggest that we add a parameter timeout to drain_console()
which defaults to 1000 ms as currently. But in the test we can set it to
50 ms. p.timeout could be changed likewise.

What are your thoughts?

Best regards

Heinrich

Hi Heinrich,

On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 12/10/19 1:58 PM, Simon Glass wrote:
> > Hi Heinrich,
> >
> > On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >>
> >> Activate UEFI unit tests on the sandbox.
> >>
> >> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> >> ---
> >>   configs/sandbox64_defconfig        | 1 +
> >>   configs/sandbox_defconfig          | 1 +
> >>   configs/sandbox_flattree_defconfig | 1 +
> >>   configs/sandbox_spl_defconfig      | 1 +
> >>   4 files changed, 4 insertions(+)
> >
> > Unfortunately this slows down the testing too much, nearly doubling
> > the time in my tests.
> >
> > I think the EFI console tests need to be modified to run in C instead
> > of all the drain_console() and p.timeout stuff. We need an effort to
> > speed up the tests, but certainly cannot make them any slower.
>
> Hello Simon,
>
> thanks for pointing at the excessive timeout.
>
> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
> we call drain_console() a lot indeed.
>
> I am using the Python tests on real hardware. The text input and output
> on the test systems uses the serial console. Whether you use C or Python
> code to feed the serial adapter will not change the time needed to drain
> the console.
>
> We expect less then 50 characters of output per test step. At 9600 baud
> draining 50 characters would require 52 ms.
>
> So I would suggest that we add a parameter timeout to drain_console()
> which defaults to 1000 ms as currently. But in the test we can set it to
> 50 ms. p.timeout could be changed likewise.
>
> What are your thoughts?

I think you might be missing my point.

I think as a matter of policy we should require that test run mostly
on the device, with just the results being output. By using C I mean
to write the test in C instead of Python - just do the same steps.
E.g. do a lot of run_command() calls and check the results. This
should be much faster.

Regards,
Simon

On 12/11/19 4:43 PM, Simon Glass wrote:
> Hi Heinrich,
>
> On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>
>> On 12/10/19 1:58 PM, Simon Glass wrote:
>>> Hi Heinrich,
>>>
>>> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>>>>
>>>> Activate UEFI unit tests on the sandbox.
>>>>
>>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
>>>> ---
>>>>    configs/sandbox64_defconfig        | 1 +
>>>>    configs/sandbox_defconfig          | 1 +
>>>>    configs/sandbox_flattree_defconfig | 1 +
>>>>    configs/sandbox_spl_defconfig      | 1 +
>>>>    4 files changed, 4 insertions(+)
>>>
>>> Unfortunately this slows down the testing too much, nearly doubling
>>> the time in my tests.
>>>
>>> I think the EFI console tests need to be modified to run in C instead
>>> of all the drain_console() and p.timeout stuff. We need an effort to
>>> speed up the tests, but certainly cannot make them any slower.
>>
>> Hello Simon,
>>
>> thanks for pointing at the excessive timeout.
>>
>> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
>> we call drain_console() a lot indeed.
>>
>> I am using the Python tests on real hardware. The text input and output
>> on the test systems uses the serial console. Whether you use C or Python
>> code to feed the serial adapter will not change the time needed to drain
>> the console.
>>
>> We expect less then 50 characters of output per test step. At 9600 baud
>> draining 50 characters would require 52 ms.
>>
>> So I would suggest that we add a parameter timeout to drain_console()
>> which defaults to 1000 ms as currently. But in the test we can set it to
>> 50 ms. p.timeout could be changed likewise.
>>
>> What are your thoughts?
>
> I think you might be missing my point.
>
> I think as a matter of policy we should require that test run mostly
> on the device, with just the results being output. By using C I mean
> to write the test in C instead of Python - just do the same steps.
> E.g. do a lot of run_command() calls and check the results. This
> should be much faster.
>

The current test checks if Unicode signs sent over the physical serial
interface are correctly interpreted in U-Boot. One reason for failure
could be that the UART driver uses bit 7 as a parity bit.

U-Boot's internal function run_command() cannot send electrical signals
inbound to the device's UART.

You may implement an additional test with a different scope. But that
should not replace the current test.

Best regards

Heinrich

Hi Heinrich,

On Wed, 11 Dec 2019 at 09:59, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 12/11/19 4:43 PM, Simon Glass wrote:
> > Hi Heinrich,
> >
> > On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >>
> >> On 12/10/19 1:58 PM, Simon Glass wrote:
> >>> Hi Heinrich,
> >>>
> >>> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
> >>>>
> >>>> Activate UEFI unit tests on the sandbox.
> >>>>
> >>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de>
> >>>> ---
> >>>>    configs/sandbox64_defconfig        | 1 +
> >>>>    configs/sandbox_defconfig          | 1 +
> >>>>    configs/sandbox_flattree_defconfig | 1 +
> >>>>    configs/sandbox_spl_defconfig      | 1 +
> >>>>    4 files changed, 4 insertions(+)
> >>>
> >>> Unfortunately this slows down the testing too much, nearly doubling
> >>> the time in my tests.
> >>>
> >>> I think the EFI console tests need to be modified to run in C instead
> >>> of all the drain_console() and p.timeout stuff. We need an effort to
> >>> speed up the tests, but certainly cannot make them any slower.
> >>
> >> Hello Simon,
> >>
> >> thanks for pointing at the excessive timeout.
> >>
> >> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex()
> >> we call drain_console() a lot indeed.
> >>
> >> I am using the Python tests on real hardware. The text input and output
> >> on the test systems uses the serial console. Whether you use C or Python
> >> code to feed the serial adapter will not change the time needed to drain
> >> the console.
> >>
> >> We expect less then 50 characters of output per test step. At 9600 baud
> >> draining 50 characters would require 52 ms.
> >>
> >> So I would suggest that we add a parameter timeout to drain_console()
> >> which defaults to 1000 ms as currently. But in the test we can set it to
> >> 50 ms. p.timeout could be changed likewise.
> >>
> >> What are your thoughts?
> >
> > I think you might be missing my point.
> >
> > I think as a matter of policy we should require that test run mostly
> > on the device, with just the results being output. By using C I mean
> > to write the test in C instead of Python - just do the same steps.
> > E.g. do a lot of run_command() calls and check the results. This
> > should be much faster.
> >
>
> The current test checks if Unicode signs sent over the physical serial
> interface are correctly interpreted in U-Boot. One reason for failure
> could be that the UART driver uses bit 7 as a parity bit.

How is this testing U-Boot? This sounds like a test of the user environment?

Even if the test passes on gitlab, if the user is using some sort of
different serial access then it might fail...?

If you want a test for 8-bit serial, you should add one.

>
> U-Boot's internal function run_command() cannot send electrical signals
> inbound to the device's UART.
>
> You may implement an additional test with a different scope. But that
> should not replace the current test.
>
> Best regards
>
> Heinrich

Regards,
Simon