Message ID | 20191109083902.27903-3-xypron.glpk@gmx.de |
---|---|
State | Changes Requested |
Delegated to: | Simon Glass |
Headers | show |
Series | sandbox: enable UEFI unit tests | expand |
Hi Heinrich, On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > > Activate UEFI unit tests on the sandbox. > > Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> > --- > configs/sandbox64_defconfig | 1 + > configs/sandbox_defconfig | 1 + > configs/sandbox_flattree_defconfig | 1 + > configs/sandbox_spl_defconfig | 1 + > 4 files changed, 4 insertions(+) Unfortunately this slows down the testing too much, nearly doubling the time in my tests. I think the EFI console tests need to be modified to run in C instead of all the drain_console() and p.timeout stuff. We need an effort to speed up the tests, but certainly cannot make them any slower. Regards, Simon
On 12/10/19 1:58 PM, Simon Glass wrote: > Hi Heinrich, > > On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: >> >> Activate UEFI unit tests on the sandbox. >> >> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> >> --- >> configs/sandbox64_defconfig | 1 + >> configs/sandbox_defconfig | 1 + >> configs/sandbox_flattree_defconfig | 1 + >> configs/sandbox_spl_defconfig | 1 + >> 4 files changed, 4 insertions(+) > > Unfortunately this slows down the testing too much, nearly doubling > the time in my tests. > > I think the EFI console tests need to be modified to run in C instead > of all the drain_console() and p.timeout stuff. We need an effort to > speed up the tests, but certainly cannot make them any slower. Hello Simon, thanks for pointing at the excessive timeout. In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex() we call drain_console() a lot indeed. I am using the Python tests on real hardware. The text input and output on the test systems uses the serial console. Whether you use C or Python code to feed the serial adapter will not change the time needed to drain the console. We expect less then 50 characters of output per test step. At 9600 baud draining 50 characters would require 52 ms. So I would suggest that we add a parameter timeout to drain_console() which defaults to 1000 ms as currently. But in the test we can set it to 50 ms. p.timeout could be changed likewise. What are your thoughts? Best regards Heinrich
Hi Heinrich, On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > > On 12/10/19 1:58 PM, Simon Glass wrote: > > Hi Heinrich, > > > > On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > >> > >> Activate UEFI unit tests on the sandbox. > >> > >> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> > >> --- > >> configs/sandbox64_defconfig | 1 + > >> configs/sandbox_defconfig | 1 + > >> configs/sandbox_flattree_defconfig | 1 + > >> configs/sandbox_spl_defconfig | 1 + > >> 4 files changed, 4 insertions(+) > > > > Unfortunately this slows down the testing too much, nearly doubling > > the time in my tests. > > > > I think the EFI console tests need to be modified to run in C instead > > of all the drain_console() and p.timeout stuff. We need an effort to > > speed up the tests, but certainly cannot make them any slower. > > Hello Simon, > > thanks for pointing at the excessive timeout. > > In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex() > we call drain_console() a lot indeed. > > I am using the Python tests on real hardware. The text input and output > on the test systems uses the serial console. Whether you use C or Python > code to feed the serial adapter will not change the time needed to drain > the console. > > We expect less then 50 characters of output per test step. At 9600 baud > draining 50 characters would require 52 ms. > > So I would suggest that we add a parameter timeout to drain_console() > which defaults to 1000 ms as currently. But in the test we can set it to > 50 ms. p.timeout could be changed likewise. > > What are your thoughts? I think you might be missing my point. I think as a matter of policy we should require that test run mostly on the device, with just the results being output. By using C I mean to write the test in C instead of Python - just do the same steps. E.g. do a lot of run_command() calls and check the results. This should be much faster. Regards, Simon
On 12/11/19 4:43 PM, Simon Glass wrote: > Hi Heinrich, > > On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: >> >> On 12/10/19 1:58 PM, Simon Glass wrote: >>> Hi Heinrich, >>> >>> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: >>>> >>>> Activate UEFI unit tests on the sandbox. >>>> >>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> >>>> --- >>>> configs/sandbox64_defconfig | 1 + >>>> configs/sandbox_defconfig | 1 + >>>> configs/sandbox_flattree_defconfig | 1 + >>>> configs/sandbox_spl_defconfig | 1 + >>>> 4 files changed, 4 insertions(+) >>> >>> Unfortunately this slows down the testing too much, nearly doubling >>> the time in my tests. >>> >>> I think the EFI console tests need to be modified to run in C instead >>> of all the drain_console() and p.timeout stuff. We need an effort to >>> speed up the tests, but certainly cannot make them any slower. >> >> Hello Simon, >> >> thanks for pointing at the excessive timeout. >> >> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex() >> we call drain_console() a lot indeed. >> >> I am using the Python tests on real hardware. The text input and output >> on the test systems uses the serial console. Whether you use C or Python >> code to feed the serial adapter will not change the time needed to drain >> the console. >> >> We expect less then 50 characters of output per test step. At 9600 baud >> draining 50 characters would require 52 ms. >> >> So I would suggest that we add a parameter timeout to drain_console() >> which defaults to 1000 ms as currently. But in the test we can set it to >> 50 ms. p.timeout could be changed likewise. >> >> What are your thoughts? > > I think you might be missing my point. > > I think as a matter of policy we should require that test run mostly > on the device, with just the results being output. By using C I mean > to write the test in C instead of Python - just do the same steps. > E.g. do a lot of run_command() calls and check the results. This > should be much faster. > The current test checks if Unicode signs sent over the physical serial interface are correctly interpreted in U-Boot. One reason for failure could be that the UART driver uses bit 7 as a parity bit. U-Boot's internal function run_command() cannot send electrical signals inbound to the device's UART. You may implement an additional test with a different scope. But that should not replace the current test. Best regards Heinrich
Hi Heinrich, On Wed, 11 Dec 2019 at 09:59, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > > On 12/11/19 4:43 PM, Simon Glass wrote: > > Hi Heinrich, > > > > On Tue, 10 Dec 2019 at 11:17, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > >> > >> On 12/10/19 1:58 PM, Simon Glass wrote: > >>> Hi Heinrich, > >>> > >>> On Sat, 9 Nov 2019 at 01:44, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote: > >>>> > >>>> Activate UEFI unit tests on the sandbox. > >>>> > >>>> Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> > >>>> --- > >>>> configs/sandbox64_defconfig | 1 + > >>>> configs/sandbox_defconfig | 1 + > >>>> configs/sandbox_flattree_defconfig | 1 + > >>>> configs/sandbox_spl_defconfig | 1 + > >>>> 4 files changed, 4 insertions(+) > >>> > >>> Unfortunately this slows down the testing too much, nearly doubling > >>> the time in my tests. > >>> > >>> I think the EFI console tests need to be modified to run in C instead > >>> of all the drain_console() and p.timeout stuff. We need an effort to > >>> speed up the tests, but certainly cannot make them any slower. > >> > >> Hello Simon, > >> > >> thanks for pointing at the excessive timeout. > >> > >> In test_efi_selftest_text_input() and test_efi_selftest_text_input_ex() > >> we call drain_console() a lot indeed. > >> > >> I am using the Python tests on real hardware. The text input and output > >> on the test systems uses the serial console. Whether you use C or Python > >> code to feed the serial adapter will not change the time needed to drain > >> the console. > >> > >> We expect less then 50 characters of output per test step. At 9600 baud > >> draining 50 characters would require 52 ms. > >> > >> So I would suggest that we add a parameter timeout to drain_console() > >> which defaults to 1000 ms as currently. But in the test we can set it to > >> 50 ms. p.timeout could be changed likewise. > >> > >> What are your thoughts? > > > > I think you might be missing my point. > > > > I think as a matter of policy we should require that test run mostly > > on the device, with just the results being output. By using C I mean > > to write the test in C instead of Python - just do the same steps. > > E.g. do a lot of run_command() calls and check the results. This > > should be much faster. > > > > The current test checks if Unicode signs sent over the physical serial > interface are correctly interpreted in U-Boot. One reason for failure > could be that the UART driver uses bit 7 as a parity bit. How is this testing U-Boot? This sounds like a test of the user environment? Even if the test passes on gitlab, if the user is using some sort of different serial access then it might fail...? If you want a test for 8-bit serial, you should add one. > > U-Boot's internal function run_command() cannot send electrical signals > inbound to the device's UART. > > You may implement an additional test with a different scope. But that > should not replace the current test. > > Best regards > > Heinrich Regards, Simon
diff --git a/configs/sandbox64_defconfig b/configs/sandbox64_defconfig index 1fea683d89..329645d362 100644 --- a/configs/sandbox64_defconfig +++ b/configs/sandbox64_defconfig @@ -21,6 +21,7 @@ CONFIG_DISPLAY_BOARDINFO_LATE=y CONFIG_CMD_CPU=y CONFIG_CMD_LICENSE=y CONFIG_CMD_BOOTZ=y +CONFIG_CMD_BOOTEFI_SELFTEST=y # CONFIG_CMD_ELF is not set CONFIG_CMD_ASKENV=y CONFIG_CMD_GREPENV=y diff --git a/configs/sandbox_defconfig b/configs/sandbox_defconfig index 20ebc68997..b6d0d3f2fa 100644 --- a/configs/sandbox_defconfig +++ b/configs/sandbox_defconfig @@ -24,6 +24,7 @@ CONFIG_ANDROID_AB=y CONFIG_CMD_CPU=y CONFIG_CMD_LICENSE=y CONFIG_CMD_BOOTZ=y +CONFIG_CMD_BOOTEFI_SELFTEST=y # CONFIG_CMD_ELF is not set CONFIG_CMD_ASKENV=y CONFIG_CMD_GREPENV=y diff --git a/configs/sandbox_flattree_defconfig b/configs/sandbox_flattree_defconfig index 898815fe53..02bbda071e 100644 --- a/configs/sandbox_flattree_defconfig +++ b/configs/sandbox_flattree_defconfig @@ -18,6 +18,7 @@ CONFIG_DISPLAY_BOARDINFO_LATE=y CONFIG_CMD_CPU=y CONFIG_CMD_LICENSE=y CONFIG_CMD_BOOTZ=y +CONFIG_CMD_BOOTEFI_SELFTEST=y # CONFIG_CMD_ELF is not set CONFIG_CMD_ASKENV=y CONFIG_CMD_GREPENV=y diff --git a/configs/sandbox_spl_defconfig b/configs/sandbox_spl_defconfig index 409b8a38d5..9c24ea89ea 100644 --- a/configs/sandbox_spl_defconfig +++ b/configs/sandbox_spl_defconfig @@ -27,6 +27,7 @@ CONFIG_SPL_ENV_SUPPORT=y CONFIG_CMD_CPU=y CONFIG_CMD_LICENSE=y CONFIG_CMD_BOOTZ=y +CONFIG_CMD_BOOTEFI_SELFTEST=y # CONFIG_CMD_ELF is not set CONFIG_CMD_ASKENV=y CONFIG_CMD_GREPENV=y
Activate UEFI unit tests on the sandbox. Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> --- configs/sandbox64_defconfig | 1 + configs/sandbox_defconfig | 1 + configs/sandbox_flattree_defconfig | 1 + configs/sandbox_spl_defconfig | 1 + 4 files changed, 4 insertions(+) -- 2.24.0.rc1