[RFC] testing: add python-crossbar tests

From: Yegor Yefremov <yegorslists@googlemail.com>

This test invokes "crossbar version" command, that checks
all dependencies found in setup.py files and prints some
system related information.

Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
---
 .../testing/tests/package/test_python_crossbar.py  | 27 ++++++++++++++++++++++
 1 file changed, 27 insertions(+)
 create mode 100644 support/testing/tests/package/test_python_crossbar.py

On Thu, Feb 15, 2018 at 2:38 PM,  <yegorslists@googlemail.com> wrote:
> From: Yegor Yefremov <yegorslists@googlemail.com>
>
> This test invokes "crossbar version" command, that checks
> all dependencies found in setup.py files and prints some
> system related information.
>
> Signed-off-by: Yegor Yefremov <yegorslists@googlemail.com>
> ---
>  .../testing/tests/package/test_python_crossbar.py  | 27 ++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
>  create mode 100644 support/testing/tests/package/test_python_crossbar.py
>
> diff --git a/support/testing/tests/package/test_python_crossbar.py b/support/testing/tests/package/test_python_crossbar.py
> new file mode 100644
> index 0000000..a67bdea
> --- /dev/null
> +++ b/support/testing/tests/package/test_python_crossbar.py
> @@ -0,0 +1,27 @@
> +from tests.package.test_python import TestPythonBase
> +
> +class TestPythonCrossbar(TestPythonBase):
> +    def version_test(self, timeout=-1):
> +        cmd = self.interpreter + " -m crossbar version"
> +        _, exit_code = self.emulator.run(cmd, timeout)
> +        self.assertEqual(exit_code, 0)
> +
> +class TestPythonPy2Crossbar(TestPythonCrossbar):
> +    config = TestPythonBase.config + \
> +"""
> +BR2_PACKAGE_PYTHON=y
> +BR2_PACKAGE_PYTHON_CROSSBAR=y
> +"""
> +    def test_run(self):
> +        self.login()
> +        self.version_test(60)
> +
> +class TestPythonPy3Crossbar(TestPythonCrossbar):
> +    config = TestPythonBase.config + \
> +"""
> +BR2_PACKAGE_PYTHON3=y
> +BR2_PACKAGE_PYTHON_CROSSBAR=y
> +"""
> +    def test_run(self):
> +        self.login()
> +        self.version_test(60)

I've tried this test locally, but it always timeouts. Even if I
specify an hour as a timeout.

If a dependency is missing (like python-bitstring), then the test also
fails with an expected message, but if all dependencies are met, the
test doesn't return.

Any idea?

Yegor

Hello,

On Thu, Feb 15, 2018 at 11:41 AM, Yegor Yefremov wrote:

[snip]
> 
> I've tried this test locally, but it always timeouts. Even if I
> specify an hour as a timeout.

I confirm that here, both test cases fail by timeout, so it should not be
related to the host.

> 
> If a dependency is missing (like python-bitstring), then the test also
> fails with an expected message, but if all dependencies are met, the
> test doesn't return.
> 
> Any idea?

I have no idea of what causes it or how to fix it.

However I can give you some ideas how to try to debug using the test infra.
I don't know how far you already explored these options, so please forgive me if
I say only obvious things.

1) run the test case locally using -k
2) look at the first line of the -run.log to get the qemu command
3) execute the qemu command and get to the VM prompt
4) check that the import of the module is not broken:
    buildroot login: root
    # python
    Python 3.6.3 (default, Feb 15 2018, 20:46:37) 
    [GCC 4.9.4] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import crossbar
    >>> crossbar.__version__
    '17.6.1-3'
    >>> 
5) check a command that should print the usage
    # python -m crossbar othervalue
    Traceback (most recent call last):
      File "usr/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 46, in _load_libc
      File "/tmp/tmp2/o/TestPythonPy3Crossbar/target/usr/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    OSError: File not found
   (omitted lines)

   NOTICE the host path (/tmp/tmp2 is my buildroot dir) being displayed inside
   the target.

6) ctrl+C to kill the VM
7) enter the <test-output>/TestPythonPy3Crossbar dir - this is the build
   directory, so you can do what we know well, for example, enter menuconfig,
   select 'strace', and call make (or do a clean build)
8) run the qemu command again with the new image
    # strace python -m crossbar version
   Wait a few minutes for many messages and look at them. These ones called my
   attention, maybe unrelated to the problem:
   (omitted lines)
    open("/lib//libc.so", O_RDONLY)         = -1 ENOENT (No such file or directory)
    open("/usrlib//libc.so", O_RDONLY)      = -1 ENOENT (No such file or directory)
    open("/lib/libc.so", O_RDONLY)          = -1 ENOENT (No such file or directory)
    open("/lib/libc.so", O_RDONLY)          = -1 ENOENT (No such file or directory)
    open("/usr/lib/libc.so", O_RDONLY)      = -1 ENOENT (No such file or directory)
    open("/usr/X11R6/lib/libc.so", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/lib//libc.so.6", O_RDONLY)       = -1 ENOENT (No such file or directory)
    open("/usrlib//libc.so.6", O_RDONLY)    = -1 ENOENT (No such file or directory)
    open("/lib/libc.so.6", O_RDONLY)        = -1 ENOENT (No such file or directory)
    open("/lib/libc.so.6", O_RDONLY)        = -1 ENOENT (No such file or directory)
    open("/usr/lib/libc.so.6", O_RDONLY)    = -1 ENOENT (No such file or directory)
    open("/usr/X11R6/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
    write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
   (omitted lines)
   See also:
    # find / -name 'libc.so*'
    /lib/libc.so.1
    /lib/libc.so.0
    # ls /lib
    ld-uClibc-1.0.25.so  libatomic.so.1       libc.so.1
    ld-uClibc.so.0       libatomic.so.1.1.0   libgcc_s.so.1
    ld-uClibc.so.1       libc.so.0            libuClibc-1.0.25.so
9) repeat steps 6 to 8 with your favorite debug tools

So... (wild guesses!)
Could it be related to not finding libc?
Could it be related to paths from host leaked to the target?

Also, did you try to rebase the test case to the previous package version just
to know if the behavior is new?

Regards,
Ricardo

Hello,

On Thu, Feb 15, 2018 at 11:38 AM, Yegor Yefremov wrote:

[snip]
> ---
>  .../testing/tests/package/test_python_crossbar.py  | 27 ++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
>  create mode 100644 support/testing/tests/package/test_python_crossbar.py

The update to .gitlab.yml can be added now that 'make .gitlab-ci.yml' is fixed
(on master).

[snip]
> +++ b/support/testing/tests/package/test_python_crossbar.py

My series fixing flake8 warnings are now mostly applied (a few items need v2).

Can you run flake8 and fix the 7 warnings? You can see the updated
test_python.py for reference.


I applied this and also
http://patchwork.ozlabs.org/patch/874421/
http://patchwork.ozlabs.org/patch/874422/
but the test still fails with timeout:
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/53345963
I did not debugged it.

Maybe I missed something?


Regards,
Ricardo

Hello,

On Thu, 15 Feb 2018 23:16:32 -0200, Ricardo Martincoski wrote:

> On Thu, Feb 15, 2018 at 11:41 AM, Yegor Yefremov wrote:
> 
> [snip]
> > 
> > I've tried this test locally, but it always timeouts. Even if I
> > specify an hour as a timeout.  
> 
> I confirm that here, both test cases fail by timeout, so it should not be
> related to the host.

I applied Yegor's patch, and ran he test. It doesn't time out here.
Here is what I have in the TestPythonPy2Crossbar-run.log:

Welcome to Buildroot
buildroot login: root
# dmesg -n 1
# echo $?
0
# python -m crossbar version
Traceback (most recent call last):
  File "usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
  File "usr/lib/python2.7/runpy.py", line 72, in _run_code
  File "usr/lib/python2.7/site-packages/crossbar/__main__.py", line 36, in <module>
  File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 479, in load_entry_point
  File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2703, in load_entry_point
  File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2320, in load
  File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2343, in require
  File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
pkg_resources.ContextualVersionConflict: (txaio 2.8.2 (/usr/lib/python2.7/site-packages), Requirement.parse('txaio>=18.8.1'), set(['autobahn']))
# echo $?
1

So it doesn't work due to some dependency issues. Is the timeout
problem going to appear once those dependency issues are solved ? Or
should I have already seen the timeout ?

Best regards,

Thomas

Hello,

On Mon, Sep 17, 2018 at 10:55 AM, Thomas Petazzoni wrote:

> On Thu, 15 Feb 2018 23:16:32 -0200, Ricardo Martincoski wrote:
> 
>> On Thu, Feb 15, 2018 at 11:41 AM, Yegor Yefremov wrote:
>> 
>> [snip]
>> > 
>> > I've tried this test locally, but it always timeouts. Even if I
>> > specify an hour as a timeout.  
>> 
>> I confirm that here, both test cases fail by timeout, so it should not be
>> related to the host.

At the time I confirmed with the patch applied on aa77030b8f and also on
(aa77030b8f +
 http://patchwork.ozlabs.org/patch/874421/ +
 http://patchwork.ozlabs.org/patch/874422/ )
and I got the timeout in both cases.

> 
> I applied Yegor's patch, and ran he test. It doesn't time out here.
> Here is what I have in the TestPythonPy2Crossbar-run.log:
> 
> Welcome to Buildroot
> buildroot login: root
> # dmesg -n 1
> # echo $?
> 0
> # python -m crossbar version
> Traceback (most recent call last):
>   File "usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>   File "usr/lib/python2.7/runpy.py", line 72, in _run_code
>   File "usr/lib/python2.7/site-packages/crossbar/__main__.py", line 36, in <module>
>   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 479, in load_entry_point
>   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2703, in load_entry_point
>   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2320, in load
>   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2343, in require
>   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
> pkg_resources.ContextualVersionConflict: (txaio 2.8.2 (/usr/lib/python2.7/site-packages), Requirement.parse('txaio>=18.8.1'), set(['autobahn']))
> # echo $?
> 1
> 
> So it doesn't work due to some dependency issues. Is the timeout
> problem going to appear once those dependency issues are solved ? Or
> should I have already seen the timeout ?

Sorry, I don't know.

Yegor, any idea? Could you bisect this? I already started:
2018.02-rc2-4-gaa77030b8f + patch = timeout
2018.08-425-gc5a33a11be + patch = dependency issue
2018.05-rc3-25-gd937f908f1 + patch = timeout
2018.05-756-g3d34f36806 + patch = (still running:
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/98611581 )

Regards,
Ricardo

Hi Ricardo,

On Tue, Sep 18, 2018 at 4:31 AM Ricardo Martincoski
<ricardo.martincoski@gmail.com> wrote:
>
> Hello,
>
> On Mon, Sep 17, 2018 at 10:55 AM, Thomas Petazzoni wrote:
>
> > On Thu, 15 Feb 2018 23:16:32 -0200, Ricardo Martincoski wrote:
> >
> >> On Thu, Feb 15, 2018 at 11:41 AM, Yegor Yefremov wrote:
> >>
> >> [snip]
> >> >
> >> > I've tried this test locally, but it always timeouts. Even if I
> >> > specify an hour as a timeout.
> >>
> >> I confirm that here, both test cases fail by timeout, so it should not be
> >> related to the host.
>
> At the time I confirmed with the patch applied on aa77030b8f and also on
> (aa77030b8f +
>  http://patchwork.ozlabs.org/patch/874421/ +
>  http://patchwork.ozlabs.org/patch/874422/ )
> and I got the timeout in both cases.
>
> >
> > I applied Yegor's patch, and ran he test. It doesn't time out here.
> > Here is what I have in the TestPythonPy2Crossbar-run.log:
> >
> > Welcome to Buildroot
> > buildroot login: root
> > # dmesg -n 1
> > # echo $?
> > 0
> > # python -m crossbar version
> > Traceback (most recent call last):
> >   File "usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> >   File "usr/lib/python2.7/runpy.py", line 72, in _run_code
> >   File "usr/lib/python2.7/site-packages/crossbar/__main__.py", line 36, in <module>
> >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 479, in load_entry_point
> >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2703, in load_entry_point
> >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2320, in load
> >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2343, in require
> >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
> > pkg_resources.ContextualVersionConflict: (txaio 2.8.2 (/usr/lib/python2.7/site-packages), Requirement.parse('txaio>=18.8.1'), set(['autobahn']))
> > # echo $?
> > 1
> >
> > So it doesn't work due to some dependency issues. Is the timeout
> > problem going to appear once those dependency issues are solved ? Or
> > should I have already seen the timeout ?
>
> Sorry, I don't know.
>
> Yegor, any idea? Could you bisect this? I already started:
> 2018.02-rc2-4-gaa77030b8f + patch = timeout
> 2018.08-425-gc5a33a11be + patch = dependency issue
> 2018.05-rc3-25-gd937f908f1 + patch = timeout
> 2018.05-756-g3d34f36806 + patch = (still running:
> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/98611581 )

Thanks for working on this issue. AFAIK crossbar abandoned Py2 support
in the newest version, so there is less to take of, but there are
still issues with Py3.7 and async keywords, that need to be resolved
first.

I don't know when I'll have time to update crossbar, so if you can
take over it would be great.

Thanks.

Regards,
Yegor

Hello,

I just remembered I've submitted a patch for a new host package long
time ago: http://patchwork.ozlabs.org/patch/896368/
Yegor mentioned that maybe it can help to reduce python-crossbar
startup time on the tests.
Thomas/Yegor, I see the patch was reviewed by Yegor but wasn't
applied. If there's something missing, please let me know.
I hope it will help.

Regards,
Asaf.
On Tue, Sep 18, 2018 at 9:20 AM Yegor Yefremov
<yegorslists@googlemail.com> wrote:
>
> Hi Ricardo,
>
> On Tue, Sep 18, 2018 at 4:31 AM Ricardo Martincoski
> <ricardo.martincoski@gmail.com> wrote:
> >
> > Hello,
> >
> > On Mon, Sep 17, 2018 at 10:55 AM, Thomas Petazzoni wrote:
> >
> > > On Thu, 15 Feb 2018 23:16:32 -0200, Ricardo Martincoski wrote:
> > >
> > >> On Thu, Feb 15, 2018 at 11:41 AM, Yegor Yefremov wrote:
> > >>
> > >> [snip]
> > >> >
> > >> > I've tried this test locally, but it always timeouts. Even if I
> > >> > specify an hour as a timeout.
> > >>
> > >> I confirm that here, both test cases fail by timeout, so it should not be
> > >> related to the host.
> >
> > At the time I confirmed with the patch applied on aa77030b8f and also on
> > (aa77030b8f +
> >  http://patchwork.ozlabs.org/patch/874421/ +
> >  http://patchwork.ozlabs.org/patch/874422/ )
> > and I got the timeout in both cases.
> >
> > >
> > > I applied Yegor's patch, and ran he test. It doesn't time out here.
> > > Here is what I have in the TestPythonPy2Crossbar-run.log:
> > >
> > > Welcome to Buildroot
> > > buildroot login: root
> > > # dmesg -n 1
> > > # echo $?
> > > 0
> > > # python -m crossbar version
> > > Traceback (most recent call last):
> > >   File "usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
> > >   File "usr/lib/python2.7/runpy.py", line 72, in _run_code
> > >   File "usr/lib/python2.7/site-packages/crossbar/__main__.py", line 36, in <module>
> > >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 479, in load_entry_point
> > >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2703, in load_entry_point
> > >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2320, in load
> > >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2343, in require
> > >   File "usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
> > > pkg_resources.ContextualVersionConflict: (txaio 2.8.2 (/usr/lib/python2.7/site-packages), Requirement.parse('txaio>=18.8.1'), set(['autobahn']))
> > > # echo $?
> > > 1
> > >
> > > So it doesn't work due to some dependency issues. Is the timeout
> > > problem going to appear once those dependency issues are solved ? Or
> > > should I have already seen the timeout ?
> >
> > Sorry, I don't know.
> >
> > Yegor, any idea? Could you bisect this? I already started:
> > 2018.02-rc2-4-gaa77030b8f + patch = timeout
> > 2018.08-425-gc5a33a11be + patch = dependency issue
> > 2018.05-rc3-25-gd937f908f1 + patch = timeout
> > 2018.05-756-g3d34f36806 + patch = (still running:
> > https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/98611581 )
>
> Thanks for working on this issue. AFAIK crossbar abandoned Py2 support
> in the newest version, so there is less to take of, but there are
> still issues with Py3.7 and async keywords, that need to be resolved
> first.
>
> I don't know when I'll have time to update crossbar, so if you can
> take over it would be great.
>
> Thanks.
>
> Regards,
> Yegor
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Hello,

On Sat, Oct 20, 2018 at 12:49 PM, Asaf Kahlon wrote:

> I just remembered I've submitted a patch for a new host package long
> time ago: http://patchwork.ozlabs.org/patch/896368/
> Yegor mentioned that maybe it can help to reduce python-crossbar
> startup time on the tests.
> Thomas/Yegor, I see the patch was reviewed by Yegor but wasn't
> applied. If there's something missing, please let me know.
> I hope it will help.

I am not against that patch but I am afraid it won't fix the scenario tested in
this patch.

Turns out that the qemu image we use don't have enough entropy at boot time.

I confirmed by running this command on the qemu:
# cat /proc/sys/kernel/random/entropy_avail
0

Because of this, crossbar -> pynacl -> libsodium hangs waiting for /dev/random,
see [1].

We could update the default kernel image used by the test infra (today 3.11.0),
or even use a custom one (4.14.x for instance) compiled in this test case, to
get the getrandom syscall. It would improve a lot the startup time but we would
yet need to use a timeout of a few minutes.

The most elegant solution IMO is to add haveged to the image built for this test
case by the test infra. This is what people with a real target with low entropy
probably would use, I guess. There is no need to update the kernel this way.

I have locally a v2 of this test case that I will send, see [2] for the test
passing on the current master, but before I send it I will wait for feedback in
another patch [3].
This is the changelog for v2:
 - call 'crossbar version' instead of 'python -m "crossbar version"'
   because it is not supported anymore, giving this message:
   /usr/bin/python3: No module named crossbar.__main__; 'crossbar' is a package and cannot be directly executed
 - add haveged to the target to provide enough entropy, avoiding
   hanging forever (on our current kernel 3.11.0 default image) or for
   a long time (when the default kernel image is upgraded to a newer
   kernel);
 - remove test for python 2 because upstream dropped support;
 - do not pass timeout parameter since version_test is now called once;
 - rewrap commit message to 72
 - update code-style (we adopted flake8 after v1 submission)
 - update .gitlab-ci.yml

[1] https://github.com/pyca/pynacl/issues/327
[2] https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/110669652
[3] http://patchwork.ozlabs.org/patch/984425/


Regards,
Ricardo

Hello,

On Sun, 21 Oct 2018 02:01:37 -0300, Ricardo Martincoski wrote:

> We could update the default kernel image used by the test infra (today 3.11.0),

We can definitely update that pre-built image if needed. There is no
special reason to have a 3.11 based image, except this being the kernel
version I had around at the time I started playing with the test
infrastructure.

> The most elegant solution IMO is to add haveged to the image built for this test
> case by the test infra. This is what people with a real target with low entropy
> probably would use, I guess. There is no need to update the kernel this way.

I also don't see any problem with adding haveged in the image.

Best regards,

Thomas