| Message ID | 20171213144911.6428-1-stefanha@redhat.com |
|---|---|
| Headers | show |
| Series | VSOCK: add vsock_test test suite | expand |
> On Dec 13, 2017, at 3:49 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > The vsock_diag.ko module already has a test suite but the core AF_VSOCK > functionality has no tests. This patch series adds several test cases that > exercise AF_VSOCK SOCK_STREAM socket semantics (send/recv, connect/accept, > half-closed connections, simultaneous connections). > > The test suite is modest but I hope to cover additional cases in the future. > My goal is to have a shared test suite so VMCI, Hyper-V, and KVM can ensure > that our transports behave the same. > > I have tested virtio-vsock. > > Jorgen: Please test the VMCI transport and let me know if anything needs to be > adjusted. See tools/testing/vsock/README for information on how to run the > test suite. > I tried running the vsock_test on VMCI, and all the tests failed in one way or another: 1) connection reset test: when the guest tries to connect to the host, we get EINVAL as the error instead of ECONNRESET. I’ll fix that. 2) client close and server close tests: On the host side, VMCI doesn’t support reading data from a socket that has been closed by the guest. When the guest closes a connection, all data is gone, and we return EOF on the host side. So the tests that try to read data after close, should not attempt that on VMCI host side. I got the tests to pass by adding a getsockname call to determine if the local CID was the host CID, and then skip the read attempt in that case. We could add a vmci flag, that would enable this behavior. 3) send_byte(fd, -EPIPE): for the VMCI transport, the close isn’t necessarily visible immediately on the peer. So in most cases, these send operations would complete with success. I was running these tests using nested virtualization, so I suspect that the problem is more likely to occur here, but I had to add a sleep to be sure to get the EPIPE error. 4) server close test: the connect would sometimes fail. This looks like an issue where we detect the peer close on the client side before we complete the connection handshake on the client side. There are two different channels used for the connection handshake and the disconnect. I’ll look into this to see what exactly is going on. 5) multiple connections tests: with the standard socket sizes, VMCI is only able to support about 100 concurrent stream connections so this test passes with MULTICONN_NFDS set to 100. Thanks, Jorgen
On Wed, Dec 20, 2017 at 02:48:43PM +0000, Jorgen S. Hansen wrote: > > > On Dec 13, 2017, at 3:49 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > > > The vsock_diag.ko module already has a test suite but the core AF_VSOCK > > functionality has no tests. This patch series adds several test cases that > > exercise AF_VSOCK SOCK_STREAM socket semantics (send/recv, connect/accept, > > half-closed connections, simultaneous connections). > > > > The test suite is modest but I hope to cover additional cases in the future. > > My goal is to have a shared test suite so VMCI, Hyper-V, and KVM can ensure > > that our transports behave the same. > > > > I have tested virtio-vsock. > > > > Jorgen: Please test the VMCI transport and let me know if anything needs to be > > adjusted. See tools/testing/vsock/README for information on how to run the > > test suite. > > > > I tried running the vsock_test on VMCI, and all the tests failed in one way or > another: Great, thank you for testing and looking into the failures! > 1) connection reset test: when the guest tries to connect to the host, we > get EINVAL as the error instead of ECONNRESET. I’ll fix that. Yay, the tests found a bug! > 2) client close and server close tests: On the host side, VMCI doesn’t > support reading data from a socket that has been closed by the > guest. When the guest closes a connection, all data is gone, and > we return EOF on the host side. So the tests that try to read data > after close, should not attempt that on VMCI host side. I got the > tests to pass by adding a getsockname call to determine if > the local CID was the host CID, and then skip the read attempt > in that case. We could add a vmci flag, that would enable > this behavior. Interesting behavior. Is there a reason for disallowing half-closed sockets on the host side? > 3) send_byte(fd, -EPIPE): for the VMCI transport, the close > isn’t necessarily visible immediately on the peer. So in most > cases, these send operations would complete with success. > I was running these tests using nested virtualization, so I > suspect that the problem is more likely to occur here, but > I had to add a sleep to be sure to get the EPIPE error. Good point, you've discovered a race condition that affects all transports. The vsock close state transition might not have occurred yet when the TCP control channel receives the "CLOSED" message. test_stream_client_close_server() needs to wait for the socket status to change before attempting send_byte(fd, -EPIPE). I guess I'll have to use vsock_diag or another kernel interface to check the socket's state. > 5) multiple connections tests: with the standard socket sizes, > VMCI is only able to support about 100 concurrent stream > connections so this test passes with MULTICONN_NFDS > set to 100. The 1000 magic number is because many distros have a maximum number of file descriptors ulimit of 1024. But it's an arbitrary number and we could lower it to 100. Is this VMCI concurrent stream limit a denial of service vector? Can an unprivileged guest userspace process open many sockets to prevent legitimate connections from other users within the same guest? Stefan
> On Jan 2, 2018, at 1:05 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote: > > On Wed, Dec 20, 2017 at 02:48:43PM +0000, Jorgen S. Hansen wrote: >> >>> On Dec 13, 2017, at 3:49 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote: >>> >>> The vsock_diag.ko module already has a test suite but the core AF_VSOCK >>> functionality has no tests. This patch series adds several test cases that >>> exercise AF_VSOCK SOCK_STREAM socket semantics (send/recv, connect/accept, >>> half-closed connections, simultaneous connections). >>> >>> The test suite is modest but I hope to cover additional cases in the future. >>> My goal is to have a shared test suite so VMCI, Hyper-V, and KVM can ensure >>> that our transports behave the same. >>> >>> I have tested virtio-vsock. >>> >>> Jorgen: Please test the VMCI transport and let me know if anything needs to be >>> adjusted. See tools/testing/vsock/README for information on how to run the >>> test suite. >>> >> >> I tried running the vsock_test on VMCI, and all the tests failed in one way or >> another: > > Great, thank you for testing and looking into the failures! > >> 1) connection reset test: when the guest tries to connect to the host, we >> get EINVAL as the error instead of ECONNRESET. I’ll fix that. > > Yay, the tests found a bug! > >> 2) client close and server close tests: On the host side, VMCI doesn’t >> support reading data from a socket that has been closed by the >> guest. When the guest closes a connection, all data is gone, and >> we return EOF on the host side. So the tests that try to read data >> after close, should not attempt that on VMCI host side. I got the >> tests to pass by adding a getsockname call to determine if >> the local CID was the host CID, and then skip the read attempt >> in that case. We could add a vmci flag, that would enable >> this behavior. > > Interesting behavior. Is there a reason for disallowing half-closed > sockets on the host side? This is a consequence of the way the underlying VMCI queue pairs are implemented. When the guest side closes a connection, it signals this to the peer by detaching from the VMCI queue pair used for the data transfer (the detach will result in an event being generated on the peer side). However, the VMCI queue pair is allocated as part of the guest memory, so when the guest detaches, that memory is reclaimed. So the host side would need to create a copy of the contents of that queue pair in kernel memory as part of the detach operation. When this was implemented, it was decided that it was better to avoid a potential large kernel memory allocation and the data copy at detach time than to maintain the half close behavior of INET. > >> 5) multiple connections tests: with the standard socket sizes, >> VMCI is only able to support about 100 concurrent stream >> connections so this test passes with MULTICONN_NFDS >> set to 100. > > The 1000 magic number is because many distros have a maximum number of > file descriptors ulimit of 1024. But it's an arbitrary number and we > could lower it to 100. > > Is this VMCI concurrent stream limit a denial of service vector? Can an > unprivileged guest userspace process open many sockets to prevent > legitimate connections from other users within the same guest? vSocket uses VMCI queue pairs for the stream, and the VMCI device only allows a limited amount of memory to be used for queue pairs per VM. So it is possible to exhaust this shared resource. The queue pairs are created as part of the connection establishment process, so it would require the user process to both create and connect the sockets to a host side endpoint (connections between guest processes will not allocate VMCI queue pairs). Thanks, Jorgen