From patchwork Fri May 20 16:44:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Martin Doucha X-Patchwork-Id: 1633995 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.cz header.i=@suse.cz header.a=rsa-sha256 header.s=susede2_rsa header.b=cU/Qqsez; dkim=fail reason="signature verification failed" header.d=suse.cz header.i=@suse.cz header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=SojZD1VN; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.linux.it (client-ip=2001:1418:10:5::2; helo=picard.linux.it; envelope-from=ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it; receiver=) Received: from picard.linux.it (picard.linux.it [IPv6:2001:1418:10:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4L4Xdr1bVlz9sG0 for ; Sat, 21 May 2022 02:45:19 +1000 (AEST) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id 510B73CAB2B for ; Fri, 20 May 2022 18:45:14 +0200 (CEST) X-Original-To: ltp@lists.linux.it Delivered-To: ltp@picard.linux.it Received: from in-3.smtp.seeweb.it (in-3.smtp.seeweb.it [217.194.8.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by picard.linux.it (Postfix) with ESMTPS id D92163C0919 for ; Fri, 20 May 2022 18:45:12 +0200 (CEST) Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by in-3.smtp.seeweb.it (Postfix) with ESMTPS id 22AF91A01019 for ; Fri, 20 May 2022 18:45:10 +0200 (CEST) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 30A0E1F8A3 for ; Fri, 20 May 2022 16:45:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1653065110; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=MPhVHIbWeKptWijPVjaFm1yhznvV3nUzFg/KqGdBEhM=; b=cU/Qqsez8trEcmSC9Fid2qRPwrhxuf461jY6FRhuJEZsE0DyHHHy+5Fu4ga7u1aVRNUhNb B8cEHfJsounj2OLNVrTkyvs0I8LOE1mZqWRluXmD2KEU7jTnyM/ORarBNmqb+GSS1NiBr8 1pHOEM/794cAn9yF9XrIVQ+jcggcPYk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1653065110; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=MPhVHIbWeKptWijPVjaFm1yhznvV3nUzFg/KqGdBEhM=; b=SojZD1VN97gnHteYbdttcUbmHltji5uIYwShPpw0iycmkT94FUeHaY4RFxt24Bwhyvjsca ZJi7a/cbQLvyWKCw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 1BD0A13AF4 for ; Fri, 20 May 2022 16:45:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id JRjjBZbFh2L+MwAAMHmgww (envelope-from ) for ; Fri, 20 May 2022 16:45:10 +0000 From: Martin Doucha To: ltp@lists.linux.it Date: Fri, 20 May 2022 18:44:55 +0200 Message-Id: <20220520164455.26910-1-mdoucha@suse.cz> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 X-Virus-Scanned: clamav-milter 0.102.4 at in-3.smtp.seeweb.it X-Virus-Status: Clean X-Spam-Status: No, score=0.1 required=7.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on in-3.smtp.seeweb.it Subject: [LTP] [PATCH] Add documentation for KVM testing X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it Sender: "ltp" Signed-off-by: Martin Doucha --- doc/kvm-test-api.txt | 384 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 384 insertions(+) create mode 100644 doc/kvm-test-api.txt diff --git a/doc/kvm-test-api.txt b/doc/kvm-test-api.txt new file mode 100644 index 000000000..25b72172d --- /dev/null +++ b/doc/kvm-test-api.txt @@ -0,0 +1,384 @@ +LTP KVM Test API +================ + +Testing KVM is more complex than other Linux features. Some KVM bugs allow +userspace code running inside the virtual machine to bypass (emulated) hardware +access restrictions and elevate its privileges inside the guest operating +system. The worst types of KVM bugs may even allow the guest code to crash or +compromise the physical host. KVM tests therefore need to be split into two +components – a KVM controller program running on the physical host and a guest +payload program running inside the VM. The cooperation of these two components +allows testing even of bugs that somehow cross the virtualization boundary. + +NOTE: See also + https://github.com/linux-test-project/ltp/wiki/Test-Writing-Guidelines[Test Writing Guidelines], + https://github.com/linux-test-project/ltp/wiki/C-Test-Case-Tutorial[C Test Case Tutorial], + https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API]. + +1. Basic KVM test structure +--------------------------- + +KVM tests are simple C source files containing both the KVM controller code +and the guest payload code separated by `#ifdef COMPILE_PAYLOAD` preprocessor +condition. The file will be compiled twice. Once to compile the payload part, +once to compile the KVM controller part and embed the payload binary inside. +The result is a single self-contained binary that'll execute the embedded +payload inside a KVM virtual machine and print results in the same format as +a normal LTP test. + +A KVM test source should start with `#include "kvm_test.h"` instead of the +usual `tst_test.h`. The `kvm_test.h` header file will include the other basic +headers appropriate for the current compilation pass. Everything else in the +source file should be enclosed in `#ifdef COMPILE_PAYLOAD ... #else ... #endif` +condition, including any other header file includes. Note that the standard +LTP headers are not available in the payload compilation pass, only the KVM +guest library headers can be included. + +.Example KVM test +[source,c] +------------------------------------------------------------------------------- +#include "kvm_test.h" + +#ifdef COMPILE_PAYLOAD + +/* Guest payload code */ + +void main(void) +{ + tst_res(TPASS, "Hello, world!"); +} + +#else /* COMPILE_PAYLOAD */ + +/* KVM controller code */ + +static struct tst_test test = { + .test_all = tst_kvm_run, + .setup = tst_kvm_setup, + .cleanup = tst_kvm_cleanup, +}; + +#endif /* COMPILE_PAYLOAD */ +------------------------------------------------------------------------------- + +The KVM controller code is a normal LTP test and needs to define an instance +of `struct tst_test` with metadata and the usual setup, cleanup, and test +functions. Basic implementation of all three functions is provided by the KVM +host library. + +On the other hand, the payload is essentially a tiny kernel that'll run +on bare virtual hardware. It cannot access any files, Linux syscalls, standard +library functions, etc. except for the small subset provided by the KVM guest +library. The payload code must define a `void main(void)` function which will +be the VM entry point of the test. + +2. KVM host library +------------------- + +The KVM host library provides helper functions for creating and running +a minimal KVM virtual machine. + +2.1 Data structures +~~~~~~~~~~~~~~~~~~~ + +[source,c] +------------------------------------------------------------------------------- +struct tst_kvm_instance { + int vm_fd, vcpu_fd; + struct kvm_run *vcpu_info; + size_t vcpu_info_size; + struct kvm_userspace_memory_region ram[MAX_KVM_MEMSLOTS]; + struct tst_kvm_result *result; +}; +------------------------------------------------------------------------------- + +`struct tst_kvm_instance` holds the file descriptors and memory buffers +of a single KVM virtual machine: + +- `int vm_fd` is the main VM file descriptor created by `ioctl(KVM_CREATE_VM)` +- `int vcpu_fd` is the virtual CPU filedescriptor created by + `ioctl(KVM_CREATE_VCPU)` +- `struct kvm_run *vcpu_info` is the VCPU state structure created by + `mmap(vcpu_fd)` +- `size_t vcpu_info_size` is the size of `vcpu_info` buffer +- `struct kvm_userspace_memory_region ram[MAX_KVM_MEMSLOTS]` is the list + of memory slots defined in this VM. Unused memory slots have zero + in the `userspace_addr` field. +- `struct tst_kvm_result *result` is a buffer for passing test result data + from the VM to the controller program, mainly `tst_res()`/`tst_brk()` flags + and messages. + +[source,c] +------------------------------------------------------------------------------- +struct tst_kvm_result { + int32_t result; + int32_t lineno; + uint64_t file_addr; + char message[0]; +}; +------------------------------------------------------------------------------- + +`struct tst_kvm_result` is used to pass test results and synchronization data +between the KVM guest and the controller program. Most often, it is used +to pass `tst_res()` and `tst_brk()` messages from the VM, but special values +can also be used to send control flow requests both ways. + +- `int32_t result` is the message type, either one of the `TPASS`, `TFAIL`, + `TWARN`, `TBROK`, `TINFO` flags or a special control flow value. Errno flags + are not supported. +- `int32_t lineno` is the line number for `tst_res()`/`tst_brk()` messages. +- `uint64_t file_addr` is the VM address of the filename string for + `tst_res()`/`tst_brk()` messages. +- `char message[0]` is the buffer for arbitrary message data, most often used + to pass `tst_res()`/`tst_brk()` message strings. + +2.2 Working with virtual machines +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The KVM host library provides default implementation of the setup, cleanup +and test functions for `struct tst_test` in cases where you do not need +to customize the VM configuration. You can either assign these functions +to the `struct tst_test` instance directly or call them from your own function +that does some additional steps. All three function must be used together. + +- `void tst_kvm_setup(void)` +- `void tst_kvm_run(void)` +- `void tst_kvm_cleanup(void)` + +Note: `tst_kvm_run()` calls `tst_free_all()`. Calling it will free all +previously allocated guarded buffers. + +- `void tst_kvm_validate_result(int value)` – Validate whether the value + returned in `struct tst_kvm_result.result` can be safely passed + to `tst_res()` or `tst_brk()`. If the value is not valid, the controller + program will be terminated with error. + +- `uint64_t tst_kvm_get_phys_address(const struct tst_kvm_instance *inst, + uint64_t addr)` – Converts pointer value (virtual address) from KVM virtual + machine `inst` to the corresponding physical address. Returns 0 if + the virtual address is not mapped to any physical address. If virtual memory + mapping is not enabled in the VM or not available on the arch at all, this + function simply returns `addr` as is. + +- `int tst_kvm_find_phys_memslot(const struct tst_kvm_instance *inst, + uint64_t paddr)` – Returns index of the memory slot in KVM virtual machine + `inst` which contains the physical address `paddr`. If the address is not + backed by a memory buffer, returns -1. + +- `int tst_kvm_find_memslot(const struct tst_kvm_instance *inst, + uint64_t addr)` – Returns index of the memory slot in KVM virtual machine + `inst` which contains the virtual address `addr`. If the virtual address + is not mapped to a valid physical address backed by a memory buffer, + returns -1. + +- `void *tst_kvm_get_memptr(const struct tst_kvm_instance *inst, + uint64_t addr)` – Converts pointer value (virtual address) from KVM virtual + machine `inst` to host-side pointer. + +- `void *tst_kvm_alloc_memory(struct tst_kvm_instance *inst, unsigned int slot, + uint64_t baseaddr, size_t size, unsigned int flags)` – Allocates a guarded + buffer of given `size` in bytes and installs it into specified memory `slot` + of the KVM virtual machine `inst` at base address `baseaddr`. The buffer + will be automatically page aligned at both ends. See the kernel + documentation of `KVM_SET_USER_MEMORY_REGION` ioctl for list of valid + `flags`. Returns pointer to page-aligned beginning of the allocated buffer. + The actual requested `baseaddr` will be located at + `ret + baseaddr % pagesize`. + +- `struct kvm_cpuid2 *tst_kvm_get_cpuid(int sysfd)` – Get a list of supported + virtual CPU features returned by `ioctl(KVM_GET_SUPPORTED_CPUID)`. + The argument must be an open file descriptor returned by `open("/dev/kvm")`. + +- `void tst_kvm_create_instance(struct tst_kvm_instance *inst, + size_t ram_size)` – Creates and fully initializes a new KVM virtual machine + with at least `ram_size` bytes of memory. The VM instance info will be + stored in `inst`. + +- `void tst_kvm_run_instance(struct tst_kvm_instance *inst)` – Executes + the program installed in KVM virtual machine `inst`. Any result messages + returned by the VM will be automatically printed to controller program output. + +- `void tst_kvm_destroy_instance(struct tst_kvm_instance *inst)` – Deletes + the KVM virtual machine `inst`. Note that the guarded buffers assigned + to the VM by `tst_kvm_create_instance()` or `tst_kvm_alloc_memory()` will + not be freed. + +The KVM host library does not provide any way to reset a VM instance back +to initial state. Running multiple iterations of the test requires destroying +the old VM instance and creating a new one. Otherwise the VM will exit +without reporting any results on the second iteration and the test will fail. +The `tst_kvm_run()` function handles this issue correctly. + +3. KVM guest library +-------------------- + +The KVM guest library provides a minimal implementation of both the LTP +test library and the standard C library functions. Do not try to include +the usual LTP or C headers in guest payload code, it will not work. + +3.1 Standard C functions +~~~~~~~~~~~~~~~~~~~~~~~~ + +`#include "kvm_test.h"` + +The functions listed below are implemented according to the C standard: + +- `void *memset(void *dest, int val, size_t size)` +- `void *memzero(void *dest, size_t size)` +- `void *memcpy(void *dest, const void *src, size_t size)` +- `char *strcpy(char *dest, const char *src)` +- `char *strcat(char *dest, const char *src)` +- `size_t strlen(const char *str)` + +3.2 LTP library functions +~~~~~~~~~~~~~~~~~~~~~~~~~ + +`#include "kvm_test.h"` + +The KVM guest library currently provides the LTP functions for reporting test +results. All standard result flags except for `T*ERRNO` are supported +with the same rules as usual. However, the printf-like formatting is not +implemented yet. + +- `void tst_res(int result, const char *message)` +- `void tst_brk(int result, const char *message) __attribute__((noreturn))` + +A handful of useful macros is also available: + +- `TST_TEST_TCONF(message)` – Generates a test program that will simply print + a `TCONF` message and exit. This is useful when the real test cannot be + built due to missing dependencies or arch limitations. + +- `ARRAY_SIZE(arr)` – Returns the number of items in statically allocated + array `arr`. + +- `LTP_ALIGN(x, a)` – Aligns integer `x` to be a multiple of `a`, which + must be a power of 2. + +3.3 Arch independent functions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`#include "kvm_test.h"` + +Memory management in KVM guest library currently uses only primitive linear +buffer for memory allocation. There are no checks whether the VM can allocate +more memory and the already allocated memory cannot be freed. + +- `void *tst_heap_alloc(size_t size)` – Allocates a block of memory on the heap. + +- `void *tst_heap_alloc_aligned(size_t size, size_t align)` – Allocates + a block of memory on the heap with the starting address aligned to given + value. + +3.4 x86 specific functions +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +`#include "kvm_test.h"` + +`#include "kvm_x86.h"` + +- `struct kvm_interrupt_frame` – Opaque arch-dependent structure which holds + interrupt frame information. Use the functions below to get individual values: + +- `uintptr_t kvm_get_interrupt_ip(const struct kvm_interrupt_frame *ifrm)` – + Get instruction pointer value from interrupt frame structure. This may be + the instruction which caused an interrupt or the one immediately after, + depending on the interrupt vector semantics. + +- `int (*tst_interrupt_callback)(void *userdata, + struct kvm_interrupt_frame *ifrm, unsigned long errcode)` – Interrupt handler + callback prototype. When an interrupt occurs, the assigned callback function + will be passed the `userdata` pointer that was given + to `tst_set_interrupt_callback()`, interrupt frame `ifrm` and the error + code `errcode` defined by the interrupt vector semantics. If the interrupt + vector does not generate an error code, `errcode` will be set to zero. + The callback function must return 0 if the interrupt was successfully + handled and test execution should resume. Non-zero return value means that + the interrupt could not be handled and the test will terminate with error. + +- `void tst_set_interrupt_callback(unsigned int vector, + tst_interrupt_callback func, void *userdata)` – Register new interrupt + handler callback function `func` for interrupt `vector`. The `userdata` + argument is an arbitrary pointer that will be passed to `func()` every time + it gets called. The previous interrupt handler callback will be removed. + Setting `func` to `NULL` will remove any existing interrupt handler + from `vector` and the interrupt will become fatal error. + +[source,c] +------------------------------------------------------------------------------- +struct page_table_entry_pae { + unsigned int present: 1; + unsigned int writable: 1; + unsigned int user_access: 1; + unsigned int write_through: 1; + unsigned int disable_cache: 1; + unsigned int accessed: 1; + unsigned int dirty: 1; + unsigned int page_type: 1; + unsigned int global: 1; + unsigned int padding: 3; + uint64_t address: 40; + unsigned int padding2: 7; + unsigned int prot_key: 4; + unsigned int noexec: 1; +} __attribute__((__packed__)); + +struct kvm_cpuid { + unsigned int eax, ebx, ecx, edx; +}; + +struct kvm_cregs { + unsigned long cr0, cr2, cr3, cr4; +}; +------------------------------------------------------------------------------- + +`struct page_table_entry_pae` is the page table entry structure for PAE and +64bit paging modes. See Intel(R) 64 and IA-32 Architectures Software +Developer's Manual, Volume 3, Chapter 4 for explanation of the fields. + +- `uintptr_t kvm_get_page_address_pae(const struct page_table_entry_pae *entry)` + – Returns the physical address of the memory page referenced by the given + page table `entry`. Depending on memory mapping changes done by the test, + the physical address may not be a valid pointer. The caller must determine + whether the address points to another page table entry or a data page, using + the known position in page table hierarchy and `entry->page_type`. Returns + zero if the `entry` does not reference any memory page. + +- `void kvm_get_cpuid(unsigned int eax, unsigned int ecx, + struct kvm_cpuid *buf)` – Executes the CPUID instruction with the given + `eax` and `ecx` arguments and stores the results in `buf`. + +- `void kvm_read_cregs(struct kvm_cregs *buf)` – Copies the current values + of control registers to `buf`. + +- `uint64_t kvm_rdmsr(unsigned int msr)` – Returns the current value + of model-specific register `msr`. + +- `void kvm_wrmsr(unsigned int msr, uint64_t value)` – Stores `value` + into model-specific register `msr`. + +- `void kvm_exit(void) __attribute__((noreturn))` – Terminate the test. + Similar to calling `exit(0)` in a regular LTP test, although `kvm_exit()` + will terminate only one iteration of the test, not the whole host process. + +See Intel(R) 64 and IA-32 Architectures Software Developer's Manual +for documentation of standard and model-specific x86 registers. + +4. KVM guest environment +------------------------ + +KVM guest payload execution begins with bootstrap code which will perform +the minimal guest environment setup required for running C code: + +- Activate the appropriate CPU execution mode (IA-32 protected mode + on 32-bit x86 or the 64-bit mode on x86_64). +- Create indentity mapping (virtual address = physical address) of the lower + 2GB memory region, even if parts of the region are not backed by any host + memory buffers. The memory region above 2GB threshold is left unmapped + except for one memory page reserved for the `struct tst_kvm_result` buffer. +- Initialize 8KB stack. +- Install default interrupt handlers for standard CPU exception vectors. + +When the environment setup is complete, bootstrap will call `void main(void)` +function implemented by the test program. To finish execution of guest payload, +the test can either return from the `main()` function or call `kvm_exit()` +at any point.